github_mirrors/vectorscan
3
0
Fork 1
mirror of https://github.com/VectorCamp/vectorscan.git synced 2025-06-28 16:41:01 +03:00
Code Issues Packages Projects Releases Wiki Activity

keep debian folder in a separate branch

Browse Source
This commit is contained in:
Konstantinos Margaritis 2022-01-21 12:06:58 +02:00
parent 6cd6957a23
commit ada4e1377f
906 changed files with 0 additions and 311729 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

336
CHANGELOG.md
View File

@ -1,336 +0,0 @@
# Hyperscan Change Log
This is a list of notable changes to Hyperscan, in reverse chronological order.
## [5.4.0] 2020-12-31
- Improvement on literal matcher "Fat Teddy" performance, including
support for Intel(R) AVX-512 Vector Byte Manipulation Instructions (Intel(R)
AVX-512 VBMI).
- Introduce a new 32-state shuffle-based DFA engine ("Sheng32"). This improves
scanning performance by leveraging AVX-512 VBMI.
- Introduce a new 64-state shuffle-based DFA engine ("Sheng64"). This improves
scanning performance by leveraging AVX-512 VBMI.
- Introduce a new shuffle-based hybrid DFA engine ("McSheng64"). This improves
scanning performance by leveraging AVX-512 VBMI.
- Improvement on exceptional state handling performance for LimEx NFA, including
support for AVX-512 VBMI.
- Improvement on lookaround performance with new models, including support for
AVX-512.
- Improvement on DFA state space efficiency.
- Optimization on decision of NFA/DFA generation.
- hsbench: add CSV dump support for hsbench.
- Bugfix for cmake error on Icelake under release mode.
- Bugfix in find_vertices_in_cycles() to avoid self-loop checking in SCC.
- Bugfix for issue #270: fix return value handling in chimera.
- Bugfix for issue #284: use correct free function in logical combination.
- Add BUILD_EXAMPLES cmake option to enable example code compilation. (#260)
- Some typo fixing. (#242, #259)
## [5.3.0] 2020-05-15
- Improvement on literal matcher "Teddy" performance, including support for
Intel(R) AVX-512 Vector Byte Manipulation Instructions (Intel(R) AVX-512
VBMI).
- Improvement on single-byte/two-byte matching performance, including support
for Intel(R) Advanced Vector Extensions 512 (Intel(R) AVX-512).
- hsbench: add hyphen support for -T option.
- tools/fuzz: add test scripts for synthetic pattern generation.
- Bugfix for acceleration path analysis in LimEx NFA.
- Bugfix for duplicate matches for Small-write engine.
- Bugfix for UTF8 checking problem for hscollider.
- Bugfix for issue #205: avoid crash of `hs_compile_lit_multi()` with clang and
ASAN.
- Bugfix for issue #211: fix error in `db_check_platform()` function.
- Bugfix for issue #217: fix cmake parsing issue of CPU arch for non-English
locale.
- Bugfix for issue #228: avoid undefined behavior when calling `close()` after
`fdopendir()` in `loadExpressions()`.
- Bugfix for issue #239: fix hyperscan compile issue under gcc-10.
- Add VLAN packets processing capability in pcap analysis script. (#214)
- Avoid extra convert instruction for "Noodle". (#221)
- Add Hyperscan version marcro in `hs.h`. (#222)
## [5.2.1] 2019-10-13
- Bugfix for issue #186: fix compile issue when `BUILD_SHARED_LIBS` is on in
release mode.
- Disable redundant move check for older compiler versions.
## [5.2.0] 2019-07-12
- Literal API: add new API `hs_compile_lit()` and `hs_compile_lit_multi()` to
process pure literal rule sets. The 2 literal APIs treat each expression text
in a literal sense without recognizing any regular grammers.
- Logical combination: add support for purely negative combinations, which
report match at EOD in case of no sub-expressions matched.
- Windows porting: support shared library (DLL) on Windows with available tools
hscheck, hsbench and hsdump.
- Bugfix for issue #148: fix uninitialized use of `scatter_unit_uX` due to
padding.
- Bugfix for issue #155: fix numerical result out of range error.
- Bugfix for issue #165: avoid corruption of pending combination report in
streaming mode.
- Bugfix for issue #174: fix scratch free issue when memory allocation fails.
## [5.1.1] 2019-04-03
- Add extra detection and handling when invalid rose programs are triggered.
- Bugfix for issue #136: fix CMake parsing of CPU architecure for GCC-9.
- Bugfix for issue #137: avoid file path impact on fat runtime build.
- Bugfix for issue #141: fix rose literal programs for multi-pattern
matching when no pattern ids are provided.
- Bugfix for issue #144: fix library install path in pkg-config files.
## [5.1.0] 2019-01-17
- Improve DFA state compression by wide-state optimization to reduce bytecode
size.
- Create specific interpreter runtime handling to boost the performance of pure
literal matching.
- Optimize original presentation of interpreter (the "Rose" engine ) to
increase overall performance.
- Bugfix for logical combinations: fix error reporting combination's match in
case of sub-expression has EOD match under streaming mode.
- Bugfix for logical combinations: fix miss reporting combination's match under
vacuous input.
- Bugfix for issue #104: fix compile error with Boost 1.68.0.
- Bugfix for issue #127: avoid pcre error for hscollider with installed PCRE
package.
- Update version of PCRE used by testing tools as a syntax and semantic
reference to PCRE 8.41 or above.
- Fix github repo address in doc.
## [5.0.0] 2018-07-09
- Introduce chimera hybrid engine of Hyperscan and PCRE, to fully support
PCRE syntax as well as to take advantage of the high performance nature of
Hyperscan.
- New API feature: logical combinations (AND, OR and NOT) of patterns in a
given pattern set.
- Windows porting: hsbench, hscheck, hscollider and hsdump tools now available
on Windows 8 or newer.
- Improve undirected graph implementation to avoid graph copy and reduce
compile time.
- Bugfix for issue #86: enable hscollider for installed PCRE package.
## [4.7.0] 2018-01-24
- Introduced hscollider pattern testing tool, for validating Hyperscan match
behaviour against PCRE.
- Introduced hscheck pattern compilation tool.
- Introduced hsdump development tool for producing information about Hyperscan
pattern compilation.
- New API feature: extended approximate matching support for Hamming distance.
- Bugfix for issue #69: Force C++ linkage in Xcode.
- Bugfix for issue #73: More documentation for `hs_close_stream()`.
- Bugfix for issue #78: Fix for fat runtime initialisation when used as a
shared library.
## [4.6.0] 2017-09-22
- New API feature: stream state compression. This allows the user to compress
and restore state for streams to reduce memory usage.
- Many improvements to literal matching performance, including more support
for Intel(R) Advanced Vector Extensions 512 (Intel(R) AVX-512).
- Compile time improvements, mainly reducing compiler memory allocation.
Also results in reduced compile time for some pattern sets.
- Bugfix for issue #62: fix error building Hyperscan using older versions of
Boost.
- Small updates to fix warnings identified by Coverity.
## [4.5.2] 2017-07-26
- Bugfix for issue #57: Treat characters between `\Q.\E` as codepoints in
UTF8 mode.
- Bugfix for issue #60: Use a portable flag for mktemp for fat runtime builds.
- Bugfix for fat runtime builds on AVX-512 capable machines with Hyperscan's
AVX-512 support disabled.
## [4.5.1] 2017-06-16
- Bugfix for issue #56: workaround for gcc-4.8 C++11 defect.
- Bugfix for literal matching table generation, reversing a regression in
performance for some literal matching cases.
- Bugfixes for hsbench, related to multicore benchmarking, portability fixes
for FreeBSD, and clarifying output results.
- CMake: removed a duplicate else branch that causes very recent (v3.9) builds
of CMake to fail.
## [4.5.0] 2017-06-09
- New API feature: approximate matching using the "edit distance" extended
parameter. This allows the user to request all matches that are a given edit
distance from an exact match for a pattern.
- Initial support for Intel(R) Advanced Vector Extensions 512 (Intel(R)
AVX-512), disabled by default. To enable it, pass `-DBUILD_AVX512=1` to
`cmake`.
- Major compile time improvements in many subsystems, reducing compile time
significantly for many large pattern sets.
- Internal reworking of literal matchers to operate on literals of at
most eight characters, with subsequent confirmation done in the Rose
interpreter. This reduces complexity and bytecode size and improves
performance for many pattern sets.
- Improve performance of the FDR literal matcher front end.
- Improve bucket assignment and other heuristics governing the FDR literal
matcher.
- Improve optimisation passes that take advantage of extended parameter
constraints (`min_offset`, etc).
- Introduce further lookaround specialisations to improve scanning performance.
- Optimise Rose interpreter construction to reduce the length of programs
generated in some situations.
- Remove the old "Rose" pattern decomposition analysis pass in favour of the
new "Violet" pass introduced in Hyperscan 4.3.0.
- In streaming mode, allow exhaustion (where the stream can no longer produce
matchers) to be detected in more situations, improving scanning performance.
- Improve parsing of control verbs (such as `(*UTF8)`) that can only occur at
the beginning of the pattern. Combinations of supported verbs in any order
are now permitted.
- Update version of PCRE used by testing tools as a syntax and semantic
reference to PCRE 8.40.
- Tuning support for Intel(R) microarchitecture code names Skylake, Skylake
Server, Goldmont.
- CMake: when building a native build with a version of GCC that doesn't
recognise the host compiler, tune for the microarch selected by
`-march=native`.
- CMake: don't fail if SQLite (which is only required to build the `hsbench`
tool) is not present.
- CMake: detect libc++ directly and use that to inform the Boost version
requirement.
- Bugfix for issue #51: make the fat runtime build wrapper less fragile.
- Bugfix for issues #46, #52: use `sqlite3_errmsg()` to allow SQLite 3.6.x to
be used. Thanks to @EaseTheWorld for the PR.
## [4.4.1] 2017-02-28
- Bugfixes to fix issues where stale data was being referenced in scratch
memory. In particular this may have resulted in `hs_close_stream()`
referencing data from other previously scanned streams. This may result in
incorrect matches being been reported.
## [4.4.0] 2017-01-20
- Introduce the "fat runtime" build. This will build several variants of the
Hyperscan scanning engine specialised for different processor feature sets,
and use the appropriate one for the host at runtime. This uses the "ifunc"
indirect function attribute provided by GCC and is currently available on
Linux only, where it is the default for release builds.
- New API function: add the `hs_valid_platform()` function. This function tests
whether the host provides the SSSE3 instruction set required by Hyperscan.
- Introduce a new standard benchmarking tool, "hsbench". This provides an easy
way to measure Hyperscan's performance for a particular set of patterns and
corpus of data to be scanned.
- Introduce a 64-bit GPR LimEx NFA model, which uses 64-bit GPRs on 64-bit
hosts and SSE registers on 32-bit hosts.
- Introduce a new DFA model ("McSheng") which is a hybrid of the existing
McClellan and Sheng models. This improves scanning performance for some
cases.
- Introduce lookaround specialisations to improve scanning performance.
- Improve the handling of long literals by moving confirmation to the Rose
interpreter and simplifying the hash table used to track them in streaming
mode.
- Improve compile time optimisation for removing redundant paths from
expression graphs.
- Build: improve support for building with MSVC toolchain.
- Reduce the size of small write DFAs used for small scans in block mode.
- Introduce a custom graph type (`ue2_graph`) used in place of the Boost Graph
Library's `adjacency_list` type. Improves compile time performance and type
safety.
- Improve scanning performance of the McClellan DFA.
- Bugfix for a very unusual SOM case where the incorrect start offset was
reported for a match.
- Bugfix for issue #37, removing execute permissions from some source files.
- Bugfix for issue #41, handle Windows line endings in pattern files.
## [4.3.2] 2016-11-15
- Bugfix for issue #39. This small change is a workaround for an issue in
Boost 1.62. The fix has been submitted to Boost for inclusion in a future
release.
## [4.3.1] 2016-08-29
- Bugfix for issue #30. In recent versions of Clang, a write to a variable was
being elided, resulting in corrupted stream state after calling
`hs_reset_stream()`.
## [4.3.0] 2016-08-24
- Introduce a new analysis pass ("Violet") used for decomposition of patterns
into literals and smaller engines.
- Introduce a new container engine ("Tamarama") for infix and suffix engines
that can be proven to run exclusively of one another. This reduces stream
state for pattern sets with many such engines.
- Introduce a new shuffle-based DFA engine ("Sheng"). This improves scanning
performance for pattern sets where small engines are generated.
- Improve the analysis used to extract extra mask information from short
literals.
- Reduced compile time spent in equivalence class analysis.
- Build: frame pointers are now only omitted for 32-bit release builds.
- Build: Workaround for C++ issues reported on FreeBSD/libc++ platforms.
(github issue #27)
- Simplify the LimEx NFA with a unified "variable shift" model, which reduces
the number of different NFA code paths to one per model size.
- Allow some anchored prefixes that may squash the literal to which they are
attached to run eagerly. This improves scanning performance for some
patterns.
- Simplify and improve EOD ("end of data") matching, using the interpreter for
all operations.
- Elide unnecessary instructions in the Rose interpreter at compile time.
- Reduce the number of inlined instantiations of the Rose interpreter in order
to reduce instruction cache pressure.
- Small improvements to literal matcher acceleration.
- Parser: ignore `\E` metacharacters that are not preceded by `\Q`. This
conforms to PCRE's behaviour, rather than returning a compile error.
- Check for misaligned memory when allocating an error structure in Hyperscan's
compile path and return an appropriate error if detected.
## [4.2.0] 2016-05-31
- Introduce an interpreter for many complex actions to replace the use of
internal reports within the core of Hyperscan (the "Rose" engine). This
improves scanning performance and reduces database size for many pattern
sets.
- Many enhancements to the acceleration framework used by NFA and DFA engines,
including more flexible multibyte implementations and more AVX2 support. This
improves scanning performance for many pattern sets.
- Improved prefiltering support for complex patterns containing very large
bounded repeats (`R{M,N}` with large `N`).
- Improve scanning performance of pattern sets with a very large number of
EOD-anchored patterns.
- Improve scanning performance of large pattern sets that use the
`HS_FLAG_SINGLEMATCH` flag.
- Improve scanning performance of pattern sets that contain a single literal by
improving the "Noodle" literal matcher.
- Small reductions in total stream state for many pattern sets.
- Improve runtime detection of AVX2 support.
- Disable -Werror for release builds, in order to behave better for packagers
and users with different compiler combinations than those that we test.
- Improve support for building on Windows with MSVC 2015 (github issue #14).
Support for Hyperscan on Windows is still experimental.
- Small updates to fix warnings identified by Coverity.
- Remove Python codegen for the "FDR" and "Teddy" literal matchers. These are
now implemented directly in C code.
- Remove the specialist "Sidecar" engine in favour of using our more general
repeat engines.
- New API function: add the `hs_expression_ext_info()` function. This is a
variant of `hs_expression_info()` that can accept patterns with extended
parameters.
- New API error value: add the `HS_SCRATCH_IN_USE` error, which is returned
when Hyperscan detects that a scratch region is already in use on entry to an
API function.
## [4.1.0] 2015-12-18
- Update version of PCRE used by testing tools as a syntax and semantic
reference to PCRE 8.38.
- Small updates to fix warnings identified by Coverity.
- Clean up and unify exception handling behaviour across GPR and SIMD NFA
models.
- Fix bug in handling of bounded repeat triggers with large gaps between them
for sparse repeat model.
- Correctly reject POSIX collating elements (`[.ch.]`, `[=ch=]`) in the parser.
These are not supported by Hyperscan.
- Add support for quoted sequences (`\Q...\E`) inside character classes.
- Simplify FDR literal matcher runtime by removing some static specialization.
- Fix handling of the POSIX `[:graph:]`, `[:print:]` and `[:punct:]` character
classes to match the behaviour of PCRE 8.38 in both standard operation and
with the UCP flag set. (Note: some bugs were fixed in this area in PCRE
8.38.) Previously Hyperscan's behaviour was the same as versions of PCRE
before 8.34.
- Improve performance when compiling pattern sets that include a large number
of similar bounded repeat constructs. (github issue #9)
## [4.0.1] 2015-10-30
- Minor cleanups to test code.
- CMake and other build system improvements.
- API update: allow `hs_reset_stream()` and `hs_reset_and_copy_stream()` to be
supplied with a NULL scratch pointer if no matches are required. This is in
line with the behaviour of `hs_close_stream()`.
- Disallow bounded repeats with a very large minimum repeat but no maximum,
i.e. {N,} for very large N.
- Reduce compile memory usage in literal set explansion for some large cases.
## [4.0.0] 2015-10-20
- Original release of Hyperscan as open-source software.

1461
CMakeLists.txt
View File

File diff suppressed because it is too large Load Diff

27
COPYING
View File

@ -1,27 +0,0 @@
Copyright (c) 2015, Intel Corporation
Copyright (c) 2019-20, VectorCamp PC
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
* Neither the name of Intel Corporation nor the names of its contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

590
Jenkinsfile vendored
View File

@ -1,590 +0,0 @@
pipeline {
agent none
stages {
stage("Build") {
failFast true
parallel {
stage("Release/SSE") {
agent { label "x86" }
stages {
stage("Git checkout") {
steps {
checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
}
}
stage("Build") {
steps {
cmakeBuild buildDir: 'build-release-SSE', buildType: 'Release', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=no -DBUILD_AVX512=no -DFAT_RUNTIME=no', installation: 'InSearchPath', steps: [[args: '--parallel 4', withCmake: true]]
}
}
stage("Unit Test") {
steps {
sh 'build-release-SSE/bin/unit-internal'
}
}
stage("Test") {
steps {
sh 'build-release-SSE/bin/unit-hyperscan'
}
}
}
}
stage("Release/AVX2") {
agent { label "x86" }
stages {
stage("Git checkout") {
steps {
checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
}
}
stage("Build") {
steps {
cmakeBuild buildDir: 'build-release-AVX2', buildType: 'Release', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=yes -DBUILD_AVX512=no -DFAT_RUNTIME=no', installation: 'InSearchPath', steps: [[args: '--parallel 4', withCmake: true]]
}
}
stage("Unit Test") {
steps {
sh 'build-release-AVX2/bin/unit-internal'
}
}
stage("Test") {
steps {
sh 'build-release-AVX2/bin/unit-hyperscan'
}
}
}
}
stage("Release/AVX512") {
agent { label "x86" }
stages {
stage("Git checkout") {
steps {
checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
}
}
stage("Build") {
steps {
cmakeBuild buildDir: 'build-release-AVX512', buildType: 'Release', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=yes -DBUILD_AVX512=yes -DFAT_RUNTIME=no', installation: 'InSearchPath', steps: [[args: '--parallel 4', withCmake: true]]
}
}
stage("Unit Test") {
steps {
sh 'build-release-AVX512/bin/unit-internal'
}
}
stage("Test") {
steps {
sh 'build-release-AVX512/bin/unit-hyperscan'
}
}
}
}
stage("Release/FAT") {
agent { label "x86" }
stages {
stage("Git checkout") {
steps {
checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
}
}
stage("Build") {
steps {
cmakeBuild buildDir: 'build-release-fat', buildType: 'Release', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=yes -DBUILD_AVX512=yes -DFAT_RUNTIME=yes', installation: 'InSearchPath', steps: [[args: '--parallel 4', withCmake: true]]
}
}
stage("Test") {
steps {
sh 'build-release-fat/bin/unit-hyperscan'
}
}
}
}
stage("Debug/SSE") {
agent { label "x86" }
stages {
stage("Git checkout") {
steps {
checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
}
}
stage("Build") {
steps {
cmakeBuild buildDir: 'build-debug-SSE', buildType: 'Debug', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=no -DBUILD_AVX512=no -DFAT_RUNTIME=no', installation: 'InSearchPath', steps: [[args: '--parallel 4', withCmake: true]]
}
}
stage("Unit Test") {
steps {
sh 'build-debug-SSE/bin/unit-internal'
}
}
stage("Test") {
steps {
sh 'build-debug-SSE/bin/unit-hyperscan'
}
}
}
}
stage("Debug/AVX2") {
agent { label "x86" }
stages {
stage("Git checkout") {
steps {
checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
}
}
stage("Build") {
steps {
cmakeBuild buildDir: 'build-debug-AVX2', buildType: 'Debug', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=yes -DBUILD_AVX512=no -DFAT_RUNTIME=no', installation: 'InSearchPath', steps: [[args: '--parallel 4', withCmake: true]]
}
}
stage("Unit Test") {
steps {
sh 'build-debug-AVX2/bin/unit-internal'
}
}
stage("Test") {
steps {
sh 'build-debug-AVX2/bin/unit-hyperscan'
}
}
}
}
stage("Debug/AVX512") {
agent { label "x86" }
stages {
stage("Git checkout") {
steps {
checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
}
}
stage("Build") {
steps {
cmakeBuild buildDir: 'build-debug-AVX512', buildType: 'Debug', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=yes -DBUILD_AVX512=yes -DFAT_RUNTIME=no', installation: 'InSearchPath', steps: [[args: '--parallel 4', withCmake: true]]
}
}
stage("Unit Test") {
steps {
sh 'build-debug-AVX512/bin/unit-internal'
}
}
stage("Test") {
steps {
sh 'build-debug-AVX512/bin/unit-hyperscan'
}
}
}
}
stage("Debug/FAT") {
agent { label "x86" }
stages {
stage("Git checkout") {
steps {
checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
}
}
stage("Build") {
steps {
cmakeBuild buildDir: 'build-debug-fat', buildType: 'Debug', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=yes -DBUILD_AVX512=yes -DFAT_RUNTIME=yes', installation: 'InSearchPath', steps: [[args: '--parallel 4', withCmake: true]]
}
}
stage("Test") {
steps {
sh 'build-debug-fat/bin/unit-hyperscan'
}
}
}
}
stage("Release/ARM") {
agent { label "arm" }
stages {
stage("Git checkout") {
steps {
checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
}
}
stage("Build") {
steps {
cmakeBuild buildDir: 'build-release-arm', buildType: 'Release', cleanBuild: true, cmakeArgs: '', installation: 'InSearchPath', steps: [[args: '--parallel 4', withCmake: true]]
}
}
stage("Unit Test") {
steps {
sh 'build-release-arm/bin/unit-internal'
}
}
stage("Test") {
steps {
sh 'build-release-arm/bin/unit-hyperscan'
}
}
}
}
stage("Debug/ARM") {
agent { label "arm" }
stages {
stage("Git checkout") {
steps {
checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
}
}
stage("Build") {
steps {
cmakeBuild buildDir: 'build-debug-arm', buildType: 'Debug', cleanBuild: true, cmakeArgs: '', installation: 'InSearchPath', steps: [[args: '--parallel 4', withCmake: true]]
}
}
stage("Unit Test") {
steps {
sh 'build-debug-arm/bin/unit-internal'
}
}
stage("Test") {
steps {
sh 'build-debug-arm/bin/unit-hyperscan'
}
}
}
}
stage("Release/Power") {
agent { label "power" }
stages {
stage("Git checkout") {
steps {
checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
}
}
stage("Build") {
steps {
cmakeBuild buildDir: 'build-release-power', buildType: 'Release', cleanBuild: true, cmakeArgs: '', installation: 'InSearchPath', steps: [[args: '--parallel 4', withCmake: true]]
}
}
stage("Unit Test") {
steps {
sh 'build-release-power/bin/unit-internal'
}
}
stage("Test") {
steps {
sh 'build-release-power/bin/unit-hyperscan'
}
}
}
}
stage("Debug/Power") {
agent { label "power" }
stages {
stage("Git checkout") {
steps {
checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
}
}
stage("Build") {
steps {
cmakeBuild buildDir: 'build-debug-power', buildType: 'Debug', cleanBuild: true, cmakeArgs: '', installation: 'InSearchPath', steps: [[args: '--parallel 4', withCmake: true]]
}
}
stage("Unit Test") {
steps {
sh 'build-debug-power/bin/unit-internal'
}
}
stage("Test") {
steps {
sh 'build-debug-power/bin/unit-hyperscan'
}
}
}
}
stage("Clang-Release/SSE") {
agent { label "x86" }
stages {
stage("Git checkout") {
steps {
checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
}
}
stage("Build") {
steps {
cmakeBuild buildDir: 'build-clang-release-SSE', buildType: 'Release', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=no -DBUILD_AVX512=no -DFAT_RUNTIME=no', installation: 'InSearchPath', steps: [[envVars: 'CC=clang CXX=clang++', args: '--parallel 4', withCmake: true]]
}
}
stage("Unit Test") {
steps {
sh 'build-clang-release-SSE/bin/unit-internal'
}
}
stage("Test") {
steps {
sh 'build-clang-release-SSE/bin/unit-hyperscan'
}
}
}
}
stage("Clang-Release/AVX2") {
agent { label "x86" }
stages {
stage("Git checkout") {
steps {
checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
}
}
stage("Build") {
steps {
cmakeBuild buildDir: 'build-clang-release-AVX2', buildType: 'Release', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=yes -DBUILD_AVX512=no -DFAT_RUNTIME=no', installation: 'InSearchPath', steps: [[envVars: 'CC=clang CXX=clang++', args: '--parallel 4', withCmake: true]]
}
}
stage("Unit Test") {
steps {
sh 'build-clang-release-AVX2/bin/unit-internal'
}
}
stage("Test") {
steps {
sh 'build-clang-release-AVX2/bin/unit-hyperscan'
}
}
}
}
stage("Clang-Release/AVX512") {
agent { label "x86" }
stages {
stage("Git checkout") {
steps {
checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
}
}
stage("Build") {
steps {
cmakeBuild buildDir: 'build-clang-release-AVX512', buildType: 'Release', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=yes -DBUILD_AVX512=yes -DFAT_RUNTIME=no', installation: 'InSearchPath', steps: [[envVars: 'CC=clang CXX=clang++', args: '--parallel 4', withCmake: true]]
}
}
stage("Unit Test") {
steps {
sh 'build-clang-release-AVX512/bin/unit-internal'
}
}
stage("Test") {
steps {
sh 'build-clang-release-AVX512/bin/unit-hyperscan'
}
}
}
}
stage("Clang-Release/FAT") {
agent { label "x86" }
stages {
stage("Git checkout") {
steps {
checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
}
}
stage("Build") {
steps {
cmakeBuild buildDir: 'build-clang-release-fat', buildType: 'Release', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=yes -DBUILD_AVX512=yes -DFAT_RUNTIME=yes', installation: 'InSearchPath', steps: [[envVars: 'CC=clang CXX=clang++', args: '--parallel 4', withCmake: true]]
}
}
stage("Test") {
steps {
sh 'build-clang-release-fat/bin/unit-hyperscan'
}
}
}
}
stage("Clang-Debug/SSE") {
agent { label "x86" }
stages {
stage("Git checkout") {
steps {
checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
}
}
stage("Build") {
steps {
cmakeBuild buildDir: 'build-clang-debug-SSE', buildType: 'Debug', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=no -DBUILD_AVX512=no -DFAT_RUNTIME=no', installation: 'InSearchPath', steps: [[envVars: 'CC=clang CXX=clang++', args: '--parallel 4', withCmake: true]]
}
}
stage("Unit Test") {
steps {
sh 'build-clang-debug-SSE/bin/unit-internal'
}
}
stage("Test") {
steps {
sh 'build-clang-debug-SSE/bin/unit-hyperscan'
}
}
}
}
stage("Clang-Debug/AVX2") {
agent { label "x86" }
stages {
stage("Git checkout") {
steps {
checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
}
}
stage("Build") {
steps {
cmakeBuild buildDir: 'build-clang-debug-AVX2', buildType: 'Debug', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=yes -DBUILD_AVX512=no -DFAT_RUNTIME=no', installation: 'InSearchPath', steps: [[envVars: 'CC=clang CXX=clang++', args: '--parallel 4', withCmake: true]]
}
}
stage("Unit Test") {
steps {
sh 'build-clang-debug-AVX2/bin/unit-internal'
}
}
stage("Test") {
steps {
sh 'build-clang-debug-AVX2/bin/unit-hyperscan'
}
}
}
}
stage("Clang-Debug/AVX512") {
agent { label "x86" }
stages {
stage("Git checkout") {
steps {
checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
}
}
stage("Build") {
steps {
cmakeBuild buildDir: 'build-clang-debug-AVX512', buildType: 'Debug', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=yes -DBUILD_AVX512=yes -DFAT_RUNTIME=no', installation: 'InSearchPath', steps: [[envVars: 'CC=clang CXX=clang++', args: '--parallel 4', withCmake: true]]
}
}
stage("Unit Test") {
steps {
sh 'build-clang-debug-AVX512/bin/unit-internal'
}
}
stage("Test") {
steps {
sh 'build-clang-debug-AVX512/bin/unit-hyperscan'
}
}
}
}
stage("Clang-Debug/FAT") {
agent { label "x86" }
stages {
stage("Git checkout") {
steps {
checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
}
}
stage("Build") {
steps {
cmakeBuild buildDir: 'build-clang-debug-fat', buildType: 'Debug', cleanBuild: true, cmakeArgs: '-DBUILD_AVX2=yes -DBUILD_AVX512=yes -DFAT_RUNTIME=yes', installation: 'InSearchPath', steps: [[envVars: 'CC=clang CXX=clang++', args: '--parallel 4', withCmake: true]]
}
}
stage("Test") {
steps {
sh 'build-clang-debug-fat/bin/unit-hyperscan'
}
}
}
}
stage("Clang-Release/ARM") {
agent { label "arm" }
stages {
stage("Git checkout") {
steps {
checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
}
}
stage("Build") {
steps {
cmakeBuild buildDir: 'build-clang-release-arm', buildType: 'Release', cleanBuild: true, cmakeArgs: '', installation: 'InSearchPath', steps: [[envVars: 'CC=clang CXX=clang++', args: '--parallel 4', withCmake: true]]
}
}
stage("Unit Test") {
steps {
sh 'build-clang-release-arm/bin/unit-internal'
}
}
stage("Test") {
steps {
sh 'build-clang-release-arm/bin/unit-hyperscan'
}
}
}
}
stage("Clang-Debug/ARM") {
agent { label "arm" }
stages {
stage("Git checkout") {
steps {
checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
}
}
stage("Build") {
steps {
cmakeBuild buildDir: 'build-debug-arm', buildType: 'Debug', cleanBuild: true, cmakeArgs: '', installation: 'InSearchPath', steps: [[envVars: 'CC=clang CXX=clang++', args: '--parallel 4', withCmake: true]]
}
}
stage("Unit Test") {
steps {
sh 'build-clang-debug-arm/bin/unit-internal'
}
}
stage("Test") {
steps {
sh 'build-clang-debug-arm/bin/unit-hyperscan'
}
}
}
}
stage("Clang-Release/Power") {
agent { label "power" }
stages {
stage("Git checkout") {
steps {
checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
}
}
stage("Build") {
steps {
cmakeBuild buildDir: 'build-clang-release-power', buildType: 'Release', cleanBuild: true, cmakeArgs: '', installation: 'InSearchPath', steps: [[envVars: 'CC=clang CXX=clang++', args: '--parallel 4', withCmake: true]]
}
}
stage("Unit Test") {
steps {
sh 'build-clang-release-power/bin/unit-internal'
}
}
stage("Test") {
steps {
sh 'build-clang-release-power/bin/unit-hyperscan'
}
}
}
}
stage("Clang-Debug/Power") {
agent { label "power" }
stages {
stage("Git checkout") {
steps {
checkout([$class: 'GitSCM', branches: [[name: '${sha1}']], extensions: [], userRemoteConfigs: [[refspec: '+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*', url: 'https://github.com/VectorCamp/vectorscan.git']]])
}
}
stage("Build") {
steps {
cmakeBuild buildDir: 'build-clang-debug-power', buildType: 'Debug', cleanBuild: true, cmakeArgs: '', installation: 'InSearchPath', steps: [[envVars: 'CC=clang CXX=clang++', args: '--parallel 4', withCmake: true]]
}
}
stage("Unit Test") {
steps {
sh 'build-clang-debug-power/bin/unit-internal'
}
}
stage("Test") {
steps {
sh 'build-clang-debug-power/bin/unit-hyperscan'
}
}
}
}
}
}
}
}

123
LICENSE
View File

@ -1,123 +0,0 @@
Hyperscan is licensed under the BSD License.
Copyright (c) 2015, Intel Corporation
Vectorscan is licensed under the BSD License.
Copyright (c) 2020, VectorCamp PC
Copyright (c) 2021, Arm Limited
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
* Neither the name of Intel Corporation nor the names of its contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
--------------------------------------------------------------------------------
This product also contains code from third parties, under the following
licenses:
Intel's Slicing-by-8 CRC32 implementation
-----------------------------------------
Copyright (c) 2004-2006, Intel Corporation
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Boost C++ Headers Library
-------------------------
Boost Software License - Version 1.0 - August 17th, 2003
Permission is hereby granted, free of charge, to any person or organization
obtaining a copy of the software and accompanying documentation covered by
this license (the "Software") to use, reproduce, display, distribute,
execute, and transmit the Software, and to prepare derivative works of the
Software, and to permit third-parties to whom the Software is furnished to
do so, all subject to the following:
The copyright notices in the Software and this entire statement, including
the above license grant, this restriction and the following disclaimer,
must be included in all copies of the Software, in whole or in part, and
all derivative works of the Software, unless such copies or derivative
works are solely in the form of machine-executable object code generated by
a source language processor.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT
SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE
FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE,
ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
DEALINGS IN THE SOFTWARE.
The Google C++ Testing Framework (Google Test)
----------------------------------------------
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of Google Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

96
README.md
View File

@ -1,96 +0,0 @@
# Vectorscan?
A fork of Intel's Hyperscan, modified to run on more platforms. Currently ARM NEON/ASIMD
is 100% functional, and Power VSX are in development. ARM SVE2 will be implemented when
harwdare becomes accessible to the developers. More platforms will follow in the future,
on demand/request.
Vectorscan will follow Intel's API and internal algorithms where possible, but will not
hesitate to make code changes where it is thought of giving better performance or better
portability. In addition, the code will be gradually simplified and made more uniform and
all architecture specific -currently Intel- #ifdefs will be removed and abstracted away.
# Why the fork?
Originally, the ARM porting was supposed to be merged into Intel's own Hyperscan, and 2
Pull Requests had been made to the project for this reason ([1], [2]). Unfortunately, the
PRs were rejected for now and the forseeable future, thus we have created Vectorscan for
our own multi-architectural and opensource collaborative needs.
# What is Hyperscan?
Hyperscan is a high-performance multiple regex matching library. It follows the
regular expression syntax of the commonly-used libpcre library, but is a
standalone library with its own C API.
Hyperscan uses hybrid automata techniques to allow simultaneous matching of
large numbers (up to tens of thousands) of regular expressions and for the
matching of regular expressions across streams of data.
Vectorscan is typically used in a DPI library stack, just like Hyperscan.
# Cross Compiling for AArch64
- To cross compile for AArch64, first adjust the variables set in cmake/setenv-arm64-cross.sh.
- `export CROSS=<arm-cross-compiler-dir>/bin/aarch64-linux-gnu-`
- `export CROSS_SYS=<arm-cross-compiler-system-dir>`
- `export BOOST_PATH=<boost-source-dir>`
- Set the environment variables:
- `source cmake/setenv-arm64-cross.sh`
- Configure Vectorscan:
- `mkdir <build-dir-name>`
- `cd <build-dir>`
- `cmake -DCROSS_COMPILE_AARCH64=1 <hyperscan-source-dir> -DCMAKE_TOOLCHAIN_FILE=<hyperscan-source-dir>/cmake/arm64-cross.cmake`
- Build Vectorscan:
- `make -jT` where T is the number of threads used to compile.
- `cmake --build . -- -j T` can also be used instead of make.
# Compiling for SVE
The following cmake variables can be set in order to target Arm's Scalable
Vector Extension. They are listed in ascending order of strength, with cmake
detecting whether the feature is available in the compiler and falling back to
a weaker version if not. Only one of these variables needs to be set as weaker
variables will be implied as set.
- `BUILD_SVE`
- `BUILD_SVE2`
- `BUILD_SVE2_BITPERM`
# Documentation
Information on building the Hyperscan library and using its API is available in
the [Developer Reference Guide](http://intel.github.io/hyperscan/dev-reference/).
# License
Vectorscan, like Hyperscan is licensed under the BSD License. See the LICENSE file in the
project repository.
# Versioning
The `master` branch on Github will always contain the most recent release of
Hyperscan. Each version released to `master` goes through QA and testing before
it is released; if you're a user, rather than a developer, this is the version
you should be using.
Further development towards the next release takes place on the `develop`
branch.
# Get Involved
The official homepage for Vectorscan is at [www.github.com/VectorCamp/vectorscan](https://www.github.com/VectorCamp/vectorscan).
# Original Hyperscan links
The official homepage for Hyperscan is at [www.hyperscan.io](https://www.hyperscan.io).
If you have questions or comments, we encourage you to [join the mailing
list](https://lists.01.org/mailman/listinfo/hyperscan). Bugs can be filed by
sending email to the list, or by creating an issue on Github.
If you wish to contact the Hyperscan team at Intel directly, without posting
publicly to the mailing list, send email to
[hyperscan@intel.com](mailto:hyperscan@intel.com).
[1]: https://github.com/intel/hyperscan/pull/272
[2]: https://github.com/intel/hyperscan/pull/287

6
benchmarks/CMakeLists.txt
View File

@ -1,6 +0,0 @@
if (NOT FAT_RUNTIME AND (BUILD_STATIC_AND_SHARED OR BUILD_STATIC_LIBS))
add_executable(benchmarks benchmarks.cpp)
set_source_files_properties(benchmarks.cpp PROPERTIES COMPILE_FLAGS
"-Wall -Wno-unused-variable")
target_link_libraries(benchmarks hs)
endif()

253
benchmarks/benchmarks.cpp
View File

@ -1,253 +0,0 @@
/*
* Copyright (c) 2020, 2021, VectorCamp PC
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#include <iostream>
#include <chrono>
#include <cstring>
#include <ctime>
#include <cstdlib>
#include <memory>
#include <functional>
#include "benchmarks.hpp"
#define MAX_LOOPS 1000000000
#define MAX_MATCHES 5
#define N 8
struct hlmMatchEntry {
size_t to;
u32 id;
hlmMatchEntry(size_t end, u32 identifier) :
to(end), id(identifier) {}
};
std::vector<hlmMatchEntry> ctxt;
static
hwlmcb_rv_t hlmSimpleCallback(size_t to, u32 id,
UNUSED struct hs_scratch *scratch) {
DEBUG_PRINTF("match @%zu = %u\n", to, id);
ctxt.push_back(hlmMatchEntry(to, id));
return HWLM_CONTINUE_MATCHING;
}
template<typename InitFunc, typename BenchFunc>
static void run_benchmarks(int size, int loops, int max_matches, bool is_reverse, MicroBenchmark &bench, InitFunc &&init, BenchFunc &&func) {
init(bench);
double total_sec = 0.0;
u64a total_size = 0;
double bw = 0.0;
double avg_bw = 0.0;
double max_bw = 0.0;
double avg_time = 0.0;
if (max_matches) {
int pos = 0;
for(int j = 0; j < max_matches - 1; j++) {
bench.buf[pos] = 'b';
pos = (j+1) *size / max_matches ;
bench.buf[pos] = 'a';
u64a actual_size = 0;
auto start = std::chrono::steady_clock::now();
for(int i = 0; i < loops; i++) {
const u8 *res = func(bench);
if (is_reverse)
actual_size += bench.buf.data() + size - res;
else
actual_size += res - bench.buf.data();
}
auto end = std::chrono::steady_clock::now();
double dt = std::chrono::duration_cast<std::chrono::microseconds>(end - start).count();
total_sec += dt;
/*convert microseconds to seconds*/
/*calculate bandwidth*/
bw = (actual_size / dt) * 1000000.0 / 1048576.0;
/*std::cout << "act_size = " << act_size << std::endl;
std::cout << "dt = " << dt << std::endl;
std::cout << "bw = " << bw << std::endl;*/
avg_bw += bw;
/*convert to MB/s*/
max_bw = std::max(bw, max_bw);
/*calculate average time*/
avg_time += total_sec / loops;
}
avg_time /= max_matches;
avg_bw /= max_matches;
total_sec /= 1000000.0;
/*convert average time to us*/
printf(KMAG "%s: %u matches, %u * %u iterations," KBLU " total elapsed time =" RST " %.3f s, "
KBLU "average time per call =" RST " %.3f μs," KBLU " max bandwidth = " RST " %.3f MB/s," KBLU " average bandwidth =" RST " %.3f MB/s \n",
bench.label, max_matches, size ,loops, total_sec, avg_time, max_bw, avg_bw);
} else {
auto start = std::chrono::steady_clock::now();
for (int i = 0; i < loops; i++) {
const u8 *res = func(bench);
}
auto end = std::chrono::steady_clock::now();
total_sec += std::chrono::duration_cast<std::chrono::microseconds>(end - start).count();
/*calculate transferred size*/
total_size = size * loops;
/*calculate average time*/
avg_time = total_sec / loops;
/*convert microseconds to seconds*/
total_sec /= 1000000.0;
/*calculate maximum bandwidth*/
max_bw = total_size / total_sec;
/*convert to MB/s*/
max_bw /= 1048576.0;
printf(KMAG "%s: no matches, %u * %u iterations," KBLU " total elapsed time =" RST " %.3f s, "
KBLU "average time per call =" RST " %.3f μs ," KBLU " bandwidth = " RST " %.3f MB/s \n",
bench.label, size ,loops, total_sec, avg_time, max_bw );
}
}
int main(){
int matches[] = {0, MAX_MATCHES};
std::vector<size_t> sizes;
for (size_t i = 0; i < N; i++) sizes.push_back(16000 << i*2);
const char charset[] = "aAaAaAaAAAaaaaAAAAaaaaAAAAAAaaaAAaaa";
for (int m = 0; m < 2; m++) {
for (size_t i = 0; i < std::size(sizes); i++) {
MicroBenchmark bench("Shufti", sizes[i]);
run_benchmarks(sizes[i], MAX_LOOPS / sizes[i], matches[m], false, bench,
[&](MicroBenchmark &b) {
b.chars.set('a');
ue2::shuftiBuildMasks(b.chars, (u8 *)&b.lo, (u8 *)&b.hi);
memset(b.buf.data(), 'b', b.size);
},
[&](MicroBenchmark &b) {
return shuftiExec(b.lo, b.hi, b.buf.data(), b.buf.data() + b.size);
}
);
}
for (size_t i = 0; i < std::size(sizes); i++) {
MicroBenchmark bench("Reverse Shufti", sizes[i]);
run_benchmarks(sizes[i], MAX_LOOPS / sizes[i], matches[m], true, bench,
[&](MicroBenchmark &b) {
b.chars.set('a');
ue2::shuftiBuildMasks(b.chars, (u8 *)&b.lo, (u8 *)&b.hi);
memset(b.buf.data(), 'b', b.size);
},
[&](MicroBenchmark &b) {
return rshuftiExec(b.lo, b.hi, b.buf.data(), b.buf.data() + b.size);
}
);
}
for (size_t i = 0; i < std::size(sizes); i++) {
MicroBenchmark bench("Truffle", sizes[i]);
run_benchmarks(sizes[i], MAX_LOOPS / sizes[i], matches[m], false, bench,
[&](MicroBenchmark &b) {
b.chars.set('a');
ue2::truffleBuildMasks(b.chars, (u8 *)&b.lo, (u8 *)&b.hi);
memset(b.buf.data(), 'b', b.size);
},
[&](MicroBenchmark &b) {
return truffleExec(b.lo, b.hi, b.buf.data(), b.buf.data() + b.size);
}
);
}
for (size_t i = 0; i < std::size(sizes); i++) {
MicroBenchmark bench("Reverse Truffle", sizes[i]);
run_benchmarks(sizes[i], MAX_LOOPS / sizes[i], matches[m], true, bench,
[&](MicroBenchmark &b) {
b.chars.set('a');
ue2::truffleBuildMasks(b.chars, (u8 *)&b.lo, (u8 *)&b.hi);
memset(b.buf.data(), 'b', b.size);
},
[&](MicroBenchmark &b) {
return rtruffleExec(b.lo, b.hi, b.buf.data(), b.buf.data() + b.size);
}
);
}
for (size_t i = 0; i < std::size(sizes); i++) {
MicroBenchmark bench("Vermicelli", sizes[i]);
run_benchmarks(sizes[i], MAX_LOOPS / sizes[i], matches[m], false, bench,
[&](MicroBenchmark &b) {
b.chars.set('a');
ue2::truffleBuildMasks(b.chars, (u8 *)&b.lo, (u8 *)&b.hi);
memset(b.buf.data(), 'b', b.size);
},
[&](MicroBenchmark &b) {
return vermicelliExec('a', 'b', b.buf.data(), b.buf.data() + b.size);
}
);
}
for (size_t i = 0; i < std::size(sizes); i++) {
MicroBenchmark bench("Reverse Vermicelli", sizes[i]);
run_benchmarks(sizes[i], MAX_LOOPS / sizes[i], matches[m], true, bench,
[&](MicroBenchmark &b) {
b.chars.set('a');
ue2::truffleBuildMasks(b.chars, (u8 *)&b.lo, (u8 *)&b.hi);
memset(b.buf.data(), 'b', b.size);
},
[&](MicroBenchmark &b) {
return rvermicelliExec('a', 'b', b.buf.data(), b.buf.data() + b.size);
}
);
}
for (size_t i = 0; i < std::size(sizes); i++) {
//we imitate the noodle unit tests
std::string str;
const size_t char_len = 5;
str.resize(char_len + 1);
for (size_t j=0; j < char_len; j++) {
srand (time(NULL));
int key = rand() % + 36 ;
str[char_len] = charset[key];
str[char_len + 1] = '\0';
}
MicroBenchmark bench("Noodle", sizes[i]);
run_benchmarks(sizes[i], MAX_LOOPS / sizes[i], matches[m], false, bench,
[&](MicroBenchmark &b) {
ctxt.clear();
memset(b.buf.data(), 'a', b.size);
u32 id = 1000;
ue2::hwlmLiteral lit(str, true, id);
b.nt = ue2::noodBuildTable(lit);
assert(b.nt != nullptr);
},
[&](MicroBenchmark &b) {
noodExec(b.nt.get(), b.buf.data(), b.size, 0, hlmSimpleCallback, &b.scratch);
return b.buf.data() + b.size;
}
);
}
}
return 0;
}

69
benchmarks/benchmarks.hpp
View File

@ -1,69 +0,0 @@
/*
* Copyright (c) 2020, 2021, VectorCamp PC
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#include "nfa/shufti.h"
#include "nfa/shufticompile.h"
#include "nfa/truffle.h"
#include "nfa/trufflecompile.h"
#include "nfa/vermicelli.hpp"
#include "hwlm/noodle_build.h"
#include "hwlm/noodle_engine.h"
#include "hwlm/noodle_internal.h"
#include "hwlm/hwlm_literal.h"
#include "util/bytecode_ptr.h"
#include "scratch.h"
/*define colour control characters*/
#define RST "\x1B[0m"
#define KRED "\x1B[31m"
#define KGRN "\x1B[32m"
#define KYEL "\x1B[33m"
#define KBLU "\x1B[34m"
#define KMAG "\x1B[35m"
#define KCYN "\x1B[36m"
#define KWHT "\x1B[37m"
class MicroBenchmark
{
public:
char const *label;
size_t size;
// Shufti/Truffle
m128 lo, hi;
ue2::CharReach chars;
std::vector<u8> buf;
// Noodle
struct hs_scratch scratch;
ue2::bytecode_ptr<noodTable> nt;
MicroBenchmark(char const *label_, size_t size_)
:label(label_), size(size_), buf(size_) {
};
};

47
chimera/CMakeLists.txt
View File

@ -1,47 +0,0 @@
# Chimera lib
include_directories(${PCRE_INCLUDE_DIRS})
# only set these after all tests are done
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} ${EXTRA_C_FLAGS}")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${EXTRA_CXX_FLAGS}")
SET(chimera_HEADERS
ch.h
ch_common.h
ch_compile.h
ch_runtime.h
)
install(FILES ${chimera_HEADERS} DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}/hs")
SET(chimera_SRCS
${chimera_HEADERS}
ch_alloc.c
ch_alloc.h
ch_compile.cpp
ch_database.c
ch_database.h
ch_internal.h
ch_runtime.c
ch_scratch.h
ch_scratch.c
)
add_library(chimera STATIC ${chimera_SRCS})
add_dependencies(chimera hs pcre)
target_link_libraries(chimera hs pcre)
install(TARGETS chimera DESTINATION ${CMAKE_INSTALL_LIBDIR})
# expand out library names for pkgconfig static link info
foreach (LIB ${CMAKE_CXX_IMPLICIT_LINK_LIBRARIES})
# this is fragile, but protects us from toolchain specific files
if (NOT EXISTS ${LIB})
set(PRIVATE_LIBS "${PRIVATE_LIBS} -l${LIB}")
endif()
endforeach()
set(PRIVATE_LIBS "${PRIVATE_LIBS} -L${LIBDIR} -lpcre")
configure_file(libch.pc.in libch.pc @ONLY) # only replace @ quoted vars
install(FILES ${CMAKE_BINARY_DIR}/chimera/libch.pc
DESTINATION "${CMAKE_INSTALL_LIBDIR}/pkgconfig")

45
chimera/ch.h
View File

@ -1,45 +0,0 @@
/*
* Copyright (c) 2018, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#ifndef CH_H_
#define CH_H_
/**
* @file
* @brief The complete Chimera API definition.
*
* Chimera is a hybrid solution of Hyperscan and PCRE.
*
* This header includes both the Chimera compiler and runtime components. See
* the individual component headers for documentation.
*/
#include "ch_compile.h"
#include "ch_runtime.h"
#endif /* CH_H_ */

109
chimera/ch_alloc.c
View File

@ -1,109 +0,0 @@
/*
* Copyright (c) 2018, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/** \file
* \brief Runtime functions for setting custom allocators.
*/
#include "ch.h"
#include "ch_common.h"
#include "ch_internal.h"
#include "hs.h"
#include "ue2common.h"
#define default_malloc malloc
#define default_free free
ch_alloc_t ch_database_alloc = default_malloc;
ch_alloc_t ch_misc_alloc = default_malloc;
ch_alloc_t ch_scratch_alloc = default_malloc;
ch_free_t ch_database_free = default_free;
ch_free_t ch_misc_free = default_free;
ch_free_t ch_scratch_free = default_free;
static
ch_alloc_t normalise_alloc(ch_alloc_t a) {
if (!a) {
return default_malloc;
} else {
return a;
}
}
static
ch_free_t normalise_free(ch_free_t f) {
if (!f) {
return default_free;
} else {
return f;
}
}
HS_PUBLIC_API
ch_error_t HS_CDECL ch_set_allocator(ch_alloc_t allocfunc,
ch_free_t freefunc) {
ch_set_database_allocator(allocfunc, freefunc);
ch_set_misc_allocator(allocfunc, freefunc);
ch_set_scratch_allocator(allocfunc, freefunc);
// Set core Hyperscan alloc/free.
hs_error_t ret = hs_set_allocator(allocfunc, freefunc);
return ret;
}
HS_PUBLIC_API
ch_error_t HS_CDECL ch_set_database_allocator(ch_alloc_t allocfunc,
ch_free_t freefunc) {
ch_database_alloc = normalise_alloc(allocfunc);
ch_database_free = normalise_free(freefunc);
// Set Hyperscan database alloc/free.
return hs_set_database_allocator(allocfunc, freefunc);
}
HS_PUBLIC_API
ch_error_t HS_CDECL ch_set_misc_allocator(ch_alloc_t allocfunc,
ch_free_t freefunc) {
ch_misc_alloc = normalise_alloc(allocfunc);
ch_misc_free = normalise_free(freefunc);
// Set Hyperscan misc alloc/free.
return hs_set_misc_allocator(allocfunc, freefunc);
}
HS_PUBLIC_API
ch_error_t HS_CDECL ch_set_scratch_allocator(ch_alloc_t allocfunc,
ch_free_t freefunc) {
ch_scratch_alloc = normalise_alloc(allocfunc);
ch_scratch_free = normalise_free(freefunc);
// Set Hyperscan scratch alloc/free.
return hs_set_scratch_allocator(allocfunc, freefunc);
}

65
chimera/ch_alloc.h
View File

@ -1,65 +0,0 @@
/*
* Copyright (c) 2018, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#ifndef CH_ALLOC_H
#define CH_ALLOC_H
#include "hs_common.h"
#include "ue2common.h"
#include "ch_common.h"
#ifdef __cplusplus
extern "C"
{
#endif
extern hs_alloc_t ch_database_alloc;
extern hs_alloc_t ch_misc_alloc;
extern hs_alloc_t ch_scratch_alloc;
extern hs_free_t ch_database_free;
extern hs_free_t ch_misc_free;
extern hs_free_t ch_scratch_free;
#ifdef __cplusplus
} /* extern C */
#endif
/** \brief Check the results of an alloc done with hs_alloc for alignment.
*
* If we have incorrect alignment, return an error. Caller should free the
* offending block. */
static really_inline
ch_error_t ch_check_alloc(const void *mem) {
ch_error_t ret = CH_SUCCESS;
if (!mem) {
ret = CH_NOMEM;
} else if (!ISALIGNED_N(mem, alignof(unsigned long long))) {
ret = CH_BAD_ALLOC;
}
return ret;
}
#endif

370
chimera/ch_common.h
View File

@ -1,370 +0,0 @@
/*
* Copyright (c) 2018-2020, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#ifndef CH_COMMON_H_
#define CH_COMMON_H_
#include "hs_common.h"
#include <stdlib.h>
/**
* @file
* @brief The Chimera common API definition.
*
* Chimera is a hybrid of Hyperscan and PCRE.
*
* This header contains functions available to both the Chimera compiler and
* runtime.
*/
#ifdef __cplusplus
extern "C"
{
#endif
struct ch_database;
/**
* A Chimera pattern database.
*
* Generated by one of the Chimera compiler functions:
* - @ref ch_compile()
* - @ref ch_compile_multi()
* - @ref ch_compile_ext_multi()
*/
typedef struct ch_database ch_database_t;
/**
* A type for errors returned by Chimera functions.
*/
typedef int ch_error_t;
/**
* Free a compiled pattern database.
*
* The free callback set by @ref ch_set_allocator()) will be used by this
* function.
*
* @param db
* A compiled pattern database. NULL may also be safely provided, in which
* case the function does nothing.
*
* @return
* @ref CH_SUCCESS on success, other values on failure.
*/
ch_error_t HS_CDECL ch_free_database(ch_database_t *db);
/**
* Utility function for identifying this release version.
*
* @return
* A string containing the version number of this release build and the
* date of the build. It is allocated statically, so it does not need to
* be freed by the caller.
*/
const char * HS_CDECL ch_version(void);
/**
* Returns the size of the given database.
*
* @param database
* Pointer to compiled expression database.
*
* @param database_size
* On success, the size of the compiled database in bytes is placed in this
* parameter.
*
* @return
* @ref CH_SUCCESS on success, other values on failure.
*/
ch_error_t HS_CDECL ch_database_size(const ch_database_t *database,
size_t *database_size);
/**
* Utility function providing information about a database.
*
* @param database
* Pointer to a compiled database.
*
* @param info
* On success, a string containing the version and platform information for
* the supplied database is placed in the parameter. The string is
* allocated using the allocator supplied in @ref hs_set_allocator()
* (or malloc() if no allocator was set) and should be freed by the caller.
*
* @return
* @ref CH_SUCCESS on success, other values on failure.
*/
ch_error_t HS_CDECL ch_database_info(const ch_database_t *database,
char **info);
/**
* The type of the callback function that will be used by Chimera to allocate
* more memory at runtime as required.
*
* If Chimera is to be used in a multi-threaded, or similarly concurrent
* environment, the allocation function will need to be re-entrant, or
* similarly safe for concurrent use.
*
* @param size
* The number of bytes to allocate.
* @return
* A pointer to the region of memory allocated, or NULL on error.
*/
typedef void *(HS_CDECL *ch_alloc_t)(size_t size);
/**
* The type of the callback function that will be used by Chimera to free
* memory regions previously allocated using the @ref ch_alloc_t function.
*
* @param ptr
* The region of memory to be freed.
*/
typedef void (HS_CDECL *ch_free_t)(void *ptr);
/**
* Set the allocate and free functions used by Chimera for allocating
* memory at runtime for stream state, scratch space, database bytecode,
* and various other data structure returned by the Chimera API.
*
* The function is equivalent to calling @ref ch_set_scratch_allocator(),
* @ref ch_set_database_allocator() and
* @ref ch_set_misc_allocator() with the provided parameters.
*
* This call will override any previous allocators that have been set.
*
* Note: there is no way to change the allocator used for temporary objects
* created during the various compile calls (@ref ch_compile() and @ref
* ch_compile_multi()).
*
* @param alloc_func
* A callback function pointer that allocates memory. This function must
* return memory suitably aligned for the largest representable data type
* on this platform.
*
* @param free_func
* A callback function pointer that frees allocated memory.
*
* @return
* @ref CH_SUCCESS on success, other values on failure.
*/
ch_error_t HS_CDECL ch_set_allocator(ch_alloc_t alloc_func,
ch_free_t free_func);
/**
* Set the allocate and free functions used by Chimera for allocating memory
* for database bytecode produced by the compile calls (@ref ch_compile() and @ref
* ch_compile_multi()).
*
* If no database allocation functions are set, or if NULL is used in place of
* both parameters, then memory allocation will default to standard methods
* (such as the system malloc() and free() calls).
*
* This call will override any previous database allocators that have been set.
*
* Note: the database allocator may also be set by calling @ref
* ch_set_allocator().
*
* Note: there is no way to change how temporary objects created during the
* various compile calls (@ref ch_compile() and @ref ch_compile_multi()) are
* allocated.
*
* @param alloc_func
* A callback function pointer that allocates memory. This function must
* return memory suitably aligned for the largest representable data type
* on this platform.
*
* @param free_func
* A callback function pointer that frees allocated memory.
*
* @return
* @ref HS_SUCCESS on success, other values on failure.
*/
ch_error_t HS_CDECL ch_set_database_allocator(ch_alloc_t alloc_func,
ch_free_t free_func);
/**
* Set the allocate and free functions used by Chimera for allocating memory
* for items returned by the Chimera API such as @ref ch_compile_error_t.
*
* If no misc allocation functions are set, or if NULL is used in place of both
* parameters, then memory allocation will default to standard methods (such as
* the system malloc() and free() calls).
*
* This call will override any previous misc allocators that have been set.
*
* Note: the misc allocator may also be set by calling @ref ch_set_allocator().
*
* @param alloc_func
* A callback function pointer that allocates memory. This function must
* return memory suitably aligned for the largest representable data type
* on this platform.
*
* @param free_func
* A callback function pointer that frees allocated memory.
*
* @return
* @ref CH_SUCCESS on success, other values on failure.
*/
ch_error_t HS_CDECL ch_set_misc_allocator(ch_alloc_t alloc_func,
ch_free_t free_func);
/**
* Set the allocate and free functions used by Chimera for allocating memory
* for scratch space by @ref ch_alloc_scratch() and @ref ch_clone_scratch().
*
* If no scratch allocation functions are set, or if NULL is used in place of
* both parameters, then memory allocation will default to standard methods
* (such as the system malloc() and free() calls).
*
* This call will override any previous scratch allocators that have been set.
*
* Note: the scratch allocator may also be set by calling @ref
* ch_set_allocator().
*
* @param alloc_func
* A callback function pointer that allocates memory. This function must
* return memory suitably aligned for the largest representable data type
* on this platform.
*
* @param free_func
* A callback function pointer that frees allocated memory.
*
* @return
* @ref CH_SUCCESS on success, other values on failure.
*/
ch_error_t HS_CDECL ch_set_scratch_allocator(ch_alloc_t alloc_func,
ch_free_t free_func);
/**
* @defgroup CH_ERROR ch_error_t values
*
* @{
*/
/**
* The engine completed normally.
*/
#define CH_SUCCESS 0
/**
* A parameter passed to this function was invalid.
*/
#define CH_INVALID (-1)
/**
* A memory allocation failed.
*/
#define CH_NOMEM (-2)
/**
* The engine was terminated by callback.
*
* This return value indicates that the target buffer was partially scanned,
* but that the callback function requested that scanning cease after a match
* was located.
*/
#define CH_SCAN_TERMINATED (-3)
/**
* The pattern compiler failed, and the @ref ch_compile_error_t should be
* inspected for more detail.
*/
#define CH_COMPILER_ERROR (-4)
/**
* The given database was built for a different version of the Chimera matcher.
*/
#define CH_DB_VERSION_ERROR (-5)
/**
* The given database was built for a different platform (i.e., CPU type).
*/
#define CH_DB_PLATFORM_ERROR (-6)
/**
* The given database was built for a different mode of operation. This error
* is returned when streaming calls are used with a non-streaming database and
* vice versa.
*/
#define CH_DB_MODE_ERROR (-7)
/**
* A parameter passed to this function was not correctly aligned.
*/
#define CH_BAD_ALIGN (-8)
/**
* The memory allocator did not correctly return memory suitably aligned for
* the largest representable data type on this platform.
*/
#define CH_BAD_ALLOC (-9)
/**
* The scratch region was already in use.
*
* This error is returned when Chimera is able to detect that the scratch
* region given is already in use by another Chimera API call.
*
* A separate scratch region, allocated with @ref ch_alloc_scratch() or @ref
* ch_clone_scratch(), is required for every concurrent caller of the Chimera
* API.
*
* For example, this error might be returned when @ref ch_scan() has been
* called inside a callback delivered by a currently-executing @ref ch_scan()
* call using the same scratch region.
*
* Note: Not all concurrent uses of scratch regions may be detected. This error
* is intended as a best-effort debugging tool, not a guarantee.
*/
#define CH_SCRATCH_IN_USE (-10)
/**
* Unexpected internal error from Hyperscan.
*
* This error indicates that there was unexpected matching behaviors from
* Hyperscan. This could be related to invalid usage of scratch space or
* invalid memory operations by users.
*
*/
#define CH_UNKNOWN_HS_ERROR (-13)
/**
* Returned when pcre_exec (called for some expressions internally from @ref
* ch_scan) failed due to a fatal error.
*/
#define CH_FAIL_INTERNAL (-32)
/** @} */
#ifdef __cplusplus
} /* extern "C" */
#endif
#endif /* CH_COMMON_H_ */

877
chimera/ch_compile.cpp
View File

@ -1,877 +0,0 @@
/*
* Copyright (c) 2018, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/** \file
* \brief Compiler front-end, including public API calls for compilation.
*/
#include "ch_compile.h"
#include "ch_alloc.h"
#include "ch_internal.h"
#include "ch_database.h"
#include "grey.h"
#include "hs_common.h"
#include "hs_internal.h"
#include "ue2common.h"
#include "util/compile_error.h"
#include "util/multibit_build.h"
#include "util/target_info.h"
#include <algorithm>
#include <cassert>
#include <cstddef>
#include <cstring>
#include <memory>
#include <ostream>
#include <sstream>
#include <limits.h>
#include <string>
#include <vector>
#include <boost/core/noncopyable.hpp>
#define PCRE_ERROR_MSG "Internal error building PCRE pattern."
using namespace std;
using namespace ue2;
static const char failureNoMemory[] = "Unable to allocate memory.";
static const char failureInternal[] = "Internal error.";
static const char failureBadAlloc[] = "Allocator returned misaligned memory.";
static const ch_compile_error_t ch_enomem
= { const_cast<char *>(failureNoMemory), 0 };
static const ch_compile_error_t ch_einternal
= { const_cast<char *>(failureInternal), 0 };
static const ch_compile_error_t ch_badalloc
= { const_cast<char *>(failureBadAlloc), 0 };
static
ch_compile_error_t *generateChimeraCompileError(const string &err,
int expression) {
ch_compile_error_t *ret =
(struct ch_compile_error *)ch_misc_alloc(sizeof(ch_compile_error_t));
if (ret) {
ch_error_t e = ch_check_alloc(ret);
if (e != CH_SUCCESS) {
ch_misc_free(ret);
return const_cast<ch_compile_error_t *>(&ch_badalloc);
}
char *msg = (char *)ch_misc_alloc(err.size() + 1);
if (msg) {
e = ch_check_alloc(msg);
if (e != HS_SUCCESS) {
ch_misc_free(msg);
return const_cast<ch_compile_error_t *>(&ch_badalloc);
}
memcpy(msg, err.c_str(), err.size() + 1);
ret->message = msg;
} else {
ch_misc_free(ret);
ret = nullptr;
}
}
if (!ret || !ret->message) {
return const_cast<ch_compile_error_t *>(&ch_enomem);
}
ret->expression = expression;
return ret;
}
static
void freeChimeraCompileError(ch_compile_error_t *error) {
if (!error) {
return;
}
if (error == &ch_enomem || error == &ch_einternal ||
error == &ch_badalloc) {
// These are not allocated.
return;
}
ch_misc_free(error->message);
ch_misc_free(error);
}
static
bool checkMode(unsigned int mode, ch_compile_error_t **comp_error) {
static const unsigned int supported = CH_MODE_GROUPS;
if (mode & ~supported) {
*comp_error =
generateChimeraCompileError("Invalid mode flag supplied.", -1);
return false;
}
return true;
}
/** \brief Throw a compile error if we're passed some unsupported flags. */
static
void checkFlags(const unsigned int flags) {
static const unsigned int supported = HS_FLAG_DOTALL
| HS_FLAG_MULTILINE
| HS_FLAG_CASELESS
| HS_FLAG_SINGLEMATCH
| HS_FLAG_UCP
| HS_FLAG_UTF8;
if (flags & ~supported) {
throw CompileError("Unrecognized flag used.");
}
}
static
bool isHyperscanSupported(const char *expression, unsigned int flags,
const hs_platform_info *platform) {
hs_database_t *db = nullptr;
hs_compile_error *comp_error = nullptr;
unsigned int id = 0;
hs_error_t err = hs_compile_multi(&expression, &flags, &id,
1, HS_MODE_BLOCK, platform, &db,
&comp_error);
if (err != HS_SUCCESS) {
assert(!db);
assert(comp_error);
DEBUG_PRINTF("unsupported: %s\n", comp_error->message);
hs_free_compile_error(comp_error);
return false;
}
assert(db);
assert(!comp_error);
hs_free_database(db);
return true;
}
static
bool writeHyperscanDatabase(char *ptr, hs_database_t *db) {
// Note: we must use our serialization calls to re-home the database.
char *serialized = nullptr;
size_t slen = 0;
hs_error_t err = hs_serialize_database(db, &serialized, &slen);
if (err != HS_SUCCESS) {
DEBUG_PRINTF("hs_serialize_database returned %d\n", err);
assert(0);
return false;
}
DEBUG_PRINTF("writing database to ptr %p\n", ptr);
// deserialize_at without the platform tests.
err = hs_deserialize_database_at(serialized, slen, (hs_database_t *)ptr);
if (err != HS_SUCCESS) {
DEBUG_PRINTF("hs_deserialize_database_at returned %d\n", err);
assert(0);
ch_misc_free(serialized);
return false;
}
ch_misc_free(serialized);
return true;
}
static
bool writeHyperscanDatabase(ch_bytecode *db, hs_database_t *hs_db) {
db->databaseOffset = ROUNDUP_CL(sizeof(*db));
char *ptr = (char *)db + db->databaseOffset;
return writeHyperscanDatabase(ptr, hs_db);
}
static
int convertFlagsToPcreOptions(unsigned int flags) {
int options = 0;
if (flags & HS_FLAG_CASELESS) {
options |= PCRE_CASELESS;
}
if (flags & HS_FLAG_DOTALL) {
options |= PCRE_DOTALL;
}
if (flags & HS_FLAG_MULTILINE) {
options |= PCRE_MULTILINE;
}
if (flags & HS_FLAG_UTF8) {
options |= PCRE_UTF8;
}
if (flags & HS_FLAG_UCP) {
options |= PCRE_UCP;
}
// All other flags are meaningless to PCRE.
return options;
}
namespace {
/** \brief Data about a single pattern. */
struct PatternData : boost::noncopyable {
PatternData(const char *pattern, u32 flags, u32 idx, u32 id_in,
unsigned mode, unsigned long int match_limit,
unsigned long int match_limit_recursion,
const hs_platform_info *platform);
~PatternData() {
pcre_free(compiled);
pcre_free(extra);
}
void buildPcre(const char *pattern, u32 flags);
size_t patternSize() const;
void writePattern(ch_pattern *pattern) const;
pcre *compiled; //!< pcre_compile output
pcre_extra *extra; //!< pcre_study output
size_t compiled_size;
int study_size;
int capture_cnt;
bool utf8;
u32 id; //!< ID from the user
u32 expr_index; //!< index in the expression array
bool singlematch; //!< pattern is in highlander mode
bool guard; //!< this pattern should be guarded by the multimatcher
u32 minWidth; //!< min match width
u32 maxWidth; //!< max match width
u32 fixedWidth; //!< fixed pattern width
unsigned long int matchLimit; //! pcre match limit
unsigned long int matchLimitRecursion; //! pcre match_limit_recursion
};
PatternData::PatternData(const char *pattern, u32 flags, u32 idx, u32 id_in,
unsigned mode, unsigned long int match_limit,
unsigned long int match_limit_recursion,
const hs_platform_info *platform)
: compiled(nullptr), extra(nullptr), id(id_in), expr_index(idx),
singlematch(flags & HS_FLAG_SINGLEMATCH),
guard(false), minWidth(0), maxWidth(UINT_MAX),
fixedWidth(UINT_MAX), matchLimit(match_limit),
matchLimitRecursion(match_limit_recursion) {
assert(pattern);
flags |= HS_FLAG_ALLOWEMPTY; /* don't hand things off to pcre for no
reason */
buildPcre(pattern, flags);
// Fetch the expression info for a prefiltering, non-singlematch version of
// this pattern, if possible.
hs_expr_info *info = nullptr;
hs_compile_error_t *error = nullptr;
u32 infoflags = (flags | HS_FLAG_PREFILTER) & ~HS_FLAG_SINGLEMATCH;
u32 rawflags = (flags | HS_FLAG_SOM_LEFTMOST) & ~HS_FLAG_SINGLEMATCH;
hs_error_t err = hs_expression_info(pattern, infoflags, &info, &error);
if (err == HS_SUCCESS) {
assert(info);
hs_expr_info *i = (hs_expr_info *)info;
minWidth = i->min_width;
maxWidth = i->max_width;
bool ordered = i->unordered_matches ? false : true;
// Only enable capturing if required
u32 captureCnt = 0;
if (mode & CH_MODE_GROUPS) {
captureCnt = capture_cnt;
}
// No need to confirm with PCRE if:
// 1) pattern is fixed width
// 2) pattern isn't vacuous as it can't combine with start of match
// 3) no capturing in this pattern
// 4) no offset adjust in this pattern as hyperscan match callback
// will arrive without order, i.e. [^a]\z has offset adjust
// 5) hyperscan compile succeeds without prefiltering
if (minWidth == maxWidth && minWidth && maxWidth != UINT_MAX &&
!captureCnt && ordered &&
isHyperscanSupported(pattern, rawflags, platform)) {
fixedWidth = maxWidth;
}
DEBUG_PRINTF("gathered info: widths=[%u,%u]\n", minWidth, maxWidth);
ch_misc_free(info);
u32 guardflags;
guardflags = flags | HS_FLAG_PREFILTER;
guard = isHyperscanSupported(pattern, guardflags, platform);
} else {
// We can't even prefilter this pattern, so we're dependent on Big Dumb
// Pcre Scans.
DEBUG_PRINTF("hs_expression_info failed, falling back to pcre\n");
hs_free_compile_error(error);
}
}
void PatternData::buildPcre(const char *pattern, u32 flags) {
int options = convertFlagsToPcreOptions(flags);
const char *errptr = nullptr;
int erroffset = 0;
compiled = pcre_compile(pattern, options, &errptr, &erroffset, nullptr);
if (!compiled) {
DEBUG_PRINTF("PCRE failed to compile: %s\n", pattern);
string err("PCRE compilation failed: ");
err += string(errptr);
err += ".";
throw CompileError(expr_index, err);
}
extra = pcre_study(compiled, PCRE_STUDY_JIT_COMPILE, &errptr);
// Note that it's OK for pcre_study to return NULL if there's nothing
// to be found, but a non-NULL error is always bad.
if (errptr) {
DEBUG_PRINTF("PCRE could not be studied: %s\n", errptr);
string err("PCRE compilation failed: ");
err += string(errptr);
err += ".";
throw CompileError(expr_index, err);
}
if (pcre_fullinfo(compiled, extra, PCRE_INFO_SIZE, &compiled_size)) {
throw CompileError(PCRE_ERROR_MSG);
}
if (!extra) {
study_size = 0;
} else {
if (pcre_fullinfo(compiled, extra, PCRE_INFO_STUDYSIZE, &study_size)) {
throw CompileError(PCRE_ERROR_MSG);
}
}
if (pcre_fullinfo(compiled, extra, PCRE_INFO_CAPTURECOUNT, &capture_cnt)) {
throw CompileError(PCRE_ERROR_MSG);
}
/* We use the pcre rather than hs to get this information as we may need it
* even in the pure unguarded pcre mode where there is no hs available. We
* can not use the compile flags due to (*UTF8) verb */
unsigned long int opts = 0; // PCRE_INFO_OPTIONS demands an unsigned long
if (pcre_fullinfo(compiled, extra, PCRE_INFO_OPTIONS, &opts)) {
throw CompileError(PCRE_ERROR_MSG);
}
utf8 = opts & PCRE_UTF8;
}
size_t PatternData::patternSize() const {
size_t len = 0;
// ch_pattern header.
len += sizeof(ch_pattern);
len = ROUNDUP_N(len, 8);
DEBUG_PRINTF("compiled pcre at %zu\n", len);
len += compiled_size;
// PCRE study data, which may be zero.
if (study_size) {
len = ROUNDUP_N(len, 8);
DEBUG_PRINTF("study at %zu\n", len);
len += (size_t)study_size;
}
DEBUG_PRINTF("pattern size %zu\n", len);
return len;
}
/** \brief Write out an ch_pattern structure, which should already be sized
* correctly according to PatternData::patternSize. */
void PatternData::writePattern(ch_pattern *pattern) const {
assert(pattern);
assert(ISALIGNED_CL(pattern));
pattern->id = id;
u32 flags = 0;
if (singlematch) {
flags |= CHIMERA_PATTERN_FLAG_SINGLEMATCH;
}
if (utf8) {
flags |= CHIMERA_PATTERN_FLAG_UTF8;
}
pattern->flags = flags;
pattern->maxWidth = maxWidth;
pattern->minWidth = minWidth == UINT_MAX ? 0 : minWidth;
pattern->fixedWidth = fixedWidth;
// Compiled PCRE pattern.
char *ptr = (char *)pattern;
ptr += ROUNDUP_N(sizeof(*pattern), 8);
DEBUG_PRINTF("compiled pcre at %zu\n", (size_t)(ptr - (char *)pattern));
memcpy(ptr, compiled, compiled_size);
ptr += compiled_size;
// PCRE match limits
pattern->extra.flags = PCRE_EXTRA_MATCH_LIMIT |
PCRE_EXTRA_MATCH_LIMIT_RECURSION;
pattern->extra.match_limit = matchLimit ? matchLimit : 10000000;
// Set to avoid segment fault
pattern->extra.match_limit_recursion =
matchLimitRecursion ? matchLimitRecursion : 1500;
// PCRE study_data.
u32 studyOffset = 0;
if (extra) {
assert(extra->study_data);
ptr = ROUNDUP_PTR(ptr, 8);
DEBUG_PRINTF("study at %zu\n", (size_t)(ptr - (char *)pattern));
memcpy(ptr, extra->study_data, study_size);
studyOffset = (size_t)(ptr - (char *)pattern);
pattern->extra.flags |= PCRE_EXTRA_STUDY_DATA;
pattern->extra.study_data = ptr;
ptr += study_size;
} else {
pattern->extra.flags &= ~PCRE_EXTRA_STUDY_DATA;
}
pattern->studyOffset = studyOffset;
size_t pcreLen = (ptr - (char *)pattern);
assert(pcreLen <= patternSize());
pattern->length = (u32)pcreLen;
// We shouldn't overrun the space we've allocated for this pattern.
assert(patternSize() >= (size_t)(ptr - (char *)pattern));
}
} // namespace
namespace ch {
static
void ch_compile_multi_int(const char *const *expressions, const unsigned *flags,
const unsigned *ids, unsigned elements,
unsigned mode, unsigned long int match_limit,
unsigned long int match_limit_recursion,
const hs_platform_info_t *platform,
ch_database_t **out) {
vector<unique_ptr<PatternData>> pcres;
pcres.reserve(elements);
vector<u32> unguarded; // indices of unguarded PCREs.
vector<const char *> multiExpr;
vector<unsigned int> multiFlags;
vector<unsigned int> multiIds;
bool allConfirm = true;
bool allSingleMatch = true;
for (unsigned int i = 0; i < elements; i++) {
const char *myExpr = expressions[i];
unsigned int myFlags = flags ? flags[i] : 0;
unsigned int myId = ids ? ids[i] : 0;
checkFlags(myFlags);
// First, build with libpcre. A build failure from libpcre will throw
// an exception up to the caller.
auto patternData =
std::make_unique<PatternData>(myExpr, myFlags, i, myId, mode, match_limit,
match_limit_recursion, platform);
pcres.push_back(move(patternData));
PatternData &curr = *pcres.back();
if (!(myFlags & HS_FLAG_SINGLEMATCH)) {
allSingleMatch = false;
}
// in the multimatch, we always run in prefilter mode and accept vacuous
// patterns.
myFlags |=
HS_FLAG_ALLOWEMPTY | HS_FLAG_PREFILTER;
if (curr.fixedWidth != UINT_MAX) {
myFlags |= HS_FLAG_SOM_LEFTMOST;
DEBUG_PRINTF("fixed width, turn off prefiltering\n");
myFlags &= ~HS_FLAG_PREFILTER;
allConfirm = false;
// Single match can't coexist with SOM.
myFlags &= ~HS_FLAG_SINGLEMATCH;
}
if (curr.guard) {
// We use the index into the PCREs array as the Hyperscan idx.
multiExpr.push_back(myExpr);
multiFlags.push_back(myFlags);
multiIds.push_back(i);
} else {
// No Hyperscan support, PCRE is unguarded.
unguarded.push_back(i);
}
}
DEBUG_PRINTF("built %zu PCREs, %zu of which are unguarded\n",
pcres.size(), unguarded.size());
// Work out our sizing for the output database.
size_t patternSize = 0;
for (unsigned int i = 0; i < elements; i++) {
size_t len = pcres[i]->patternSize();
patternSize += ROUNDUP_CL(len);
}
DEBUG_PRINTF("pcre bytecode takes %zu bytes\n", patternSize);
bool noMulti = multiExpr.empty();
size_t multiSize = 0;
hs_database *multidb = nullptr;
if (!noMulti) {
hs_compile_error_t *hs_comp_error = nullptr;
hs_error_t err = hs_compile_multi(&multiExpr[0], &multiFlags[0],
&multiIds[0], multiExpr.size(),
HS_MODE_BLOCK, platform, &multidb,
&hs_comp_error);
if (err != HS_SUCCESS) {
assert(hs_comp_error);
DEBUG_PRINTF("hs_compile_multi returned error: %s\n",
hs_comp_error->message);
assert(0);
hs_free_compile_error(hs_comp_error);
throw CompileError("Internal error.");
}
assert(multidb);
err = hs_database_size(multidb, &multiSize);
if (err != HS_SUCCESS) {
assert(0);
throw CompileError("Internal error.");
}
DEBUG_PRINTF("built hyperscan database with len %zu bytes\n", multiSize);
}
size_t bytecodeLen = sizeof(ch_bytecode) +
multiSize + alignof(u32) +
(sizeof(u32) * unguarded.size()) +
(sizeof(u32) * elements) +
patternSize +
128; // padding for alignment
size_t totalSize = sizeof(ch_database) + bytecodeLen;
DEBUG_PRINTF("allocating %zu bytes for database\n", totalSize);
char *ptr = (char *)ch_database_alloc(totalSize);
if (ch_check_alloc(ptr) != CH_SUCCESS) {
ch_database_free(ptr);
throw std::bad_alloc();
}
memset(ptr, 0, totalSize);
// First, the header.
ch_database *hydb = (ch_database *)ptr;
hydb->magic = CH_DB_MAGIC;
hydb->version = HS_VERSION_32BIT;
hydb->length = bytecodeLen;
// Then, the bytecode.
size_t shift = (size_t)hydb->bytes & 0x3f;
hydb->bytecode = offsetof(struct ch_database, bytes) - shift;
ch_bytecode *db = (ch_bytecode *)((char *)hydb + hydb->bytecode);
db->patternCount = elements;
db->activeSize = mmbit_size(elements);
db->flags = 0;
db->length = bytecodeLen;
if (noMulti) {
db->flags |= CHIMERA_FLAG_NO_MULTIMATCH;
}
if (mode & CH_MODE_GROUPS) {
db->flags |= CHIMERA_FLAG_GROUPS;
}
if (allConfirm) {
db->flags |= CHIMERA_FLAG_ALL_CONFIRM;
}
if (allSingleMatch) {
db->flags |= CHIMERA_FLAG_ALL_SINGLE;
}
// Find and set the max ovector size by looking at the capture count for
// each pcre.
u32 maxCaptureGroups = 0;
for (unsigned int i = 0; i < elements; i++) {
maxCaptureGroups = max(maxCaptureGroups, (u32)pcres[i]->capture_cnt);
}
db->maxCaptureGroups = maxCaptureGroups;
DEBUG_PRINTF("max capture groups is %u\n", maxCaptureGroups);
if (!noMulti) {
DEBUG_PRINTF("write hyperscan database\n");
// Write Hyperscan database directly after the header struct, then free it.
if (!writeHyperscanDatabase(db, multidb)) {
ch_database_free(hydb);
hs_free_database(multidb);
throw CompileError("Internal error.");
}
hs_free_database(multidb);
} else {
db->databaseOffset = ROUNDUP_CL(sizeof(*db));
}
// Then, write our unguarded PCRE list.
db->unguardedCount = unguarded.size();
db->unguardedOffset = ROUNDUP_N(db->databaseOffset + multiSize, 4);
ptr = (char *)db + db->unguardedOffset;
copy(unguarded.begin(), unguarded.end(), (u32 *)ptr);
// Then, write all our compiled PCRE patterns and the lookup table for
// them.
db->patternOffset = db->unguardedOffset + unguarded.size() * sizeof(u32);
u32 *patternOffset = (u32 *)((char *)db + db->patternOffset);
u32 offset = ROUNDUP_CL(db->patternOffset + elements * sizeof(u32));
for (unsigned int i = 0; i < elements; i++) {
*patternOffset = offset;
size_t len = pcres[i]->patternSize();
ptr = (char *)db + offset;
struct ch_pattern *pattern = (struct ch_pattern *)ptr;
pcres[i]->writePattern(pattern);
DEBUG_PRINTF("wrote pcre %u into offset %u, len %zu\n", i, offset, len);
offset += ROUNDUP_CL(len);
patternOffset++;
}
assert(offset <= totalSize);
assert(hydb->magic == CH_DB_MAGIC);
DEBUG_PRINTF("built hybrid database, size %zu bytes\n", totalSize);
DEBUG_PRINTF("offset=%u\n", offset);
*out = hydb;
}
} // namespace ch
extern "C" HS_PUBLIC_API
ch_error_t HS_CDECL ch_compile(const char *expression, unsigned flags,
unsigned mode,
const hs_platform_info_t *platform,
ch_database_t **db,
ch_compile_error_t **comp_error) {
if (!comp_error) {
if (db) {
db = nullptr;
}
// nowhere to write the string, but we can still report an error code
return CH_COMPILER_ERROR;
}
if (!db) {
*comp_error =
generateChimeraCompileError("Invalid parameter: db is NULL", -1);
return CH_COMPILER_ERROR;
}
if (!expression) {
*db = nullptr;
*comp_error =
generateChimeraCompileError("Invalid parameter: expressions is\
NULL", -1);
return CH_COMPILER_ERROR;
}
if (!checkMode(mode, comp_error)) {
*db = nullptr;
assert(*comp_error); // set by checkMode
return CH_COMPILER_ERROR;
}
try {
unsigned id = 0; // single expressions get zero as an ID
// Internal function to do all the work, now that we've handled all the
// argument checking.
ch::ch_compile_multi_int(&expression, &flags, &id, 1, mode, 0, 0,
platform, db);
}
catch (const CompileError &e) {
// Compiler error occurred
*db = nullptr;
*comp_error = generateChimeraCompileError(e.reason, e.hasIndex ?
(int)e.index : -1);
return CH_COMPILER_ERROR;
}
catch (std::bad_alloc &) {
*db = nullptr;
*comp_error = const_cast<ch_compile_error_t *>(&ch_enomem);
return CH_COMPILER_ERROR;
}
catch (...) {
assert(!"Internal error, unexpected exception");
*db = nullptr;
*comp_error = const_cast<ch_compile_error_t *>(&ch_einternal);
return CH_COMPILER_ERROR;
}
DEBUG_PRINTF("success!\n");
return CH_SUCCESS;
}
extern "C" HS_PUBLIC_API
ch_error_t HS_CDECL ch_compile_multi(const char *const *expressions,
const unsigned *flags, const unsigned *ids,
unsigned elements, unsigned mode,
const hs_platform_info_t *platform,
ch_database_t **db,
ch_compile_error_t **comp_error) {
if (!comp_error) {
if (db) {
db = nullptr;
}
// nowhere to write the string, but we can still report an error code
return CH_COMPILER_ERROR;
}
if (!db) {
*comp_error =
generateChimeraCompileError("Invalid parameter: db is NULL", -1);
return CH_COMPILER_ERROR;
}
if (!expressions) {
*db = nullptr;
*comp_error =
generateChimeraCompileError("Invalid parameter: expressions is\
NULL", -1);
return CH_COMPILER_ERROR;
}
if (!elements) {
*db = nullptr;
*comp_error = generateChimeraCompileError("Invalid parameter:\
elements is zero", -1);
return CH_COMPILER_ERROR;
}
if (!checkMode(mode, comp_error)) {
*db = nullptr;
assert(*comp_error); // set by checkMode
return CH_COMPILER_ERROR;
}
try {
// Internal function to do all the work, now that we've handled all the
// argument checking.
ch::ch_compile_multi_int(expressions, flags, ids, elements, mode, 0, 0,
platform, db);
}
catch (const CompileError &e) {
// Compiler error occurred
*db = nullptr;
*comp_error = generateChimeraCompileError(e.reason, e.hasIndex ?
(int)e.index : -1);
return CH_COMPILER_ERROR;
}
catch (std::bad_alloc &) {
*db = nullptr;
*comp_error = const_cast<ch_compile_error_t *>(&ch_enomem);
return CH_COMPILER_ERROR;
}
catch (...) {
assert(!"Internal error, unexpected exception");
*db = nullptr;
*comp_error = const_cast<ch_compile_error_t *>(&ch_einternal);
return CH_COMPILER_ERROR;
}
DEBUG_PRINTF("success!\n");
return CH_SUCCESS;
}
extern "C" HS_PUBLIC_API
ch_error_t HS_CDECL ch_compile_ext_multi(
const char *const *expressions,
const unsigned *flags,
const unsigned *ids,
unsigned elements, unsigned mode,
unsigned long int match_limit,
unsigned long int match_limit_recursion,
const hs_platform_info_t *platform,
ch_database_t **db,
ch_compile_error_t **comp_error) {
if (!comp_error) {
if (db) {
db = nullptr;
}
// nowhere to write the string, but we can still report an error code
return CH_COMPILER_ERROR;
}
if (!db) {
*comp_error =
generateChimeraCompileError("Invalid parameter: db is NULL", -1);
return CH_COMPILER_ERROR;
}
if (!expressions) {
*db = nullptr;
*comp_error =
generateChimeraCompileError("Invalid parameter: expressions is\
NULL", -1);
return CH_COMPILER_ERROR;
}
if (!elements) {
*db = nullptr;
*comp_error = generateChimeraCompileError("Invalid parameter:\
elements is zero", -1);
return CH_COMPILER_ERROR;
}
if (!checkMode(mode, comp_error)) {
*db = nullptr;
assert(*comp_error); // set by checkMode
return CH_COMPILER_ERROR;
}
try {
// Internal function to do all the work, now that we've handled all the
// argument checking.
ch::ch_compile_multi_int(expressions, flags, ids, elements, mode,
match_limit, match_limit_recursion, platform,
db);
}
catch (const CompileError &e) {
// Compiler error occurred
*db = nullptr;
*comp_error = generateChimeraCompileError(e.reason, e.hasIndex ?
(int)e.index : -1);
return CH_COMPILER_ERROR;
}
catch (std::bad_alloc &) {
*db = nullptr;
*comp_error = const_cast<ch_compile_error_t *>(&ch_enomem);
return CH_COMPILER_ERROR;
}
catch (...) {
assert(!"Internal error, unexpected exception");
*db = nullptr;
*comp_error = const_cast<ch_compile_error_t *>(&ch_einternal);
return CH_COMPILER_ERROR;
}
DEBUG_PRINTF("success!\n");
return CH_SUCCESS;
}
extern "C" HS_PUBLIC_API
ch_error_t HS_CDECL ch_free_compile_error(ch_compile_error_t *error) {
freeChimeraCompileError(error);
return CH_SUCCESS;
}

394
chimera/ch_compile.h
View File

@ -1,394 +0,0 @@
/*
* Copyright (c) 2018, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#ifndef CH_COMPILE_H_
#define CH_COMPILE_H_
/**
* @file
* @brief The Chimera compiler API definition.
*
* Chimera is a hybrid solution of Hyperscan and PCRE.
*
* This header contains functions for compiling regular expressions into
* Chimera databases that can be used by the Chimera runtime.
*/
#include "ch_common.h"
#include "hs_compile.h"
#ifdef __cplusplus
extern "C"
{
#endif
/**
* A type containing error details that is returned by the compile calls (@ref
* ch_compile() and @ref ch_compile_multi() on failure. The caller may inspect
* the values returned in this type to determine the cause of failure.
*/
typedef struct ch_compile_error {
/**
* A human-readable error message describing the error.
*/
char *message;
/**
* The zero-based number of the expression that caused the error (if this
* can be determined). If the error is not specific to an expression, then
* this value will be less than zero.
*/
int expression;
} ch_compile_error_t;
/**
* The basic regular expression compiler.
*
* This is the function call with which an expression is compiled into a
* Chimera database which can be passed to the runtime function (
* @ref ch_scan())
*
* @param expression
* The NULL-terminated expression to parse. Note that this string must
* represent ONLY the pattern to be matched, with no delimiters or flags;
* any global flags should be specified with the @a flags argument. For
* example, the expression `/abc?def/i` should be compiled by providing
* `abc?def` as the @a expression, and @ref CH_FLAG_CASELESS as the @a
* flags.
*
* @param flags
* Flags which modify the behaviour of the expression. Multiple flags may
* be used by ORing them together. Valid values are:
* - CH_FLAG_CASELESS - Matching will be performed case-insensitively.
* - CH_FLAG_DOTALL - Matching a `.` will not exclude newlines.
* - CH_FLAG_MULTILINE - `^` and `$` anchors match any newlines in data.
* - CH_FLAG_SINGLEMATCH - Only one match will be generated for the
* expression per stream.
* - CH_FLAG_UTF8 - Treat this pattern as a sequence of UTF-8 characters.
* - CH_FLAG_UCP - Use Unicode properties for character classes.
*
* @param mode
* Compiler mode flag that affect the database as a whole for capturing
* groups. One of CH_MODE_NOGROUPS or CH_MODE_GROUPS must be supplied.
* See @ref CH_MODE_FLAG for more details.
*
* @param platform
* If not NULL, the platform structure is used to determine the target
* platform for the database. If NULL, a database suitable for running
* on the current host platform is produced.
*
* @param db
* On success, a pointer to the generated database will be returned in
* this parameter, or NULL on failure. The caller is responsible for
* deallocating the buffer using the @ref ch_free_database() function.
*
* @param compile_error
* If the compile fails, a pointer to a @ref ch_compile_error_t will be
* returned, providing details of the error condition. The caller is
* responsible for deallocating the buffer using the @ref
* ch_free_compile_error() function.
*
* @return
* @ref CH_SUCCESS is returned on successful compilation; @ref
* CH_COMPILER_ERROR on failure, with details provided in the error
* parameter.
*/
ch_error_t HS_CDECL ch_compile(const char *expression, unsigned int flags,
unsigned int mode,
const hs_platform_info_t *platform,
ch_database_t **db,
ch_compile_error_t **compile_error);
/**
* The multiple regular expression compiler.
*
* This is the function call with which a set of expressions is compiled into a
* database which can be passed to the runtime function (@ref ch_scan()).
* Each expression can be labelled with a unique integer which is passed into
* the match callback to identify the pattern that has matched.
*
* @param expressions
* Array of NULL-terminated expressions to compile. Note that (as for @ref
* ch_compile()) these strings must contain only the pattern to be
* matched, with no delimiters or flags. For example, the expression
* `/abc?def/i` should be compiled by providing `abc?def` as the first
* string in the @a expressions array, and @ref CH_FLAG_CASELESS as the
* first value in the @a flags array.
*
* @param flags
* Array of flags which modify the behaviour of each expression. Multiple
* flags may be used by ORing them together. Specifying the NULL pointer
* in place of an array will set the flags value for all patterns to zero.
* Valid values are:
* - CH_FLAG_CASELESS - Matching will be performed case-insensitively.
* - CH_FLAG_DOTALL - Matching a `.` will not exclude newlines.
* - CH_FLAG_MULTILINE - `^` and `$` anchors match any newlines in data.
* - CH_FLAG_SINGLEMATCH - Only one match will be generated by patterns
* with this match id per stream.
* - CH_FLAG_UTF8 - Treat this pattern as a sequence of UTF-8 characters.
* - CH_FLAG_UCP - Use Unicode properties for character classes.
*
* @param ids
* An array of integers specifying the ID number to be associated with the
* corresponding pattern in the expressions array. Specifying the NULL
* pointer in place of an array will set the ID value for all patterns to
* zero.
*
* @param elements
* The number of elements in the input arrays.
*
* @param mode
* Compiler mode flag that affect the database as a whole for capturing
* groups. One of CH_MODE_NOGROUPS or CH_MODE_GROUPS must be supplied.
* See @ref CH_MODE_FLAG for more details.
*
* @param platform
* If not NULL, the platform structure is used to determine the target
* platform for the database. If NULL, a database suitable for running
* on the current host platform is produced.
*
* @param db
* On success, a pointer to the generated database will be returned in
* this parameter, or NULL on failure. The caller is responsible for
* deallocating the buffer using the @ref ch_free_database() function.
*
* @param compile_error
* If the compile fails, a pointer to a @ref ch_compile_error_t will be
* returned, providing details of the error condition. The caller is
* responsible for deallocating the buffer using the @ref
* ch_free_compile_error() function.
*
* @return
* @ref CH_SUCCESS is returned on successful compilation; @ref
* CH_COMPILER_ERROR on failure, with details provided in the @a error
* parameter.
*
*/
ch_error_t HS_CDECL ch_compile_multi(const char *const *expressions,
const unsigned int *flags,
const unsigned int *ids,
unsigned int elements, unsigned int mode,
const hs_platform_info_t *platform,
ch_database_t **db,
ch_compile_error_t **compile_error);
/**
* The multiple regular expression compiler with extended match limits support.
*
* This is the function call with which a set of expressions is compiled into a
* database in the same way as @ref ch_compile_multi(), but allows additional
* parameters to be specified via match_limit and match_limit_recursion to
* define match limits for PCRE runtime.
*
* @param expressions
* Array of NULL-terminated expressions to compile. Note that (as for @ref
* ch_compile()) these strings must contain only the pattern to be
* matched, with no delimiters or flags. For example, the expression
* `/abc?def/i` should be compiled by providing `abc?def` as the first
* string in the @a expressions array, and @ref CH_FLAG_CASELESS as the
* first value in the @a flags array.
*
* @param flags
* Array of flags which modify the behaviour of each expression. Multiple
* flags may be used by ORing them together. Specifying the NULL pointer
* in place of an array will set the flags value for all patterns to zero.
* Valid values are:
* - CH_FLAG_CASELESS - Matching will be performed case-insensitively.
* - CH_FLAG_DOTALL - Matching a `.` will not exclude newlines.
* - CH_FLAG_MULTILINE - `^` and `$` anchors match any newlines in data.
* - CH_FLAG_SINGLEMATCH - Only one match will be generated by patterns
* with this match id per stream.
* - CH_FLAG_UTF8 - Treat this pattern as a sequence of UTF-8 characters.
* - CH_FLAG_UCP - Use Unicode properties for character classes.
*
* @param ids
* An array of integers specifying the ID number to be associated with the
* corresponding pattern in the expressions array. Specifying the NULL
* pointer in place of an array will set the ID value for all patterns to
* zero.
*
* @param elements
* The number of elements in the input arrays.
*
* @param mode
* Compiler mode flag that affect the database as a whole for capturing
* groups. One of CH_MODE_NOGROUPS or CH_MODE_GROUPS must be supplied.
* See @ref CH_MODE_FLAG for more details.
*
* @param match_limit
* A limit from pcre_extra on the amount of match function called in PCRE
* to limit backtracking that can take place.
*
* @param match_limit_recursion
* A limit from pcre_extra on the recursion depth of match function
* in PCRE.
*
* @param platform
* If not NULL, the platform structure is used to determine the target
* platform for the database. If NULL, a database suitable for running
* on the current host platform is produced.
*
* @param db
* On success, a pointer to the generated database will be returned in
* this parameter, or NULL on failure. The caller is responsible for
* deallocating the buffer using the @ref ch_free_database() function.
*
* @param compile_error
* If the compile fails, a pointer to a @ref ch_compile_error_t will be
* returned, providing details of the error condition. The caller is
* responsible for deallocating the buffer using the @ref
* ch_free_compile_error() function.
*
* @return
* @ref CH_SUCCESS is returned on successful compilation; @ref
* CH_COMPILER_ERROR on failure, with details provided in the @a error
* parameter.
*
*/
ch_error_t HS_CDECL ch_compile_ext_multi(const char *const *expressions,
const unsigned int *flags,
const unsigned int *ids,
unsigned int elements,
unsigned int mode,
unsigned long int match_limit,
unsigned long int match_limit_recursion,
const hs_platform_info_t *platform,
ch_database_t **db,
ch_compile_error_t **compile_error);
/**
* Free an error structure generated by @ref ch_compile(), @ref
* ch_compile_multi().
*
* @param error
* The @ref ch_compile_error_t to be freed. NULL may also be safely
* provided.
*
* @return
* @ref CH_SUCCESS on success, other values on failure.
*/
ch_error_t HS_CDECL ch_free_compile_error(ch_compile_error_t *error);
/**
* @defgroup CH_PATTERN_FLAG Pattern flags
*
* @{
*/
/**
* Compile flag: Set case-insensitive matching.
*
* This flag sets the expression to be matched case-insensitively by default.
* The expression may still use PCRE tokens (notably `(?i)` and
* `(?-i)`) to switch case-insensitive matching on and off.
*/
#define CH_FLAG_CASELESS 1
/**
* Compile flag: Matching a `.` will not exclude newlines.
*
* This flag sets any instances of the `.` token to match newline characters as
* well as all other characters. The PCRE specification states that the `.`
* token does not match newline characters by default, so without this flag the
* `.` token will not cross line boundaries.
*/
#define CH_FLAG_DOTALL 2
/**
* Compile flag: Set multi-line anchoring.
*
* This flag instructs the expression to make the `^` and `$` tokens match
* newline characters as well as the start and end of the stream. If this flag
* is not specified, the `^` token will only ever match at the start of a
* stream, and the `$` token will only ever match at the end of a stream within
* the guidelines of the PCRE specification.
*/
#define CH_FLAG_MULTILINE 4
/**
* Compile flag: Set single-match only mode.
*
* This flag sets the expression's match ID to match at most once, only the
* first match for each invocation of @ref ch_scan() will be returned.
*
*/
#define CH_FLAG_SINGLEMATCH 8
/**
* Compile flag: Enable UTF-8 mode for this expression.
*
* This flag instructs Chimera to treat the pattern as a sequence of UTF-8
* characters. The results of scanning invalid UTF-8 sequences with a Chimera
* library that has been compiled with one or more patterns using this flag are
* undefined.
*/
#define CH_FLAG_UTF8 32
/**
* Compile flag: Enable Unicode property support for this expression.
*
* This flag instructs Chimera to use Unicode properties, rather than the
* default ASCII interpretations, for character mnemonics like `\w` and `\s` as
* well as the POSIX character classes. It is only meaningful in conjunction
* with @ref CH_FLAG_UTF8.
*/
#define CH_FLAG_UCP 64
/** @} */
/**
* @defgroup CH_MODE_FLAG Compile mode flags
*
* The mode flags are used as values for the mode parameter of the various
* compile calls (@ref ch_compile(), @ref ch_compile_multi().
*
* By default, the matcher will only supply the start and end offsets of the
* match when the match callback is called. Using mode flag @ref CH_MODE_GROUPS
* will also fill the `captured' array with the start and end offsets of all
* the capturing groups specified by the pattern that has matched.
*
* @{
*/
/**
* Compiler mode flag: Disable capturing groups.
*/
#define CH_MODE_NOGROUPS 0
/**
* Compiler mode flag: Enable capturing groups.
*/
#define CH_MODE_GROUPS 1048576
/** @} */
#ifdef __cplusplus
} /* extern "C" */
#endif
#endif /* CH_COMPILE_H_ */

126
chimera/ch_database.c
View File

@ -1,126 +0,0 @@
/*
* Copyright (c) 2018, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/** \file
* \brief Chimera: database construction, etc.
*/
#include <limits.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "allocator.h"
#include "database.h"
#include "hs.h"
#include "ch.h"
#include "hs_internal.h"
#include "ch_common.h"
#include "ch_alloc.h"
#include "ch_database.h"
#include "ch_internal.h"
static really_inline
int db_correctly_aligned(const void *db) {
return ISALIGNED_N(db, alignof(unsigned long long));
}
HS_PUBLIC_API
ch_error_t HS_CDECL ch_free_database(ch_database_t *hydb) {
if (hydb && hydb->magic != CH_DB_MAGIC) {
return CH_INVALID;
}
ch_database_free(hydb);
return CH_SUCCESS;
}
HS_PUBLIC_API
ch_error_t HS_CDECL ch_database_size(const ch_database_t *hydb, size_t *size) {
if (!size) {
return CH_INVALID;
}
ch_error_t ret = hydbIsValid(hydb);
if (unlikely(ret != CH_SUCCESS)) {
return ret;
}
*size = sizeof(struct ch_database) + hydb->length;
return CH_SUCCESS;
}
/** \brief Identifier prepended to database info. */
static const char CHIMERA_IDENT[] = "Chimera ";
HS_PUBLIC_API
ch_error_t HS_CDECL ch_database_info(const ch_database_t *hydb, char **info) {
if (!info) {
return CH_INVALID;
}
*info = NULL;
if (!hydb || !db_correctly_aligned(hydb) || hydb->magic != CH_DB_MAGIC) {
return HS_INVALID;
}
const struct ch_bytecode *bytecode = ch_get_bytecode(hydb);
char noMulti = (bytecode->flags & CHIMERA_FLAG_NO_MULTIMATCH);
if (noMulti) {
size_t len = strlen(CHIMERA_IDENT);
*info = ch_misc_alloc(len + 1);
if (!(*info)) {
return CH_INVALID;
}
memcpy((*info), CHIMERA_IDENT, len);
(*info)[len] = '\0';
return CH_SUCCESS;
}
char *hsinfo = NULL;
hs_error_t ret = hs_database_info(getHyperscanDatabase(bytecode), &hsinfo);
if (ret != HS_SUCCESS) {
assert(!hsinfo);
return ret;
}
size_t hybridlen = strlen(CHIMERA_IDENT);
size_t hslen = strlen(hsinfo);
*info = ch_misc_alloc(hybridlen + hslen + 1);
if (!(*info)) {
ch_misc_free(hsinfo);
return CH_INVALID;
}
memcpy((*info), CHIMERA_IDENT, hybridlen);
memcpy((*info) + hybridlen, hsinfo, hslen);
(*info)[hybridlen + hslen] = '\0';
ch_misc_free(hsinfo);
return CH_SUCCESS;
}

158
chimera/ch_database.h
View File

@ -1,158 +0,0 @@
/*
* Copyright (c) 2018, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/** \file
* \brief Runtime code for ch_database manipulation.
*/
#ifndef CH_DATABASE_H_
#define CH_DATABASE_H_
#ifdef __cplusplus
extern "C"
{
#endif
#define PCRE_STATIC
#include <pcre.h>
#include "ch_compile.h" // for CH_MODE_ flags
#include "ue2common.h"
#include "hs_version.h"
#include "hs.h"
#define CH_DB_MAGIC 0xdedededeU //!< Magic number stored in \ref ch_database
/** \brief Main Chimera database header. */
struct ch_database {
u32 magic; //!< must be \ref CH_DB_MAGIC
u32 version; //!< release version
u32 length; //!< total allocated length in bytes
u32 reserved0; //!< unused
u32 reserved1; //!< unused
u32 bytecode; //!< offset relative to db start
u32 padding[16]; //!< padding for alignment of rest of bytecode
char bytes[];
};
/** \brief Chimera bytecode header, which follows the \ref ch_database and is
* always 64-byte aligned. */
struct ch_bytecode {
u32 length; //!< length of bytecode including this header struct
u32 flags; //!< whole-database flags (CHIMERA_FLAG_NO_MULTIMATCH,
// CHIMERA_FLAG_GROUPS)
u32 patternCount; //!< total number of patterns
u32 activeSize; //!< size of mmbit to store active pattern ids
u32 databaseOffset; //!< offset for database following \ref ch_bytecode
// header
u32 patternOffset; //!< points to an array of u32 offsets, each pointing to
// a \ref ch_pattern
u32 unguardedOffset; //!< pointer to a list of unguarded pattern indices
u32 unguardedCount; //!< number of unguarded patterns
u32 maxCaptureGroups; //!< max number of capture groups used by any pattern
};
/** \brief Per-pattern header.
*
* struct is followed in bytecode by:
* 1. pcre bytecode (always present)
* 2. pcre study data (sometimes)
*/
struct ch_pattern {
u32 id; //!< pattern ID to report to the user
u32 flags; //!< per-pattern flags (e.g. \ref CHIMERA_PATTERN_FLAG_UTF8)
u32 maxWidth; //!< maximum width of a match, or UINT_MAX for inf.
u32 minWidth; //!< minimum width of a match.
u32 fixedWidth;//!< pattern has fixed width.
u32 studyOffset; //!< offset relative to struct start of study data,
// or zero if there is none
u32 length; //!< length of struct plus pcre bytecode and study data
pcre_extra extra; //!< pcre_extra struct, used to store study data ptr for
// the currently-running pcre at runtime.
};
static really_inline
const void *ch_get_bytecode(const struct ch_database *db) {
assert(db);
const void *bytecode = (const char *)db + db->bytecode;
assert(ISALIGNED_16(bytecode));
return bytecode;
}
struct hs_database;
static really_inline
const struct hs_database *getHyperscanDatabase(const struct ch_bytecode *db) {
assert(db);
const char *ptr = (const char *)db;
const struct hs_database *hs_db;
hs_db = (const struct hs_database *)(ptr + db->databaseOffset);
assert(ISALIGNED_CL(hs_db));
return hs_db;
}
static really_inline
const u32 *getUnguarded(const struct ch_bytecode *db) {
assert(db);
const char *ptr = (const char *)db;
const u32 *unguarded = (const u32 *)(ptr + db->unguardedOffset);
assert(ISALIGNED_N(unguarded, sizeof(u32)));
return unguarded;
}
static really_inline
const struct ch_pattern *getPattern(const struct ch_bytecode *db, u32 i) {
assert(db);
assert(i < db->patternCount);
const char *ptr = (const char *)db;
const u32 *patternOffset = (const u32 *)(ptr + db->patternOffset);
assert(patternOffset[i] < db->length);
return (const struct ch_pattern *)(ptr + patternOffset[i]);
}
static really_inline
ch_error_t hydbIsValid(const struct ch_database *hydb) {
if (!hydb || hydb->magic != CH_DB_MAGIC) {
DEBUG_PRINTF("bad magic (%u != %u)\n", hydb->magic, CH_DB_MAGIC);
return CH_INVALID;
}
if (hydb->version != HS_VERSION_32BIT) {
DEBUG_PRINTF("bad version\n");
return CH_DB_VERSION_ERROR;
}
return CH_SUCCESS;
}
#ifdef __cplusplus
} /* extern "C" */
#endif
#endif /* CH_DATABASE_H_ */

44
chimera/ch_internal.h
View File

@ -1,44 +0,0 @@
/*
* Copyright (c) 2018, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/** \file
* \brief Chimera: data structures and internals.
*/
#ifndef CH_INTERNAL_H
#define CH_INTERNAL_H
#define CHIMERA_FLAG_NO_MULTIMATCH 1 //!< Don't run a multimatch scan
#define CHIMERA_FLAG_GROUPS 2 //!< Return capturing groups
#define CHIMERA_FLAG_ALL_CONFIRM 4 //!< All patterns need confirm
#define CHIMERA_FLAG_ALL_SINGLE 8 //!< All patterns need only one match
#define CHIMERA_PATTERN_FLAG_SINGLEMATCH 1 //!< only report the first match
#define CHIMERA_PATTERN_FLAG_UTF8 2 //!< pattern is in UTF-8 mode
#endif

643
chimera/ch_runtime.c
View File

@ -1,643 +0,0 @@
/*
* Copyright (c) 2018-2020, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/** \file
* \brief Chimera: main runtime.
*/
#include <limits.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "ch.h"
#include "hs.h"
#include "hs_internal.h"
#include "ue2common.h"
#include "ch_database.h"
#include "ch_internal.h"
#include "ch_scratch.h"
#include "util/multibit.h"
#include "util/unicode_def.h"
typedef struct queue_item PQ_T;
static
char PQ_COMP(PQ_T *pqc_items, int a, int b) {
if ((pqc_items)[a].to != (pqc_items)[b].to) {
return (pqc_items)[a].to < (pqc_items)[b].to;
} else if ((pqc_items)[a].from != (pqc_items)[b].from) {
return (pqc_items)[a].from < (pqc_items)[b].from;
} else {
return (pqc_items)[a].id < (pqc_items)[b].id;
}
}
static
char PQ_COMP_B(PQ_T *pqc_items, int a, PQ_T b_fixed) {
if ((pqc_items)[a].to != (b_fixed).to) {
return (pqc_items)[a].to < (b_fixed).to;
} else if ((pqc_items)[a].from != (b_fixed).from) {
return (pqc_items)[a].from < (b_fixed).from;
} else {
return (pqc_items)[a].id < b_fixed.id;
}
}
#include "util/pqueue.h"
static really_inline
void pq_insert_with(struct match_pq *pq, int from, int to, u32 id) {
DEBUG_PRINTF("inserting pattern%u in pq at %u\n", id, to);
struct queue_item temp = {
.from = from,
.to = to,
.id = id,
};
pq_insert(pq->item, pq->size, temp);
++pq->size;
}
static really_inline
void pq_pop_nice(struct match_pq *pq) {
pq_pop(pq->item, pq->size);
pq->size--;
}
/** dummy event handler for use when user does not provide one */
static
int HS_CDECL null_onEvent(UNUSED unsigned id, UNUSED unsigned long long from,
UNUSED unsigned long long to, UNUSED unsigned flags,
UNUSED unsigned size, UNUSED const ch_capture_t *captured,
UNUSED void *ctxt) {
return 0;
}
/** \brief Chimera runtime context. */
struct HybridContext {
const char *data; //!< buffer being scanned
u32 length; //!< length of data buffer
u32 valid_utf8_highwater; //!< UTF-8 has been validated up to here.
const struct ch_bytecode *db;
struct ch_scratch *scratch;
struct match_pq *pq;
/** \brief user-supplied match callback */
int (HS_CDECL *match_callback)(unsigned int id, unsigned long long from,
unsigned long long to, unsigned int flags,
unsigned int size, const ch_capture_t *capture,
void *ctx);
/** \brief user-supplied error callback */
int (HS_CDECL *error_callback)(ch_error_event_t error_type, unsigned int id,
void *info, void *ctx);
/** \brief user-supplied context */
void *context;
};
// Internal PCRE func.
extern int _pcre_valid_utf(const unsigned char *, int, int *);
/** UTF-8 validity check. Returns >0 if the given region of the data is valid
* UTF-8, 0 otherwise. */
static
char isValidUTF8(struct HybridContext *hyctx, u32 end) {
assert(hyctx);
if (hyctx->valid_utf8_highwater >= end) {
return 1; // Already validated.
}
const unsigned char *data =
(const unsigned char *)hyctx->data + hyctx->valid_utf8_highwater;
int validate_len = end - hyctx->valid_utf8_highwater;
DEBUG_PRINTF("validating %d bytes\n", validate_len);
int erroroffset = 0;
if (_pcre_valid_utf(data, validate_len, &erroroffset)) {
DEBUG_PRINTF("UTF8 invalid at offset %d\n", erroroffset);
return 0;
}
hyctx->valid_utf8_highwater = end;
return 1;
}
static
const pcre *getPcre(const struct ch_pattern *pattern) {
const char *ptr = (const char *)pattern;
const pcre *p = (const pcre *)(ptr + ROUNDUP_N(sizeof(*pattern), 8));
assert(ISALIGNED_N(p, 8));
return p;
}
/** \brief Fill the Chimera groups array from a pcre_exec ovector. */
static
void fillGroupsFromOvector(ch_capture_t *groups, int numPairs, int *ovector) {
assert(groups);
assert(ISALIGNED_N(groups, alignof(ch_capture_t)));
DEBUG_PRINTF("filling %d groups (@ %p) from pcre ovector\n",
numPairs, groups);
for (int i = 0; i < numPairs * 2; i += 2) {
if (ovector[i] == -1) {
groups->flags = CH_CAPTURE_FLAG_INACTIVE;
} else {
groups->flags = CH_CAPTURE_FLAG_ACTIVE;
assert(ovector[i] <= ovector[i + 1]);
groups->from = ovector[i];
groups->to = ovector[i + 1];
}
++groups;
}
}
static
ch_error_t handlePcreNonMatch(const struct ch_pattern *pattern, int rv,
ch_error_event_handler onError,
void *userContext) {
assert(rv < 0);
if (rv == PCRE_ERROR_NOMATCH) {
DEBUG_PRINTF("no match found by libpcre\n");
return CH_SUCCESS;
} else if (rv == PCRE_ERROR_MATCHLIMIT) {
DEBUG_PRINTF("pcre hit match limit\n");
if (onError) {
return onError(CH_ERROR_MATCHLIMIT, pattern->id, NULL,
userContext);
}
return CH_SUCCESS;
} else if (rv == PCRE_ERROR_RECURSIONLIMIT) {
DEBUG_PRINTF("pcre hit recursion limit\n");
if (onError) {
return onError(CH_ERROR_RECURSIONLIMIT, pattern->id, NULL,
userContext);
}
return CH_SUCCESS;
}
// All other errors not handled above are fatal.
return CH_FAIL_INTERNAL;
}
static
ch_error_t scanPcre(struct HybridContext *hyctx, UNUSED unsigned int length,
unsigned int offset, u32 id) {
const char *data = hyctx->data;
unsigned int full_length = hyctx->length;
ch_error_event_handler onError = hyctx->error_callback;
void *userContext = hyctx->context;
const struct ch_pattern *pattern = getPattern(hyctx->db, id);
const pcre *p = getPcre(pattern);
// Set up the PCRE extra block.
const pcre_extra *extra = &pattern->extra;
int startoffset = offset;
int *ovector = hyctx->scratch->ovector;
int ovectorSize = (hyctx->scratch->maxCaptureGroups + 1) * 3;
assert(ovectorSize >= 2);
DEBUG_PRINTF("scanning %u bytes, pattern %u, startoffset %d\n",
length, id, startoffset);
int options = 0;
if (pattern->flags & CHIMERA_PATTERN_FLAG_UTF8) {
// We do our own UTF-8 validation.
options |= PCRE_NO_UTF8_CHECK;
if (!isValidUTF8(hyctx, full_length)) {
return handlePcreNonMatch(pattern, PCRE_ERROR_BADUTF8, onError,
userContext);
}
}
int rv = pcre_exec(p, extra, data, full_length, startoffset, options,
ovector, ovectorSize);
DEBUG_PRINTF("pcre return code is %d\n", rv);
// Handle all non-match or error cases, all of which involve us
// terminating the loop.
if (rv < 0) {
return handlePcreNonMatch(pattern, rv, onError, userContext);
}
// We've found a match, and we should always have room for at least the
// start and end offsets in our ovector. Pass this info to the user.
assert(rv >= 1);
assert(rv < ovectorSize);
int from = ovector[0];
int to = ovector[1];
DEBUG_PRINTF("match %d -> %d\n", from, to);
struct ch_patterndata *pd = hyctx->scratch->patternData + id;
if (hyctx->db->flags & CHIMERA_FLAG_GROUPS) {
fillGroupsFromOvector(pd->match, rv, ovector);
} else {
rv = 0;
}
pd->groupCount = (u32)rv;
// Insert new matched item to the queue
pq_insert_with(hyctx->pq, from, to, id);
// Next scan starts at the first codepoint after the match. It's
// possible that we have a vacuous match, in which case we must step
// past it to ensure that we always progress.
if (from != to) {
startoffset = to;
} else if (pattern->flags & CHIMERA_PATTERN_FLAG_UTF8) {
startoffset = to + 1;
while (startoffset < (int)full_length &&
((data[startoffset] & 0xc0) == UTF_CONT_BYTE_HEADER)) {
++startoffset;
}
} else {
startoffset = to + 1;
}
pd->scanStart = startoffset;
DEBUG_PRINTF("new offset %u\n", pd->scanStart);
return CH_SUCCESS;
}
static
ch_error_t catchupPcre(struct HybridContext *hyctx, unsigned int id,
unsigned long long from, unsigned long long to) {
ch_match_event_handler onEvent = hyctx->match_callback;
void *userContext = hyctx->context;
DEBUG_PRINTF("priority queue size %u\n", hyctx->pq->size);
while (hyctx->pq->size) {
u32 num_item = hyctx->pq->size;
struct queue_item *item = pq_top(hyctx->pq->item);
size_t top_from = item->from;
size_t top_to = item->to;
u32 top_id = item->id;
if (top_to > to) {
pq_insert_with(hyctx->pq, from, to, id);
break;
}
pq_pop_nice(hyctx->pq);
const struct ch_pattern *pattern = getPattern(hyctx->db, top_id);
struct ch_patterndata *pd = hyctx->scratch->patternData + top_id;
// Report match for pattern
DEBUG_PRINTF("trigger match@%zu\n", top_to);
ch_callback_t cbrv =
onEvent(pattern->id, top_from, top_to, 0 /* flags */,
pd->groupCount, pd->match, userContext);
if (cbrv == CH_CALLBACK_TERMINATE) {
DEBUG_PRINTF("user callback told us to terminate scanning\n");
return CH_SCAN_TERMINATED;
} else if (cbrv == CH_CALLBACK_SKIP_PATTERN) {
DEBUG_PRINTF("user callback told us to skip this pattern\n");
pd->scanStart = hyctx->length;
}
if (top_id == id) {
break;
}
// Push a new match to replace the old one
unsigned int start = pd->scanStart;
unsigned int len = hyctx->length - pd->scanStart;
if (hyctx->length >= pd->scanStart &&
!(pattern->flags & CHIMERA_PATTERN_FLAG_SINGLEMATCH)) {
DEBUG_PRINTF("get a new match item\n");
int ret = scanPcre(hyctx, len, start, top_id);
if (ret == CH_CALLBACK_TERMINATE) {
DEBUG_PRINTF("user callback told us to terminate scanning\n");
return CH_SCAN_TERMINATED;
} else if (ret == CH_CALLBACK_SKIP_PATTERN) {
DEBUG_PRINTF("user callback told us to skip this pattern\n");
pd->scanStart = hyctx->length;
ret = CH_SUCCESS;
} else if (ret == CH_FAIL_INTERNAL) {
return ret;
}
// No further match is found
if (hyctx->pq->size == num_item - 1) {
pd->scanStart = hyctx->length;
}
}
}
return CH_SUCCESS;
}
/** \brief Callback used for internal Hyperscan multi-matcher. */
static
int HS_CDECL multiCallback(unsigned int id, unsigned long long from,
unsigned long long to, UNUSED unsigned int flags,
void *ctx) {
assert(ctx);
struct HybridContext *hyctx = ctx;
DEBUG_PRINTF("match for ID %u at offset %llu\n", id, to);
assert(id < hyctx->db->patternCount);
const struct ch_pattern *pattern = getPattern(hyctx->db, id);
struct ch_patterndata *pd = hyctx->scratch->patternData + id;
char needConfirm = pattern->fixedWidth == ~0U;
if (needConfirm &&
mmbit_isset(hyctx->scratch->active, hyctx->db->patternCount, id)) {
if ((hyctx->db->flags & CHIMERA_FLAG_ALL_CONFIRM) &&
mmbit_all(hyctx->scratch->active, hyctx->db->patternCount)) {
return 1;
}
return 0;
}
// Store the fact that we've seen this bit.
char already = mmbit_set(hyctx->scratch->active,
hyctx->db->patternCount, id);
DEBUG_PRINTF("match from %u to %llu\n", pd->scanStart, to);
if (!already) {
pd->scanStart = 0;
} else if (to < pd->scanStart + pattern->minWidth) {
return 0;
} else if (pattern->flags & CHIMERA_PATTERN_FLAG_SINGLEMATCH) {
if ((hyctx->db->flags & CHIMERA_FLAG_ALL_SINGLE) &&
mmbit_all(hyctx->scratch->active, hyctx->db->patternCount)) {
return 1;
}
// Note: we may have unordered match from Hyperscan,
// thus possibly get to < pd->scanStart.
return 0;
}
int ret = HS_SUCCESS;
unsigned int start = pd->scanStart;
unsigned int len = hyctx->length - pd->scanStart;
assert(hyctx->length >= pd->scanStart);
const char *data = hyctx->data;
if (needConfirm) {
DEBUG_PRINTF("run confirm for the first time\n");
ret = scanPcre(hyctx, len, start, id);
hyctx->scratch->ret = ret;
if (ret == CH_CALLBACK_TERMINATE) {
DEBUG_PRINTF("user callback told us to terminate scanning\n");
return HS_SCAN_TERMINATED;
} else if (ret == CH_CALLBACK_SKIP_PATTERN) {
DEBUG_PRINTF("user callback told us to skip this pattern\n");
pd->scanStart = hyctx->length;
ret = HS_SUCCESS;
hyctx->scratch->ret = ret;
} else if (ret == CH_FAIL_INTERNAL) {
return ret;
}
} else {
if (already) {
DEBUG_PRINTF("catch up with new matches\n");
ret = catchupPcre(hyctx, id, from, to);
hyctx->scratch->ret = ret;
if (pd->scanStart >= hyctx->length) {
return ret;
}
}
int startoffset = 0;
// Next scan starts at the first codepoint after the match. It's
// possible that we have a vacuous match, in which case we must step
// past it to ensure that we always progress.
if (from != to) {
startoffset = to;
} else if (pattern->flags & CHIMERA_PATTERN_FLAG_UTF8) {
startoffset = to + 1;
while (startoffset < (int)hyctx->length &&
((data[startoffset] & 0xc0) == UTF_CONT_BYTE_HEADER)) {
++startoffset;
}
} else {
startoffset = to + 1;
}
pd->scanStart = startoffset;
int rv = 0;
if (hyctx->db->flags & CHIMERA_FLAG_GROUPS) {
ch_capture_t *groups = pd->match;
groups->flags = CH_CAPTURE_FLAG_ACTIVE;
groups->from = from;
groups->to = to;
rv = 1;
}
pd->groupCount = (u32)rv;
pq_insert_with(hyctx->pq, from, to, id);
}
return ret;
}
static
hs_error_t scanHyperscan(struct HybridContext *hyctx, const char *data,
unsigned int length) {
DEBUG_PRINTF("scanning %u bytes with Hyperscan\n", length);
const struct ch_bytecode *hydb = hyctx->db;
const hs_database_t *db = getHyperscanDatabase(hydb);
hs_scratch_t *scratch = hyctx->scratch->multi_scratch;
hs_error_t err = hs_scan(db, data, length, 0, scratch, multiCallback,
hyctx);
return err;
}
/** \brief Init match priority queue.
*
* Add a first match offset for each pattern that is not supported by Hyperscan
* with prefiltering.
*/
static really_inline
ch_error_t initQueue(struct HybridContext *hyctx, struct match_pq *pq) {
const struct ch_bytecode *db = hyctx->db;
u8 *active = hyctx->scratch->active;
mmbit_clear(active, db->patternCount);
// Init match queue size
pq->size = 0;
unsigned int length = hyctx->length;
const u32 *unguarded = getUnguarded(db);
for (u32 i = 0; i < db->unguardedCount; i++) {
u32 patternId = unguarded[i];
DEBUG_PRINTF("switch on unguarded pcre %u\n", patternId);
mmbit_set(active, db->patternCount, patternId);
DEBUG_PRINTF("get a new match item\n");
int ret = scanPcre(hyctx, length, 0, patternId);
struct ch_patterndata *pd = hyctx->scratch->patternData + patternId;
if (ret == CH_CALLBACK_TERMINATE) {
DEBUG_PRINTF("user callback told us to terminate scanning\n");
return CH_SCAN_TERMINATED;
} else if (ret == CH_CALLBACK_SKIP_PATTERN) {
DEBUG_PRINTF("user callback told us to skip this pattern\n");
pd->scanStart = length;
ret = CH_SUCCESS;
} else if (ret == CH_FAIL_INTERNAL) {
return ret;
}
}
return CH_SUCCESS;
}
static really_inline
ch_error_t ch_scan_i(const ch_database_t *hydb,
const char *data, unsigned int length,
UNUSED unsigned int flags,
ch_scratch_t *scratch,
ch_match_event_handler onEvent,
ch_error_event_handler onError,
void *userContext) {
if (unlikely(!hydb || !scratch || !data)) {
DEBUG_PRINTF("args invalid\n");
return CH_INVALID;
}
ch_error_t ret = hydbIsValid(hydb);
if (ret != CH_SUCCESS) {
DEBUG_PRINTF("database invalid\n");
return ret;
}
if (!ISALIGNED_CL(scratch)) {
DEBUG_PRINTF("bad alignment %p\n", scratch);
return CH_INVALID;
}
if (scratch->magic != CH_SCRATCH_MAGIC) {
DEBUG_PRINTF("scratch invalid\n");
return CH_INVALID;
}
if (unlikely(markScratchInUse(scratch))) {
return CH_SCRATCH_IN_USE;
}
// Hyperscan underlying scratch and database validity will be checked by
// the hs_scan() call, so no need to do it here.
// PCRE takes the data region length in as an int, so this limits our block
// size to INT_MAX.
if (length > INT_MAX) {
DEBUG_PRINTF("length invalid\n");
unmarkScratchInUse(scratch);
return CH_INVALID;
}
const struct ch_bytecode *db = ch_get_bytecode(hydb);
scratch->pq.size = 0;
scratch->ret = CH_SUCCESS;
// Firstly, we run Hyperscan in block mode and add its matches into the
// active list for subsequent confirmation with pcre.
struct HybridContext hyctx = {
.data = data,
.length = length,
.valid_utf8_highwater = 0,
.db = db,
.scratch = scratch,
.pq = &scratch->pq,
.match_callback = onEvent ? onEvent : null_onEvent,
.error_callback = onError,
.context = userContext
};
// Init priority queue.
ret = initQueue(&hyctx, &scratch->pq);
if (ret != CH_SUCCESS) {
DEBUG_PRINTF("Chimera returned error %d\n", ret);
unmarkScratchInUse(scratch);
return ret;
}
if (!(db->flags & CHIMERA_FLAG_NO_MULTIMATCH)) {
ret = scanHyperscan(&hyctx, data, length);
// Errors from pcre scan.
if (scratch->ret == CH_CALLBACK_TERMINATE) {
DEBUG_PRINTF("Pcre terminates scan\n");
unmarkScratchInUse(scratch);
return CH_SCAN_TERMINATED;
} else if (scratch->ret != CH_SUCCESS) {
DEBUG_PRINTF("Pcre internal error\n");
unmarkScratchInUse(scratch);
return scratch->ret;
}
// Errors from Hyperscan scan. Note Chimera could terminate
// Hyperscan callback on purpose so this is not counted as an error.
if (ret != HS_SUCCESS && ret != HS_SCAN_TERMINATED) {
assert(scratch->ret == CH_SUCCESS);
DEBUG_PRINTF("Hyperscan returned error %d\n", ret);
unmarkScratchInUse(scratch);
return ret;
}
}
DEBUG_PRINTF("Flush priority queue\n");
// Catch up with PCRE and make up id and offsets as we don't really care
// about their values
ret = catchupPcre(&hyctx, ~0U, length, length);
if (ret != CH_SUCCESS) {
DEBUG_PRINTF("PCRE catch up returned error %d\n", ret);
unmarkScratchInUse(scratch);
return ret;
}
unmarkScratchInUse(scratch);
return CH_SUCCESS;
}
HS_PUBLIC_API
ch_error_t HS_CDECL ch_scan(const ch_database_t *hydb, const char *data,
unsigned int length, unsigned int flags,
ch_scratch_t *scratch,
ch_match_event_handler onEvent,
ch_error_event_handler onError, void *userContext) {
ch_error_t ret = ch_scan_i(hydb, data, length, flags, scratch, onEvent,
onError, userContext);
return ret;
}
HS_PUBLIC_API
const char * HS_CDECL ch_version(void) {
return HS_VERSION_STRING;
}

378
chimera/ch_runtime.h
View File

@ -1,378 +0,0 @@
/*
* Copyright (c) 2018, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#ifndef CH_RUNTIME_H_
#define CH_RUNTIME_H_
#include <stdlib.h>
/**
* @file
* @brief The Chimera runtime API definition.
*
* Chimera is a hybrid of Hyperscan and PCRE regular expression engine.
*
* This header contains functions for using compiled Chimera databases for
* scanning data at runtime.
*/
#include "hs_common.h"
#ifdef __cplusplus
extern "C"
{
#endif
struct ch_scratch;
/**
* A Chimera scratch space.
*/
typedef struct ch_scratch ch_scratch_t;
/**
* Callback return value used to tell the Chimera matcher what to do after
* processing this match.
*/
typedef int ch_callback_t;
/**
* @defgroup CH_CALLBACK ch_callback_t values
*
* @{
*/
/**
* Continue matching.
*/
#define CH_CALLBACK_CONTINUE 0
/**
* Terminate matching.
*/
#define CH_CALLBACK_TERMINATE 1
/**
* Skip remaining matches for this ID and continue.
*/
#define CH_CALLBACK_SKIP_PATTERN 2
/** @} */
/**
* Type used to differentiate the errors raised with the @ref
* ch_error_event_handler callback.
*/
typedef int ch_error_event_t;
/**
* @defgroup CH_ERROR_EVENT ch_error_event_t values
*
* @{
*/
/**
* PCRE hits its match limit and reports PCRE_ERROR_MATCHLIMIT.
*/
#define CH_ERROR_MATCHLIMIT 1
/**
* PCRE hits its recursion limit and reports PCRE_ERROR_RECURSIONLIMIT.
*/
#define CH_ERROR_RECURSIONLIMIT 2
/** @} */
/**
* Structure representing a captured subexpression within a match. An array of
* these structures corresponding to capture groups in order is passed to the
* callback on match, with active structures identified by the
* CH_CAPTURE_FLAG_ACTIVE flag.
*/
typedef struct ch_capture {
/**
* The flags indicating if this structure is active.
*/
unsigned int flags;
/**
* offset at which this capture group begins.
*/
unsigned long long from; /*< offset at which this capture group begins. */
/**
* offset at which this capture group ends.
*/
unsigned long long to;
} ch_capture_t;
/**
* @defgroup CH_CAPTURE ch_capture_t flags
*
* These flags are used in @ref ch_capture_t::flags to indicate if this
* structure is active.
*
* @{
*/
/**
* Flag indicating that a particular capture group is inactive, used in @ref
* ch_capture_t::flags.
*/
#define CH_CAPTURE_FLAG_INACTIVE 0
/**
* Flag indicating that a particular capture group is active, used in @ref
* ch_capture_t::flags.
*/
#define CH_CAPTURE_FLAG_ACTIVE 1
/** @} */
/**
* Definition of the match event callback function type.
*
* A callback function matching the defined type must be provided by the
* application calling the @ref ch_scan()
*
* This callback function will be invoked whenever a match is located in the
* target data during the execution of a scan. The details of the match are
* passed in as parameters to the callback function, and the callback function
* should return a value indicating whether or not matching should continue on
* the target data. If no callbacks are desired from a scan call, NULL may be
* provided in order to suppress match production.
*
* @param id
* The ID number of the expression that matched. If the expression was a
* single expression compiled with @ref ch_compile(), this value will be
* zero.
*
* @param from
* The offset of the first byte that matches the expression.
*
* @param to
* The offset after the last byte that matches the expression.
*
* @param flags
* This is provided for future use and is unused at present.
*
* @param size
* The number of valid entries pointed to by the captured parameter.
*
* @param captured
* A pointer to an array of @ref ch_capture_t structures that
* contain the start and end offsets of entire pattern match and
* each captured subexpression.
*
* @param ctx
* The pointer supplied by the user to the @ref ch_scan() function.
*
* @return
* The callback can return @ref CH_CALLBACK_TERMINATE to stop matching.
* Otherwise, a return value of @ref CH_CALLBACK_CONTINUE will continue,
* with the current pattern if configured to produce multiple matches per
* pattern, while a return value of @ref CH_CALLBACK_SKIP_PATTERN will
* cease matching this pattern but continue matching the next pattern.
*/
typedef ch_callback_t (HS_CDECL *ch_match_event_handler)(unsigned int id,
unsigned long long from,
unsigned long long to,
unsigned int flags,
unsigned int size,
const ch_capture_t *captured,
void *ctx);
/**
* Definition of the Chimera error event callback function type.
*
* A callback function matching the defined type may be provided by the
* application calling the @ref ch_scan function. This callback function
* will be invoked when an error event occurs during matching; this indicates
* that some matches for a given expression may not be reported.
*
* @param error_type
* The type of error event that occurred. Currently these errors
* correspond to resource limits on PCRE backtracking
* @ref CH_ERROR_MATCHLIMIT and @ref CH_ERROR_RECURSIONLIMIT.
*
* @param id
* The ID number of the expression that matched.
*
* @param info
* Event-specific data, for future use. Currently unused.
*
* @param ctx
* The context pointer supplied by the user to the @ref ch_scan
* function.
*
* @return
* The callback can return @ref CH_CALLBACK_SKIP_PATTERN to cease matching
* this pattern but continue matching the next pattern. Otherwise, we stop
* matching for all patterns with @ref CH_CALLBACK_TERMINATE.
*/
typedef ch_callback_t (HS_CDECL *ch_error_event_handler)(
ch_error_event_t error_type,
unsigned int id, void *info,
void *ctx);
/**
* The block regular expression scanner.
*
* This is the function call in which the actual pattern matching takes place
* for block-mode pattern databases.
*
* @param db
* A compiled pattern database.
*
* @param data
* Pointer to the data to be scanned.
*
* @param length
* The number of bytes to scan.
*
* @param flags
* Flags modifying the behaviour of this function. This parameter is
* provided for future use and is unused at present.
*
* @param scratch
* A per-thread scratch space allocated by @ref ch_alloc_scratch() for this
* database.
*
* @param onEvent
* Pointer to a match event callback function. If a NULL pointer is given,
* no matches will be returned.
*
* @param onError
* Pointer to a error event callback function. If a NULL pointer is given,
* @ref CH_ERROR_MATCHLIMIT and @ref CH_ERROR_RECURSIONLIMIT errors will
* be ignored and match will continue.
*
* @param context
* The user defined pointer which will be passed to the callback function.
*
* @return
* Returns @ref CH_SUCCESS on success; @ref CH_SCAN_TERMINATED if the
* match callback indicated that scanning should stop; other values on
* error.
*/
ch_error_t HS_CDECL ch_scan(const ch_database_t *db, const char *data,
unsigned int length, unsigned int flags,
ch_scratch_t *scratch,
ch_match_event_handler onEvent,
ch_error_event_handler onError,
void *context);
/**
* Allocate a "scratch" space for use by Chimera.
*
* This is required for runtime use, and one scratch space per thread, or
* concurrent caller, is required. Any allocator callback set by @ref
* ch_set_scratch_allocator() or @ref ch_set_allocator() will be used by this
* function.
*
* @param db
* The database, as produced by @ref ch_compile().
*
* @param scratch
* On first allocation, a pointer to NULL should be provided so a new
* scratch can be allocated. If a scratch block has been previously
* allocated, then a pointer to it should be passed back in to see if it
* is valid for this database block. If a new scratch block is required,
* the original will be freed and the new one returned, otherwise the
* previous scratch block will be returned. On success, the scratch block
* will be suitable for use with the provided database in addition to any
* databases that original scratch space was suitable for.
*
* @return
* @ref CH_SUCCESS on successful allocation; @ref CH_NOMEM if the
* allocation fails. Other errors may be returned if invalid parameters
* are specified.
*/
ch_error_t HS_CDECL ch_alloc_scratch(const ch_database_t *db,
ch_scratch_t **scratch);
/**
* Allocate a scratch space that is a clone of an existing scratch space.
*
* This is useful when multiple concurrent threads will be using the same set
* of compiled databases, and another scratch space is required. Any allocator
* callback set by @ref ch_set_scratch_allocator() or @ref ch_set_allocator()
* will be used by this function.
*
* @param src
* The existing @ref ch_scratch_t to be cloned.
*
* @param dest
* A pointer to the new scratch space will be returned here.
*
* @return
* @ref CH_SUCCESS on success; @ref CH_NOMEM if the allocation fails.
* Other errors may be returned if invalid parameters are specified.
*/
ch_error_t HS_CDECL ch_clone_scratch(const ch_scratch_t *src,
ch_scratch_t **dest);
/**
* Provides the size of the given scratch space.
*
* @param scratch
* A per-thread scratch space allocated by @ref ch_alloc_scratch() or @ref
* ch_clone_scratch().
*
* @param scratch_size
* On success, the size of the scratch space in bytes is placed in this
* parameter.
*
* @return
* @ref CH_SUCCESS on success, other values on failure.
*/
ch_error_t HS_CDECL ch_scratch_size(const ch_scratch_t *scratch,
size_t *scratch_size);
/**
* Free a scratch block previously allocated by @ref ch_alloc_scratch() or @ref
* ch_clone_scratch().
*
* The free callback set by @ref ch_set_scratch_allocator() or @ref
* ch_set_allocator() will be used by this function.
*
* @param scratch
* The scratch block to be freed. NULL may also be safely provided.
*
* @return
* @ref CH_SUCCESS on success, other values on failure.
*/
ch_error_t HS_CDECL ch_free_scratch(ch_scratch_t *scratch);
#ifdef __cplusplus
} /* extern "C" */
#endif
#endif /* CH_RUNTIME_H_ */

316
chimera/ch_scratch.c
View File

@ -1,316 +0,0 @@
/*
* Copyright (c) 2018, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/** \file
* \brief Chimera: scratch space alloc.
*/
#include <string.h>
#include "allocator.h"
#include "ch.h"
#include "hs.h"
#include "hs_internal.h"
#include "ue2common.h"
#include "ch_alloc.h"
#include "ch_internal.h"
#include "ch_scratch.h"
#include "ch_database.h"
static
size_t getPatternDataSize(const ch_scratch_t *s) {
size_t numCapturingStructs =
s->patternCount * (s->maxCaptureGroups + 1);
return (sizeof(struct ch_patterndata) * s->patternCount) +
alignof(struct ch_capture) + // padding
(sizeof(struct ch_capture) * numCapturingStructs);
}
static
void initPatternData(const ch_scratch_t *s) {
// ch_capture array is aligned, directly after the patterndata array.
char *ptr = (char *)s->patternData +
(sizeof(struct ch_patterndata) * s->patternCount);
struct ch_capture *cap = (struct ch_capture *)
(ROUNDUP_PTR(ptr, alignof(struct ch_capture)));
for (u32 i = 0; i < s->patternCount; i++) {
struct ch_patterndata *pd = &s->patternData[i];
pd->match = cap;
DEBUG_PRINTF("pattern %u: pd=%p, match=%p\n", i, pd, pd->match);
cap += (s->maxCaptureGroups + 1);
}
}
static
ch_error_t alloc_scratch(const ch_scratch_t *proto, ch_scratch_t **scratch) {
size_t ovectorSize = (proto->maxCaptureGroups + 1) * sizeof(int) * 3;
size_t capturedSize =
sizeof(struct ch_capture) * (proto->maxCaptureGroups + 1);
size_t patternDataSize = getPatternDataSize(proto);
size_t activeSize = proto->activeSize;
size_t queueSize = proto->patternCount * sizeof(struct queue_item);
// max padding for alignment below.
size_t padding = alignof(int) + alignof(struct ch_capture) +
alignof(struct ch_patterndata) +
alignof(struct queue_item);
size_t allocSize = sizeof(ch_scratch_t) + ovectorSize + capturedSize +
patternDataSize + activeSize + queueSize + padding
+ 256; /* padding for cacheline alignment */
ch_scratch_t *s;
ch_scratch_t *s_tmp = ch_scratch_alloc(allocSize);
ch_error_t err = ch_check_alloc(s_tmp);
if (err != CH_SUCCESS) {
ch_scratch_free(s_tmp);
*scratch = NULL;
return err;
}
memset(s_tmp, 0, allocSize);
s = ROUNDUP_PTR(s_tmp, 64);
// Set ordinary members.
*s = *proto;
s->magic = CH_SCRATCH_MAGIC;
s->in_use = 0;
s->scratch_alloc = (char *)s_tmp;
// Set pointers internal to allocation.
char *ptr = (char *)s + sizeof(*s);
ptr = ROUNDUP_PTR(ptr, alignof(int));
s->ovector = (int *)ptr;
ptr += ovectorSize;
ptr = ROUNDUP_PTR(ptr, alignof(struct ch_capture));
s->captured = (struct ch_capture *)ptr;
ptr += capturedSize;
ptr = ROUNDUP_PTR(ptr, alignof(struct ch_patterndata));
s->patternData = (struct ch_patterndata *)ptr;
ptr += patternDataSize;
// Pre-fill pattern data, setting captureOffsets
initPatternData(s);
ptr = ROUNDUP_PTR(ptr, alignof(struct queue_item));
s->pq.item = (struct queue_item *)ptr;
ptr += queueSize;
s->active = (u8 *)ptr;
// Store size.
s->scratchSize = allocSize;
// We should never overrun our allocation.
assert((ptr + activeSize) - (char *)s <= (ptrdiff_t)allocSize);
*scratch = s;
return CH_SUCCESS;
}
HS_PUBLIC_API
ch_error_t HS_CDECL ch_alloc_scratch(const ch_database_t *hydb,
ch_scratch_t **scratch) {
if (!hydb || !scratch) {
DEBUG_PRINTF("invalid args\n");
return CH_INVALID;
}
DEBUG_PRINTF("hydb=%p, &scratch=%p\n", hydb, scratch);
ch_error_t rv = hydbIsValid(hydb);
if (rv != CH_SUCCESS) {
DEBUG_PRINTF("invalid database\n");
return rv;
}
if (*scratch != NULL) {
/* has to be aligned before we can do anything with it */
if (!ISALIGNED_CL(*scratch)) {
return CH_INVALID;
}
if ((*scratch)->magic != CH_SCRATCH_MAGIC) {
return CH_INVALID;
}
if (markScratchInUse(*scratch)) {
return CH_SCRATCH_IN_USE;
}
}
// We allocate a prototype of the scratch header to do our sizing with.
ch_scratch_t *proto;
ch_scratch_t *proto_tmp = ch_scratch_alloc(sizeof(ch_scratch_t) + 256);
ch_error_t proto_ret = ch_check_alloc(proto_tmp);
if (proto_ret != CH_SUCCESS) {
ch_scratch_free(proto_tmp);
ch_scratch_free(*scratch);
*scratch = NULL;
return proto_ret;
}
proto = ROUNDUP_PTR(proto_tmp, 64);
int resize = 0;
if (*scratch) {
*proto = **scratch;
} else {
memset(proto, 0, sizeof(*proto));
resize = 1;
}
proto->scratch_alloc = (char *)proto_tmp;
const struct ch_bytecode *db = ch_get_bytecode(hydb);
if (db->maxCaptureGroups > proto->maxCaptureGroups) {
proto->maxCaptureGroups = db->maxCaptureGroups;
resize = 1;
}
if (db->patternCount > proto->patternCount) {
proto->patternCount = db->patternCount;
proto->activeSize = db->activeSize;
resize = 1;
}
if (resize) {
if (*scratch) {
ch_scratch_free((*scratch)->scratch_alloc);
}
ch_error_t alloc_ret = alloc_scratch(proto, scratch);
ch_scratch_free(proto_tmp);
if (alloc_ret != CH_SUCCESS) {
*scratch = NULL;
return alloc_ret;
}
} else {
ch_scratch_free(proto_tmp);
unmarkScratchInUse(*scratch);
}
if (db->flags & CHIMERA_FLAG_NO_MULTIMATCH) {
return CH_SUCCESS;
}
// We may still have to realloc the underlying Hyperscan scratch.
rv = hs_alloc_scratch(getHyperscanDatabase(db),
&(*scratch)->multi_scratch);
if (rv != HS_SUCCESS) {
DEBUG_PRINTF("hs_alloc_scratch for multi_scratch failed\n");
hs_free_scratch((*scratch)->multi_scratch);
ch_scratch_free((*scratch)->scratch_alloc);
*scratch = NULL;
return rv;
}
return CH_SUCCESS;
}
HS_PUBLIC_API
ch_error_t HS_CDECL ch_clone_scratch(const ch_scratch_t *src,
ch_scratch_t **dest) {
if (!dest || !src || !ISALIGNED_CL(src) ||
src->magic != CH_SCRATCH_MAGIC) {
DEBUG_PRINTF("scratch invalid\n");
return CH_INVALID;
}
ch_error_t ret = alloc_scratch(src, dest);
if (ret != CH_SUCCESS) {
DEBUG_PRINTF("alloc_scratch failed\n");
*dest = NULL;
return ret;
}
if (src->multi_scratch) {
(*dest)->multi_scratch = NULL;
ret = hs_clone_scratch(src->multi_scratch, &(*dest)->multi_scratch);
if (ret != HS_SUCCESS) {
DEBUG_PRINTF("hs_clone_scratch(multi_scratch,...) failed\n");
ch_scratch_free(*dest);
return ret;
}
}
return CH_SUCCESS;
}
HS_PUBLIC_API
ch_error_t HS_CDECL ch_free_scratch(ch_scratch_t *scratch) {
ch_error_t ret = CH_SUCCESS;
if (scratch) {
/* has to be aligned before we can do anything with it */
if (!ISALIGNED_CL(scratch)) {
return CH_INVALID;
}
if (scratch->magic != CH_SCRATCH_MAGIC) {
return CH_INVALID;
}
if (markScratchInUse(scratch)) {
return CH_SCRATCH_IN_USE;
}
if (scratch->multi_scratch) {
ret = hs_free_scratch(scratch->multi_scratch);
}
scratch->magic = 0;
assert(scratch->scratch_alloc);
DEBUG_PRINTF("scratch %p is really at %p : freeing\n", scratch,
scratch->scratch_alloc);
ch_scratch_free(scratch->scratch_alloc);
}
return ret;
}
/** Not public, but used for info from our internal tools. Note that in the
* hybrid matcher the scratch is definitely not a contiguous memory region. */
HS_PUBLIC_API
ch_error_t HS_CDECL ch_scratch_size(const ch_scratch_t *scratch, size_t *size) {
ch_error_t ret = CH_SUCCESS;
if (!size || !scratch || !ISALIGNED_CL(scratch) ||
scratch->magic != CH_SCRATCH_MAGIC) {
return CH_INVALID;
} else {
size_t multi_size = 0;
if (scratch->multi_scratch) {
ret = hs_scratch_size(scratch->multi_scratch, &multi_size);
}
if (ret) {
multi_size = 0;
}
*size = scratch->scratchSize + multi_size;
}
return ret;
}

119
chimera/ch_scratch.h
View File

@ -1,119 +0,0 @@
/*
* Copyright (c) 2018, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/** \file
* \brief Scratch and associated data structures.
*
* This header gets pulled into many places (many deep, slow to compile
* places). Try to keep the included headers under control.
*/
#ifndef CH_SCRATCH_H_
#define CH_SCRATCH_H_
#include "ch_common.h"
#include "ch_runtime.h"
#ifdef __cplusplus
extern "C"
{
#endif
#define CH_SCRATCH_MAGIC 0x554F4259 //!< Magic number stored in \ref ch_scratch
struct queue_item {
int from; /** \brief used to store the start location. */
int to; /** \brief used to store the current location. */
u32 id; /**< pattern index. */
};
struct match_pq {
struct queue_item *item;
u32 size; /**< current size of the priority queue */
};
/** \brief Information about a pattern stored at runtime when a match is
* encountered. */
struct ch_patterndata {
struct ch_capture *match; //!< buffered group info
u32 groupCount; //!< number of capturing groups
u32 scanStart; //!< start of match window (still to be single-scanned).
};
/** \brief Scratch space header for Chimera. */
struct ch_scratch {
u32 magic; //!< must be \ref CH_SCRATCH_MAGIC
u8 in_use; /**< non-zero when being used by an API call. */
struct hs_scratch *multi_scratch; //!< for hyperscan scatch.
int *ovector; //!< maximally-sized ovector for PCRE usage.
struct ch_capture *captured; //!< max-sized capture group struct.
u8 *active; //!< active multibit.
struct ch_patterndata *patternData; //!< per-pattern match data, indexed by
// pattern ID.
struct match_pq pq; //!< priority queue to ensure matching ordering
u32 patternCount; //!< number of patterns, used to size active multibit
u32 activeSize; //!< size of active multibit
u32 maxCaptureGroups; //!< largest num of capturing groups required
u32 scratchSize; //!< size of allocation
int ret; //!< return value in Hyperscan callback
char *scratch_alloc; /* user allocated scratch object */
};
/**
* \brief Mark scratch as in use.
*
* Returns non-zero if it was already in use, zero otherwise.
*/
static really_inline
char markScratchInUse(struct ch_scratch *scratch) {
DEBUG_PRINTF("marking scratch as in use\n");
assert(scratch && scratch->magic == CH_SCRATCH_MAGIC);
if (scratch->in_use) {
DEBUG_PRINTF("scratch already in use!\n");
return 1;
}
scratch->in_use = 1;
return 0;
}
/**
* \brief Mark scratch as no longer in use.
*/
static really_inline
void unmarkScratchInUse(struct ch_scratch *scratch) {
DEBUG_PRINTF("marking scratch as not in use\n");
assert(scratch && scratch->magic == CH_SCRATCH_MAGIC);
assert(scratch->in_use == 1);
scratch->in_use = 0;
}
#ifdef __cplusplus
} /* extern "C" */
#endif
#endif /* CH_SCRATCH_H_ */

12
chimera/libch.pc.in
View File

@ -1,12 +0,0 @@
prefix=@CMAKE_INSTALL_PREFIX@
exec_prefix=@CMAKE_INSTALL_PREFIX@
libdir=@CMAKE_INSTALL_PREFIX@/@CMAKE_INSTALL_LIBDIR@
includedir=@CMAKE_INSTALL_PREFIX@/@CMAKE_INSTALL_INCLUDEDIR@
Name: libch
Description: Intel(R) Chimera Library
Version: @HS_VERSION@
Requires.private: libhs
Libs: -L${libdir} -lchimera
Libs.private: @PRIVATE_LIBS@
Cflags: -I${includedir}/hs

196
cmake/arch.cmake
View File

@ -1,196 +0,0 @@
# detect architecture features
#
# must be called after determining where compiler intrinsics are defined
if (HAVE_C_X86INTRIN_H)
set (INTRIN_INC_H "x86intrin.h")
elseif (HAVE_C_INTRIN_H)
set (INTRIN_INC_H "intrin.h")
elseif (HAVE_C_ARM_NEON_H)
set (INTRIN_INC_H "arm_neon.h")
set (FAT_RUNTIME OFF)
elseif (HAVE_C_PPC64EL_ALTIVEC_H)
set (INTRIN_INC_H "altivec.h")
set (FAT_RUNTIME OFF)
else()
message (FATAL_ERROR "No intrinsics header found")
endif ()
if (ARCH_ARM32 OR ARCH_AARCH64)
CHECK_C_SOURCE_COMPILES("#include <${INTRIN_INC_H}>
int main() {
int32x4_t a = vdupq_n_s32(1);
(void)a;
}" HAVE_NEON)
endif ()
if (ARCH_AARCH64)
set(PREV_FLAGS "${CMAKE_C_FLAGS}")
if (BUILD_SVE2_BITPERM)
set(CMAKE_C_FLAGS "-march=${GNUCC_ARCH} ${CMAKE_C_FLAGS}")
CHECK_C_SOURCE_COMPILES("#include <arm_sve.h>
int main() {
svuint8_t a = svbext(svdup_u8(1), svdup_u8(2));
(void)a;
}" HAVE_SVE2_BITPERM)
if (HAVE_SVE2_BITPERM)
add_definitions(-DHAVE_SVE2_BITPERM)
endif ()
endif()
if (BUILD_SVE2)
set(CMAKE_C_FLAGS "-march=${GNUCC_ARCH} ${CMAKE_C_FLAGS}")
CHECK_C_SOURCE_COMPILES("#include <arm_sve.h>
int main() {
svuint8_t a = svbsl(svdup_u8(1), svdup_u8(2), svdup_u8(3));
(void)a;
}" HAVE_SVE2)
endif()
if (HAVE_SVE2 OR HAVE_SVE2_BITPERM)
add_definitions(-DHAVE_SVE2)
endif ()
if (BUILD_SVE)
set(CMAKE_C_FLAGS "-march=${GNUCC_ARCH} ${CMAKE_C_FLAGS}")
CHECK_C_SOURCE_COMPILES("#include <arm_sve.h>
int main() {
svuint8_t a = svdup_u8(1);
(void)a;
}" HAVE_SVE)
endif ()
if (HAVE_SVE OR HAVE_SVE2 OR HAVE_SVE2_BITPERM)
add_definitions(-DHAVE_SVE)
endif ()
set(CMAKE_C_FLAGS "${PREV_FLAGS}")
endif()
if (BUILD_AVX512)
CHECK_C_COMPILER_FLAG(${SKYLAKE_FLAG} HAS_ARCH_SKYLAKE)
if (NOT HAS_ARCH_SKYLAKE)
message (FATAL_ERROR "AVX512 not supported by compiler")
endif ()
endif ()
if (BUILD_AVX512VBMI)
CHECK_C_COMPILER_FLAG(${ICELAKE_FLAG} HAS_ARCH_ICELAKE)
if (NOT HAS_ARCH_ICELAKE)
message (FATAL_ERROR "AVX512VBMI not supported by compiler")
endif ()
endif ()
if (FAT_RUNTIME)
if (NOT DEFINED(BUILD_AVX2))
set(BUILD_AVX2 TRUE)
endif ()
# test the highest level microarch to make sure everything works
if (BUILD_AVX512)
if (BUILD_AVX512VBMI)
set (CMAKE_REQUIRED_FLAGS "${CMAKE_C_FLAGS} ${EXTRA_C_FLAGS} ${ICELAKE_FLAG}")
else ()
set (CMAKE_REQUIRED_FLAGS "${CMAKE_C_FLAGS} ${EXTRA_C_FLAGS} ${SKYLAKE_FLAG}")
endif (BUILD_AVX512VBMI)
elseif (BUILD_AVX2)
set (CMAKE_REQUIRED_FLAGS "${CMAKE_C_FLAGS} ${EXTRA_C_FLAGS} -march=core-avx2 -mavx2")
elseif ()
set (CMAKE_REQUIRED_FLAGS "${CMAKE_C_FLAGS} ${EXTRA_C_FLAGS} -march=core-i7 -mssse3")
endif ()
else (NOT FAT_RUNTIME)
# if not fat runtime, then test given cflags
set (CMAKE_REQUIRED_FLAGS "${CMAKE_C_FLAGS} ${EXTRA_C_FLAGS} ${ARCH_C_FLAGS}")
endif ()
if (ARCH_IA32 OR ARCH_X86_64)
# ensure we have the minimum of SSE4.2 - call a SSE4.2 intrinsic
CHECK_C_SOURCE_COMPILES("#include <${INTRIN_INC_H}>
int main() {
__m128i a = _mm_set1_epi8(1);
(void)_mm_shuffle_epi8(a, a);
}" HAVE_SSE42)
# now look for AVX2
CHECK_C_SOURCE_COMPILES("#include <${INTRIN_INC_H}>
#if !defined(__AVX2__)
#error no avx2
#endif
int main(){
__m256i z = _mm256_setzero_si256();
(void)_mm256_xor_si256(z, z);
}" HAVE_AVX2)
# and now for AVX512
CHECK_C_SOURCE_COMPILES("#include <${INTRIN_INC_H}>
#if !defined(__AVX512BW__)
#error no avx512bw
#endif
int main(){
__m512i z = _mm512_setzero_si512();
(void)_mm512_abs_epi8(z);
}" HAVE_AVX512)
# and now for AVX512VBMI
CHECK_C_SOURCE_COMPILES("#include <${INTRIN_INC_H}>
#if !defined(__AVX512VBMI__)
#error no avx512vbmi
#endif
int main(){
__m512i a = _mm512_set1_epi8(0xFF);
__m512i idx = _mm512_set_epi64(3ULL, 2ULL, 1ULL, 0ULL, 7ULL, 6ULL, 5ULL, 4ULL);
(void)_mm512_permutexvar_epi8(idx, a);
}" HAVE_AVX512VBMI)
elseif (ARCH_ARM32 OR ARCH_AARCH64)
CHECK_C_SOURCE_COMPILES("#include <${INTRIN_INC_H}>
int main() {
int32x4_t a = vdupq_n_s32(1);
(void)a;
}" HAVE_NEON)
elseif (ARCH_PPC64EL)
CHECK_C_SOURCE_COMPILES("#include <${INTRIN_INC_H}>
int main() {
vector int a = vec_splat_s32(1);
(void)a;
}" HAVE_VSX)
else ()
message (FATAL_ERROR "Unsupported architecture")
endif ()
if (FAT_RUNTIME)
if ((ARCH_IA32 OR ARCH_X86_64) AND NOT HAVE_SSE42)
message(FATAL_ERROR "SSE4.2 support required to build fat runtime")
endif ()
if ((ARCH_IA32 OR ARCH_X86_64) AND BUILD_AVX2 AND NOT HAVE_AVX2)
message(FATAL_ERROR "AVX2 support required to build fat runtime")
endif ()
if ((ARCH_IA32 OR ARCH_X86_64) AND BUILD_AVX512 AND NOT HAVE_AVX512)
message(FATAL_ERROR "AVX512 support requested but not supported")
endif ()
if ((ARCH_IA32 OR ARCH_X86_64) AND BUILD_AVX512VBMI AND NOT HAVE_AVX512VBMI)
message(FATAL_ERROR "AVX512VBMI support requested but not supported")
endif ()
else (NOT FAT_RUNTIME)
if ((ARCH_IA32 OR ARCH_X86_64) AND NOT BUILD_AVX2)
message(STATUS "Building without AVX2 support")
endif ()
if ((ARCH_IA32 OR ARCH_X86_64) AND NOT HAVE_AVX512)
message(STATUS "Building without AVX512 support")
endif ()
if ((ARCH_IA32 OR ARCH_X86_64) AND NOT HAVE_AVX512VBMI)
message(STATUS "Building without AVX512VBMI support")
endif ()
if ((ARCH_IA32 OR ARCH_X86_64) AND NOT HAVE_SSE42)
message(FATAL_ERROR "A minimum of SSE4.2 compiler support is required")
endif ()
if ((ARCH_ARM32 OR ARCH_AARCH64) AND NOT HAVE_NEON)
message(FATAL_ERROR "NEON support required for ARM support")
endif ()
if (ARCH_PPPC64EL AND NOT HAVE_VSX)
message(FATAL_ERROR "VSX support required for Power support")
endif ()
endif ()
unset (PREV_FLAGS)
unset (CMAKE_REQUIRED_FLAGS)
unset (INTRIN_INC_H)

22
cmake/arm64-cross.cmake
View File

@ -1,22 +0,0 @@
set(CMAKE_SYSTEM_NAME "Linux")
set(CMAKE_SYSTEM_PROCESSOR "aarch64")
# specify the cross compiler
set(CMAKE_C_COMPILER "$ENV{CROSS}gcc")
set(CMAKE_CXX_COMPILER "$ENV{CROSS}g++")
# where is the target environment
set(CMAKE_SYSROOT $ENV{CROSS_SYS})
set(Boost_INCLUDE_DIR $ENV{BOOST_PATH})
# for libraries and headers in the target directories
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
set(THREADS_PTHREAD_ARG "2" CACHE STRING "Result from TRY_RUN" FORCE)
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -falign-functions=16 -falign-jumps=16 -falign-labels=16 -falign-loops=16" CACHE STRING "" FORCE)
set(GNUCC_ARCH "armv8.2-a+fp16+simd+rcpc+dotprod+crypto")
set(TUNE_FLAG "neoverse-n1")

13
cmake/attrib.cmake
View File

@ -1,13 +0,0 @@
# tests for compiler properties
# set -Werror so we can't ignore unused attribute warnings
set (CMAKE_REQUIRED_FLAGS "-Werror")
CHECK_C_SOURCE_COMPILES("
int foo(int) __attribute__ ((ifunc(\"foo_i\")));
int f1(int i) { return i; }
void (*foo_i()) { return f1; }
int main(void) { return 0; }
" HAS_C_ATTR_IFUNC)
unset(CMAKE_REQUIRED_FLAGS)

56
cmake/backtrace.cmake
View File

@ -1,56 +0,0 @@
# The `backtrace' function is available on Linux via glibc, and on FreeBSD if
# the 'libexecinfo' package is installed.
CHECK_C_SOURCE_COMPILES(
"#include <stdlib.h>\n#include <execinfo.h>\nint main () { backtrace(NULL, 0); }"
BACKTRACE_LIBC)
if(BACKTRACE_LIBC)
set(HAVE_BACKTRACE TRUE)
set(BACKTRACE_CFLAGS "")
set(BACKTRACE_LDFLAGS "")
endif()
if(NOT BACKTRACE_LIBC)
# FreeBSD 10 has backtrace but requires libexecinfo
list(INSERT CMAKE_REQUIRED_LIBRARIES 0 "-lexecinfo")
CHECK_C_SOURCE_COMPILES(
"#include <stdlib.h>\n#include <execinfo.h>\nint main () { backtrace(NULL, 0); }"
BACKTRACE_LIBEXECINFO)
list(REMOVE_ITEM CMAKE_REQUIRED_LIBRARIES "-lexecinfo")
if(BACKTRACE_LIBEXECINFO)
set(HAVE_BACKTRACE TRUE)
set(BACKTRACE_CFLAGS "")
set(BACKTRACE_LDFLAGS "-lexecinfo")
else()
# older FreeBSD requires it from ports
list(INSERT CMAKE_REQUIRED_INCLUDES 0 "/usr/local/include")
list(INSERT CMAKE_REQUIRED_LIBRARIES 0 "-L/usr/local/lib -lexecinfo")
CHECK_C_SOURCE_COMPILES(
"#include <stdlib.h>\n#include <execinfo.h>\nint main () { backtrace(NULL, 0); }"
BACKTRACE_LIBEXECINFO_LOCAL)
list(REMOVE_ITEM CMAKE_REQUIRED_INCLUDES 0 "/usr/local/include")
list(REMOVE_ITEM CMAKE_REQUIRED_LIBRARIES "-L/usr/local/lib -lexecinfo")
if(BACKTRACE_LIBEXECINFO_LOCAL)
set(HAVE_BACKTRACE TRUE)
set(BACKTRACE_CFLAGS "-I/usr/local/include")
set(BACKTRACE_LDFLAGS "-L/usr/local/lib -lexecinfo")
endif()
endif()
endif()
if(HAVE_BACKTRACE)
CHECK_C_COMPILER_FLAG(-rdynamic HAS_RDYNAMIC)
if(HAS_RDYNAMIC)
list(INSERT BACKTRACE_LDFLAGS 0 -rdynamic)
endif()
else()
set(BACKTRACE_CFLAGS "")
set(BACKTRACE_LDFLAGS "")
endif()
# cmake scope fun
set(HAVE_BACKTRACE ${HAVE_BACKTRACE} CACHE BOOL INTERNAL)
set(BACKTRACE_CFLAGS ${BACKTRACE_CFLAGS} CACHE STRING INTERNAL)
set(BACKTRACE_LDFLAGS ${BACKTRACE_LDFLAGS} CACHE STRING INTERNAL)

72
cmake/boost.cmake
View File

@ -1,72 +0,0 @@
# Various checks related to Boost
set(BOOST_USE_STATIC_LIBS OFF)
set(BOOST_USE_MULTITHREADED OFF)
set(BOOST_USE_STATIC_RUNTIME OFF)
if (HAVE_LIBCPP)
# we need a more recent boost for libc++
set(BOOST_MINVERSION 1.61.0)
else ()
set(BOOST_MINVERSION 1.57.0)
endif ()
set(BOOST_NO_BOOST_CMAKE ON)
unset(Boost_INCLUDE_DIR CACHE)
# we might have boost in tree, so provide a hint and try again
set(BOOST_INCLUDEDIR "${PROJECT_SOURCE_DIR}/include")
find_package(Boost ${BOOST_MINVERSION} QUIET)
if(NOT Boost_FOUND)
# otherwise check for Boost installed on the system
unset(BOOST_INCLUDEDIR)
find_package(Boost ${BOOST_MINVERSION} QUIET)
if(NOT Boost_FOUND)
message(FATAL_ERROR "Boost ${BOOST_MINVERSION} or later not found. Either install system packages if available, extract Boost headers to ${CMAKE_SOURCE_DIR}/include, or set the CMake BOOST_ROOT variable.")
endif()
endif()
message(STATUS "Boost version: ${Boost_MAJOR_VERSION}.${Boost_MINOR_VERSION}.${Boost_SUBMINOR_VERSION}")
# Boost 1.62 has a bug that we've patched around, check if it is required
if (Boost_VERSION EQUAL 106200)
set (CMAKE_REQUIRED_INCLUDES ${BOOST_INCLUDEDIR} "${PROJECT_SOURCE_DIR}/include")
set (BOOST_REV_TEST "
#include <boost/graph/graph_concepts.hpp>
#include <boost/graph/adjacency_list.hpp>
#include <boost/graph/reverse_graph.hpp>
#include <boost/concept/assert.hpp>
int main(int,char*[])
{
using namespace boost;
// Check const reverse_graph
{
typedef adjacency_list< vecS, vecS, bidirectionalS,
property<vertex_color_t, int>,
property<edge_weight_t, int>,
property<graph_name_t, std::string>
> AdjList;
typedef reverse_graph<AdjList> Graph;
BOOST_CONCEPT_ASSERT(( BidirectionalGraphConcept<Graph> ));
}
return 0;
}
")
CHECK_CXX_SOURCE_COMPILES("${BOOST_REV_TEST}" BOOST_REVGRAPH_OK)
if (NOT BOOST_REVGRAPH_OK)
message(STATUS "trying patched")
CHECK_CXX_SOURCE_COMPILES("
#include <boost-patched/graph/reverse_graph.hpp>
${BOOST_REV_TEST}" BOOST_REVGRAPH_PATCH)
endif()
if (NOT BOOST_REVGRAPH_OK AND NOT BOOST_REVGRAPH_PATCH)
message(FATAL_ERROR "Something is wrong with this copy of boost::reverse_graph")
endif()
unset (CMAKE_REQUIRED_INCLUDES)
else ()
unset(BOOST_REVGRAPH_OK CACHE)
unset(BOOST_REVGRAPH_PATCH CACHE)
endif () # Boost 1.62.0

28
cmake/build_wrapper.sh
View File

@ -1,28 +0,0 @@
#!/bin/sh -e
# This is used for renaming symbols for the fat runtime, don't call directly
# TODO: make this a lot less fragile!
cleanup () {
rm -f ${SYMSFILE} ${KEEPSYMS}
}
PREFIX=$1
KEEPSYMS_IN=$2
shift 2
# $@ contains the actual build command
OUT=$(echo "$@" | rev | cut -d ' ' -f 2- | rev | sed 's/.* -o \(.*\.o\).*/\1/')
trap cleanup INT QUIT EXIT
SYMSFILE=$(mktemp -p /tmp ${PREFIX}_rename.syms.XXXXX)
KEEPSYMS=$(mktemp -p /tmp keep.syms.XXXXX)
# find the libc used by gcc
LIBC_SO=$("$@" --print-file-name=libc.so.6)
cp ${KEEPSYMS_IN} ${KEEPSYMS}
# get all symbols from libc and turn them into patterns
nm -f p -g -D ${LIBC_SO} | sed -s 's/\([^ @]*\).*/^\1$/' >> ${KEEPSYMS}
# build the object
"$@"
# rename the symbols in the object
nm -f p -g ${OUT} | cut -f1 -d' ' | grep -v -f ${KEEPSYMS} | sed -e "s/\(.*\)/\1\ ${PREFIX}_\1/" >> ${SYMSFILE}
if test -s ${SYMSFILE}
then
objcopy --redefine-syms=${SYMSFILE} ${OUT}
fi

142
cmake/config.h.in
View File

@ -1,142 +0,0 @@
/* used by cmake */
#ifndef CONFIG_H_
#define CONFIG_H_
/* "Define if the build is 32 bit" */
#cmakedefine ARCH_32_BIT
/* "Define if the build is 64 bit" */
#cmakedefine ARCH_64_BIT
/* "Define if building for IA32" */
#cmakedefine ARCH_IA32
/* "Define if building for EM64T" */
#cmakedefine ARCH_X86_64
/* "Define if building for ARM32" */
#cmakedefine ARCH_ARM32
/* "Define if building for AARCH64" */
#cmakedefine ARCH_AARCH64
/* "Define if building for PPC64EL" */
#cmakedefine ARCH_PPC64EL
/* "Define if cross compiling for AARCH64" */
#cmakedefine CROSS_COMPILE_AARCH64
/* Define if building SVE for AARCH64. */
#cmakedefine BUILD_SVE
/* Define if building SVE2 for AARCH64. */
#cmakedefine BUILD_SVE2
/* Define if building SVE2+BITPERM for AARCH64. */
#cmakedefine BUILD_SVE2_BITPERM
/* internal build, switch on dump support. */
#cmakedefine DUMP_SUPPORT
/* Define if building "fat" runtime. */
#cmakedefine FAT_RUNTIME
/* Define if building AVX2 in the fat runtime. */
#cmakedefine BUILD_AVX2
/* Define if building AVX-512 in the fat runtime. */
#cmakedefine BUILD_AVX512
/* Define if building AVX512VBMI in the fat runtime. */
#cmakedefine BUILD_AVX512VBMI
/* Define to 1 if `backtrace' works. */
#cmakedefine HAVE_BACKTRACE
/* C compiler has __builtin_assume_aligned */
#cmakedefine HAVE_CC_BUILTIN_ASSUME_ALIGNED
/* C++ compiler has __builtin_assume_aligned */
#cmakedefine HAVE_CXX_BUILTIN_ASSUME_ALIGNED
/* C++ compiler has x86intrin.h */
#cmakedefine HAVE_CXX_X86INTRIN_H
/* C compiler has x86intrin.h */
#cmakedefine HAVE_C_X86INTRIN_H
/* C++ compiler has intrin.h */
#cmakedefine HAVE_CXX_INTRIN_H
/* C compiler has intrin.h */
#cmakedefine HAVE_C_INTRIN_H
/* C compiler has arm_neon.h */
#cmakedefine HAVE_C_ARM_NEON_H
/* C compiler has arm_sve.h */
#cmakedefine HAVE_C_ARM_SVE_H
/* C compiler has arm_neon.h */
#cmakedefine HAVE_C_PPC64EL_ALTIVEC_H
/* Define to 1 if you have the declaration of `pthread_setaffinity_np', and to
0 if you don't. */
#cmakedefine HAVE_DECL_PTHREAD_SETAFFINITY_NP
#cmakedefine HAVE_PTHREAD_NP_H
/* Define to 1 if you have the `malloc_info' function. */
#cmakedefine HAVE_MALLOC_INFO
/* Define to 1 if you have the `memmem' function. */
#cmakedefine HAVE_MEMMEM
/* Define to 1 if you have a working `mmap' system call. */
#cmakedefine HAVE_MMAP
/* Define to 1 if `posix_memalign' works. */
#cmakedefine HAVE_POSIX_MEMALIGN
/* Define to 1 if you have the `setrlimit' function. */
#cmakedefine HAVE_SETRLIMIT
/* Define to 1 if you have the `shmget' function. */
#cmakedefine HAVE_SHMGET
/* Define to 1 if you have the `sigaction' function. */
#cmakedefine HAVE_SIGACTION
/* Define to 1 if you have the `sigaltstack' function. */
#cmakedefine HAVE_SIGALTSTACK
/* Define if the sqlite3_open_v2 call is available */
#cmakedefine HAVE_SQLITE3_OPEN_V2
/* Define to 1 if you have the <unistd.h> header file. */
#cmakedefine HAVE_UNISTD_H
/* Define to 1 if you have the `_aligned_malloc' function. */
#cmakedefine HAVE__ALIGNED_MALLOC
/* Define if compiler has __builtin_constant_p */
#cmakedefine HAVE__BUILTIN_CONSTANT_P
/* Optimize, inline critical functions */
#cmakedefine HS_OPTIMIZE
#cmakedefine HS_VERSION
#cmakedefine HS_MAJOR_VERSION
#cmakedefine HS_MINOR_VERSION
#cmakedefine HS_PATCH_VERSION
#cmakedefine BUILD_DATE
/* define if this is a release build. */
#cmakedefine RELEASE_BUILD
/* define if reverse_graph requires patch for boost 1.62.0 */
#cmakedefine BOOST_REVGRAPH_PATCH
#endif /* CONFIG_H_ */

18
cmake/formatdate.py
View File

@ -1,18 +0,0 @@
#!/usr/bin/env python
from __future__ import print_function
import os
import sys
import datetime
def usage():
print("Usage:", os.path.basename(sys.argv[0]), "<seconds from epoch>")
if len(sys.argv) != 2:
usage()
sys.exit(1)
ts = sys.argv[1]
build_date = datetime.datetime.utcfromtimestamp(int(ts))
print(build_date.strftime("%Y-%m-%d"))

11
cmake/keep.syms.in
View File

@ -1,11 +0,0 @@
# names to exclude
hs_misc_alloc
hs_misc_free
hs_free_scratch
hs_stream_alloc
hs_stream_free
hs_scratch_alloc
hs_scratch_free
hs_database_alloc
hs_database_free
^_

63
cmake/pcre.cmake
View File

@ -1,63 +0,0 @@
# first look in pcre-$version or pcre subdirs
if (PCRE_SOURCE)
# either provided on cmdline or we've seen it already
set (PCRE_BUILD_SOURCE TRUE)
elseif (EXISTS ${PROJECT_SOURCE_DIR}/pcre-${PCRE_REQUIRED_VERSION})
set (PCRE_SOURCE ${PROJECT_SOURCE_DIR}/pcre-${PCRE_REQUIRED_VERSION})
set (PCRE_BUILD_SOURCE TRUE)
elseif (EXISTS ${PROJECT_SOURCE_DIR}/pcre)
set (PCRE_SOURCE ${PROJECT_SOURCE_DIR}/pcre)
set (PCRE_BUILD_SOURCE TRUE)
endif()
if (PCRE_BUILD_SOURCE)
if (NOT IS_ABSOLUTE ${PCRE_SOURCE})
set(PCRE_SOURCE "${CMAKE_BINARY_DIR}/${PCRE_SOURCE}")
endif ()
set (saved_INCLUDES "${CMAKE_REQUIRED_INCLUDES}")
set (CMAKE_REQUIRED_INCLUDES "${CMAKE_REQUIRED_INCLUDES} ${PCRE_SOURCE}")
if (PCRE_CHECKED)
set(PCRE_INCLUDE_DIRS ${PCRE_SOURCE} ${PROJECT_BINARY_DIR}/pcre)
set(PCRE_LDFLAGS -L"${LIBDIR}" -lpcre)
# already processed this file and set up pcre building
return()
endif ()
# first, check version number
CHECK_C_SOURCE_COMPILES("#include <pcre.h.generic>
#if PCRE_MAJOR != ${PCRE_REQUIRED_MAJOR_VERSION} || PCRE_MINOR < ${PCRE_REQUIRED_MINOR_VERSION}
#error Incorrect pcre version
#endif
main() {}" CORRECT_PCRE_VERSION)
set (CMAKE_REQUIRED_INCLUDES "${saved_INCLUDES}")
if (NOT CORRECT_PCRE_VERSION)
unset(CORRECT_PCRE_VERSION CACHE)
message(STATUS "Incorrect version of pcre - version ${PCRE_REQUIRED_VERSION} or above is required")
return ()
else()
message(STATUS "PCRE version ${PCRE_REQUIRED_VERSION} or above - building from source.")
endif()
# PCRE compile options
option(PCRE_BUILD_PCRECPP OFF)
option(PCRE_BUILD_PCREGREP OFF)
option(PCRE_SHOW_REPORT OFF)
set(PCRE_SUPPORT_UNICODE_PROPERTIES ON CACHE BOOL "Build pcre with unicode")
add_subdirectory(${PCRE_SOURCE} ${PROJECT_BINARY_DIR}/pcre EXCLUDE_FROM_ALL)
set(PCRE_INCLUDE_DIRS ${PCRE_SOURCE} ${PROJECT_BINARY_DIR}/pcre)
set(PCRE_LDFLAGS -L"${LIBDIR}" -lpcre)
else ()
# pkgconf should save us
find_package(PkgConfig)
pkg_check_modules(PCRE libpcre>=${PCRE_REQUIRED_VERSION})
if (PCRE_FOUND)
set(CORRECT_PCRE_VERSION TRUE)
message(STATUS "PCRE version ${PCRE_REQUIRED_VERSION} or above")
else ()
message(STATUS "PCRE version ${PCRE_REQUIRED_VERSION} or above not found")
return ()
endif ()
endif (PCRE_BUILD_SOURCE)

24
cmake/platform.cmake
View File

@ -1,24 +0,0 @@
# determine compiler
if (CMAKE_CXX_COMPILER_ID MATCHES "Clang")
set(CMAKE_COMPILER_IS_CLANG TRUE)
endif()
# determine the target arch
if (CROSS_COMPILE_AARCH64)
set(ARCH_AARCH64 TRUE)
set(ARCH_64_BIT TRUE)
message(STATUS "Cross compiling for aarch64")
else()
# really only interested in the preprocessor here
CHECK_C_SOURCE_COMPILES("#if !(defined(__x86_64__) || defined(_M_X64))\n#error not 64bit\n#endif\nint main(void) { return 0; }" ARCH_X86_64)
CHECK_C_SOURCE_COMPILES("#if !(defined(__i386__) || defined(_M_IX86))\n#error not 32bit\n#endif\nint main(void) { return 0; }" ARCH_IA32)
CHECK_C_SOURCE_COMPILES("#if !defined(__ARM_ARCH_ISA_A64)\n#error not 64bit\n#endif\nint main(void) { return 0; }" ARCH_AARCH64)
CHECK_C_SOURCE_COMPILES("#if !defined(__ARM_ARCH_ISA_ARM)\n#error not 32bit\n#endif\nint main(void) { return 0; }" ARCH_ARM32)
CHECK_C_SOURCE_COMPILES("#if !defined(__PPC64__) && !(defined(__LITTLE_ENDIAN__) && defined(__VSX__))\n#error not ppc64el\n#endif\nint main(void) { return 0; }" ARCH_PPC64EL)
if (ARCH_X86_64 OR ARCH_AARCH64 OR ARCH_PPC64EL)
set(ARCH_64_BIT TRUE)
else()
set(ARCH_32_BIT TRUE)
endif()
endif()

16
cmake/ragel.cmake
View File

@ -1,16 +0,0 @@
# function for doing all the dirty work in turning a .rl into C++
function(ragelmaker src_rl)
get_filename_component(src_dir ${src_rl} PATH) # old cmake needs PATH
get_filename_component(src_file ${src_rl} NAME_WE)
set(rl_out ${CMAKE_CURRENT_BINARY_DIR}/${src_dir}/${src_file}.cpp)
add_custom_command(
OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/${src_dir}/${src_file}.cpp
COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_CURRENT_BINARY_DIR}/${src_dir}
COMMAND ${RAGEL} ${CMAKE_CURRENT_SOURCE_DIR}/${src_rl} -o ${rl_out}
DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/${src_rl}
)
add_custom_target(ragel_${src_file} DEPENDS ${rl_out})
set_source_files_properties(${rl_out} PROPERTIES GENERATED TRUE)
endfunction(ragelmaker)

19
cmake/setenv-arm64-cross.sh
View File

@ -1,19 +0,0 @@
#!/bin/bash
export BOOST_VERSION=1_57_0
export BOOST_DOT_VERSION=${BOOST_VERSION//_/.}
export CROSS=<arm-cross-compiler-dir>/bin/aarch64-linux-gnu-
export CROSS_SYS=<arm-cross-compiler-system-dir>
# if [ ! -d "boost_$BOOST_VERSION" ];
# then
# wget -O boost_$BOOST_VERSION.tar.gz https://sourceforge.net/projects/boost/files/boost/$BOOST_DOT_VERSION/boost_$BOOST_VERSION.tar.gz/download
# tar xf boost_$BOOST_VERSION.tar.gz
# fi
if [ ! -d "pcre-8.41" ];
then
wget -O pcre-8.41.tar.bz2 https://ftp.pcre.org/pub/pcre/pcre-8.41.tar.bz2
tar xf pcre-8.41.tar.bz2
export PCRE_SOURCE=1
fi
export BOOST_PATH=<boost-source-dir>

51
cmake/sqlite3.cmake
View File

@ -1,51 +0,0 @@
#
# a lot of noise to find sqlite
#
option(SQLITE_PREFER_STATIC "Build sqlite3 statically instead of using an installed lib" OFF)
if(NOT SQLITE_PREFER_STATIC)
find_package(PkgConfig QUIET)
# first check for sqlite on the system
pkg_check_modules(SQLITE3 sqlite3)
endif()
if (NOT SQLITE3_FOUND)
message(STATUS "looking for sqlite3 in source tree")
# look in the source tree
if (EXISTS "${PROJECT_SOURCE_DIR}/sqlite3/sqlite3.h" AND
EXISTS "${PROJECT_SOURCE_DIR}/sqlite3/sqlite3.c")
message(STATUS " found sqlite3 in source tree")
set(SQLITE3_FOUND TRUE)
set(SQLITE3_BUILD_SOURCE TRUE)
set(SQLITE3_INCLUDE_DIRS "${PROJECT_SOURCE_DIR}/sqlite3")
set(SQLITE3_LDFLAGS sqlite3_static)
else()
message(STATUS " no sqlite3 in source tree")
endif()
endif()
# now do version checks
if (SQLITE3_FOUND)
list(INSERT CMAKE_REQUIRED_INCLUDES 0 "${SQLITE3_INCLUDE_DIRS}")
CHECK_C_SOURCE_COMPILES("#include <sqlite3.h>\n#if SQLITE_VERSION_NUMBER >= 3008007 && SQLITE_VERSION_NUMBER < 3008010\n#error broken sqlite\n#endif\nint main() {return 0;}" SQLITE_VERSION_OK)
if (NOT SQLITE_VERSION_OK)
message(FATAL_ERROR "sqlite3 is broken from 3.8.7 to 3.8.10 - please find a working version")
endif()
if (NOT SQLITE3_BUILD_SOURCE)
set(_SAVED_FLAGS ${CMAKE_REQUIRED_FLAGS})
list(INSERT CMAKE_REQUIRED_LIBRARIES 0 ${SQLITE3_LDFLAGS})
CHECK_SYMBOL_EXISTS(sqlite3_open_v2 sqlite3.h HAVE_SQLITE3_OPEN_V2)
list(REMOVE_ITEM CMAKE_REQUIRED_INCLUDES "${SQLITE3_INCLUDE_DIRS}")
list(REMOVE_ITEM CMAKE_REQUIRED_LIBRARIES ${SQLITE3_LDFLAGS})
else()
if (NOT TARGET sqlite3_static)
# build sqlite as a static lib to compile into our test programs
add_library(sqlite3_static STATIC "${PROJECT_SOURCE_DIR}/sqlite3/sqlite3.c")
set_target_properties(sqlite3_static PROPERTIES COMPILE_FLAGS "-Wno-error -Wno-extra -Wno-unused -Wno-cast-qual -DSQLITE_OMIT_LOAD_EXTENSION")
endif()
endif()
endif()
# that's enough about sqlite

35
doc/dev-reference/CMakeLists.txt
View File

@ -1,35 +0,0 @@
find_program(DOXYGEN doxygen)
if (DOXYGEN STREQUAL DOXYGEN-NOTFOUND)
message(STATUS "Doxygen not found, unable to generate API reference")
else()
configure_file("${CMAKE_CURRENT_SOURCE_DIR}/hyperscan.doxyfile.in"
"${CMAKE_CURRENT_BINARY_DIR}/hyperscan.doxyfile" @ONLY)
add_custom_target(dev-reference-doxygen
${DOXYGEN} ${CMAKE_CURRENT_BINARY_DIR}/hyperscan.doxyfile
COMMENT "Building doxygen XML for API reference")
endif()
find_program(SPHINX_BUILD sphinx-build)
if (SPHINX_BUILD STREQUAL SPHINX_BUILD-NOTFOUND)
message(STATUS "Sphinx not found, unable to generate developer reference")
else()
set(SPHINX_BUILD_DIR "${CMAKE_CURRENT_BINARY_DIR}/_build")
set(SPHINX_CACHE_DIR "${CMAKE_CURRENT_BINARY_DIR}/_doctrees")
set(SPHINX_HTML_DIR "${CMAKE_CURRENT_BINARY_DIR}/html")
configure_file("${CMAKE_CURRENT_SOURCE_DIR}/conf.py.in"
"${CMAKE_CURRENT_BINARY_DIR}/conf.py" @ONLY)
add_custom_target(dev-reference
${SPHINX_BUILD}
-b html
-c "${CMAKE_CURRENT_BINARY_DIR}"
-d "${SPHINX_CACHE_DIR}"
"${CMAKE_CURRENT_SOURCE_DIR}"
"${SPHINX_HTML_DIR}"
DEPENDS dev-reference-doxygen
COMMENT "Building HTML dev reference with Sphinx")
endif()

10
doc/dev-reference/_static/hyperscan.css
View File

@ -1,10 +0,0 @@
/* Differentiate the way we display regex fragments. */
.regexp {
color: darkred !important;
}
/* Avoid (the alabaster theme default) Goudy Old Style, which renders in
* italics on some Mac/Safari systems. */
body {
font-family: 'minion pro', 'bell mt', Georgia, 'Hiragino Mincho Pro', serif;
}

53
doc/dev-reference/api_constants.rst
View File

@ -1,53 +0,0 @@
.. _api_constants:
########################
API Reference: Constants
########################
***********
Error Codes
***********
.. doxygengroup:: HS_ERROR
:content-only:
:no-link:
*****************
hs_expr_ext flags
*****************
.. doxygengroup:: HS_EXT_FLAG
:content-only:
:no-link:
*************
Pattern flags
*************
.. doxygengroup:: HS_PATTERN_FLAG
:content-only:
:no-link:
*************************
CPU feature support flags
*************************
.. doxygengroup:: HS_CPU_FEATURES_FLAG
:content-only:
:no-link:
****************
CPU tuning flags
****************
.. doxygengroup:: HS_TUNE_FLAG
:content-only:
:no-link:
******************
Compile mode flags
******************
.. doxygengroup:: HS_MODE_FLAG
:content-only:
:no-link:

29
doc/dev-reference/api_files.rst
View File

@ -1,29 +0,0 @@
.. _api_files:
####################
API Reference: Files
####################
**********
File: hs.h
**********
.. doxygenfile:: hs.h
*****************
File: hs_common.h
*****************
.. doxygenfile:: hs_common.h
******************
File: hs_compile.h
******************
.. doxygenfile:: hs_compile.h
******************
File: hs_runtime.h
******************
.. doxygenfile:: hs_runtime.h

333
doc/dev-reference/chimera.rst
View File

@ -1,333 +0,0 @@
.. _chimera:
#######
Chimera
#######
This section describes Chimera library.
************
Introduction
************
Chimera is a software regular expression matching engine that is a hybrid of
Hyperscan and PCRE. The design goals of Chimera are to fully support PCRE
syntax as well as to take advantage of the high performance nature of Hyperscan.
Chimera inherits the design guideline of Hyperscan with C APIs for compilation
and scanning.
The Chimera API itself is composed of two major components:
===========
Compilation
===========
These functions take a group of regular expressions, along with identifiers and
option flags, and compile them into an immutable database that can be used by
the Chimera scanning API. This compilation process performs considerable
analysis and optimization work in order to build a database that will match
the given expressions efficiently.
See :ref:`chcompile` for more details
========
Scanning
========
Once a Chimera database has been created, it can be used to scan data in memory.
Chimera only supports block mode in which we scan a single contiguous block in
memory.
Matches are delivered to the application via a user-supplied callback function
that is called synchronously for each match.
For a given database, Chimera provides several guarantees:
* No memory allocations occur at runtime with the exception of scratch space
allocation, it should be done ahead of time for performance-critical
applications:
- **Scratch space**: temporary memory used for internal data at scan time.
Structures in scratch space do not persist beyond the end of a single scan
call.
* The size of the scratch space required for a given database is fixed and
determined at database compile time. This means that the memory requirement
of the application are known ahead of time, and the scratch space can be
pre-allocated if required for performance reasons.
* Any pattern that has successfully been compiled by the Chimera compiler can
be scanned against any input. There could be internal resource limits or
other limitations caused by PCRE at runtime that could cause a scan call to
return an error.
.. note:: Chimera is designed to have the same matching behavior as PCRE,
including greedy/ungreedy, capturing, etc. Chimera reports both
**start offset** and **end offset** for each match like PCRE. Different
from the fashion of reporting all matches in Hyperscan, Chimera only reports
non-overlapping matches. For example, the pattern :regexp:`/foofoo/` will
match ``foofoofoofoo`` at offsets (0, 6) and (6, 12).
.. note:: Since Chimera is a hybrid of Hyperscan and PCRE in order to support
full PCRE syntax, there will be extra performance overhead compared to
Hyperscan-only solution. Please always use Hyperscan for better performance
unless you must need full PCRE syntax support.
See :ref:`chruntime` for more details
************
Requirements
************
The PCRE library (http://pcre.org/) version 8.41 is required for Chimera.
.. note:: Since Chimera needs to reference PCRE internal function, please place PCRE source
directory under Hyperscan root directory in order to build Chimera.
Beside this, both hardware and software requirements of Chimera are the same to Hyperscan.
See :ref:`hardware` and :ref:`software` for more details.
.. note:: Building Hyperscan will automatically generate Chimera library.
Currently only static library is supported for Chimera, so please
use static build type when configure CMake build options.
.. _chcompile:
******************
Compiling Patterns
******************
===================
Building a Database
===================
The Chimera compiler API accepts regular expressions and converts them into a
compiled pattern database that can then be used to scan data.
The API provides two functions that compile regular expressions into
databases:
#. :c:func:`ch_compile`: compiles a single expression into a pattern database.
#. :c:func:`ch_compile_multi`: compiles an array of expressions into a pattern
database. All of the supplied patterns will be scanned for concurrently at
scan time, with user-supplied identifiers returned when they match.
#. :c:func:`ch_compile_ext_multi`: compiles an array of expressions as above,
but allows PCRE match limits to be specified for each expression.
Compilation allows the Chimera library to analyze the given pattern(s) and
pre-determine how to scan for these patterns in an optimized fashion using
Hyperscan and PCRE.
===============
Pattern Support
===============
Chimera fully supports the pattern syntax used by the PCRE library ("libpcre"),
described at <http://www.pcre.org/>.The version of PCRE used to validate
Chimera's interpretation of this syntax is 8.41.
=========
Semantics
=========
Chimera supports the exact same semantics of PCRE library. Moreover, it supports
multiple simultaneous pattern matching like Hyperscan and the multiple matches
will be reported in order by end offset.
.. _chruntime:
*********************
Scanning for Patterns
*********************
Chimera provides scan function with ``ch_scan``.
================
Handling Matches
================
``ch_scan`` will call a user-supplied callback function when a match
is found. This function has the following signature:
.. doxygentypedef:: ch_match_event_handler
:outline:
:no-link:
The *id* argument will be set to the identifier for the matching expression
provided at compile time, and the *from* argument will be set to the
start-offset of the match the *to* argument will be set to the end-offset
of the match. The *captured* stores offsets of entire pattern match as well as
captured subexpressions. The *size* will be set to the number of valid entries in
the *captured*.
The match callback function has the capability to continue or halt scanning
by returning different values.
See :c:type:`ch_match_event_handler` for more information.
=======================
Handling Runtime Errors
=======================
``ch_scan`` will call a user-supplied callback function when a runtime error
occurs in libpcre. This function has the following signature:
.. doxygentypedef:: ch_error_event_handler
:outline:
:no-link:
The *id* argument will be set to the identifier for the matching expression
provided at compile time.
The match callback function has the capability to either halt scanning or
continue scanning for the next pattern.
See :c:type:`ch_error_event_handler` for more information.
=============
Scratch Space
=============
While scanning data, Chimera needs a small amount of temporary memory to store
on-the-fly internal data. This amount is unfortunately too large to fit on the
stack, particularly for embedded applications, and allocating memory dynamically
is too expensive, so a pre-allocated "scratch" space must be provided to the
scanning functions.
The function :c:func:`ch_alloc_scratch` allocates a large enough region of
scratch space to support a given database. If the application uses multiple
databases, only a single scratch region is necessary: in this case, calling
:c:func:`ch_alloc_scratch` on each database (with the same ``scratch`` pointer)
will ensure that the scratch space is large enough to support scanning against
any of the given databases.
While the Chimera library is re-entrant, the use of scratch spaces is not.
For example, if by design it is deemed necessary to run recursive or nested
scanning (say, from the match callback function), then an additional scratch
space is required for that context.
In the absence of recursive scanning, only one such space is required per thread
and can (and indeed should) be allocated before data scanning is to commence.
In a scenario where a set of expressions are compiled by a single "main"
thread and data will be scanned by multiple "worker" threads, the convenience
function :c:func:`ch_clone_scratch` allows multiple copies of an existing
scratch space to be made for each thread (rather than forcing the caller to pass
all the compiled databases through :c:func:`ch_alloc_scratch` multiple times).
For example:
.. code-block:: c
ch_error_t err;
ch_scratch_t *scratch_prototype = NULL;
err = ch_alloc_scratch(db, &scratch_prototype);
if (err != CH_SUCCESS) {
printf("ch_alloc_scratch failed!");
exit(1);
}
ch_scratch_t *scratch_thread1 = NULL;
ch_scratch_t *scratch_thread2 = NULL;
err = ch_clone_scratch(scratch_prototype, &scratch_thread1);
if (err != CH_SUCCESS) {
printf("ch_clone_scratch failed!");
exit(1);
}
err = ch_clone_scratch(scratch_prototype, &scratch_thread2);
if (err != CH_SUCCESS) {
printf("ch_clone_scratch failed!");
exit(1);
}
ch_free_scratch(scratch_prototype);
/* Now two threads can both scan against database db,
each with its own scratch space. */
=================
Custom Allocators
=================
By default, structures used by Chimera at runtime (scratch space, etc) are
allocated with the default system allocators, usually
``malloc()`` and ``free()``.
The Chimera API provides a facility for changing this behaviour to support
applications that use custom memory allocators.
These functions are:
- :c:func:`ch_set_database_allocator`, which sets the allocate and free functions
used for compiled pattern databases.
- :c:func:`ch_set_scratch_allocator`, which sets the allocate and free
functions used for scratch space.
- :c:func:`ch_set_misc_allocator`, which sets the allocate and free functions
used for miscellaneous data, such as compile error structures and
informational strings.
The :c:func:`ch_set_allocator` function can be used to set all of the custom
allocators to the same allocate/free pair.
************************
API Reference: Constants
************************
===========
Error Codes
===========
.. doxygengroup:: CH_ERROR
:content-only:
:no-link:
=============
Pattern flags
=============
.. doxygengroup:: CH_PATTERN_FLAG
:content-only:
:no-link:
==================
Compile mode flags
==================
.. doxygengroup:: CH_MODE_FLAG
:content-only:
:no-link:
********************
API Reference: Files
********************
==========
File: ch.h
==========
.. doxygenfile:: ch.h
=================
File: ch_common.h
=================
.. doxygenfile:: ch_common.h
==================
File: ch_compile.h
==================
.. doxygenfile:: ch_compile.h
==================
File: ch_runtime.h
==================
.. doxygenfile:: ch_runtime.h

633
doc/dev-reference/compilation.rst
View File

@ -1,633 +0,0 @@
.. include:: <isonum.txt>
.. _compilation:
##################
Compiling Patterns
##################
*******************
Building a Database
*******************
The Hyperscan compiler API accepts regular expressions and converts them into a
compiled pattern database that can then be used to scan data.
The API provides three functions that compile regular expressions into
databases:
#. :c:func:`hs_compile`: compiles a single expression into a pattern database.
#. :c:func:`hs_compile_multi`: compiles an array of expressions into a pattern
database. All of the supplied patterns will be scanned for concurrently at
scan time, with user-supplied identifiers returned when they match.
#. :c:func:`hs_compile_ext_multi`: compiles an array of expressions as above,
but allows :ref:`extparam` to be specified for each expression.
Compilation allows the Hyperscan library to analyze the given pattern(s) and
pre-determine how to scan for these patterns in an optimized fashion that would
be far too expensive to compute at run-time.
When compiling expressions, a decision needs to be made whether the resulting
compiled patterns are to be used in a streaming, block or vectored mode:
- **Streaming mode**: the target data to be scanned is a continuous stream, not
all of which is available at once; blocks of data are scanned in sequence and
matches may span multiple blocks in a stream. In streaming mode, each stream
requires a block of memory to store its state between scan calls.
- **Block mode**: the target data is a discrete, contiguous block which can be
scanned in one call and does not require state to be retained.
- **Vectored mode**: the target data consists of a list of non-contiguous
blocks that are available all at once. As for block mode, no retention of
state is required.
To compile patterns to be used in streaming mode, the ``mode`` parameter of
:c:func:`hs_compile` must be set to :c:member:`HS_MODE_STREAM`; similarly,
block mode requires the use of :c:member:`HS_MODE_BLOCK` and vectored mode
requires the use of :c:member:`HS_MODE_VECTORED`. A pattern database compiled
for one mode (streaming, block or vectored) can only be used in that mode. The
version of Hyperscan used to produce a compiled pattern database must match the
version of Hyperscan used to scan with it.
Hyperscan provides support for targeting a database at a particular CPU
platform; see :ref:`instr_specialization` for details.
=====================
Compile Pure Literals
=====================
Pure literal is a special case of regular expression. A character sequence is
regarded as a pure literal if and only if each character is read and
interpreted independently. No syntax association happens between any adjacent
characters.
For example, given an expression written as :regexp:`/bc?/`. We could say it is
a regular expression, with the meaning that character ``b`` followed by nothing
or by one character ``c``. On the other view, we could also say it is a pure
literal expression, with the meaning that this is a character sequence of 3-byte
length, containing characters ``b``, ``c`` and ``?``. In regular case, the
question mark character ``?`` has a particular syntax role called 0-1 quantifier,
which has a syntax association with the character ahead of it. Similar
characters exist in regular grammar like ``[``, ``]``, ``(``, ``)``, ``{``,
``}``, ``-``, ``*``, ``+``, ``\``, ``|``, ``/``, ``:``, ``^``, ``.``, ``$``.
While in pure literal case, all these meta characters lost extra meanings
expect for that they are just common ASCII codes.
Hyperscan is initially designed to process common regular expressions. It is
hence embedded with a complex parser to do comprehensive regular grammar
interpretation. Particularly, the identification of above meta characters is the
basic step for the interpretation of far more complex regular grammars.
However in real cases, patterns may not always be regular expressions. They
could just be pure literals. Problem will come if the pure literals contain
regular meta characters. Supposing fed directly into traditional Hyperscan
compile API, all these meta characters will be interpreted in predefined ways,
which is unnecessary and the result is totally out of expectation. To avoid
such misunderstanding by traditional API, users have to preprocess these
literal patterns by converting the meta characters into some other formats:
either by adding a backslash ``\`` before certain meta characters, or by
converting all the characters into a hexadecimal representation.
In ``v5.2.0``, Hyperscan introduces 2 new compile APIs for pure literal patterns:
#. :c:func:`hs_compile_lit`: compiles a single pure literal into a pattern
database.
#. :c:func:`hs_compile_lit_multi`: compiles an array of pure literals into a
pattern database. All of the supplied patterns will be scanned for
concurrently at scan time, with user-supplied identifiers returned when they
match.
These 2 APIs are designed for use cases where all patterns contained in the
target rule set are pure literals. Users can pass the initial pure literal
content directly into these APIs without worrying about writing regular meta
characters in their patterns. No preprocessing work is needed any more.
For new APIs, the ``length`` of each literal pattern is a newly added parameter.
Hyperscan needs to locate the end position of the input expression via clearly
knowing each literal's length, not by simply identifying character ``\0`` of a
string.
Supported flags: :c:member:`HS_FLAG_CASELESS`, :c:member:`HS_FLAG_SINGLEMATCH`,
:c:member:`HS_FLAG_SOM_LEFTMOST`.
.. note:: We don't support literal compilation API with :ref:`extparam`. And
for runtime implementation, traditional runtime APIs can still be
used to match pure literal patterns.
.. note:: If the target rule set contains at least one regular expression,
please use traditional compile APIs :c:func:`hs_compile`,
:c:func:`hs_compile_multi` and :c:func:`hs_compile_ext_multi`.
The new literal APIs introduced here are designed for rule sets
containing only pure literal expressions.
***************
Pattern Support
***************
Hyperscan supports the pattern syntax used by the PCRE library ("libpcre"),
described at <http://www.pcre.org/>. However, not all constructs available in
libpcre are supported. The use of unsupported constructs will result in
compilation errors.
The version of PCRE used to validate Hyperscan's interpretation of this syntax
is 8.41 or above.
====================
Supported Constructs
====================
The following regex constructs are supported by Hyperscan:
* Literal characters and strings, with all libpcre quoting and character
escapes.
* Character classes such as :regexp:`.` (dot), :regexp:`[abc]`, and
:regexp:`[^abc]`, as well as the predefined character classes :regexp:`\\s`,
:regexp:`\\d`, :regexp:`\\w`, :regexp:`\\v`, and :regexp:`\\h` and their
negated counterparts (:regexp:`\\S`, :regexp:`\\D`, :regexp:`\\W`,
:regexp:`\\V`, and :regexp:`\\H`).
* The POSIX named character classes :regexp:`[[:xxx:]]` and negated named
character classes :regexp:`[[:^xxx:]]`.
* Unicode character properties, such as :regexp:`\\p{L}`, :regexp:`\\P{Sc}`,
:regexp:`\\p{Greek}`.
* Quantifiers:
* Quantifiers such as :regexp:`?`, :regexp:`*` and :regexp:`+` are supported
when applied to arbitrary supported sub-expressions.
* Bounded repeat qualifiers such as :regexp:`{n}`, :regexp:`{m,n}`,
:regexp:`{n,}` are supported with limitations.
* For arbitrary repeated sub-patterns: *n* and *m* should be either small
or infinite, e.g. :regexp:`(a|b){4}`, :regexp:`(ab?c?d){4,10}` or
:regexp:`(ab(cd)*){6,}`.
* For single-character width sub-patterns such as :regexp:`[^\\a]` or
:regexp:`.` or :regexp:`x`, nearly all repeat counts are supported, except
where repeats are extremely large (maximum bound greater than 32767).
Stream states may be very large for large bounded repeats, e.g.
:regexp:`a.{2000}b`. Note: such sub-patterns may be considerably
cheaper if at the beginning or end of patterns and especially if the
:c:member:`HS_FLAG_SINGLEMATCH` flag is on for that pattern.
* Lazy modifiers (:regexp:`?` appended to another quantifier, e.g.
:regexp:`\\w+?`) are supported but ignored (as Hyperscan reports all
matches).
* Parenthesization, including the named and unnamed capturing and
non-capturing forms. However, capturing is ignored.
* Alternation with the :regexp:`|` symbol, as in :regexp:`foo|bar`.
* The anchors :regexp:`^`, :regexp:`$`, :regexp:`\\A`, :regexp:`\\Z` and
:regexp:`\\z`.
* Option modifiers:
These allow behaviour to be switched on (with :regexp:`(?<option>)`) and off
(with :regexp:`(?-<option>)`) for a sub-pattern. The supported options are:
* :regexp:`i`: Case-insensitive matching, as per
:c:member:`HS_FLAG_CASELESS`.
* :regexp:`m`: Multi-line matching, as per :c:member:`HS_FLAG_MULTILINE`.
* :regexp:`s`: Interpret ``.`` as "any character", as per
:c:member:`HS_FLAG_DOTALL`.
* :regexp:`x`: Extended syntax, which will ignore most whitespace in the
pattern for compatibility with libpcre's ``PCRE_EXTENDED`` option.
For example, the expression :regexp:`foo(?i)bar(?-i)baz` will switch on
case-insensitive matching *only* for the ``bar`` portion of the match.
* The :regexp:`\\b` and :regexp:`\\B` zero-width assertions (word boundary and
'not word boundary', respectively).
* Comments in :regexp:`(?# comment)` syntax.
* The :regexp:`(*UTF8)` and :regexp:`(*UCP)` control verbs at the beginning of a
pattern, used to enable UTF-8 and UCP mode.
.. note:: Bounded-repeat quantifiers with large repeat counts of arbitrary
expressions (e.g. :regexp:`([a-z]|bc*d|xy?z){1000,5000}`) will result in a
"Pattern too large" error at pattern compile time.
.. note:: At this time, not all patterns can be successfully compiled with the
:c:member:`HS_FLAG_SOM_LEFTMOST` flag, which enables per-pattern support for
:ref:`som`. The patterns that support this flag are a subset of patterns that
can be successfully compiled with Hyperscan; notably, many bounded repeat
forms that can be compiled with Hyperscan without the Start of Match flag
enabled cannot be compiled with the flag enabled.
======================
Unsupported Constructs
======================
The following regex constructs are not supported by Hyperscan:
* Backreferences and capturing sub-expressions.
* Arbitrary zero-width assertions.
* Subroutine references and recursive patterns.
* Conditional patterns.
* Backtracking control verbs.
* The :regexp:`\\C` "single-byte" directive (which breaks UTF-8 sequences).
* The :regexp:`\\R` newline match.
* The :regexp:`\\K` start of match reset directive.
* Callouts and embedded code.
* Atomic grouping and possessive quantifiers.
.. _semantics:
*********
Semantics
*********
While Hyperscan follows libpcre syntax, it provides different semantics. The
major departures from libpcre semantics are motivated by the requirements of
streaming and multiple simultaneous pattern matching.
The major departures from libpcre semantics are:
#. **Multiple pattern matching**: Hyperscan allows matches to be reported for
several patterns simultaneously. This is not equivalent to separating the
patterns by :regexp:`|` in libpcre, which evaluates alternations
left-to-right.
#. **Lack of ordering**: the multiple matches that Hyperscan produces are not
guaranteed to be ordered, although they will always fall within the bounds of
the current scan.
#. **End offsets only**: Hyperscan's default behaviour is only to report the end
offset of a match. Reporting of the start offset can be enabled with
per-expression flags at pattern compile time. See :ref:`som` for details.
#. **"All matches" reported**: scanning :regexp:`/foo.*bar/` against
``fooxyzbarbar`` will return two matches from Hyperscan -- at the points
corresponding to the ends of ``fooxyzbar`` and ``fooxyzbarbar``. In contrast,
libpcre semantics by default would report only one match at ``fooxyzbarbar``
(greedy semantics) or, if non-greedy semantics were switched on, one match at
``fooxyzbar``. This means that switching between greedy and non-greedy
semantics is a no-op in Hyperscan.
To support libpcre quantifier semantics while accurately reporting streaming
matches at the time they occur is impossible. For example, consider the pattern
above, :regexp:`/foo.*bar/`, in streaming mode, against the following
stream (three blocks scanned in sequence):
============= ======= ========
block 1 block 2 block 3
============= ======= ========
``fooxyzbar`` ``baz`` ``qbar``
============= ======= ========
Since the :regexp:`.*` repeat in the pattern is a *greedy* repeat in libpcre, it
must match as much as possible without causing the rest of the pattern to fail.
However, in streaming mode, this would require knowledge of data in the stream
beyond the current block being scanned.
In this example, the match at offset 9 in the first block is only the correct
match (under libpcre semantics) if there is no ``bar`` in a subsequent block --
as in block 3 -- which would constitute a better match for the pattern.
.. _som:
==============
Start of Match
==============
In standard operation, Hyperscan will only provide the end offset of a match
when the match callback is called. If the :c:member:`HS_FLAG_SOM_LEFTMOST` flag
is specified for a particular pattern, then the same set of matches is
returned, but each match will also provide the leftmost possible start offset
corresponding to its end offset.
Using the SOM flag entails a number of trade-offs and limitations:
* Reduced pattern support: For many patterns, tracking SOM is complex and can
result in Hyperscan failing to compile a pattern with a "Pattern too
large" error, even if the pattern is supported in normal operation.
* Increased stream state: At scan time, state space is required to track
potential SOM offsets, and this must be stored in persistent stream state in
streaming mode. Accordingly, SOM will generally increase the stream state
required to match a pattern.
* Performance overhead: Similarly, there is generally a performance cost
associated with tracking SOM.
* Incompatible features: Some other Hyperscan pattern flags (such as
:c:member:`HS_FLAG_SINGLEMATCH` and :c:member:`HS_FLAG_PREFILTER`) can not be
used in combination with SOM. Specifying them together with
:c:member:`HS_FLAG_SOM_LEFTMOST` will result in a compilation error.
In streaming mode, the amount of precision delivered by SOM can be controlled
with the SOM horizon flags. These instruct Hyperscan to deliver accurate SOM
information within a certain distance of the end offset, and return a special
start offset of :c:member:`HS_OFFSET_PAST_HORIZON` otherwise. Specifying a
small or medium SOM horizon will usually reduce the stream state required for a
given database.
.. note:: In streaming mode, the start offset returned for a match may refer to
a point in the stream *before* the current block being scanned. Hyperscan
provides no facility for accessing earlier blocks; if the calling application
needs to inspect historical data, then it must store it itself.
.. _extparam:
===================
Extended Parameters
===================
In some circumstances, more control over the matching behaviour of a pattern is
required than can be specified easily using regular expression syntax. For
these scenarios, Hyperscan provides the :c:func:`hs_compile_ext_multi` function
that allows a set of "extended parameters" to be set on a per-pattern basis.
Extended parameters are specified using an :c:type:`hs_expr_ext_t` structure,
which provides the following fields:
* ``flags``: Flags governing which of the other fields in the structure are
used.
* ``min_offset``: The minimum end offset in the data stream at which this
expression should match successfully.
* ``max_offset``: The maximum end offset in the data stream at which this
expression should match successfully.
* ``min_length``: The minimum match length (from start to end) required to
successfully match this expression.
* ``edit_distance``: Match this expression within a given Levenshtein distance.
* ``hamming_distance``: Match this expression within a given Hamming distance.
These parameters either allow the set of matches produced by a pattern to be
constrained at compile time (rather than relying on the application to process
unwanted matches at runtime), or allow matching a pattern approximately (within
a given edit distance) to produce more matches.
For example, the pattern :regexp:`/foo.*bar/` when given a ``min_offset`` of 10
and a ``max_offset`` of 15 will not produce matches when scanned against
``foobar`` or ``foo0123456789bar`` but will produce a match against the data
streams ``foo0123bar`` or ``foo0123456bar``.
Similarly, the pattern :regexp:`/foobar/` when given an ``edit_distance`` of 2
will produce matches when scanned against ``foobar``, ``f00bar``, ``fooba``,
``fobr``, ``fo_baz``, ``foooobar``, and anything else that lies within edit
distance of 2 (as defined by Levenshtein distance).
When the same pattern :regexp:`/foobar/` is given a ``hamming_distance`` of 2,
it will produce matches when scanned against ``foobar``, ``boofar``,
``f00bar``, and anything else with at most two characters substituted from the
original pattern. For more details, see the :ref:`approximate_matching`
section.
=================
Prefiltering Mode
=================
Hyperscan provides a per-pattern flag, :c:member:`HS_FLAG_PREFILTER`, which can
be used to implement a prefilter for a pattern than Hyperscan would not
ordinarily support.
This flag instructs Hyperscan to compile an "approximate" version of this
pattern for use in a prefiltering application, even if Hyperscan does not
support the pattern in normal operation.
The set of matches returned when this flag is used is guaranteed to be a
superset of the matches specified by the non-prefiltering expression.
If the pattern contains pattern constructs not supported by Hyperscan (such as
zero-width assertions, back-references or conditional references) these
constructs will be replaced internally with broader constructs that may match
more often.
For example, the pattern :regexp:`/(\\w+) again \\1/` contains the
back-reference :regexp:`\\1`. In prefiltering mode, this pattern might be
approximated by having its back-reference replaced with its referent, forming
:regexp:`/\\w+ again \\w+/`.
Furthermore, in prefiltering mode Hyperscan may simplify a pattern that would
otherwise return a "Pattern too large" error at compile time, or for performance
reasons (subject to the matching guarantee above).
It is generally expected that the application will subsequently confirm
prefilter matches with another regular expression matcher that can provide exact
matches for the pattern.
.. note:: The use of this flag in combination with Start of Match mode (using
the :c:member:`HS_FLAG_SOM_LEFTMOST` flag) is not currently supported and
will result in a pattern compilation error.
.. _instr_specialization:
******************************
Instruction Set Specialization
******************************
Hyperscan is able to make use of several modern instruction set features found
on x86 processors to provide improvements in scanning performance.
Some of these features are selected when the library is built; for example,
Hyperscan will use the native ``POPCNT`` instruction on processors where it is
available and the library has been optimized for the host architecture.
.. note:: By default, the Hyperscan runtime is built with the ``-march=native``
compiler flag and (where possible) will make use of all instructions known by
the host's C compiler.
To use some instruction set features, however, Hyperscan must build a
specialized database to support them. This means that the target platform must
be specified at pattern compile time.
The Hyperscan compiler API functions all accept an optional
:c:type:`hs_platform_info_t` argument, which describes the target platform
for the database to be built. If this argument is NULL, the database will be
targeted at the current host platform.
The :c:type:`hs_platform_info_t` structure has two fields:
#. ``tune``: This allows the application to specify information about the target
platform which may be used to guide the optimisation process of the compile.
Use of this field does not limit the processors that the resulting database
can run on, but may impact the performance of the resulting database.
#. ``cpu_features``: This allows the application to specify a mask of CPU
features that may be used on the target platform. For example,
:c:member:`HS_CPU_FEATURES_AVX2` can be specified for Intel\ |reg| Advanced
Vector Extensions 2 (Intel\ |reg| AVX2) instruction set support. If a flag
for a particular CPU feature is specified, the database will not be usable on
a CPU without that feature.
An :c:type:`hs_platform_info_t` structure targeted at the current host can be
built with the :c:func:`hs_populate_platform` function.
See :ref:`api_constants` for the full list of CPU tuning and feature flags.
.. _approximate_matching:
********************
Approximate matching
********************
Hyperscan provides an experimental approximate matching mode, which will match
patterns within a given edit distance. The exact matching behavior is defined as
follows:
#. **Edit distance** is defined as Levenshtein distance. That is, there are
three possible edit types considered: insertion, removal and substitution.
A more formal description can be found on
`Wikipedia <https://en.wikipedia.org/wiki/Levenshtein_distance>`__.
#. **Hamming distance** is the number of positions by which two strings of
equal length differ. That is, it is the number of substitutions required to
convert one string to the other. There are no insertions or removals when
approximate matching using a Hamming distance. A more formal description can
be found on
`Wikipedia <https://en.wikipedia.org/wiki/Hamming_distance>`__.
#. **Approximate matching** will match all *corpora* within a given edit or
Hamming distance. That is, given a pattern, approximate matching will match
anything that can be edited to arrive at a corpus that exactly matches the
original pattern.
#. **Matching semantics** are exactly the same as described in :ref:`semantics`.
Here are a few examples of approximate matching:
* Pattern :regexp:`/foo/` can match ``foo`` when using regular Hyperscan
matching behavior. With approximate matching within edit distance 2, the
pattern will produce matches when scanned against ``foo``, ``foooo``, ``f00``,
``f``, and anything else that lies within edit distance 2 of matching corpora
for the original pattern (``foo`` in this case).
* Pattern :regexp:`/foo(bar)+/` with edit distance 1 will match ``foobarbar``,
``foobarb0r``, ``fooarbar``, ``foobarba``, ``f0obarbar``, ``fobarbar`` and
anything else that lies within edit distance 1 of matching corpora for the
original pattern (``foobarbar`` in this case).
* Pattern :regexp:`/foob?ar/` with edit distance 2 will match ``fooar``,
``foo``, ``fabar``, ``oar`` and anything else that lies within edit distance 2
of matching corpora for the original pattern (``fooar`` in this case).
Currently, there are trade-offs and limitations that come with approximate
matching support. Here they are, in a nutshell:
* Reduced pattern support:
* For many patterns, approximate matching is complex and can result in
Hyperscan failing to compile a pattern with a "Pattern too large" error,
even if the pattern is supported in normal operation.
* Additionally, some patterns cannot be approximately matched because they
reduce to so-called "vacuous" patterns (patterns that match everything). For
example, pattern :regexp:`/foo/` with edit distance 3, if implemented,
would reduce to matching zero-length buffers. Such patterns will result in a
"Pattern cannot be approximately matched" compile error. Approximate
matching within a Hamming distance does not remove symbols, so will not
reduce to a vacuous pattern.
* Finally, due to the inherent complexities of defining matching behavior,
approximate matching implements a reduced subset of regular expression
syntax. Approximate matching does not support UTF-8 (and other
multibyte character encodings), and word boundaries (that is, ``\b``, ``\B``
and other equivalent constructs). Patterns containing unsupported constructs
will result in "Pattern cannot be approximately matched" compile error.
* When using approximate matching in conjunction with SOM, all of the
restrictions of SOM also apply. See :ref:`som` for more
details.
* Increased stream state/byte code size requirements: due to approximate
matching byte code being inherently larger and more complex than exact
matching, the corresponding requirements also increase.
* Performance overhead: similarly, there is generally a performance cost
associated with approximate matching, both due to increased matching
complexity, and due to the fact that it will produce more matches.
Approximate matching is always disabled by default, and can be enabled on a
per-pattern basis by using an extended parameter described in :ref:`extparam`.
.. _logical_combinations:
********************
Logical Combinations
********************
For situations when a user requires behaviour that depends on the presence or
absence of matches from groups of patterns, Hyperscan provides support for the
logical combination of patterns in a given pattern set, with three operators:
``NOT``, ``AND`` and ``OR``.
The logical value of such a combination is based on each expression's matching
status at a given offset. The matching status of any expression has a boolean
value: *false* if the expression has not yet matched or *true* if the expression
has already matched. In particular, the value of a ``NOT`` operation at a given
offset is *true* if the expression it refers to is *false* at this offset.
For example, ``NOT 101`` means that expression 101 has not yet matched at this
offset.
A logical combination is passed to Hyperscan at compile time as an expression.
This combination expression will raise matches at every offset where one of its
sub-expressions matches and the logical value of the whole expression is *true*.
To illustrate, here is an example combination expression: ::
((301 OR 302) AND 303) AND (304 OR NOT 305)
If expression 301 matches at offset 10, the logical value of 301 is *true*
while the other patterns' values are *false*. Hence, the whole combination's value is
*false*.
Then expression 303 matches at offset 20. Now the values of 301 and 303 are
*true* while the other patterns' values are still *false*. In this case, the
combination's value is *true*, so the combination expression raises a match at
offset 20.
Finally, expression 305 has matches at offset 30. Now the values of 301, 303 and 305
are *true* while the other patterns' values are still *false*. In this case, the
combination's value is *false* and no match is raised.
**Using Logical Combinations**
In logical combination syntax, an expression is written as infix notation, it
consists of operands, operators and parentheses. The operands are expression
IDs, and operators are ``!`` (NOT), ``&`` (AND) or ``|`` (OR). For example, the
combination described in the previous section would be written as: ::
((301 | 302) & 303) & (304 | !305)
In a logical combination expression:
* The priority of operators are ``!`` > ``&`` > ``|``. For example:
- ``A&B|C`` is treated as ``(A&B)|C``,
- ``A|B&C`` is treated as ``A|(B&C)``,
- ``A&!B`` is treated as ``A&(!B)``.
* Extra parentheses are allowed. For example:
- ``(A)&!(B)`` is the same as ``A&!B``,
- ``(A&B)|C`` is the same as ``A&B|C``.
* Whitespace is ignored.
To use a logical combination expression, it must be passed to one of the
Hyperscan compile functions (:c:func:`hs_compile_multi`,
:c:func:`hs_compile_ext_multi`) along with the :c:member:`HS_FLAG_COMBINATION` flag,
which identifies the pattern as a logical combination expression. The patterns
referred to in the logical combination expression must be compiled together in
the same pattern set as the combination expression.
When an expression has the :c:member:`HS_FLAG_COMBINATION` flag set, it ignores
all other flags except the :c:member:`HS_FLAG_SINGLEMATCH` flag and the
:c:member:`HS_FLAG_QUIET` flag.
Hyperscan will accept logical combination expressions at compile time that
evaluate to *true* when no patterns have matched, and report the match for
combination at end of data if no patterns have matched; for example: ::
!101
!101|102
!101&!102
!(101&102)
Patterns that are referred to as operands within a logical combination (for
example, 301 through 305 in the examples above) may also use the
:c:member:`HS_FLAG_QUIET` flag to silence the reporting of individual matches
for those patterns. In the absence of this flag, all matches (for
both individual patterns and their logical combinations) will be reported.
When an expression has both the :c:member:`HS_FLAG_COMBINATION` flag and the
:c:member:`HS_FLAG_QUIET` flag set, no matches for this logical combination
will be reported.

275
doc/dev-reference/conf.py.in
View File

@ -1,275 +0,0 @@
# -*- coding: utf-8 -*-
#
# Hyperscan documentation build configuration file, created by
# sphinx-quickstart on Tue Sep 29 15:59:19 2015.
#
# This file is execfile()d with the current directory set to its
# containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.
import sys
import os
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#sys.path.insert(0, os.path.abspath('.'))
# -- General configuration ------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = ['breathe']
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix of source filenames.
source_suffix = '.rst'
# The encoding of source files.
#source_encoding = 'utf-8-sig'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = u'Hyperscan'
copyright = u'2015-2018, Intel Corporation'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
version = '@HS_MAJOR_VERSION@.@HS_MINOR_VERSION@'
# The full version, including alpha/beta/rc tags.
release = '@HS_VERSION@'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#language = None
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
#today = ''
# Else, today_fmt is used as the format for a strftime call.
#today_fmt = '%B %d, %Y'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = ['_build']
# The reST default role (used for this markup: `text`) to use for all
# documents.
#default_role = None
# If true, '()' will be appended to :func: etc. cross-reference text.
#add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
#add_module_names = True
# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
#show_authors = False
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# A list of ignored prefixes for module index sorting.
#modindex_common_prefix = []
# If true, keep warnings as "system message" paragraphs in the built documents.
#keep_warnings = False
# -- Options for HTML output ----------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
html_theme = 'alabaster'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
html_theme_options = {
# Change some style colors; these are used for admonitions
'pink_1' : '#e0f8ff',
'pink_2' : '#e0f8ff'
}
# Add any paths that contain custom themes here, relative to this directory.
#html_theme_path = []
# The name for this set of Sphinx documents. If None, it defaults to
# "<project> v<release> documentation".
#html_title = None
# A shorter title for the navigation bar. Default is the same as html_title.
#html_short_title = None
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
#html_logo = None
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
#html_favicon = None
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['@CMAKE_CURRENT_SOURCE_DIR@/_static']
# Add any extra paths that contain custom files (such as robots.txt or
# .htaccess) here, relative to this directory. These files are copied
# directly to the root of the documentation.
#html_extra_path = []
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
#html_last_updated_fmt = '%b %d, %Y'
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
#html_use_smartypants = True
# Custom sidebar templates, maps document names to template names.
html_sidebars = {
'**': ['globaltoc.html', 'searchbox.html']
}
# Additional templates that should be rendered to pages, maps page names to
# template names.
#html_additional_pages = {}
# If false, no module index is generated.
#html_domain_indices = True
# If false, no index is generated.
#html_use_index = True
# If true, the index is split into individual pages for each letter.
#html_split_index = False
# If true, links to the reST sources are added to the pages.
html_show_sourcelink = False
# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
#html_show_sphinx = True
# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
#html_show_copyright = True
# If true, an OpenSearch description file will be output, and all pages will
# contain a <link> tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
#html_use_opensearch = ''
# This is the file name suffix for HTML files (e.g. ".xhtml").
#html_file_suffix = None
# Output file base name for HTML help builder.
htmlhelp_basename = 'Hyperscandoc'
# -- Options for LaTeX output ---------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
#'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
#'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
#'preamble': '',
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
('index', 'Hyperscan.tex', u'Hyperscan Documentation',
u'Intel Corporation', 'manual'),
]
# The name of an image file (relative to this directory) to place at the top of
# the title page.
#latex_logo = None
# For "manual" documents, if this is true, then toplevel headings are parts,
# not chapters.
#latex_use_parts = False
# If true, show page references after internal links.
#latex_show_pagerefs = False
# If true, show URL addresses after external links.
#latex_show_urls = False
# Documents to append as an appendix to all manuals.
#latex_appendices = []
# If false, no module index is generated.
#latex_domain_indices = True
# -- Options for manual page output ---------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
('index', 'hyperscan', u'Hyperscan Documentation',
[u'Intel Corporation'], 1)
]
# If true, show URL addresses after external links.
#man_show_urls = False
# -- Options for Texinfo output -------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
('index', 'Hyperscan', u'Hyperscan Documentation',
u'Intel Corporation', 'Hyperscan', 'High-performance regular expression matcher.',
'Miscellaneous'),
]
# Documents to append as an appendix to all manuals.
#texinfo_appendices = []
# If false, no module index is generated.
#texinfo_domain_indices = True
# How to display URL addresses: 'footnote', 'no', or 'inline'.
#texinfo_show_urls = 'footnote'
# If true, do not generate a @detailmenu in the "Top" node's menu.
#texinfo_no_detailmenu = False
# -- Options for Breathe doxygen import -----------------------------------
breathe_projects = { "hyperscan": "doxygen_xml" }
breathe_default_project = "hyperscan"
breathe_domain_by_extension = {"h" : "c"}
# -- Add some customisation -----------------------------------------------
def setup(app):
app.add_stylesheet("hyperscan.css") # Custom stylesheet for e.g. :regex:

33
doc/dev-reference/copyright.rst
View File

@ -1,33 +0,0 @@
.. include:: <isonum.txt>
#########
Copyright
#########
No license (express or implied, by estoppel or otherwise) to any intellectual
property rights is granted by this document.
Intel disclaims all express and implied warranties, including without
limitation, the implied warranties of merchantability, fitness for a particular
purpose, and non-infringement, as well as any warranty arising from course of
performance, course of dealing, or usage in trade.
This document contains information on products, services and/or processes in
development. All information provided here is subject to change without
notice. Contact your Intel representative to obtain the latest forecast,
schedule, specifications and roadmaps.
The products and services described may contain defects or errors known as
errata which may cause deviations from published specifications. Current
characterized errata are available on request.
Copies of documents which have an order number and are referenced in this
document, or other Intel literature, may be obtained by calling 1-800-548-4725,
or go to: <http://www.intel.com/design/literature.htm>.
Intel, and the Intel logo, are trademarks of Intel Corporation in the U.S.
and/or other countries.
\*Other names and brands may be claimed as the property of others.
Copyright |copy| 2015-2018, Intel Corporation. All rights reserved.

309
doc/dev-reference/getting_started.rst
View File

@ -1,309 +0,0 @@
.. include:: <isonum.txt>
###############
Getting Started
###############
Very Quick Start
****************
#. Clone Hyperscan ::
cd <where-you-want-hyperscan-source>
git clone git://github.com/intel/hyperscan
#. Configure Hyperscan
Ensure that you have the correct :ref:`dependencies <software>` present,
and then:
::
cd <where-you-want-to-build-hyperscan>
mkdir <build-dir>
cd <build-dir>
cmake [-G <generator>] [options] <hyperscan-source-path>
Known working generators:
* ``Unix Makefiles`` --- make-compatible makefiles (default on Linux/FreeBSD/Mac OS X)
* ``Ninja`` --- `Ninja <http://martine.github.io/ninja/>`_ build files.
* ``Visual Studio 15 2017`` --- Visual Studio projects
Generators that might work include:
* ``Xcode`` --- OS X Xcode projects.
#. Build Hyperscan
Depending on the generator used:
* ``cmake --build .`` --- will build everything
* ``make -j<jobs>`` --- use makefiles in parallel
* ``ninja`` --- use Ninja build
* ``MsBuild.exe`` --- use Visual Studio MsBuild
* etc.
#. Check Hyperscan
Run the Hyperscan unit tests: ::
bin/unit-hyperscan
Requirements
************
.. _hardware:
Hardware
========
Hyperscan will run on x86 processors in 64-bit (Intel\ |reg| 64 Architecture) and
32-bit (IA-32 Architecture) modes.
Hyperscan is a high performance software library that takes advantage of recent
Intel architecture advances. At a minimum, support for Supplemental Streaming
SIMD Extensions 3 (SSSE3) is required, which should be available on any modern
x86 processor.
Additionally, Hyperscan can make use of:
* Intel Streaming SIMD Extensions 4.2 (SSE4.2)
* the POPCNT instruction
* Bit Manipulation Instructions (BMI, BMI2)
* Intel Advanced Vector Extensions 2 (Intel AVX2)
if present.
These can be determined at library compile time, see :ref:`target_arch`.
.. _software:
Software
========
As a software library, Hyperscan doesn't impose any particular runtime
software requirements, however to build the Hyperscan library we require a
modern C and C++ compiler -- in particular, Hyperscan requires C99 and C++11
compiler support. The supported compilers are:
* GCC, v4.8.1 or higher
* Clang, v3.4 or higher (with libstdc++ or libc++)
* Intel C++ Compiler v15 or higher
* Visual C++ 2017 Build Tools
Examples of operating systems that Hyperscan is known to work on include:
Linux:
* Ubuntu 14.04 LTS or newer
* RedHat/CentOS 7 or newer
FreeBSD:
* 10.0 or newer
Windows:
* 8 or newer
Mac OS X:
* 10.8 or newer, using XCode/Clang
Hyperscan *may* compile and run on other platforms, but there is no guarantee.
We currently have experimental support for Windows using Intel C++ Compiler
or Visual Studio 2017.
In addition, the following software is required for compiling the Hyperscan library:
======================================================= =========== ======================================
Dependency Version Notes
======================================================= =========== ======================================
`CMake <http://www.cmake.org/>`_ >=2.8.11
`Ragel <http://www.colm.net/open-source/ragel/>`_ 6.9
`Python <http://www.python.org/>`_ 2.7
`Boost <http://boost.org/>`_ >=1.57 Boost headers required
`Pcap <http://tcpdump.org>`_ >=0.8 Optional: needed for example code only
======================================================= =========== ======================================
Most of these dependencies can be provided by the package manager on the build
system (e.g. Debian/Ubuntu/RedHat packages, FreeBSD ports, etc). However,
ensure that the correct version is present. As for Windows, in order to have
Ragel, you may use Cygwin to build it from source.
Boost Headers
-------------
Compiling Hyperscan depends on a recent version of the Boost C++ header
library. If the Boost libraries are installed on the build machine in the
usual paths, CMake will find them. If the Boost libraries are not installed,
the location of the Boost source tree can be specified during the CMake
configuration step using the ``BOOST_ROOT`` variable (described below).
Another alternative is to put a copy of (or a symlink to) the boost
subdirectory in ``<hyperscan-source-path>/include/boost``.
For example: for the Boost-1.59.0 release: ::
ln -s boost_1_59_0/boost <hyperscan-source-path>/include/boost
As Hyperscan uses the header-only parts of Boost, it is not necessary to
compile the Boost libraries.
CMake Configuration
===================
When CMake is invoked, it generates build files using the given options.
Options are passed to CMake in the form ``-D<variable name>=<value>``.
Common options for CMake include:
+------------------------+----------------------------------------------------+
| Variable | Description |
+========================+====================================================+
| CMAKE_C_COMPILER | C compiler to use. Default is /usr/bin/cc. |
+------------------------+----------------------------------------------------+
| CMAKE_CXX_COMPILER | C++ compiler to use. Default is /usr/bin/c++. |
+------------------------+----------------------------------------------------+
| CMAKE_INSTALL_PREFIX | Install directory for ``install`` target |
+------------------------+----------------------------------------------------+
| CMAKE_BUILD_TYPE | Define which kind of build to generate. |
| | Valid options are Debug, Release, RelWithDebInfo, |
| | and MinSizeRel. Default is RelWithDebInfo. |
+------------------------+----------------------------------------------------+
| BUILD_SHARED_LIBS | Build Hyperscan as a shared library instead of |
| | the default static library. |
+------------------------+----------------------------------------------------+
| BUILD_STATIC_AND_SHARED| Build both static and shared Hyperscan libs. |
| | Default off. |
+------------------------+----------------------------------------------------+
| BOOST_ROOT | Location of Boost source tree. |
+------------------------+----------------------------------------------------+
| DEBUG_OUTPUT | Enable very verbose debug output. Default off. |
+------------------------+----------------------------------------------------+
| FAT_RUNTIME | Build the :ref:`fat runtime<fat_runtime>`. Default |
| | true on Linux, not available elsewhere. |
+------------------------+----------------------------------------------------+
For example, to generate a ``Debug`` build: ::
cd <build-dir>
cmake -DCMAKE_BUILD_TYPE=Debug <hyperscan-source-path>
Build Type
----------
CMake determines a number of features for a build based on the Build Type.
Hyperscan defaults to ``RelWithDebInfo``, i.e. "release with debugging
information". This is a performance optimized build without runtime assertions
but with debug symbols enabled.
The other types of builds are:
* ``Release``: as above, but without debug symbols
* ``MinSizeRel``: a stripped release build
* ``Debug``: used when developing Hyperscan. Includes runtime assertions
(which has a large impact on runtime performance), and will also enable
some other build features like building internal unit
tests.
.. _target_arch:
Target Architecture
-------------------
Unless using the :ref:`fat runtime<fat_runtime>`, by default Hyperscan will be
compiled to target the instruction set of the processor of the machine that
being used for compilation. This is done via the use of ``-march=native``. The
result of this means that a library built on one machine may not work on a
different machine if they differ in supported instruction subsets.
To override the use of ``-march=native``, set appropriate flags for the
compiler in ``CFLAGS`` and ``CXXFLAGS`` environment variables before invoking
CMake, or ``CMAKE_C_FLAGS`` and ``CMAKE_CXX_FLAGS`` on the CMake command line. For
example, to set the instruction subsets up to ``SSE4.2`` using GCC 4.8: ::
cmake -DCMAKE_C_FLAGS="-march=corei7" \
-DCMAKE_CXX_FLAGS="-march=corei7" <hyperscan-source-path>
For more information, refer to :ref:`instr_specialization`.
.. _fat_runtime:
Fat Runtime
-----------
A feature introduced in Hyperscan v4.4 is the ability for the Hyperscan
library to dispatch the most appropriate runtime code for the host processor.
This feature is called the "fat runtime", as a single Hyperscan library
contains multiple copies of the runtime code for different instruction sets.
.. note::
The fat runtime feature is only available on Linux. Release builds of
Hyperscan will default to having the fat runtime enabled where supported.
When building the library with the fat runtime, the Hyperscan runtime code
will be compiled multiple times for these different instruction sets, and
these compiled objects are combined into one library. There are no changes to
how user applications are built against this library.
When applications are executed, the correct version of the runtime is selected
for the machine that it is running on. This is done using a ``CPUID`` check
for the presence of the instruction set, and then an indirect function is
resolved so that the right version of each API function is used. There is no
impact on function call performance, as this check and resolution is performed
by the ELF loader once when the binary is loaded.
If the Hyperscan library is used on x86 systems without ``SSSE3``, the runtime
API functions will resolve to functions that return :c:member:`HS_ARCH_ERROR`
instead of potentially executing illegal instructions. The API function
:c:func:`hs_valid_platform` can be used by application writers to determine if
the current platform is supported by Hyperscan.
As of this release, the variants of the runtime that are built, and the CPU
capability that is required, are the following:
+--------------+---------------------------------+---------------------------+
| Variant | CPU Feature Flag(s) Required | gcc arch flag |
+==============+=================================+===========================+
| Core 2 | ``SSSE3`` | ``-march=core2`` |
+--------------+---------------------------------+---------------------------+
| Core i7 | ``SSE4_2`` and ``POPCNT`` | ``-march=corei7`` |
+--------------+---------------------------------+---------------------------+
| AVX 2 | ``AVX2`` | ``-march=core-avx2`` |
+--------------+---------------------------------+---------------------------+
| AVX 512 | ``AVX512BW`` (see note below) | ``-march=skylake-avx512`` |
+--------------+---------------------------------+---------------------------+
| AVX 512 VBMI | ``AVX512VBMI`` (see note below) | ``-march=icelake-server`` |
+--------------+---------------------------------+---------------------------+
.. note::
Hyperscan v4.5 adds support for AVX-512 instructions - in particular the
``AVX-512BW`` instruction set that was introduced on Intel "Skylake" Xeon
processors - however the AVX-512 runtime variant is **not** enabled by
default in fat runtime builds as not all toolchains support AVX-512
instruction sets. To build an AVX-512 runtime, the CMake variable
``BUILD_AVX512`` must be enabled manually during configuration. For
example: ::
cmake -DBUILD_AVX512=on <...>
Hyperscan v5.3 adds support for AVX512VBMI instructions - in particular the
``AVX512VBMI`` instruction set that was introduced on Intel "Icelake" Xeon
processors - however the AVX512VBMI runtime variant is **not** enabled by
default in fat runtime builds as not all toolchains support AVX512VBMI
instruction sets. To build an AVX512VBMI runtime, the CMake variable
``BUILD_AVX512VBMI`` must be enabled manually during configuration. For
example: ::
cmake -DBUILD_AVX512VBMI=on <...>
As the fat runtime requires compiler, libc, and binutils support, at this time
it will only be enabled for Linux builds where the compiler supports the
`indirect function "ifunc" function attribute
<https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-indirect-functions-3321>`_.
This attribute should be available on all supported versions of GCC, and
recent versions of Clang and ICC. There is currently no operating system
support for this feature on non-Linux systems.

2383
doc/dev-reference/hyperscan.doxyfile.in
View File

File diff suppressed because it is too large Load Diff

23
doc/dev-reference/index.rst
View File

@ -1,23 +0,0 @@
###############################################
Hyperscan |version| Developer's Reference Guide
###############################################
-------
|today|
-------
.. toctree::
:maxdepth: 2
copyright
preface
intro
getting_started
compilation
runtime
serialization
performance
tools
api_constants
api_files
chimera

85
doc/dev-reference/intro.rst
View File

@ -1,85 +0,0 @@
.. include:: <isonum.txt>
.. _intro:
############
Introduction
############
Hyperscan is a software regular expression matching engine designed with
high performance and flexibility in mind. It is implemented as a library that
exposes a straightforward C API.
The Hyperscan API itself is composed of two major components:
***********
Compilation
***********
These functions take a group of regular expressions, along with identifiers and
option flags, and compile them into an immutable database that can be used by
the Hyperscan scanning API. This compilation process performs considerable
analysis and optimization work in order to build a database that will match the
given expressions efficiently.
If a pattern cannot be built into a database for any reason (such as the use of
an unsupported expression construct, or the overflowing of a resource limit),
an error will be returned by the pattern compiler.
Compiled databases can be serialized and relocated, so that they can be stored
to disk or moved between hosts. They can also be targeted to particular
platform features (for example, the use of Intel\ |reg| Advanced Vector Extensions
2 (Intel\ |reg| AVX2) instructions).
See :ref:`compilation` for more detail.
********
Scanning
********
Once a Hyperscan database has been created, it can be used to scan data in
memory. Hyperscan provides several scanning modes, depending on whether the
data to be scanned is available as a single contiguous block, whether it is
distributed amongst several blocks in memory at the same time, or whether it is
to be scanned as a sequence of blocks in a stream.
Matches are delivered to the application via a user-supplied callback function
that is called synchronously for each match.
For a given database, Hyperscan provides several guarantees:
* No memory allocations occur at runtime with the exception of two
fixed-size allocations, both of which should be done ahead of time for
performance-critical applications:
- **Scratch space**: temporary memory used for internal data at scan time.
Structures in scratch space do not persist beyond the end of a single scan
call.
- **Stream state**: in streaming mode only, some state space is required to
store data that persists between scan calls for each stream. This allows
Hyperscan to track matches that span multiple blocks of data.
* The sizes of the scratch space and stream state (in streaming mode) required
for a given database are fixed and determined at database compile time. This
means that the memory requirements of the application are known ahead of
time, and these structures can be pre-allocated if required for performance
reasons.
* Any pattern that has successfully been compiled by the Hyperscan compiler can
be scanned against any input. There are no internal resource limits or other
limitations at runtime that could cause a scan call to return an error.
See :ref:`runtime` for more detail.
*****
Tools
*****
Some utilities for testing and benchmarking Hyperscan are included with the
library. See :ref:`tools` for more information.
************
Example Code
************
Some simple example code demonstrating the use of the Hyperscan API is
available in the ``examples/`` subdirectory of the Hyperscan distribution.

345
doc/dev-reference/performance.rst
View File

@ -1,345 +0,0 @@
.. _perf:
##########################
Performance Considerations
##########################
Hyperscan supports a wide range of patterns in all three scanning modes. It is
capable of extremely high levels of performance, but certain patterns can
reduce performance markedly.
The following guidelines will help construct patterns and pattern sets that
will perform better:
*****************************
Regular expression constructs
*****************************
.. tip:: Do not hand-optimize regular expression constructs.
Quite a large number of regular expressions can be written in multiple ways.
For example, caseless matching of :regexp:`/abc/` can be written as:
* :regexp:`/[Aa][Bb][Cc]/`
* :regexp:`/(A|a)(B|b)(C|c)/`
* :regexp:`/(?i)abc(?-i)/`
* :regexp:`/abc/i`
Hyperscan is capable of handling all these constructs. Unless there is a
specific reason otherwise, do not rewrite patterns from one form to another.
As another example, matching of :regexp:`/foo(bar|baz)(frotz)?/` can be
equivalently written as:
* :regexp:`/foobarfrotz|foobazfrotz|foobar|foobaz/`
This change will not improve performance or reduce overheads.
*************
Library usage
*************
.. tip:: Do not hand-optimize library usage.
The Hyperscan library is capable of dealing with small writes, unusually large
and small pattern sets, etc. Unless there is a specific performance problem
with some usage of the library, it is best to use Hyperscan in a simple and
direct fashion. For example, it is unlikely for there to be much benefit in
buffering input to the library into larger blocks unless streaming writes are
tiny (say, 1-2 bytes at a time).
Unlike many other pattern matching products, Hyperscan will run faster with
small numbers of patterns and slower with large numbers of patterns in a smooth
fashion (as opposed to, typically, running at a moderate speed up to some fixed
limit then either breaking or running half as fast).
Hyperscan also provides high-throughput matching with a single thread of
control per core; if a database runs at 3.0 Gbps in Hyperscan it means that a
3000-bit block of data will be scanned in 1 microsecond in a single thread of
control, not that it is required to scan 22 3000-bit blocks of data in 22
microseconds. Thus, it is not usually necessary to buffer data to supply
Hyperscan with available parallelism.
********************
Block-based matching
********************
.. tip:: Prefer block-based matching to streaming matching where possible.
Whenever input data appears in discrete records, or already requires some sort
of transformation (e.g. URI normalization) that requires all the data to be
accumulated before processing, it should be scanned in block rather than in
streaming mode.
Unnecessary use of streaming mode reduces the number of optimizations that can
be applied in Hyperscan and may make some patterns run slower.
If there is a mixture of 'block' and 'streaming' mode patterns, these should be
scanned in separate databases except in the case that the streaming patterns
vastly outnumber the block mode patterns.
*********************
Unnecessary databases
*********************
.. tip:: Avoid unnecessary 'union' databases.
If there are 5 different types of network traffic T1 through T5 that must
be scanned against 5 different signature sets, it will be far more efficient to
construct 5 separate databases and scan traffic against the appropriate one
than it will be to merge all 5 signature sets and remove inappropriate matches
after the fact.
This will be true even in the case where there is substantial overlap among the
signatures. Only if the common subset of the signatures is overwhelmingly large
(say, 90% of the signatures appear in all 5 traffic types) should a database
that merges all 5 signature sets be considered, and only then if there are no
performance issues with specific patterns that appear outside the common
subset.
******************************
Allocate scratch ahead of time
******************************
.. tip:: Do not allocate scratch space for your pattern database just before
calling a scan function. Instead, do it just after the pattern database is
compiled or deserialized.
Scratch allocation is not necessarily a cheap operation. Since it is the first
time (after compilation or deserialization) that a pattern database is used,
Hyperscan performs some validation checks inside :c:func:`hs_alloc_scratch` and
must also allocate memory.
Therefore, it is important to ensure that :c:func:`hs_alloc_scratch` is not
called in the application's scanning path just before :c:func:`hs_scan` (for
example).
Instead, scratch should be allocated immediately after a pattern database is
compiled or deserialized, then retained for later scanning operations.
***********************************************
Allocate one scratch space per scanning context
***********************************************
.. tip:: A scratch space can be allocated so that it can be used with any one of
a number of databases. Each concurrent scan operation (such as a thread)
needs its own scratch space.
The :c:func:`hs_alloc_scratch` function can accept an existing scratch space and
"grow" it to support scanning with another pattern database. This means that
instead of allocating one scratch space for every database used by an
application, one can call :c:func:`hs_alloc_scratch` with a pointer to the same
:c:type:`hs_scratch_t` and it will be sized appropriately for use with any of
the given databases. For example:
.. code-block:: c
hs_database_t *db1 = buildDatabaseOne();
hs_database_t *db2 = buildDatabaseTwo();
hs_database_t *db3 = buildDatabaseThree();
hs_error_t err;
hs_scratch_t *scratch = NULL;
err = hs_alloc_scratch(db1, &scratch);
if (err != HS_SUCCESS) {
printf("hs_alloc_scratch failed!");
exit(1);
}
err = hs_alloc_scratch(db2, &scratch);
if (err != HS_SUCCESS) {
printf("hs_alloc_scratch failed!");
exit(1);
}
err = hs_alloc_scratch(db3, &scratch);
if (err != HS_SUCCESS) {
printf("hs_alloc_scratch failed!");
exit(1);
}
/* scratch may now be used to scan against any of
the databases db1, db2, db3. */
*****************
Anchored patterns
*****************
.. tip:: If a pattern is meant to appear at the start of data, be sure to
anchor it.
Anchored patterns (:regexp:`/^.../`) are far simpler to match than other
patterns, especially patterns anchored to the start of the buffer (or stream, in
streaming mode). Anchoring patterns to the end of the buffer results in less of
a performance gain, especially in streaming mode.
There are a variety of ways to anchor a pattern to a particular offset:
- The :regexp:`^` and :regexp:`\\A` constructs anchor the pattern to the start
of the buffer. For example, :regexp:`/^foo/` can *only* match at offset 3.
- The :regexp:`$`, :regexp:`\\z` and :regexp:`\\Z` constructs anchor the pattern
to the end of the buffer. For example, :regexp:`/foo\\z/` can only match when
the data buffer being scanned ends in ``foo``. (It should be noted that
:regexp:`$` and :regexp:`\\Z` will also match before a newline at the end of
the buffer, so :regexp:`/foo\\z/` would match against either ``abc foo`` or
``abc foo\n``.)
- The ``min_offset`` and ``max_offset`` extended parameters may also be used to
constrain where a pattern could match. For example, the pattern
:regexp:`/foo/` with a ``max_offset`` of 10 will only match at offsets less
than or equal to 10 in the buffer. (This pattern could also be written as
:regexp:`/^.{0,7}foo/`, compiled with the :c:member:`HS_FLAG_DOTALL` flag).
*******************
Matching everywhere
*******************
.. tip:: Avoid patterns that match everywhere, and remember that our semantics
are 'match everywhere, end of match only'.
Pattern that match everywhere will run slowly due to the sheer number of
matches that they return.
Patterns like :regexp:`/.*/` in an automata-based matcher will match before and
after every single character position, so a buffer with 100 characters will
return 101 matches. Greedy pattern matchers such as libpcre will return a
single match in this case, but our semantics is to return all matches. This is
likely to be very expensive for our code and for the client code of the
library.
Another result of our semantics ("match everywhere") is that patterns that have
optional start or ending sections -- for example :regexp:`/x?abcd*/` -- may not
perform as expected.
Firstly, the :regexp:`x?` portion of the pattern is unnecessary, as it will not
affect the match results.
Secondly, the above pattern will match 'more' than :regexp:`/abc/` but
:regexp:`/abc/` will always detect any input data that will be matched by
:regexp:`/x?abcd*/` -- it will just produce fewer matches.
For example, input data ``0123abcdddd`` will match :regexp:`/abc/` once but
:regexp:`/abcd*/` five times (at ``abc``, ``abcd``, ``abcdd``, ``abcddd``, and
``abcdddd``).
*********************************
Bounded repeats in streaming mode
*********************************
.. tip:: Bounded repeats are expensive in streaming mode.
A bounded repeat construction such as :regexp:`/X.{1000,1001}abcd/` is extremely
expensive in streaming mode, of necessity. It requires us to take action on
each ``X`` character (itself expensive, relative to searching for longer strings)
and potentially record a history of hundreds of offsets where ``X`` occurred in
case the ``X`` and ``abcd`` characters are separated by a stream boundary.
Heavy and unnecessary use of bounded repeats should be avoided, especially
where other parts of a signature are quite specific. For example, a virus
signature that matches a virus payload may be sufficient without including a
prefix that includes, for example, a 2-character Windows executable prefix and
a bounded repeat beforehand.
***************
Prefer literals
***************
.. tip:: Where possible, prefer patterns which 'require' literals, especially
longer literals, and in streaming mode, prefer signatures that 'require'
literals earlier in the pattern.
Patterns which must match on a literal will run faster than patterns that do
not. For example:
- :regexp:`/\\wab\\d*\\w\\w\\w/` will run faster than
- :regexp:`/\\w\\w\\d*\\w\\w/`, or, for that matter
- :regexp:`/\\w(abc)?\\d*\\w\\w\\w/` (this contains a literal but it need
not appear in the input).
Even implicit literals are better than none: :regexp:`/[0-2][3-5].*\\w\\w/`
still effectively contains 9 2-character literals. No hand-optimization of this
case is required; this pattern will not run faster if rewritten as:
:regexp:`/(03|04|05|13|14|15|23|24|25).*\\w\\w/`.
Under all circumstances it is better to use longer literals than shorter ones.
A database consisting of 100 14-character literals will scan considerably
faster than one consisting of 100 4-character literals and return fewer
positives.
Additionally, in streaming mode, a signature that contains a longer literal
early in the pattern is preferred to one that does not.
For example: :regexp:`/b\\w*foobar/` is not as good a pattern as
:regexp:`/blah\\w*foobar/`.
The disparity between these patterns is much smaller in block mode.
Longer literals anywhere in the pattern are still preferred in streaming mode.
For example, both of the above patterns are stronger and will scan faster than
:regexp:`/b\\w*fo/` even in streaming mode.
**************
"Dot all" mode
**************
.. tip:: Use "dot all" mode where possible.
Not using the :c:member:`HS_FLAG_DOTALL` pattern flag can be expensive, as
implicitly, it means that patterns of the form :regexp:`/A.*B/` become
:regexp:`/A[^\\n]*B/`.
It is likely that scanning tasks without the DOTALL flag are better done 'line
at a time', with the newline sequences marking the beginning and end of each
block.
This will be true in most use-cases (an exception being where the DOTALL flag
is off but the pattern contains either explicit newlines or constructs such as
:regexp:`\\s` that implicitly match a newline character).
*****************
Single-match flag
*****************
.. tip:: Consider using the single-match flag to limit matches to one match per
pattern only if possible.
If only one match per pattern is required, use the flag provided to indicate
this (:c:member:`HS_FLAG_SINGLEMATCH`). This flag can allow a number of
optimizations to be applied, allowing both performance improvements and state
space reductions when streaming.
However, there is some overhead associated with tracking whether each pattern in
the pattern set has matched, and some applications with infrequent matches may
see reduced performance when the single-match flag is used.
********************
Start of Match flag
********************
.. tip:: Do not request Start of Match information if it is not not needed.
Start of Match (SOM) information can be expensive to gather and can require
large amounts of stream state to store in streaming mode. As such, SOM
information should only be requested with the :c:member:`HS_FLAG_SOM_LEFTMOST`
flag for patterns that require it.
SOM information is not generally expected to be cheaper (in either performance
terms or in stream state overhead) than the use of bounded repeats.
Consequently, :regexp:`/foo.*bar/L` with a check on start of match values after
the callback is considerably more expensive and general than
:regexp:`/foo.{300}bar/`.
Similarly, the :c:member:`hs_expr_ext::min_length` extended parameter can be
used to specify a lower bound on the length of the matches for a pattern. Using
this facility may be more lightweight in some circumstances than using the SOM
flag and post-confirming match length in the calling application.
********************
Approximate matching
********************
.. tip:: Approximate matching is an experimental feature.
There is generally a performance impact associated with approximate matching due
to the reduced specificity of the matches. This impact may vary significantly
depending on the pattern and edit distance.

47
doc/dev-reference/preface.rst
View File

@ -1,47 +0,0 @@
#######
Preface
#######
********
Overview
********
Hyperscan is a regular expression engine designed to offer high performance, the
ability to match multiple expressions simultaneously and flexibility in
scanning operation.
Patterns are provided to a compilation interface which generates an immutable
pattern database. The scan interface then can be used to scan a target data
buffer for the given patterns, returning any matching results from that data
buffer. Hyperscan also provides a streaming mode, in which matches that span
several blocks in a stream are detected.
This document is designed to facilitate code-level integration of the Hyperscan
library with existing or new applications.
:ref:`intro` is a short overview of the Hyperscan library, with more detail on
the Hyperscan API provided in the subsequent sections: :ref:`compilation` and
:ref:`runtime`.
:ref:`perf` provides details on various factors which may impact the
performance of a Hyperscan integration.
:ref:`api_constants` and :ref:`api_files` provides a detailed summary of the
Hyperscan Application Programming Interface (API).
********
Audience
********
This guide is aimed at developers interested in integrating Hyperscan into an
application. For information on building the Hyperscan library, see the Quick
Start Guide.
***********
Conventions
***********
* Text in a ``fixed-width font`` refers to a code element, e.g. type name;
function or method name.
* Text in a :regexp:`coloured fixed-width font` refers to a regular
expression or a part of a regular expression.

242
doc/dev-reference/runtime.rst
View File

@ -1,242 +0,0 @@
.. _runtime:
#####################
Scanning for Patterns
#####################
Hyperscan provides three different scanning modes, each with its own scan
function beginning with ``hs_scan``. In addition, streaming mode has a number
of other API functions for managing stream state.
****************
Handling Matches
****************
All of these functions will call a user-supplied callback function when a match
is found. This function has the following signature:
.. doxygentypedef:: match_event_handler
:outline:
:no-link:
The *id* argument will be set to the identifier for the matching expression
provided at compile time, and the *to* argument will be set to the end-offset
of the match. If SOM was requested for the pattern (see :ref:`som`), the
*from* argument will be set to the leftmost possible start-offset for the match.
The match callback function has the capability to halt scanning
by returning a non-zero value.
See :c:type:`match_event_handler` for more information.
**************
Streaming Mode
**************
The core of the Hyperscan streaming runtime API consists of functions to open,
scan, and close Hyperscan data streams:
* :c:func:`hs_open_stream`: allocates and initializes a new stream for scanning.
* :c:func:`hs_scan_stream`: scans a block of data in a given stream, raising
matches as they are detected.
* :c:func:`hs_close_stream`: completes scanning of a given stream (raising any
matches that occur at the end of the stream) and frees the stream state. After
a call to :c:func:`hs_close_stream`, the stream handle is invalid and should
not be used again for any purpose.
Any matches detected in the data as it is scanned are returned to the calling
application via a function pointer callback.
The match callback function has the capability to halt scanning of the current
data stream by returning a non-zero value. In streaming mode, the result of
this is that the stream is then left in a state where no more data can be
scanned, and any subsequent calls to :c:func:`hs_scan_stream` for that stream
will return immediately with :c:member:`HS_SCAN_TERMINATED`. The caller must
still call :c:func:`hs_close_stream` to complete the clean-up process for that
stream.
Streams exist in the Hyperscan library so that pattern matching state can be
maintained across multiple blocks of target data -- without maintaining this
state, it would not be possible to detect patterns that span these blocks of
data. This, however, does come at the cost of requiring an amount of storage
per-stream (the size of this storage is fixed at compile time), and a slight
performance penalty in some cases to manage the state.
While Hyperscan does always support a strict ordering of multiple matches,
streaming matches will not be delivered at offsets before the current stream
write, with the exception of zero-width asserts, where constructs such as
:regexp:`\\b` and :regexp:`$` can cause a match on the final character of a
stream write to be delayed until the next stream write or stream close
operation.
=================
Stream Management
=================
In addition to :c:func:`hs_open_stream`, :c:func:`hs_scan_stream`, and
:c:func:`hs_close_stream`, the Hyperscan API provides a number of other
functions for the management of streams:
* :c:func:`hs_reset_stream`: resets a stream to its initial state; this is
equivalent to calling :c:func:`hs_close_stream` but will not free the memory
used for stream state.
* :c:func:`hs_copy_stream`: constructs a (newly allocated) duplicate of a
stream.
* :c:func:`hs_reset_and_copy_stream`: constructs a duplicate of a stream into
another, resetting the destination stream first. This call avoids the
allocation done by :c:func:`hs_copy_stream`.
==================
Stream Compression
==================
A stream object is allocated as a fixed size region of memory which has been
sized to ensure that no memory allocations are required during scan
operations. When the system is under memory pressure, it may be useful to reduce
the memory consumed by streams that are not expected to be used soon. The
Hyperscan API provides calls for translating a stream to and from a compressed
representation for this purpose. The compressed representation differs from the
full stream object as it does not reserve space for components which are not
required given the current stream state. The Hyperscan API functions for this
functionality are:
* :c:func:`hs_compress_stream`: fills the provided buffer with a compressed
representation of the stream and returns the number of bytes consumed by the
compressed representation. If the buffer is not large enough to hold the
compressed representation, :c:member:`HS_INSUFFICIENT_SPACE` is returned along
with the required size. This call does not modify the original stream in any
way: it may still be written to with :c:func:`hs_scan_stream`, used as part of
the various reset calls to reinitialise its state, or
:c:func:`hs_close_stream` may be called to free its resources.
* :c:func:`hs_expand_stream`: creates a new stream based on a buffer containing
a compressed representation.
* :c:func:`hs_reset_and_expand_stream`: constructs a stream based on a buffer
containing a compressed representation on top of an existing stream, resetting
the existing stream first. This call avoids the allocation done by
:c:func:`hs_expand_stream`.
Note: it is not recommended to use stream compression between every call to scan
for performance reasons as it takes time to convert between the compressed
representation and a standard stream.
**********
Block Mode
**********
The block mode runtime API consists of a single function: :c:func:`hs_scan`. Using
the compiled patterns this function identifies matches in the target data,
using a function pointer callback to communicate with the application.
This single :c:func:`hs_scan` function is essentially equivalent to calling
:c:func:`hs_open_stream`, making a single call to :c:func:`hs_scan_stream`, and
then :c:func:`hs_close_stream`, except that block mode operation does not
incur all the stream related overhead.
*************
Vectored Mode
*************
The vectored mode runtime API, like the block mode API, consists of a single
function: :c:func:`hs_scan_vector`. This function accepts an array of data
pointers and lengths, facilitating the scanning in sequence of a set of data
blocks that are not contiguous in memory.
From the caller's perspective, this mode will produce the same matches as if
the set of data blocks were (a) scanned in sequence with a series of streaming
mode scans, or (b) copied in sequence into a single block of memory and then
scanned in block mode.
*************
Scratch Space
*************
While scanning data, Hyperscan needs a small amount of temporary memory to store
on-the-fly internal data. This amount is unfortunately too large to fit on the
stack, particularly for embedded applications, and allocating memory dynamically
is too expensive, so a pre-allocated "scratch" space must be provided to the
scanning functions.
The function :c:func:`hs_alloc_scratch` allocates a large enough region of
scratch space to support a given database. If the application uses multiple
databases, only a single scratch region is necessary: in this case, calling
:c:func:`hs_alloc_scratch` on each database (with the same ``scratch`` pointer)
will ensure that the scratch space is large enough to support scanning against
any of the given databases.
While the Hyperscan library is re-entrant, the use of scratch spaces is not.
For example, if by design it is deemed necessary to run recursive or nested
scanning (say, from the match callback function), then an additional scratch
space is required for that context.
In the absence of recursive scanning, only one such space is required per thread
and can (and indeed should) be allocated before data scanning is to commence.
In a scenario where a set of expressions are compiled by a single "main"
thread and data will be scanned by multiple "worker" threads, the convenience
function :c:func:`hs_clone_scratch` allows multiple copies of an existing
scratch space to be made for each thread (rather than forcing the caller to pass
all the compiled databases through :c:func:`hs_alloc_scratch` multiple times).
For example:
.. code-block:: c
hs_error_t err;
hs_scratch_t *scratch_prototype = NULL;
err = hs_alloc_scratch(db, &scratch_prototype);
if (err != HS_SUCCESS) {
printf("hs_alloc_scratch failed!");
exit(1);
}
hs_scratch_t *scratch_thread1 = NULL;
hs_scratch_t *scratch_thread2 = NULL;
err = hs_clone_scratch(scratch_prototype, &scratch_thread1);
if (err != HS_SUCCESS) {
printf("hs_clone_scratch failed!");
exit(1);
}
err = hs_clone_scratch(scratch_prototype, &scratch_thread2);
if (err != HS_SUCCESS) {
printf("hs_clone_scratch failed!");
exit(1);
}
hs_free_scratch(scratch_prototype);
/* Now two threads can both scan against database db,
each with its own scratch space. */
*****************
Custom Allocators
*****************
By default, structures used by Hyperscan at runtime (scratch space, stream
state, etc) are allocated with the default system allocators, usually
``malloc()`` and ``free()``.
The Hyperscan API provides a facility for changing this behaviour to support
applications that use custom memory allocators.
These functions are:
- :c:func:`hs_set_database_allocator`, which sets the allocate and free functions
used for compiled pattern databases.
- :c:func:`hs_set_scratch_allocator`, which sets the allocate and free
functions used for scratch space.
- :c:func:`hs_set_stream_allocator`, which sets the allocate and free functions
used for stream state in streaming mode.
- :c:func:`hs_set_misc_allocator`, which sets the allocate and free functions
used for miscellaneous data, such as compile error structures and
informational strings.
The :c:func:`hs_set_allocator` function can be used to set all of the custom
allocators to the same allocate/free pair.

67
doc/dev-reference/serialization.rst
View File

@ -1,67 +0,0 @@
.. _serialization:
#############
Serialization
#############
For some applications, compiling Hyperscan pattern databases immediately prior
to use is not an appropriate design. Some users may wish to:
* Compile pattern databases on a different host;
* Persist compiled databases to storage and only re-compile pattern databases
when the patterns change;
* Control the region of memory in which the compiled database is located.
Hyperscan pattern databases are not completely flat in memory: they contain
pointers and have specific alignment requirements. Therefore, they cannot be
copied (or otherwise relocated) directly. To enable these use cases, Hyperscan
provides functionality for serializing and deserializing compiled pattern
databases.
The API provides the following functions:
#. :c:func:`hs_serialize_database`: serializes a pattern database into a
flat relocatable buffer of bytes.
#. :c:func:`hs_deserialize_database`: reconstructs a newly allocated pattern
database from the output of :c:func:`hs_serialize_database`.
#. :c:func:`hs_deserialize_database_at`: reconstructs a pattern
database at a given memory location from the output of
:c:func:`hs_serialize_database`.
#. :c:func:`hs_serialized_database_size`: given a serialized pattern database,
returns the size of the memory block required by the database when
deserialized.
#. :c:func:`hs_serialized_database_info`: given a serialized pattern database,
returns a string containing information about the database. This call is
analogous to :c:func:`hs_database_info`.
.. note:: Hyperscan performs both version and platform compatibility checks
upon deserialization. The :c:func:`hs_deserialize_database` and
:c:func:`hs_deserialize_database_at` functions will only permit the
deserialization of databases compiled with (a) the same version of Hyperscan
and (b) platform features supported by the current host platform. See
:ref:`instr_specialization` for more information on platform specialization.
===================
The Runtime Library
===================
The main Hyperscan library (``libhs``) contains both the compiler and runtime
portions of the library. This means that in order to support the Hyperscan
compiler, which is written in C++, it requires C++ linkage and has a
dependency on the C++ standard library.
Many embedded applications require only the scanning ("runtime") portion of the
Hyperscan library. In these cases, pattern compilation generally takes place on
another host, and serialized pattern databases are delivered to the application
for use.
To support these applications without requiring the C++ dependency, a
runtime-only version of the Hyperscan library, called ``libhs_runtime``, is also
distributed. This library does not depend on the C++ standard library and
provides all Hyperscan functions other that those used to compile databases.

265
doc/dev-reference/tools.rst
View File

@ -1,265 +0,0 @@
.. _tools:
#####
Tools
#####
This section describes the set of utilities included with the Hyperscan library.
********************
Quick Check: hscheck
********************
The ``hscheck`` tool allows the user to quickly check whether Hyperscan supports
a group of patterns. If a pattern is rejected by Hyperscan's compiler, the
compile error is provided on standard output.
For example, given the following three patterns (the last of which contains a
syntax error) in a file called ``/tmp/test``::
1:/foo.*bar/
2:/abc|def|ghi/
3:/((foo|bar)/
... the ``hscheck`` tool will produce the following output::
$ bin/hscheck -e /tmp/test
OK: 1:/foo.*bar/
OK: 2:/abc|def|ghi/
FAIL (compile): 3:/((foo|bar)/: Missing close parenthesis for group started at index 0.
SUMMARY: 1 of 3 failed.
********************
Benchmarker: hsbench
********************
The ``hsbench`` tool provides an easy way to measure Hyperscan's performance
for a particular set of patterns and corpus of data to be scanned.
Patterns are supplied in the format described below in
:ref:`tools_pattern_format`, while the corpus must be provided in the form of a
`corpus database`: this is a simple SQLite database format intended to allow for
easy control of how a corpus is broken into blocks and streams.
.. note:: A group of Python scripts for constructing corpora databases from
various input types, such as PCAP network traffic captures or text files, can
be found in the Hyperscan source tree in ``tools/hsbench/scripts``.
Running hsbench
===============
Given a file full of patterns specified with ``-e`` and a corpus database
specified with ``-c``, ``hsbench`` will perform a single-threaded benchmark and
produce output like this::
$ hsbench -e /tmp/patterns -c /tmp/corpus.db
Signatures: /tmp/patterns
Hyperscan info: Version: 4.3.1 Features: AVX2 Mode: STREAM
Expression count: 200
Bytecode size: 342,540 bytes
Database CRC: 0x6cd6b67c
Stream state size: 252 bytes
Scratch size: 18,406 bytes
Compile time: 0.153 seconds
Peak heap usage: 78,073,856 bytes
Time spent scanning: 0.600 seconds
Corpus size: 72,138,183 bytes (63,946 blocks in 8,891 streams)
Scan matches: 81 (0.001 matches/kilobyte)
Overall block rate: 2,132,004.45 blocks/sec
Overall throughput: 19,241.10 Mbit/sec
By default, the corpus is scanned twenty times, and the overall performance
reported is computed based the total number of bytes scanned in the time it
takes to perform all twenty scans. The number of repeats can be changed with the
``-n`` argument, and the results of each scan will be displayed if the
``--per-scan`` argument is specified.
To benchmark Hyperscan on more than one core, you can supply a list of cores
with the ``-T`` argument, which will instruct ``hsbench`` to start one
benchmark thread per core given and compute the throughput from the time taken
to complete all of them.
.. tip:: For single-threaded benchmarks on multi-processor systems, we recommend
using a utility like ``taskset`` to lock the hsbench process to one core and
minimize jitter due to the operating system's scheduler.
*******************************
Correctness Testing: hscollider
*******************************
The ``hscollider`` tool, or Pattern Collider, provides a way to verify
Hyperscan's matching behaviour. It does this by compiling and scanning patterns
(either singly or in groups) against known corpora and comparing the results
against another engine (the "ground truth"). Two sources of ground truth for
comparison are available:
* The PCRE library (http://pcre.org/).
* An NFA simulation run on Hyperscan's compile-time graph representation. This
is used if PCRE cannot support the pattern or if PCRE execution fails due to
a resource limit.
Much of Hyperscan's testing infrastructure is built on ``hscollider``, and the
tool is designed to take advantage of multiple cores and provide considerable
flexibility in controlling the test. These options are described in the help
(``hscollider -h``) and include:
* Testing in streaming, block or vectored mode.
* Testing corpora at different alignments in memory.
* Testing patterns in groups of varying size.
* Manipulating stream state or scratch space between tests.
* Cross-compilation and serialization/deserialization of databases.
* Synthetic generation of corpora given a pattern set.
Using hscollider to debug a pattern
===================================
One common use-case for ``hscollider`` is to determine whether Hyperscan will
match a pattern in the expected location, and whether this accords with PCRE's
behaviour for the same case.
Here is an example. We put our pattern in a file in Hyperscan's pattern
format::
$ cat /tmp/pat
1:/hatstand.*badgerbrush/
We put the corpus to be scanned in another file, with the same numeric
identifier at the start to indicate that it should match pattern 1::
$ cat /tmp/corpus
1:__hatstand__hatstand__badgerbrush_badgerbrush
Then we can run ``hscollider`` with its verbosity turned up (``-vv``) so that
individual matches are displayed in the output::
$ bin/ue2collider -e /tmp/pat -c /tmp/corpus -Z 0 -T 1 -vv
ue2collider: The Pattern Collider Mark II
Number of threads: 1 (1 scanner, 1 generator)
Expression path: /tmp/pat
Signature files: none
Mode of operation: block mode
UE2 scan alignment: 0
Corpora read from file: /tmp/corpus
Running single-pattern/single-compile test for 1 expressions.
PCRE Match @ (2,45)
PCRE Match @ (2,33)
PCRE Match @ (12,45)
PCRE Match @ (12,33)
UE2 Match @ (0,33) for 1
UE2 Match @ (0,45) for 1
Scan call returned 0
PASSED: id 1, alignment 0, corpus 0 (matched pcre:2, ue2:2)
Thread 0 processed 1 units.
Summary:
Mode: Single/Block
=========
Expressions processed: 1
Corpora processed: 1
Expressions with failures: 0
Corpora generation failures: 0
Compilation failures: pcre:0, ng:0, ue2:0
Matching failures: pcre:0, ng:0, ue2:0
Match differences: 0
No ground truth: 0
Total match differences: 0
Total elapsed time: 0.00522815 secs.
We can see from this output that both PCRE and Hyperscan find matches ending at
offset 33 and 45, and so ``hscollider`` considers this test case to have
passed.
(In the example command line above, ``-Z 0`` instructs us to only test at
corpus alignment 0, and ``-T 1`` instructs us to only use one thread.)
.. note:: In default operation, PCRE produces only one match for a scan, unlike
Hyperscan's automata semantics. The ``hscollider`` tool uses libpcre's
"callout" functionality to match Hyperscan's semantics.
Running a larger scan test
==========================
A set of patterns for testing purposes are distributed with Hyperscan, and these
can be tested via ``hscollider`` on an in-tree build. Two CMake targets are
provided to do this easily:
================================= =====================================
Make Target Description
================================= =====================================
``make collide_quick_test`` Tests all patterns in streaming mode.
``make collide_quick_test_block`` Tests all patterns in block mode.
================================= =====================================
*****************
Debugging: hsdump
*****************
When built in debug mode (using the CMake directive ``CMAKE_BUILD_TYPE`` set to
``Debug``), Hyperscan includes support for dumping information about its
internals during pattern compilation with the ``hsdump`` tool.
This information is mostly of use to Hyperscan developers familiar with the
library's internal structure, but can be used to diagnose issues with patterns
and provide more information in bug reports.
.. _tools_pattern_format:
**************
Pattern Format
**************
All of the Hyperscan tools accept patterns in the same format, read from plain
text files with one pattern per line. Each line looks like this:
* ``<integer id>:/<regex>/<flags>``
For example::
1:/hatstand.*teakettle/s
2:/(hatstand|teakettle)/iH
3:/^.{10,20}hatstand/m
The integer ID is the value that will be reported when a match is found by
Hyperscan and must be unique.
The pattern itself is a regular expression in PCRE syntax; see
:ref:`compilation` for more information on supported features.
The flags are single characters that map to Hyperscan flags as follows:
========= ================================= ===========
Character API Flag Description
========= ================================= ===========
``i`` :c:member:`HS_FLAG_CASELESS` Case-insensitive matching
``s`` :c:member:`HS_FLAG_DOTALL` Dot (``.``) will match newlines
``m`` :c:member:`HS_FLAG_MULTILINE` Multi-line anchoring
``H`` :c:member:`HS_FLAG_SINGLEMATCH` Report match ID at most once
``V`` :c:member:`HS_FLAG_ALLOWEMPTY` Allow patterns that can match against empty buffers
``8`` :c:member:`HS_FLAG_UTF8` UTF-8 mode
``W`` :c:member:`HS_FLAG_UCP` Unicode property support
``P`` :c:member:`HS_FLAG_PREFILTER` Prefiltering mode
``L`` :c:member:`HS_FLAG_SOM_LEFTMOST` Leftmost start of match reporting
``C`` :c:member:`HS_FLAG_COMBINATION` Logical combination of patterns
``Q`` :c:member:`HS_FLAG_QUIET` Quiet at matching
========= ================================= ===========
In addition to the set of flags above, :ref:`extparam` can be supplied
for each pattern. These are supplied after the flags as ``key=value`` pairs
between braces, separated by commas. For example::
1:/hatstand.*teakettle/s{min_offset=50,max_offset=100}
All Hyperscan tools will accept a pattern file (or a directory containing
pattern files) with the ``-e`` argument. If no further arguments constraining
the pattern set are given, all patterns in those files are used.
To select a subset of the patterns, a single ID can be supplied with the ``-z``
argument, or a file containing a set of IDs can be supplied with the ``-s``
argument.

27
examples/CMakeLists.txt
View File

@ -1,27 +0,0 @@
find_library(PCAP_LIBRARY pcap)
if (NOT PCAP_LIBRARY)
message(STATUS "Could not find libpcap - some examples will not be built")
endif()
add_executable(simplegrep simplegrep.c)
set_source_files_properties(simplegrep.c PROPERTIES COMPILE_FLAGS
"-Wall -Wno-unused-parameter")
target_link_libraries(simplegrep hs)
if (PCAP_LIBRARY)
add_executable(pcapscan pcapscan.cc)
set_source_files_properties(pcapscan.cc PROPERTIES COMPILE_FLAGS
"-Wall -Wno-unused-parameter")
target_link_libraries(pcapscan hs pcap)
endif()
if (PCAP_LIBRARY)
add_executable(patbench patbench.cc)
set_source_files_properties(patbench.cc PROPERTIES COMPILE_FLAGS
"-Wall -Wno-unused-parameter")
target_link_libraries(patbench hs pcap)
endif()
install(FILES simplegrep.c pcapscan.cc patbench.cc README.md
DESTINATION ${CMAKE_INSTALL_DOCDIR}/examples)

155
examples/README.md
View File

@ -1,155 +0,0 @@
Hyperscan Example Code
======================
Copyright (C) 2015 Intel Corporation. All rights reserved.
The files in this directory contain example code demonstrating the use of the
Hyperscan regular expression matching library. The examples have been
constructed to be useful utility programs, but they have been simplified
somewhat, so generally contain "shortcuts" that one would not take if building
a "real" system.
The examples each contain a short description in a comment at the top of the
file, including build instructions.
---
Example 1: simplegrep
---------------------
The first example program (`simplegrep.c`) is modelled on the ubiquitous grep
tool to search a file for a single regular expression. 'simplegrep' does the
same, but eschews a lot of grep's complexity: it is unable to read data from
`stdin`, and doesn't support grep's plethora of command-line arguments.
This code is intended to be simple portable C99.
simplegrep demonstrates the following Hyperscan concepts:
- Single pattern compilation: As simplegrep can scan for one pattern only, it
uses the `hs_compile` function instead of the multi-pattern variant:
`hs_compile_multi`.
- Block mode pattern-matching: simplegrep will search a single data buffer
for the given pattern, so it has no need to set up and tear down streams.
(See the next section for a streaming mode example)
- Scratch space allocation and use: Hyperscan requires a small amount of
temporary memory that is used in the `hs_scan` call. The caller needs to
guarantee that only one instance of `hs_scan` is using the scratch space at a
time, but there is no requirement that the same scratch area be used on
consecutive calls to `hs_scan`. Given that it is expensive to allocate the
scratch space, one would typically allocate all necessary scratch space at
system startup and reuse it throughout execution of the program.
Example 2: pcapscan
-------------------
The second example program (`pcapscan.cc`) is a very simple packet scanning
benchmark. It scans a given PCAP file full of network traffic against a group
of regular expressions and returns some coarse performance measurements. This
example provides a quick way to examine the performance achievable on a
particular combination of platform, pattern set and input data.
In block mode, pcapscan scans each packet individually against a Hyperscan
database. In streaming mode, pcapscan assigns packets to flows using a
rudimentary connection tracker, then scans the packets in each flow with
Hyperscan's streaming mode interface. This demonstrates the use of streaming
mode operation to detect matches that straddle packet boundaries.
**Note**: the flow assignment implemented here is intended as a simple demo; it
merely ensures that packets with the same 5-tuple are written to the same
stream in the order in which they appear in the PCAP file. No packet
re-ordering or connection state tracking (as you would expect to find in a real
network scanning application) is done.
pcapscan introduces the following Hyperscan concepts:
- Multi-pattern compilation: Unlike simplegrep, pcapscan requires a file of
expressions as input instead of a single pattern. pcapscan will read this
file in, one pattern per line, and use it as input to the `hs_compile_multi`
function. This function generates a pattern database that will match all the
input patterns in parallel.
- Streamed pattern-matching: pcapscan uses the `hs_scan_stream` function
(instead of the block-mode `hs_scan` call) to allow it to identify matches
that occur in a stream of data, even if they straddle the boundaries between blocks.
Streaming mode operation has a number of unique properties:
- Stream state that persists for the lifetime of the stream must be allocated
with the `hs_open_stream` function before scanning can take place.
Similarly, it must be freed with `hs_close_stream` after it is no longer
needed. Each stream being scanned concurrently requires its own stream
state.
- In streaming mode, a non-zero return from the user-specified event-handler
function has consequences for the rest of that stream's lifetime: when a
non-zero return occurs, it signals that no more of the stream should be
scanned. Consequently if the user makes a subsequent call to
`hs_scan_stream` on a stream whose processing was terminated in this way,
hs_scan_stream will return `HS_SCAN_TERMINATED`. This case has not been
demonstrated in pcapscan, as its callback always returns 0.
- Match handling during stream shutdown: As matches may occur when the
`hs_close_stream` function is called, it too must be provided with scratch
space in order to perform this match processing. Similarly, the user must
be prepared to be issued match event callbacks during the `hs_close_stream`
call. For this reason, we advise that stream shutdown be an integral part
of the system design.
Example 3: patbench
-------------------
This program allows users to detect which signatures may be the most expensive
in a set of patterns. It is designed for use with small to medium pattern set
sizes (e.g. 5-500). If used with very large pattern sets it may take a very
long time - the number of recompiles done is `g * O(lg2(n))` where `g` is the
number of generations and `n` is the number of patterns (assuming that `n >>
g`).
This utility will return a cumulative series of removed patterns. The first
generation will find and remove a single pattern. The second generation will
begin with the first pattern removed and find another pattern to remove, etc.
So if we have 100 patterns and 15 generations, the final generation's score
will be a run over 85 patterns.
This utility is probabilistic. It is possible that the pattern removed in a
generation is not a particularly expensive pattern. To reduce noise in the
results use 'taskset' and set the number of repeats to a level that still
completes in reasonable time (this will reduce the effect of random measurement
noise).
The criterion for performance can be altered by use of the `-C<x>` flag where
`<x>` can be `t,r,s,c,b`, selecting pattern matching throughput, scratch size,
stream state size (only available in streaming mode), compile time and bytecode
size respectively.
This utility will also not produce good results if all the patterns are roughly
equally expensive.
### Factor Group Size:
If there are multiple expensive patterns that are very similar on the
left-hand-side or identical, this utility will typically not find these groups
unless the `-F` flag is used to search for a group size that is equal to or
larger than the size of the group of similar patterns.
Otherwise, removing a portion of the similar patterns will have no or almost no
effect, and the search procedure used relies on the ability to remove all of
the similar patterns in at least one search case, something which will only
happen if the `factor_group_size` is large enough.
This alters the operation of the tool so that instead of trying to find the
single pattern whose removal has the most effect by binary search (the default
with `factor_group_size == 1`), we attempt to find the N patterns whose removal
has the most effect by searching over `N + 1` evenly sized groups, removing
only `1/(N + 1)` of the search signatures per iteration.
Note that the number of recompiles done greatly increases with increased factor
group size. For example, with `factor_group_size = 1`, we do `g * 2 * lg2(n)`
recompiles, while with `factor_group_size = 4`, we do `g * 4 * log(5/4)(n)`.
Informally the number of generations we require goes up as we eliminate a
smaller number of signatures and the we have to do more work per generation.

905
examples/patbench.cc
View File

@ -1,905 +0,0 @@
/*
* Copyright (c) 2015-2017, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/*
* Hyperscan pattern benchmarker.
*
* This program allows users to detect which signatures may be the most
* expensive in a set of patterns. It is designed for use with small to medium
* pattern set sizes (e.g. 5-500). If used with very large pattern sets it may
* take a very long time - the number of recompiles done is g * O(lg2(n)) where
* g is the number of generations and n is the number of patterns (assuming
* that n >> g).
*
* This utility will return a cumulative series of removed patterns. The first
* generation will find and remove a single pattern. The second generation will
* begin with the first pattern removed and find another pattern to remove,
* etc. So if we have 100 patterns and 15 generations, the final generation's
* score will be a run over 85 patterns.
*
* This utility is probabilistic. It is possible that the pattern removed in a
* generation is not a particularly expensive pattern. To reduce noise in the
* results use 'taskset' and set the number of repeats to a level that still
* completes in reasonable time (this will reduce the effect of random
* measurement noise).
*
* The criterion for performance can be altered by use of the -C<x> flag where
* <x> can be t,r,s,c,b, selecting pattern matching throughput, scratch size,
* stream state size (only available in streaming mode), compile time and
* bytecode size respectively.
*
* This utility will also not produce good results if all the patterns are
* roughly equally expensive.
*
* Factor Group Size:
*
* If there are multiple expensive patterns that are very similar on the
* left-hand-side or identical, this utility will typically not find these
* groups unless the -F flag is used to search for a group size that is equal
* to or larger than the size of the group of similar patterns.
*
* Otherwise, removing a portion of the similar patterns will have no or almost
* no effect, and the search procedure used relies on the ability to remove all
* of the similar patterns in at least one search case, something which will
* only happen if the factor_group_size is large enough.
*
* This alters the operation of our tool so that instead of trying to find the
* single pattern whose removal has the most effect by binary search (the
* default with factor_group_size == 1), we attempt to find the N patterns
* whose removal has the most effect by searching over N+1 evenly sized groups,
* removing only 1/(N+1) of the search signatures per iteration.
*
* Note that the number of recompiles done greatly increases with increased
* factor group size. For example, with factor_group_size = 1, we do g * 2 *
* lg2(n) recompiles, while with factor_group_size = 4, we do g * 4 *
* log(5/4)(n). Informally the number of generations we require goes up as we
* eliminate a smaller number of signatures and the we have to do more work per
* generation.
*
*
* Build instructions:
*
* g++ -o patbench patbench.cc $(pkg-config --cflags --libs libhs) -lpcap
*
* Usage:
*
* ./patbench [ -n repeats] [ -G generations] [ -C criterion ]
* [ -F factor_group_size ] [ -N | -S ] <pattern file> <pcap file>
*
* -n repeats sets the number of times the PCAP is repeatedly scanned
* with the pattern
* -G generations sets the number of generations that the algorithm is
* run for
* -N sets non-streaming mode, -S sets streaming mode (default)
* -F sets the factor group size (must be >0); this allows the detection
* of multiple interacting factors
*
* -C sets the "criterion", which can be either:
* t throughput (the default) - this requires a pcap file
* r scratch size
* s stream state size
* c compile time
* b bytecode size
*
* We recommend the use of a utility like 'taskset' on multiprocessor hosts to
* lock execution to a single processor: this will remove processor migration
* by the scheduler as a source of noise in the results.
*
*/
#include <random>
#include <algorithm>
#include <cstring>
#include <chrono>
#include <fstream>
#include <iomanip>
#include <iostream>
#include <set>
#include <string>
#include <vector>
#include <unordered_map>
#include <unistd.h>
// We use the BSD primitives throughout as they exist on both BSD and Linux.
#define __FAVOR_BSD
#include <netinet/in.h>
#include <netinet/in_systm.h>
#include <netinet/ip.h>
#include <netinet/tcp.h>
#include <netinet/udp.h>
#include <netinet/ip_icmp.h>
#include <net/ethernet.h>
#include <arpa/inet.h>
#include <pcap.h>
#include <hs.h>
using std::cerr;
using std::cout;
using std::endl;
using std::ifstream;
using std::string;
using std::unordered_map;
using std::vector;
using std::set;
using std::min;
using std::max;
using std::copy;
using std::random_device;
using std::mt19937;
enum Criterion {
CRITERION_THROUGHPUT,
CRITERION_BYTECODE_SIZE,
CRITERION_COMPILE_TIME,
CRITERION_STREAM_STATE,
CRITERION_SCRATCH_SIZE
};
static bool higher_is_better(Criterion c) {
return c == CRITERION_THROUGHPUT;
}
static void print_criterion(Criterion c, double val) {
std::ios::fmtflags f(cout.flags());
switch (c) {
case CRITERION_THROUGHPUT:
cout << std::fixed << std::setprecision(3) << val << " Megabits/s";
break;
case CRITERION_COMPILE_TIME:
cout << std::fixed << std::setprecision(3) << val << " seconds";
break;
case CRITERION_BYTECODE_SIZE:
case CRITERION_STREAM_STATE:
case CRITERION_SCRATCH_SIZE:
default:
cout << static_cast<size_t>(val) << " bytes";
break;
}
cout.flags(f);
}
// Key for identifying a stream in our pcap input data, using data from its IP
// headers.
struct FiveTuple {
unsigned int protocol;
unsigned int srcAddr;
unsigned int srcPort;
unsigned int dstAddr;
unsigned int dstPort;
// Construct a FiveTuple from a TCP or UDP packet.
FiveTuple(const struct ip *iphdr) {
// IP fields
protocol = iphdr->ip_p;
srcAddr = iphdr->ip_src.s_addr;
dstAddr = iphdr->ip_dst.s_addr;
// UDP/TCP ports
const struct udphdr *uh = (const struct udphdr *)
(((const char *)iphdr) + (iphdr->ip_hl * 4));
srcPort = uh->uh_sport;
dstPort = uh->uh_dport;
}
bool operator==(const FiveTuple &a) const {
return protocol == a.protocol && srcAddr == a.srcAddr &&
srcPort == a.srcPort && dstAddr == a.dstAddr &&
dstPort == a.dstPort;
}
};
// A *very* simple hash function, used when we create an unordered_map of
// FiveTuple objects.
struct FiveTupleHash {
size_t operator()(const FiveTuple &x) const {
return x.srcAddr ^ x.dstAddr ^ x.protocol ^ x.srcPort ^ x.dstPort;
}
};
// Helper function. See end of file.
static bool payloadOffset(const unsigned char *pkt_data, unsigned int *offset,
unsigned int *length);
// Match event handler: called every time Hyperscan finds a match.
static
int onMatch(unsigned int id, unsigned long long from, unsigned long long to,
unsigned int flags, void *ctx) {
// Our context points to a size_t storing the match count
size_t *matches = (size_t *)ctx;
(*matches)++;
return 0; // continue matching
}
// Simple timing class
class Clock {
public:
void start() {
time_start = std::chrono::system_clock::now();
}
void stop() {
time_end = std::chrono::system_clock::now();
}
double seconds() const {
std::chrono::duration<double> delta = time_end - time_start;
return delta.count();
}
private:
std::chrono::time_point<std::chrono::system_clock> time_start, time_end;
};
// Class wrapping all state associated with the benchmark
class Benchmark {
private:
// Packet data to be scanned
vector<string> packets;
// Stream ID for each packet
vector<size_t> stream_ids;
// Map used to construct stream_ids
unordered_map<FiveTuple, size_t, FiveTupleHash> stream_map;
// Hyperscan compiled database
hs_database_t *db = nullptr;
// Hyperscan temporary scratch space
hs_scratch_t *scratch = nullptr;
// Vector of Hyperscan stream state
vector<hs_stream_t *> streams;
// Count of matches found while scanning
size_t matchCount = 0;
public:
~Benchmark() {
hs_free_scratch(scratch);
hs_free_database(db);
}
// Initialisation; after this call, Benchmark owns the database and will
// ensure it is freed.
void setDatabase(hs_database_t *hs_db) {
hs_free_database(db); // Free previous database.
db = hs_db;
// (Re)allocate scratch to ensure that it is large enough to handle the
// database.
hs_error_t err = hs_alloc_scratch(db, &scratch);
if (err != HS_SUCCESS) {
cerr << "ERROR: could not allocate scratch space. Exiting." << endl;
exit(-1);
}
}
const hs_database_t *getDatabase() const {
return db;
}
size_t getScratchSize() const {
size_t scratch_size;
hs_error_t err = hs_scratch_size(scratch, &scratch_size);
if (err != HS_SUCCESS) {
cerr << "ERROR: could not query scratch space size. Exiting."
<< endl;
exit(-1);
}
return scratch_size;
}
// Read a set of streams from a pcap file
bool readStreams(const char *pcapFile) {
// Open PCAP file for input
char errbuf[PCAP_ERRBUF_SIZE];
pcap_t *pcapHandle = pcap_open_offline(pcapFile, errbuf);
if (pcapHandle == nullptr) {
cerr << "ERROR: Unable to open pcap file \"" << pcapFile
<< "\": " << errbuf << endl;
return false;
}
struct pcap_pkthdr pktHeader;
const unsigned char *pktData;
while ((pktData = pcap_next(pcapHandle, &pktHeader)) != nullptr) {
unsigned int offset = 0, length = 0;
if (!payloadOffset(pktData, &offset, &length)) {
continue;
}
// Valid TCP or UDP packet
const struct ip *iphdr = (const struct ip *)(pktData
+ sizeof(struct ether_header));
const char *payload = (const char *)pktData + offset;
size_t id = stream_map.insert(std::make_pair(FiveTuple(iphdr),
stream_map.size())).first->second;
packets.push_back(string(payload, length));
stream_ids.push_back(id);
}
pcap_close(pcapHandle);
return !packets.empty();
}
// Return the number of bytes scanned
size_t bytes() const {
size_t sum = 0;
for (const auto &packet : packets) {
sum += packet.size();
}
return sum;
}
// Return the number of matches found.
size_t matches() const {
return matchCount;
}
// Clear the number of matches found.
void clearMatches() {
matchCount = 0;
}
// Open a Hyperscan stream for each stream in stream_ids
void openStreams() {
streams.resize(stream_map.size());
for (auto &stream : streams) {
hs_error_t err = hs_open_stream(db, 0, &stream);
if (err != HS_SUCCESS) {
cerr << "ERROR: Unable to open stream. Exiting." << endl;
exit(-1);
}
}
}
// Close all open Hyperscan streams (potentially generating any
// end-anchored matches)
void closeStreams() {
for (auto &stream : streams) {
hs_error_t err =
hs_close_stream(stream, scratch, onMatch, &matchCount);
if (err != HS_SUCCESS) {
cerr << "ERROR: Unable to close stream. Exiting." << endl;
exit(-1);
}
}
}
// Scan each packet (in the ordering given in the PCAP file) through
// Hyperscan using the streaming interface.
void scanStreams() {
for (size_t i = 0; i != packets.size(); ++i) {
const std::string &pkt = packets[i];
hs_error_t err = hs_scan_stream(streams[stream_ids[i]],
pkt.c_str(), pkt.length(), 0,
scratch, onMatch, &matchCount);
if (err != HS_SUCCESS) {
cerr << "ERROR: Unable to scan packet. Exiting." << endl;
exit(-1);
}
}
}
// Scan each packet (in the ordering given in the PCAP file) through
// Hyperscan using the block-mode interface.
void scanBlock() {
for (size_t i = 0; i != packets.size(); ++i) {
const std::string &pkt = packets[i];
hs_error_t err = hs_scan(db, pkt.c_str(), pkt.length(), 0,
scratch, onMatch, &matchCount);
if (err != HS_SUCCESS) {
cerr << "ERROR: Unable to scan packet. Exiting." << endl;
exit(-1);
}
}
}
};
// helper function - see end of file
static void parseFile(const char *filename, vector<string> &patterns,
vector<unsigned> &flags, vector<unsigned> &ids,
vector<string> &originals);
class Sigdata {
vector<unsigned> flags;
vector<unsigned> ids;
vector<string> patterns;
vector<string> originals;
public:
Sigdata() {}
Sigdata(const char *filename) {
parseFile(filename, patterns, flags, ids, originals);
}
const string &get_original(unsigned index) const {
return originals[index];
}
hs_database_t *compileDatabase(unsigned mode, double *compileTime) const {
hs_database_t *db = nullptr;
hs_compile_error_t *compileErr;
// Turn our vector of strings into a vector of char*'s to pass in to
// hs_compile_multi. (This is just using the vector of strings as
// dynamic storage.)
vector<const char *> cstrPatterns;
cstrPatterns.reserve(patterns.size());
for (const auto &pattern : patterns) {
cstrPatterns.push_back(pattern.c_str());
}
Clock clock;
clock.start();
hs_error_t err = hs_compile_multi(cstrPatterns.data(), flags.data(),
ids.data(), cstrPatterns.size(), mode,
nullptr, &db, &compileErr);
clock.stop();
if (err != HS_SUCCESS) {
if (compileErr->expression < 0) {
// The error does not refer to a particular expression.
cerr << "ERROR: " << compileErr->message << endl;
} else {
cerr << "ERROR: Pattern '"
<< patterns[compileErr->expression]
<< "' failed with error '" << compileErr->message << "'"
<< endl;
}
// As the compileErr pointer points to dynamically allocated memory,
// if we get an error, we must be sure to release it. This is not
// necessary when no error is detected.
hs_free_compile_error(compileErr);
exit(-1);
}
*compileTime = clock.seconds();
return db;
}
unsigned size() const {
return patterns.size();
}
Sigdata cloneExclude(const set<unsigned> &excludeIndexSet) const {
Sigdata c;
for (unsigned i = 0, e = size(); i != e; ++i) {
if (excludeIndexSet.find(i) == excludeIndexSet.end()) {
c.flags.push_back(flags[i]);
c.ids.push_back(ids[i]);
c.patterns.push_back(patterns[i]);
c.originals.push_back(originals[i]);
}
}
return c;
}
};
static
void usage(const char *) {
cerr << "Usage:" << endl << endl;
cerr << " patbench [-n repeats] [ -G generations] [ -C criterion ]" << endl
<< " [ -F factor_group_size ] [ -N | -S ] "
<< "<pattern file> <pcap file>" << endl << endl
<< " -n repeats sets the number of times the PCAP is repeatedly "
"scanned" << endl << " with the pattern." << endl
<< " -G generations sets the number of generations that the "
"algorithm is" << endl << " run for." << endl
<< " -N sets non-streaming mode, -S sets streaming mode (default)."
<< endl << " -F sets the factor group size (must be >0); this "
"allows the detection" << endl
<< " of multiple interacting factors." << endl << "" << endl
<< " -C sets the 'criterion', which can be either:" << endl
<< " t throughput (the default) - this requires a pcap file"
<< endl << " r scratch size" << endl
<< " s stream state size" << endl
<< " c compile time" << endl << " b bytecode size"
<< endl << endl
<< "We recommend the use of a utility like 'taskset' on "
"multiprocessor hosts to" << endl
<< "lock execution to a single processor: this will remove processor "
"migration" << endl
<< "by the scheduler as a source of noise in the results." << endl;
}
static
double measure_stream_time(Benchmark &bench, unsigned int repeatCount) {
Clock clock;
bench.clearMatches();
clock.start();
for (unsigned int i = 0; i < repeatCount; i++) {
bench.openStreams();
bench.scanStreams();
bench.closeStreams();
}
clock.stop();
double secsScan = clock.seconds();
return secsScan;
}
static
double measure_block_time(Benchmark &bench, unsigned int repeatCount) {
Clock clock;
bench.clearMatches();
clock.start();
for (unsigned int i = 0; i < repeatCount; i++) {
bench.scanBlock();
}
clock.stop();
double secsScan = clock.seconds();
return secsScan;
}
static
double eval_set(Benchmark &bench, Sigdata &sigs, unsigned int mode,
unsigned repeatCount, Criterion criterion,
bool diagnose = true) {
double compileTime = 0;
bench.setDatabase(sigs.compileDatabase(mode, &compileTime));
switch (criterion) {
case CRITERION_BYTECODE_SIZE: {
size_t dbSize;
hs_error_t err = hs_database_size(bench.getDatabase(), &dbSize);
if (err != HS_SUCCESS) {
cerr << "ERROR: could not retrieve bytecode size" << endl;
exit(1);
}
return dbSize;
}
case CRITERION_COMPILE_TIME:
return compileTime;
case CRITERION_STREAM_STATE: {
size_t streamStateSize;
hs_error_t err = hs_stream_size(bench.getDatabase(), &streamStateSize);
if (err != HS_SUCCESS) {
cerr << "ERROR: could not retrieve stream state size" << endl;
exit(1);
}
return streamStateSize;
}
case CRITERION_SCRATCH_SIZE:
return bench.getScratchSize();
case CRITERION_THROUGHPUT:
default:
break; // do nothing - we are THROUGHPUT
}
double scan_time;
if (mode == HS_MODE_NOSTREAM) {
scan_time = measure_block_time(bench, repeatCount);
} else {
scan_time = measure_stream_time(bench, repeatCount);
}
size_t bytes = bench.bytes();
size_t matches = bench.matches();
if (diagnose) {
std::ios::fmtflags f(cout.flags());
cout << "Scan time " << std::fixed << std::setprecision(3) << scan_time
<< " sec, Scanned " << bytes * repeatCount << " bytes, Throughput "
<< std::fixed << std::setprecision(3)
<< (bytes * 8 * repeatCount) / (scan_time * 1000000)
<< " Mbps, Matches " << matches << endl;
cout.flags(f);
}
return (bytes * 8 * repeatCount) / (scan_time * 1000000);
}
// Main entry point.
int main(int argc, char **argv) {
unsigned int repeatCount = 1;
unsigned int mode = HS_MODE_STREAM;
Criterion criterion = CRITERION_THROUGHPUT;
unsigned int gen_max = 10;
unsigned int factor_max = 1;
// Process command line arguments.
int opt;
while ((opt = getopt(argc, argv, "SNn:G:F:C:")) != -1) {
switch (opt) {
case 'F':
factor_max = atoi(optarg);
break;
case 'G':
gen_max = atoi(optarg);
break;
case 'S':
mode = HS_MODE_STREAM;
break;
case 'N':
mode = HS_MODE_NOSTREAM;
break;
case 'C':
switch (optarg[0]) {
case 't':
criterion = CRITERION_THROUGHPUT;
break;
case 'b':
criterion = CRITERION_BYTECODE_SIZE;
break;
case 'c':
criterion = CRITERION_COMPILE_TIME;
break;
case 's':
criterion = CRITERION_STREAM_STATE;
break;
case 'r':
criterion = CRITERION_SCRATCH_SIZE;
break;
default:
cerr << "Unrecognised criterion: " << optarg[0] << endl;
usage(argv[0]);
exit(-1);
}
break;
case 'n':
repeatCount = atoi(optarg);
break;
default:
usage(argv[0]);
exit(-1);
}
}
if (argc - optind != ((criterion == CRITERION_THROUGHPUT) ? 2 : 1)) {
usage(argv[0]);
exit(-1);
}
const char *patternFile = argv[optind];
const char *pcapFile = argv[optind + 1];
// Read our input PCAP file in
Benchmark bench;
if (criterion == CRITERION_THROUGHPUT) {
if (!bench.readStreams(pcapFile)) {
cerr << "Unable to read packets from PCAP file. Exiting." << endl;
exit(-1);
}
}
if ((criterion == CRITERION_STREAM_STATE) && (mode != HS_MODE_STREAM)) {
cerr << "Cannot evaluate stream state for block mode compile. Exiting."
<< endl;
exit(-1);
}
cout << "Base signatures: " << patternFile;
if (pcapFile) {
cout << "\tPCAP input file: " << pcapFile
<< "\tRepeat count: " << repeatCount;
}
if (mode == HS_MODE_STREAM) {
cout << "\tMode: streaming";
} else {
cout << "\tMode: block";
}
cout << endl;
Sigdata sigs(patternFile);
// calculate and show a baseline
eval_set(bench, sigs, mode, repeatCount, criterion);
set<unsigned> work_sigs, exclude;
for (unsigned i = 0; i < sigs.size(); ++i) {
work_sigs.insert(i);
}
double score_base =
eval_set(bench, sigs, mode, repeatCount, criterion, false);
bool maximize = higher_is_better(criterion);
cout << "Number of signatures: " << sigs.size() << endl;
cout << "Base performance: ";
print_criterion(criterion, score_base);
cout << endl;
unsigned generations = min(gen_max, (sigs.size() - 1) / factor_max);
cout << "Cutting signatures cumulatively for " << generations
<< " generations" << endl;
for (unsigned gen = 0; gen < generations; ++gen) {
cout << "Generation " << gen << " ";
set<unsigned> s(work_sigs.begin(), work_sigs.end());
double best = maximize ? 0 : 1000000000000.0;
unsigned count = 0;
while (s.size() > factor_max) {
count++;
cout << "." << std::flush;
vector<unsigned> sv(s.begin(), s.end());
random_device rng;
mt19937 urng(rng());
shuffle(sv.begin(), sv.end(), urng);
unsigned groups = factor_max + 1;
for (unsigned current_group = 0; current_group < groups;
current_group++) {
unsigned sz = sv.size();
unsigned lo = (current_group * sz) / groups;
unsigned hi = ((current_group + 1) * sz) / groups;
set<unsigned> s_part1(sv.begin(), sv.begin() + lo);
set<unsigned> s_part2(sv.begin() + hi, sv.end());
set<unsigned> s_tmp = s_part1;
s_tmp.insert(s_part2.begin(), s_part2.end());
set<unsigned> tmp = s_tmp;
tmp.insert(exclude.begin(), exclude.end());
Sigdata sigs_tmp = sigs.cloneExclude(tmp);
double score = eval_set(bench, sigs_tmp, mode, repeatCount,
criterion, false);
if ((current_group == 0) ||
(!maximize ? (score < best) : (score > best))) {
s = s_tmp;
best = score;
}
}
}
for (unsigned i = count; i < 16; i++) {
cout << " ";
}
std::ios::fmtflags out_f(cout.flags());
cout << "Performance: ";
print_criterion(criterion, best);
cout << " (" << std::fixed << std::setprecision(3) << (best / score_base)
<< "x) after cutting:" << endl;
cout.flags(out_f);
// s now has factor_max signatures
for (const auto &found : s) {
exclude.insert(found);
work_sigs.erase(found);
cout << sigs.get_original(found) << endl;
}
cout << endl;
}
return 0;
}
/**
* Helper function to locate the offset of the first byte of the payload in the
* given ethernet frame. Offset into the packet, and the length of the payload
* are returned in the arguments @a offset and @a length.
*/
static
bool payloadOffset(const unsigned char *pkt_data, unsigned int *offset,
unsigned int *length) {
const ip *iph = (const ip *)(pkt_data + sizeof(ether_header));
const tcphdr *th = nullptr;
// Ignore packets that aren't IPv4
if (iph->ip_v != 4) {
return false;
}
// Ignore fragmented packets.
if (iph->ip_off & htons(IP_MF | IP_OFFMASK)) {
return false;
}
// IP header length, and transport header length.
unsigned int ihlen = iph->ip_hl * 4;
unsigned int thlen = 0;
switch (iph->ip_p) {
case IPPROTO_TCP:
th = (const tcphdr *)((const char *)iph + ihlen);
thlen = th->th_off * 4;
break;
case IPPROTO_UDP:
thlen = sizeof(udphdr);
break;
default:
return false;
}
*offset = sizeof(ether_header) + ihlen + thlen;
*length = sizeof(ether_header) + ntohs(iph->ip_len) - *offset;
return *length != 0;
}
static unsigned parseFlags(const string &flagsStr) {
unsigned flags = 0;
for (const auto &c : flagsStr) {
switch (c) {
case 'i':
flags |= HS_FLAG_CASELESS; break;
case 'm':
flags |= HS_FLAG_MULTILINE; break;
case 's':
flags |= HS_FLAG_DOTALL; break;
case 'H':
flags |= HS_FLAG_SINGLEMATCH; break;
case 'V':
flags |= HS_FLAG_ALLOWEMPTY; break;
case '8':
flags |= HS_FLAG_UTF8; break;
case 'W':
flags |= HS_FLAG_UCP; break;
case '\r': // stray carriage-return
break;
default:
cerr << "Unsupported flag \'" << c << "\'" << endl;
exit(-1);
}
}
return flags;
}
static void parseFile(const char *filename, vector<string> &patterns,
vector<unsigned> &flags, vector<unsigned> &ids,
vector<string> &originals) {
ifstream inFile(filename);
if (!inFile.good()) {
cerr << "ERROR: Can't open pattern file \"" << filename << "\"" << endl;
exit(-1);
}
for (unsigned i = 1; !inFile.eof(); ++i) {
string line;
getline(inFile, line);
// if line is empty, or a comment, we can skip it
if (line.empty() || line[0] == '#') {
continue;
}
// otherwise, it should be ID:PCRE, e.g.
// 10001:/foobar/is
size_t colonIdx = line.find_first_of(':');
if (colonIdx == string::npos) {
cerr << "ERROR: Could not parse line " << i << endl;
exit(-1);
}
// we should have an unsigned int as an ID, before the colon
unsigned id = std::stoi(line.substr(0, colonIdx).c_str());
// rest of the expression is the PCRE
const string expr(line.substr(colonIdx + 1));
size_t flagsStart = expr.find_last_of('/');
if (flagsStart == string::npos) {
cerr << "ERROR: no trailing '/' char" << endl;
exit(-1);
}
string pcre(expr.substr(1, flagsStart - 1));
string flagsStr(expr.substr(flagsStart + 1, expr.size() - flagsStart));
unsigned flag = parseFlags(flagsStr);
originals.push_back(line);
patterns.push_back(pcre);
flags.push_back(flag);
ids.push_back(id);
}
}

681
examples/pcapscan.cc
View File

@ -1,681 +0,0 @@
/*
* Copyright (c) 2015-2016, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/*
* Hyperscan example program 2: pcapscan
*
* This example is a very simple packet scanning benchmark. It scans a given
* PCAP file full of network traffic against a group of regular expressions and
* returns some coarse performance measurements. This example provides a quick
* way to examine the performance achievable on a particular combination of
* platform, pattern set and input data.
*
* Build instructions:
*
* g++ -std=c++11 -O2 -o pcapscan pcapscan.cc $(pkg-config --cflags --libs libhs) -lpcap
*
* Usage:
*
* ./pcapscan [-n repeats] <pattern file> <pcap file>
*
* We recommend the use of a utility like 'taskset' on multiprocessor hosts to
* pin execution to a single processor: this will remove processor migration
* by the scheduler as a source of noise in the results.
*
*/
#include <cstring>
#include <chrono>
#include <fstream>
#include <iomanip>
#include <iostream>
#include <string>
#include <unordered_map>
#include <vector>
#include <unistd.h>
// We use the BSD primitives throughout as they exist on both BSD and Linux.
#define __FAVOR_BSD
#include <netinet/in.h>
#include <netinet/in_systm.h>
#include <netinet/ip.h>
#include <netinet/tcp.h>
#include <netinet/udp.h>
#include <netinet/ip_icmp.h>
#include <net/ethernet.h>
#include <arpa/inet.h>
#include <pcap.h>
#include <hs.h>
using std::cerr;
using std::cout;
using std::endl;
using std::ifstream;
using std::string;
using std::unordered_map;
using std::vector;
// Key for identifying a stream in our pcap input data, using data from its IP
// headers.
struct FiveTuple {
unsigned int protocol;
unsigned int srcAddr;
unsigned int srcPort;
unsigned int dstAddr;
unsigned int dstPort;
// Construct a FiveTuple from a TCP or UDP packet.
FiveTuple(const struct ip *iphdr) {
// IP fields
protocol = iphdr->ip_p;
srcAddr = iphdr->ip_src.s_addr;
dstAddr = iphdr->ip_dst.s_addr;
// UDP/TCP ports
const struct udphdr *uh =
(const struct udphdr *)(((const char *)iphdr) + (iphdr->ip_hl * 4));
srcPort = uh->uh_sport;
dstPort = uh->uh_dport;
}
bool operator==(const FiveTuple &a) const {
return protocol == a.protocol && srcAddr == a.srcAddr &&
srcPort == a.srcPort && dstAddr == a.dstAddr &&
dstPort == a.dstPort;
}
};
// A *very* simple hash function, used when we create an unordered_map of
// FiveTuple objects.
struct FiveTupleHash {
size_t operator()(const FiveTuple &x) const {
return x.srcAddr ^ x.dstAddr ^ x.protocol ^ x.srcPort ^ x.dstPort;
}
};
// Helper function. See end of file.
static bool payloadOffset(const unsigned char *pkt_data, unsigned int *offset,
unsigned int *length);
// Match event handler: called every time Hyperscan finds a match.
static
int onMatch(unsigned int id, unsigned long long from, unsigned long long to,
unsigned int flags, void *ctx) {
// Our context points to a size_t storing the match count
size_t *matches = (size_t *)ctx;
(*matches)++;
return 0; // continue matching
}
// Simple timing class
class Clock {
public:
void start() {
time_start = std::chrono::system_clock::now();
}
void stop() {
time_end = std::chrono::system_clock::now();
}
double seconds() const {
std::chrono::duration<double> delta = time_end - time_start;
return delta.count();
}
private:
std::chrono::time_point<std::chrono::system_clock> time_start, time_end;
};
// Class wrapping all state associated with the benchmark
class Benchmark {
private:
// Packet data to be scanned.
vector<string> packets;
// The stream ID to which each packet belongs
vector<size_t> stream_ids;
// Map used to construct stream_ids
unordered_map<FiveTuple, size_t, FiveTupleHash> stream_map;
// Hyperscan compiled database (streaming mode)
const hs_database_t *db_streaming;
// Hyperscan compiled database (block mode)
const hs_database_t *db_block;
// Hyperscan temporary scratch space (used in both modes)
hs_scratch_t *scratch;
// Vector of Hyperscan stream state (used in streaming mode)
vector<hs_stream_t *> streams;
// Count of matches found during scanning
size_t matchCount;
public:
Benchmark(const hs_database_t *streaming, const hs_database_t *block)
: db_streaming(streaming), db_block(block), scratch(nullptr),
matchCount(0) {
// Allocate enough scratch space to handle either streaming or block
// mode, so we only need the one scratch region.
hs_error_t err = hs_alloc_scratch(db_streaming, &scratch);
if (err != HS_SUCCESS) {
cerr << "ERROR: could not allocate scratch space. Exiting." << endl;
exit(-1);
}
// This second call will increase the scratch size if more is required
// for block mode.
err = hs_alloc_scratch(db_block, &scratch);
if (err != HS_SUCCESS) {
cerr << "ERROR: could not allocate scratch space. Exiting." << endl;
exit(-1);
}
}
~Benchmark() {
// Free scratch region
hs_free_scratch(scratch);
}
// Read a set of streams from a pcap file
bool readStreams(const char *pcapFile) {
// Open PCAP file for input
char errbuf[PCAP_ERRBUF_SIZE];
pcap_t *pcapHandle = pcap_open_offline(pcapFile, errbuf);
if (pcapHandle == nullptr) {
cerr << "ERROR: Unable to open pcap file \"" << pcapFile
<< "\": " << errbuf << endl;
return false;
}
struct pcap_pkthdr pktHeader;
const unsigned char *pktData;
while ((pktData = pcap_next(pcapHandle, &pktHeader)) != nullptr) {
unsigned int offset = 0, length = 0;
if (!payloadOffset(pktData, &offset, &length)) {
continue;
}
// Valid TCP or UDP packet
const struct ip *iphdr = (const struct ip *)(pktData
+ sizeof(struct ether_header));
const char *payload = (const char *)pktData + offset;
size_t id = stream_map.insert(std::make_pair(FiveTuple(iphdr),
stream_map.size())).first->second;
packets.push_back(string(payload, length));
stream_ids.push_back(id);
}
pcap_close(pcapHandle);
return !packets.empty();
}
// Return the number of bytes scanned
size_t bytes() const {
size_t sum = 0;
for (const auto &packet : packets) {
sum += packet.size();
}
return sum;
}
// Return the number of matches found.
size_t matches() const {
return matchCount;
}
// Clear the number of matches found.
void clearMatches() {
matchCount = 0;
}
// Open a Hyperscan stream for each stream in stream_ids
void openStreams() {
streams.resize(stream_map.size());
for (auto &stream : streams) {
hs_error_t err = hs_open_stream(db_streaming, 0, &stream);
if (err != HS_SUCCESS) {
cerr << "ERROR: Unable to open stream. Exiting." << endl;
exit(-1);
}
}
}
// Close all open Hyperscan streams (potentially generating any
// end-anchored matches)
void closeStreams() {
for (auto &stream : streams) {
hs_error_t err = hs_close_stream(stream, scratch, onMatch,
&matchCount);
if (err != HS_SUCCESS) {
cerr << "ERROR: Unable to close stream. Exiting." << endl;
exit(-1);
}
}
}
// Scan each packet (in the ordering given in the PCAP file) through
// Hyperscan using the streaming interface.
void scanStreams() {
for (size_t i = 0; i != packets.size(); ++i) {
const std::string &pkt = packets[i];
hs_error_t err = hs_scan_stream(streams[stream_ids[i]],
pkt.c_str(), pkt.length(), 0,
scratch, onMatch, &matchCount);
if (err != HS_SUCCESS) {
cerr << "ERROR: Unable to scan packet. Exiting." << endl;
exit(-1);
}
}
}
// Scan each packet (in the ordering given in the PCAP file) through
// Hyperscan using the block-mode interface.
void scanBlock() {
for (size_t i = 0; i != packets.size(); ++i) {
const std::string &pkt = packets[i];
hs_error_t err = hs_scan(db_block, pkt.c_str(), pkt.length(), 0,
scratch, onMatch, &matchCount);
if (err != HS_SUCCESS) {
cerr << "ERROR: Unable to scan packet. Exiting." << endl;
exit(-1);
}
}
}
// Display some information about the compiled database and scanned data.
void displayStats() {
size_t numPackets = packets.size();
size_t numStreams = stream_map.size();
size_t numBytes = bytes();
hs_error_t err;
cout << numPackets << " packets in " << numStreams
<< " streams, totalling " << numBytes << " bytes." << endl;
cout << "Average packet length: " << numBytes / numPackets << " bytes."
<< endl;
cout << "Average stream length: " << numBytes / numStreams << " bytes."
<< endl;
cout << endl;
size_t dbStream_size = 0;
err = hs_database_size(db_streaming, &dbStream_size);
if (err == HS_SUCCESS) {
cout << "Streaming mode Hyperscan database size : "
<< dbStream_size << " bytes." << endl;
} else {
cout << "Error getting streaming mode Hyperscan database size"
<< endl;
}
size_t dbBlock_size = 0;
err = hs_database_size(db_block, &dbBlock_size);
if (err == HS_SUCCESS) {
cout << "Block mode Hyperscan database size : "
<< dbBlock_size << " bytes." << endl;
} else {
cout << "Error getting block mode Hyperscan database size"
<< endl;
}
size_t stream_size = 0;
err = hs_stream_size(db_streaming, &stream_size);
if (err == HS_SUCCESS) {
cout << "Streaming mode Hyperscan stream state size: "
<< stream_size << " bytes (per stream)." << endl;
} else {
cout << "Error getting stream state size" << endl;
}
}
};
// helper function - see end of file
static void parseFile(const char *filename, vector<string> &patterns,
vector<unsigned> &flags, vector<unsigned> &ids);
static hs_database_t *buildDatabase(const vector<const char *> &expressions,
const vector<unsigned> flags,
const vector<unsigned> ids,
unsigned int mode) {
hs_database_t *db;
hs_compile_error_t *compileErr;
hs_error_t err;
Clock clock;
clock.start();
err = hs_compile_multi(expressions.data(), flags.data(), ids.data(),
expressions.size(), mode, nullptr, &db, &compileErr);
clock.stop();
if (err != HS_SUCCESS) {
if (compileErr->expression < 0) {
// The error does not refer to a particular expression.
cerr << "ERROR: " << compileErr->message << endl;
} else {
cerr << "ERROR: Pattern '" << expressions[compileErr->expression]
<< "' failed compilation with error: " << compileErr->message
<< endl;
}
// As the compileErr pointer points to dynamically allocated memory, if
// we get an error, we must be sure to release it. This is not
// necessary when no error is detected.
hs_free_compile_error(compileErr);
exit(-1);
}
cout << "Hyperscan " << (mode == HS_MODE_STREAM ? "streaming" : "block")
<< " mode database compiled in " << clock.seconds() << " seconds."
<< endl;
return db;
}
/**
* This function will read in the file with the specified name, with an
* expression per line, ignoring lines starting with '#' and build a Hyperscan
* database for it.
*/
static void databasesFromFile(const char *filename,
hs_database_t **db_streaming,
hs_database_t **db_block) {
// hs_compile_multi requires three parallel arrays containing the patterns,
// flags and ids that we want to work with. To achieve this we use
// vectors and new entries onto each for each valid line of input from
// the pattern file.
vector<string> patterns;
vector<unsigned> flags;
vector<unsigned> ids;
// do the actual file reading and string handling
parseFile(filename, patterns, flags, ids);
// Turn our vector of strings into a vector of char*'s to pass in to
// hs_compile_multi. (This is just using the vector of strings as dynamic
// storage.)
vector<const char*> cstrPatterns;
for (const auto &pattern : patterns) {
cstrPatterns.push_back(pattern.c_str());
}
cout << "Compiling Hyperscan databases with " << patterns.size()
<< " patterns." << endl;
*db_streaming = buildDatabase(cstrPatterns, flags, ids, HS_MODE_STREAM);
*db_block = buildDatabase(cstrPatterns, flags, ids, HS_MODE_BLOCK);
}
static void usage(const char *prog) {
cerr << "Usage: " << prog << " [-n repeats] <pattern file> <pcap file>" << endl;
}
// Main entry point.
int main(int argc, char **argv) {
unsigned int repeatCount = 1;
// Process command line arguments.
int opt;
while ((opt = getopt(argc, argv, "n:")) != -1) {
switch (opt) {
case 'n':
repeatCount = atoi(optarg);
break;
default:
usage(argv[0]);
exit(-1);
}
}
if (argc - optind != 2) {
usage(argv[0]);
exit(-1);
}
const char *patternFile = argv[optind];
const char *pcapFile = argv[optind + 1];
// Read our pattern set in and build Hyperscan databases from it.
cout << "Pattern file: " << patternFile << endl;
hs_database_t *db_streaming, *db_block;
databasesFromFile(patternFile, &db_streaming, &db_block);
// Read our input PCAP file in
Benchmark bench(db_streaming, db_block);
cout << "PCAP input file: " << pcapFile << endl;
if (!bench.readStreams(pcapFile)) {
cerr << "Unable to read packets from PCAP file. Exiting." << endl;
exit(-1);
}
if (repeatCount != 1) {
cout << "Repeating PCAP scan " << repeatCount << " times." << endl;
}
bench.displayStats();
Clock clock;
// Streaming mode scans.
double secsStreamingScan = 0.0, secsStreamingOpenClose = 0.0;
for (unsigned int i = 0; i < repeatCount; i++) {
// Open streams.
clock.start();
bench.openStreams();
clock.stop();
secsStreamingOpenClose += clock.seconds();
// Scan all our packets in streaming mode.
clock.start();
bench.scanStreams();
clock.stop();
secsStreamingScan += clock.seconds();
// Close streams.
clock.start();
bench.closeStreams();
clock.stop();
secsStreamingOpenClose += clock.seconds();
}
// Collect data from streaming mode scans.
size_t bytes = bench.bytes();
double tputStreamScanning = (bytes * 8 * repeatCount) / secsStreamingScan;
double tputStreamOverhead = (bytes * 8 * repeatCount) / (secsStreamingScan + secsStreamingOpenClose);
size_t matchesStream = bench.matches();
double matchRateStream = matchesStream / ((bytes * repeatCount) / 1024.0); // matches per kilobyte
// Scan all our packets in block mode.
bench.clearMatches();
clock.start();
for (unsigned int i = 0; i < repeatCount; i++) {
bench.scanBlock();
}
clock.stop();
double secsScanBlock = clock.seconds();
// Collect data from block mode scans.
double tputBlockScanning = (bytes * 8 * repeatCount) / secsScanBlock;
size_t matchesBlock = bench.matches();
double matchRateBlock = matchesBlock / ((bytes * repeatCount) / 1024.0); // matches per kilobyte
cout << endl << "Streaming mode:" << endl << endl;
cout << " Total matches: " << matchesStream << endl;
cout << std::fixed << std::setprecision(4);
cout << " Match rate: " << matchRateStream << " matches/kilobyte" << endl;
cout << std::fixed << std::setprecision(2);
cout << " Throughput (with stream overhead): "
<< tputStreamOverhead/1000000 << " megabits/sec" << endl;
cout << " Throughput (no stream overhead): "
<< tputStreamScanning/1000000 << " megabits/sec" << endl;
cout << endl << "Block mode:" << endl << endl;
cout << " Total matches: " << matchesBlock << endl;
cout << std::fixed << std::setprecision(4);
cout << " Match rate: " << matchRateBlock << " matches/kilobyte" << endl;
cout << std::fixed << std::setprecision(2);
cout << " Throughput: "
<< tputBlockScanning/1000000 << " megabits/sec" << endl;
cout << endl;
if (bytes < (2*1024*1024)) {
cout << endl << "WARNING: Input PCAP file is less than 2MB in size." << endl
<< "This test may have been too short to calculate accurate results." << endl;
}
// Close Hyperscan databases
hs_free_database(db_streaming);
hs_free_database(db_block);
return 0;
}
/**
* Helper function to locate the offset of the first byte of the payload in the
* given ethernet frame. Offset into the packet, and the length of the payload
* are returned in the arguments @a offset and @a length.
*/
static bool payloadOffset(const unsigned char *pkt_data, unsigned int *offset,
unsigned int *length) {
const ip *iph = (const ip *)(pkt_data + sizeof(ether_header));
const tcphdr *th = nullptr;
// Ignore packets that aren't IPv4
if (iph->ip_v != 4) {
return false;
}
// Ignore fragmented packets.
if (iph->ip_off & htons(IP_MF|IP_OFFMASK)) {
return false;
}
// IP header length, and transport header length.
unsigned int ihlen = iph->ip_hl * 4;
unsigned int thlen = 0;
switch (iph->ip_p) {
case IPPROTO_TCP:
th = (const tcphdr *)((const char *)iph + ihlen);
thlen = th->th_off * 4;
break;
case IPPROTO_UDP:
thlen = sizeof(udphdr);
break;
default:
return false;
}
*offset = sizeof(ether_header) + ihlen + thlen;
*length = sizeof(ether_header) + ntohs(iph->ip_len) - *offset;
return *length != 0;
}
static unsigned parseFlags(const string &flagsStr) {
unsigned flags = 0;
for (const auto &c : flagsStr) {
switch (c) {
case 'i':
flags |= HS_FLAG_CASELESS; break;
case 'm':
flags |= HS_FLAG_MULTILINE; break;
case 's':
flags |= HS_FLAG_DOTALL; break;
case 'H':
flags |= HS_FLAG_SINGLEMATCH; break;
case 'V':
flags |= HS_FLAG_ALLOWEMPTY; break;
case '8':
flags |= HS_FLAG_UTF8; break;
case 'W':
flags |= HS_FLAG_UCP; break;
case '\r': // stray carriage-return
break;
default:
cerr << "Unsupported flag \'" << c << "\'" << endl;
exit(-1);
}
}
return flags;
}
static void parseFile(const char *filename, vector<string> &patterns,
vector<unsigned> &flags, vector<unsigned> &ids) {
ifstream inFile(filename);
if (!inFile.good()) {
cerr << "ERROR: Can't open pattern file \"" << filename << "\"" << endl;
exit(-1);
}
for (unsigned i = 1; !inFile.eof(); ++i) {
string line;
getline(inFile, line);
// if line is empty, or a comment, we can skip it
if (line.empty() || line[0] == '#') {
continue;
}
// otherwise, it should be ID:PCRE, e.g.
// 10001:/foobar/is
size_t colonIdx = line.find_first_of(':');
if (colonIdx == string::npos) {
cerr << "ERROR: Could not parse line " << i << endl;
exit(-1);
}
// we should have an unsigned int as an ID, before the colon
unsigned id = std::stoi(line.substr(0, colonIdx).c_str());
// rest of the expression is the PCRE
const string expr(line.substr(colonIdx + 1));
size_t flagsStart = expr.find_last_of('/');
if (flagsStart == string::npos) {
cerr << "ERROR: no trailing '/' char" << endl;
exit(-1);
}
string pcre(expr.substr(1, flagsStart - 1));
string flagsStr(expr.substr(flagsStart + 1, expr.size() - flagsStart));
unsigned flag = parseFlags(flagsStr);
patterns.push_back(pcre);
flags.push_back(flag);
ids.push_back(id);
}
}

221
examples/simplegrep.c
View File

@ -1,221 +0,0 @@
/*
* Copyright (c) 2015, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/*
* Hyperscan example program 1: simplegrep
*
* This is a simple example of Hyperscan's most basic functionality: it will
* search a given input file for a pattern supplied as a command-line argument.
* It is intended to demonstrate correct usage of the hs_compile and hs_scan
* functions of Hyperscan.
*
* Patterns are scanned in 'DOTALL' mode, which is equivalent to PCRE's '/s'
* modifier. This behaviour can be changed by modifying the "flags" argument to
* hs_compile.
*
* Build instructions:
*
* gcc -o simplegrep simplegrep.c $(pkg-config --cflags --libs libhs)
*
* Usage:
*
* ./simplegrep <pattern> <input file>
*
* Example:
*
* ./simplegrep int simplegrep.c
*
*/
#include <errno.h>
#include <limits.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <hs.h>
/**
* This is the function that will be called for each match that occurs. @a ctx
* is to allow you to have some application-specific state that you will get
* access to for each match. In our simple example we're just going to use it
* to pass in the pattern that was being searched for so we can print it out.
*/
static int eventHandler(unsigned int id, unsigned long long from,
unsigned long long to, unsigned int flags, void *ctx) {
printf("Match for pattern \"%s\" at offset %llu\n", (char *)ctx, to);
return 0;
}
/**
* Fill a data buffer from the given filename, returning it and filling @a
* length with its length. Returns NULL on failure.
*/
static char *readInputData(const char *inputFN, unsigned int *length) {
FILE *f = fopen(inputFN, "rb");
if (!f) {
fprintf(stderr, "ERROR: unable to open file \"%s\": %s\n", inputFN,
strerror(errno));
return NULL;
}
/* We use fseek/ftell to get our data length, in order to keep this example
* code as portable as possible. */
if (fseek(f, 0, SEEK_END) != 0) {
fprintf(stderr, "ERROR: unable to seek file \"%s\": %s\n", inputFN,
strerror(errno));
fclose(f);
return NULL;
}
long dataLen = ftell(f);
if (dataLen < 0) {
fprintf(stderr, "ERROR: ftell() failed: %s\n", strerror(errno));
fclose(f);
return NULL;
}
if (fseek(f, 0, SEEK_SET) != 0) {
fprintf(stderr, "ERROR: unable to seek file \"%s\": %s\n", inputFN,
strerror(errno));
fclose(f);
return NULL;
}
/* Hyperscan's hs_scan function accepts length as an unsigned int, so we
* limit the size of our buffer appropriately. */
if ((unsigned long)dataLen > UINT_MAX) {
dataLen = UINT_MAX;
printf("WARNING: clipping data to %ld bytes\n", dataLen);
} else if (dataLen == 0) {
fprintf(stderr, "ERROR: input file \"%s\" is empty\n", inputFN);
fclose(f);
return NULL;
}
char *inputData = malloc(dataLen);
if (!inputData) {
fprintf(stderr, "ERROR: unable to malloc %ld bytes\n", dataLen);
fclose(f);
return NULL;
}
char *p = inputData;
size_t bytesLeft = dataLen;
while (bytesLeft) {
size_t bytesRead = fread(p, 1, bytesLeft, f);
bytesLeft -= bytesRead;
p += bytesRead;
if (ferror(f) != 0) {
fprintf(stderr, "ERROR: fread() failed\n");
free(inputData);
fclose(f);
return NULL;
}
}
fclose(f);
*length = (unsigned int)dataLen;
return inputData;
}
int main(int argc, char *argv[]) {
if (argc != 3) {
fprintf(stderr, "Usage: %s <pattern> <input file>\n", argv[0]);
return -1;
}
char *pattern = argv[1];
char *inputFN = argv[2];
/* First, we attempt to compile the pattern provided on the command line.
* We assume 'DOTALL' semantics, meaning that the '.' meta-character will
* match newline characters. The compiler will analyse the given pattern and
* either return a compiled Hyperscan database, or an error message
* explaining why the pattern didn't compile.
*/
hs_database_t *database;
hs_compile_error_t *compile_err;
if (hs_compile(pattern, HS_FLAG_DOTALL, HS_MODE_BLOCK, NULL, &database,
&compile_err) != HS_SUCCESS) {
fprintf(stderr, "ERROR: Unable to compile pattern \"%s\": %s\n",
pattern, compile_err->message);
hs_free_compile_error(compile_err);
return -1;
}
/* Next, we read the input data file into a buffer. */
unsigned int length;
char *inputData = readInputData(inputFN, &length);
if (!inputData) {
hs_free_database(database);
return -1;
}
/* Finally, we issue a call to hs_scan, which will search the input buffer
* for the pattern represented in the bytecode. Note that in order to do
* this, scratch space needs to be allocated with the hs_alloc_scratch
* function. In typical usage, you would reuse this scratch space for many
* calls to hs_scan, but as we're only doing one, we'll be allocating it
* and deallocating it as soon as our matching is done.
*
* When matches occur, the specified callback function (eventHandler in
* this file) will be called. Note that although it is reminiscent of
* asynchronous APIs, Hyperscan operates synchronously: all matches will be
* found, and all callbacks issued, *before* hs_scan returns.
*
* In this example, we provide the input pattern as the context pointer so
* that the callback is able to print out the pattern that matched on each
* match event.
*/
hs_scratch_t *scratch = NULL;
if (hs_alloc_scratch(database, &scratch) != HS_SUCCESS) {
fprintf(stderr, "ERROR: Unable to allocate scratch space. Exiting.\n");
free(inputData);
hs_free_database(database);
return -1;
}
printf("Scanning %u bytes with Hyperscan\n", length);
if (hs_scan(database, inputData, length, 0, scratch, eventHandler,
pattern) != HS_SUCCESS) {
fprintf(stderr, "ERROR: Unable to scan input buffer. Exiting.\n");
hs_free_scratch(scratch);
free(inputData);
hs_free_database(database);
return -1;
}
/* Scanning is complete, any matches have been handled, so now we just
* clean up and exit.
*/
hs_free_scratch(scratch);
free(inputData);
hs_free_database(database);
return 0;
}

43
hs.def
View File

@ -1,43 +0,0 @@
; Hyperscan DLL export definitions
LIBRARY hs
EXPORTS
hs_alloc_scratch
hs_clone_scratch
hs_close_stream
hs_compile
hs_compile_ext_multi
hs_compile_multi
hs_compress_stream
hs_copy_stream
hs_database_info
hs_database_size
hs_deserialize_database
hs_deserialize_database_at
hs_expand_stream
hs_expression_ext_info
hs_expression_info
hs_free_compile_error
hs_free_database
hs_free_scratch
hs_open_stream
hs_populate_platform
hs_reset_and_copy_stream
hs_reset_and_expand_stream
hs_reset_stream
hs_scan
hs_scan_stream
hs_scan_vector
hs_scratch_size
hs_serialize_database
hs_serialized_database_info
hs_serialized_database_size
hs_set_allocator
hs_set_database_allocator
hs_set_misc_allocator
hs_set_scratch_allocator
hs_set_stream_allocator
hs_stream_size
hs_valid_platform
hs_version

36
hs_runtime.def
View File

@ -1,36 +0,0 @@
; Hyperscan DLL export definitions
LIBRARY hs_runtime
EXPORTS
hs_alloc_scratch
hs_clone_scratch
hs_close_stream
hs_compress_stream
hs_copy_stream
hs_database_info
hs_database_size
hs_deserialize_database
hs_deserialize_database_at
hs_expand_stream
hs_free_database
hs_free_scratch
hs_open_stream
hs_reset_and_copy_stream
hs_reset_and_expand_stream
hs_reset_stream
hs_scan
hs_scan_stream
hs_scan_vector
hs_scratch_size
hs_serialize_database
hs_serialized_database_info
hs_serialized_database_size
hs_set_allocator
hs_set_database_allocator
hs_set_misc_allocator
hs_set_scratch_allocator
hs_set_stream_allocator
hs_stream_size
hs_valid_platform
hs_version

501
include/boost-patched/graph/dominator_tree.hpp
View File

@ -1,501 +0,0 @@
//=======================================================================
// Copyright (C) 2005-2009 Jongsoo Park <jongsoo.park -at- gmail.com>
//
// Distributed under the Boost Software License, Version 1.0.
// (See accompanying file LICENSE_1_0.txt or copy at
// http://www.boost.org/LICENSE_1_0.txt)
//=======================================================================
#ifndef BOOST_GRAPH_DOMINATOR_HPP
#define BOOST_GRAPH_DOMINATOR_HPP
#include <boost/config.hpp>
#include <deque>
#include <set>
#include <boost/graph/depth_first_search.hpp>
#include <boost/concept/assert.hpp>
// Dominator tree computation
// NOTE: This file contains modifications from the distributed Boost version to
// correctly support supplying a vertex index map to the algorithm. To
// differentiate it, it has been moved into the boost_ue2 namespace.
namespace boost_ue2 {
using namespace boost;
namespace detail {
/**
* An extended time_stamper which also records vertices for each dfs number
*/
template<class TimeMap, class VertexVector, class TimeT, class Tag>
class time_stamper_with_vertex_vector
: public base_visitor<
time_stamper_with_vertex_vector<TimeMap, VertexVector, TimeT, Tag> >
{
public :
typedef Tag event_filter;
time_stamper_with_vertex_vector(TimeMap timeMap, VertexVector& v,
TimeT& t)
: timeStamper_(timeMap, t), v_(v) { }
template<class Graph>
void
operator()(const typename property_traits<TimeMap>::key_type& v,
const Graph& g)
{
timeStamper_(v, g);
v_[timeStamper_.m_time] = v;
}
private :
time_stamper<TimeMap, TimeT, Tag> timeStamper_;
VertexVector& v_;
};
/**
* A convenient way to create a time_stamper_with_vertex_vector
*/
template<class TimeMap, class VertexVector, class TimeT, class Tag>
time_stamper_with_vertex_vector<TimeMap, VertexVector, TimeT, Tag>
stamp_times_with_vertex_vector(TimeMap timeMap, VertexVector& v, TimeT& t,
Tag)
{
return time_stamper_with_vertex_vector<TimeMap, VertexVector, TimeT,
Tag>(timeMap, v, t);
}
template<class Graph, class IndexMap, class TimeMap, class PredMap,
class DomTreePredMap>
class dominator_visitor
{
typedef typename graph_traits<Graph>::vertex_descriptor Vertex;
typedef typename graph_traits<Graph>::vertices_size_type VerticesSizeType;
public :
/**
* @param g [in] the target graph of the dominator tree
* @param entry [in] the entry node of g
* @param indexMap [in] the vertex index map for g
* @param domTreePredMap [out] the immediate dominator map
* (parent map in dominator tree)
*/
dominator_visitor(const Graph& g, const Vertex& entry,
const IndexMap& indexMap,
DomTreePredMap domTreePredMap)
: semi_(num_vertices(g)),
ancestor_(num_vertices(g), graph_traits<Graph>::null_vertex()),
samedom_(ancestor_),
best_(semi_),
semiMap_(make_iterator_property_map(semi_.begin(),
indexMap)),
ancestorMap_(make_iterator_property_map(ancestor_.begin(),
indexMap)),
bestMap_(make_iterator_property_map(best_.begin(),
indexMap)),
buckets_(num_vertices(g)),
bucketMap_(make_iterator_property_map(buckets_.begin(),
indexMap)),
entry_(entry),
domTreePredMap_(domTreePredMap),
numOfVertices_(num_vertices(g)),
samedomMap(make_iterator_property_map(samedom_.begin(),
indexMap))
{
}
void
operator()(const Vertex& n, const TimeMap& dfnumMap,
const PredMap& parentMap, const Graph& g)
{
if (n == entry_) return;
const Vertex p(get(parentMap, n));
Vertex s(p);
// 1. Calculate the semidominator of n,
// based on the semidominator thm.
// * Semidominator thm. : To find the semidominator of a node n,
// consider all predecessors v of n in the CFG (Control Flow Graph).
// - If v is a proper ancestor of n in the spanning tree
// (so dfnum(v) < dfnum(n)), then v is a candidate for semi(n)
// - If v is a non-ancestor of n (so dfnum(v) > dfnum(n))
// then for each u that is an ancestor of v (or u = v),
// Let semi(u) be a candidate for semi(n)
// of all these candidates, the one with lowest dfnum is
// the semidominator of n.
// For each predecessor of n
typename graph_traits<Graph>::in_edge_iterator inItr, inEnd;
for (boost::tie(inItr, inEnd) = in_edges(n, g); inItr != inEnd; ++inItr)
{
const Vertex v = source(*inItr, g);
// To deal with unreachable nodes
if (get(dfnumMap, v) < 0 || get(dfnumMap, v) >= numOfVertices_)
continue;
Vertex s2;
if (get(dfnumMap, v) <= get(dfnumMap, n))
s2 = v;
else
s2 = get(semiMap_, ancestor_with_lowest_semi_(v, dfnumMap));
if (get(dfnumMap, s2) < get(dfnumMap, s))
s = s2;
}
put(semiMap_, n, s);
// 2. Calculation of n's dominator is deferred until
// the path from s to n has been linked into the forest
get(bucketMap_, s).push_back(n);
get(ancestorMap_, n) = p;
get(bestMap_, n) = n;
// 3. Now that the path from p to v has been linked into
// the spanning forest, these lines calculate the dominator of v,
// based on the dominator thm., or else defer the calculation
// until y's dominator is known
// * Dominator thm. : On the spanning-tree path below semi(n) and
// above or including n, let y be the node
// with the smallest-numbered semidominator. Then,
//
// idom(n) = semi(n) if semi(y)=semi(n) or
// idom(y) if semi(y) != semi(n)
typename std::deque<Vertex>::iterator buckItr;
for (buckItr = get(bucketMap_, p).begin();
buckItr != get(bucketMap_, p).end();
++buckItr)
{
const Vertex v(*buckItr);
const Vertex y(ancestor_with_lowest_semi_(v, dfnumMap));
if (get(semiMap_, y) == get(semiMap_, v))
put(domTreePredMap_, v, p);
else
put(samedomMap, v, y);
}
get(bucketMap_, p).clear();
}
protected :
/**
* Evaluate function in Tarjan's path compression
*/
const Vertex
ancestor_with_lowest_semi_(const Vertex& v, const TimeMap& dfnumMap)
{
const Vertex a(get(ancestorMap_, v));
if (get(ancestorMap_, a) != graph_traits<Graph>::null_vertex())
{
const Vertex b(ancestor_with_lowest_semi_(a, dfnumMap));
put(ancestorMap_, v, get(ancestorMap_, a));
if (get(dfnumMap, get(semiMap_, b)) <
get(dfnumMap, get(semiMap_, get(bestMap_, v))))
put(bestMap_, v, b);
}
return get(bestMap_, v);
}
std::vector<Vertex> semi_, ancestor_, samedom_, best_;
PredMap semiMap_, ancestorMap_, bestMap_;
std::vector< std::deque<Vertex> > buckets_;
iterator_property_map<typename std::vector<std::deque<Vertex> >::iterator,
IndexMap> bucketMap_;
const Vertex& entry_;
DomTreePredMap domTreePredMap_;
const VerticesSizeType numOfVertices_;
public :
PredMap samedomMap;
};
} // namespace detail
/**
* @brief Build dominator tree using Lengauer-Tarjan algorithm.
* It takes O((V+E)log(V+E)) time.
*
* @pre dfnumMap, parentMap and verticesByDFNum have dfs results corresponding
* indexMap.
* If dfs has already run before,
* this function would be good for saving computations.
* @pre Unreachable nodes must be masked as
* graph_traits<Graph>::null_vertex in parentMap.
* @pre Unreachable nodes must be masked as
* (std::numeric_limits<VerticesSizeType>::max)() in dfnumMap.
*
* @param domTreePredMap [out] : immediate dominator map (parent map
* in dom. tree)
*
* @note reference Appel. p. 452~453. algorithm 19.9, 19.10.
*
* @todo : Optimization in Finding Dominators in Practice, Loukas Georgiadis
*/
template<class Graph, class IndexMap, class TimeMap, class PredMap,
class VertexVector, class DomTreePredMap>
void
lengauer_tarjan_dominator_tree_without_dfs
(const Graph& g,
const typename graph_traits<Graph>::vertex_descriptor& entry,
const IndexMap& indexMap,
TimeMap dfnumMap, PredMap parentMap, VertexVector& verticesByDFNum,
DomTreePredMap domTreePredMap)
{
// Typedefs and concept check
typedef typename graph_traits<Graph>::vertex_descriptor Vertex;
typedef typename graph_traits<Graph>::vertices_size_type VerticesSizeType;
BOOST_CONCEPT_ASSERT(( BidirectionalGraphConcept<Graph> ));
const VerticesSizeType numOfVertices = num_vertices(g);
if (numOfVertices == 0) return;
// 1. Visit each vertex in reverse post order and calculate sdom.
detail::dominator_visitor<Graph, IndexMap, TimeMap, PredMap, DomTreePredMap>
visitor(g, entry, indexMap, domTreePredMap);
VerticesSizeType i;
for (i = 0; i < numOfVertices; ++i)
{
const Vertex u(verticesByDFNum[numOfVertices - 1 - i]);
if (u != graph_traits<Graph>::null_vertex())
visitor(u, dfnumMap, parentMap, g);
}
// 2. Now all the deferred dominator calculations,
// based on the second clause of the dominator thm., are performed
for (i = 0; i < numOfVertices; ++i)
{
const Vertex n(verticesByDFNum[i]);
if (n == entry || n == graph_traits<Graph>::null_vertex())
continue;
Vertex u = get(visitor.samedomMap, n);
if (u != graph_traits<Graph>::null_vertex())
{
put(domTreePredMap, n, get(domTreePredMap, u));
}
}
}
/**
* Unlike lengauer_tarjan_dominator_tree_without_dfs,
* dfs is run in this function and
* the result is written to dfnumMap, parentMap, vertices.
*
* If the result of dfs required after this algorithm,
* this function can eliminate the need of rerunning dfs.
*/
template<class Graph, class IndexMap, class TimeMap, class PredMap,
class VertexVector, class DomTreePredMap>
void
lengauer_tarjan_dominator_tree
(const Graph& g,
const typename graph_traits<Graph>::vertex_descriptor& entry,
const IndexMap& indexMap,
TimeMap dfnumMap, PredMap parentMap, VertexVector& verticesByDFNum,
DomTreePredMap domTreePredMap)
{
// Typedefs and concept check
typedef typename graph_traits<Graph>::vertices_size_type VerticesSizeType;
BOOST_CONCEPT_ASSERT(( BidirectionalGraphConcept<Graph> ));
// 1. Depth first visit
const VerticesSizeType numOfVertices = num_vertices(g);
if (numOfVertices == 0) return;
VerticesSizeType time =
(std::numeric_limits<VerticesSizeType>::max)();
std::vector<default_color_type>
colors(numOfVertices, color_traits<default_color_type>::white());
depth_first_visit
(g, entry,
make_dfs_visitor
(make_pair(record_predecessors(parentMap, on_tree_edge()),
detail::stamp_times_with_vertex_vector
(dfnumMap, verticesByDFNum, time, on_discover_vertex()))),
make_iterator_property_map(colors.begin(), indexMap));
// 2. Run main algorithm.
lengauer_tarjan_dominator_tree_without_dfs(g, entry, indexMap, dfnumMap,
parentMap, verticesByDFNum,
domTreePredMap);
}
/**
* Use vertex_index as IndexMap and make dfnumMap, parentMap, verticesByDFNum
* internally.
* If we don't need the result of dfs (dfnumMap, parentMap, verticesByDFNum),
* this function would be more convenient one.
*/
template<class Graph, class DomTreePredMap>
void
lengauer_tarjan_dominator_tree
(const Graph& g,
const typename graph_traits<Graph>::vertex_descriptor& entry,
DomTreePredMap domTreePredMap)
{
// typedefs
typedef typename graph_traits<Graph>::vertex_descriptor Vertex;
typedef typename graph_traits<Graph>::vertices_size_type VerticesSizeType;
typedef typename property_map<Graph, vertex_index_t>::const_type IndexMap;
typedef
iterator_property_map<typename std::vector<VerticesSizeType>::iterator,
IndexMap> TimeMap;
typedef
iterator_property_map<typename std::vector<Vertex>::iterator, IndexMap>
PredMap;
// Make property maps
const VerticesSizeType numOfVertices = num_vertices(g);
if (numOfVertices == 0) return;
const IndexMap indexMap = get(vertex_index, g);
std::vector<VerticesSizeType> dfnum(numOfVertices, 0);
TimeMap dfnumMap(make_iterator_property_map(dfnum.begin(), indexMap));
std::vector<Vertex> parent(numOfVertices,
graph_traits<Graph>::null_vertex());
PredMap parentMap(make_iterator_property_map(parent.begin(), indexMap));
std::vector<Vertex> verticesByDFNum(parent);
// Run main algorithm
lengauer_tarjan_dominator_tree(g, entry,
indexMap, dfnumMap, parentMap,
verticesByDFNum, domTreePredMap);
}
/**
* Muchnick. p. 182, 184
*
* using iterative bit vector analysis
*/
template<class Graph, class IndexMap, class DomTreePredMap>
void
iterative_bit_vector_dominator_tree
(const Graph& g,
const typename graph_traits<Graph>::vertex_descriptor& entry,
const IndexMap& indexMap,
DomTreePredMap domTreePredMap)
{
typedef typename graph_traits<Graph>::vertex_descriptor Vertex;
typedef typename graph_traits<Graph>::vertex_iterator vertexItr;
typedef typename graph_traits<Graph>::vertices_size_type VerticesSizeType;
typedef
iterator_property_map<typename std::vector< std::set<Vertex> >::iterator,
IndexMap> vertexSetMap;
BOOST_CONCEPT_ASSERT(( BidirectionalGraphConcept<Graph> ));
// 1. Finding dominator
// 1.1. Initialize
const VerticesSizeType numOfVertices = num_vertices(g);
if (numOfVertices == 0) return;
vertexItr vi, viend;
boost::tie(vi, viend) = vertices(g);
const std::set<Vertex> N(vi, viend);
bool change = true;
std::vector< std::set<Vertex> > dom(numOfVertices, N);
vertexSetMap domMap(make_iterator_property_map(dom.begin(), indexMap));
get(domMap, entry).clear();
get(domMap, entry).insert(entry);
while (change)
{
change = false;
for (boost::tie(vi, viend) = vertices(g); vi != viend; ++vi)
{
if (*vi == entry) continue;
std::set<Vertex> T(N);
typename graph_traits<Graph>::in_edge_iterator inItr, inEnd;
for (boost::tie(inItr, inEnd) = in_edges(*vi, g); inItr != inEnd; ++inItr)
{
const Vertex p = source(*inItr, g);
std::set<Vertex> tempSet;
std::set_intersection(T.begin(), T.end(),
get(domMap, p).begin(),
get(domMap, p).end(),
std::inserter(tempSet, tempSet.begin()));
T.swap(tempSet);
}
T.insert(*vi);
if (T != get(domMap, *vi))
{
change = true;
get(domMap, *vi).swap(T);
}
} // end of for (boost::tie(vi, viend) = vertices(g)
} // end of while(change)
// 2. Build dominator tree
for (boost::tie(vi, viend) = vertices(g); vi != viend; ++vi)
get(domMap, *vi).erase(*vi);
Graph domTree(numOfVertices);
for (boost::tie(vi, viend) = vertices(g); vi != viend; ++vi)
{
if (*vi == entry) continue;
// We have to iterate through copied dominator set
const std::set<Vertex> tempSet(get(domMap, *vi));
typename std::set<Vertex>::const_iterator s;
for (s = tempSet.begin(); s != tempSet.end(); ++s)
{
typename std::set<Vertex>::iterator t;
for (t = get(domMap, *vi).begin(); t != get(domMap, *vi).end(); )
{
typename std::set<Vertex>::iterator old_t = t;
++t; // Done early because t may become invalid
if (*old_t == *s) continue;
if (get(domMap, *s).find(*old_t) != get(domMap, *s).end())
get(domMap, *vi).erase(old_t);
}
}
}
for (boost::tie(vi, viend) = vertices(g); vi != viend; ++vi)
{
if (*vi != entry && get(domMap, *vi).size() == 1)
{
Vertex temp = *get(domMap, *vi).begin();
put(domTreePredMap, *vi, temp);
}
}
}
template<class Graph, class DomTreePredMap>
void
iterative_bit_vector_dominator_tree
(const Graph& g,
const typename graph_traits<Graph>::vertex_descriptor& entry,
DomTreePredMap domTreePredMap)
{
typename property_map<Graph, vertex_index_t>::const_type
indexMap = get(vertex_index, g);
iterative_bit_vector_dominator_tree(g, entry, indexMap, domTreePredMap);
}
} // namespace boost
#endif // BOOST_GRAPH_DOMINATOR_HPP

27
include/boost-patched/graph/reverse_graph.hpp
View File

@ -1,27 +0,0 @@
#ifndef REVERSE_GRAPH_PATCHED_H_
#define REVERSE_GRAPH_PATCHED_H_
#include <boost/version.hpp>
#include <boost/graph/reverse_graph.hpp>
#if defined(BOOST_REVGRAPH_PATCH)
// Boost 1.62.0 does not implement degree() in reverse_graph which is required
// by BidirectionalGraph, so add it.
namespace boost {
template <class BidirectionalGraph, class GRef>
inline typename graph_traits<BidirectionalGraph>::degree_size_type
degree(const typename graph_traits<BidirectionalGraph>::vertex_descriptor u,
const reverse_graph<BidirectionalGraph,GRef>& g)
{
return degree(u, g.m_g);
}
} // namespace boost
#endif // Boost 1.62.0
#endif

10
libhs.pc.in
View File

@ -1,10 +0,0 @@
prefix=@CMAKE_INSTALL_PREFIX@
exec_prefix=@CMAKE_INSTALL_PREFIX@
libdir=@CMAKE_INSTALL_PREFIX@/@CMAKE_INSTALL_LIBDIR@
includedir=@CMAKE_INSTALL_PREFIX@/@CMAKE_INSTALL_INCLUDEDIR@
Name: libhs
Description: Intel(R) Hyperscan Library
Version: @HS_VERSION@
Libs: -L${libdir} -lhs
Cflags: -I${includedir}/hs

113
src/alloc.c
View File

@ -1,113 +0,0 @@
/*
* Copyright (c) 2015-2017, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/** \file
* \brief Runtime functions for setting custom allocators.
*/
#include <stdlib.h>
#include <string.h>
#include "allocator.h"
#define default_malloc malloc
#define default_free free
hs_alloc_t hs_database_alloc = default_malloc;
hs_alloc_t hs_misc_alloc = default_malloc;
hs_alloc_t hs_scratch_alloc = default_malloc;
hs_alloc_t hs_stream_alloc = default_malloc;
hs_free_t hs_database_free = default_free;
hs_free_t hs_misc_free = default_free;
hs_free_t hs_scratch_free = default_free;
hs_free_t hs_stream_free = default_free;
static
hs_alloc_t normalise_alloc(hs_alloc_t a) {
if (!a) {
return default_malloc;
} else {
return a;
}
}
static
hs_free_t normalise_free(hs_free_t f) {
if (!f) {
return default_free;
} else {
return f;
}
}
HS_PUBLIC_API
hs_error_t HS_CDECL hs_set_allocator(hs_alloc_t allocfunc, hs_free_t freefunc) {
hs_set_database_allocator(allocfunc, freefunc);
hs_set_misc_allocator(allocfunc, freefunc);
hs_set_stream_allocator(allocfunc, freefunc);
hs_set_scratch_allocator(allocfunc, freefunc);
return HS_SUCCESS;
}
HS_PUBLIC_API
hs_error_t HS_CDECL hs_set_database_allocator(hs_alloc_t allocfunc,
hs_free_t freefunc) {
hs_database_alloc = normalise_alloc(allocfunc);
hs_database_free = normalise_free(freefunc);
return HS_SUCCESS;
}
HS_PUBLIC_API
hs_error_t HS_CDECL hs_set_misc_allocator(hs_alloc_t allocfunc,
hs_free_t freefunc) {
hs_misc_alloc = normalise_alloc(allocfunc);
hs_misc_free = normalise_free(freefunc);
return HS_SUCCESS;
}
HS_PUBLIC_API
hs_error_t HS_CDECL hs_set_scratch_allocator(hs_alloc_t allocfunc,
hs_free_t freefunc) {
hs_scratch_alloc = normalise_alloc(allocfunc);
hs_scratch_free = normalise_free(freefunc);
return HS_SUCCESS;
}
HS_PUBLIC_API
hs_error_t HS_CDECL hs_set_stream_allocator(hs_alloc_t allocfunc,
hs_free_t freefunc) {
hs_stream_alloc = normalise_alloc(allocfunc);
hs_stream_free = normalise_free(freefunc);
return HS_SUCCESS;
}

66
src/allocator.h
View File

@ -1,66 +0,0 @@
/*
* Copyright (c) 2015, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#ifndef ALLOCATOR_H
#define ALLOCATOR_H
#include "hs_common.h"
#include "ue2common.h"
#ifdef __cplusplus
extern "C"
{
#endif
extern hs_alloc_t hs_database_alloc;
extern hs_alloc_t hs_misc_alloc;
extern hs_alloc_t hs_scratch_alloc;
extern hs_alloc_t hs_stream_alloc;
extern hs_free_t hs_database_free;
extern hs_free_t hs_misc_free;
extern hs_free_t hs_scratch_free;
extern hs_free_t hs_stream_free;
#ifdef __cplusplus
} /* extern C */
#endif
/** \brief Check the results of an alloc done with hs_alloc for alignment.
*
* If we have incorrect alignment, return an error. Caller should free the
* offending block. */
static really_inline
hs_error_t hs_check_alloc(const void *mem) {
hs_error_t ret = HS_SUCCESS;
if (!mem) {
ret = HS_NOMEM;
} else if (!ISALIGNED_N(mem, alignof(unsigned long long))) {
ret = HS_BAD_ALLOC;
}
return ret;
}
#endif

311
src/compiler/asserts.cpp
View File

@ -1,311 +0,0 @@
/*
* Copyright (c) 2015-2017, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/** \file
* \brief Convert temporary assert vertices (from construction method) to
* edge-based flags.
*
* This pass converts the temporary assert vertices created by the Glushkov
* construction process above (vertices with special assertions flags) into
* edges between those vertices' neighbours in the graph.
*
* These edges have the appropriate flags applied to them -- a path (u,t,v)
* through an assert vertex t will be replaced with the edge (u,v) with the
* assertion flags from t.
*
* Edges with mutually incompatible flags (such as the conjunction of
* word-to-word and word-to-nonword) are dropped.
*/
#include "asserts.h"
#include "compiler/compiler.h"
#include "nfagraph/ng.h"
#include "nfagraph/ng_prune.h"
#include "nfagraph/ng_redundancy.h"
#include "nfagraph/ng_util.h"
#include "parser/position.h" // for POS flags
#include "util/compile_error.h"
#include "util/graph_range.h"
#include <queue>
#include <set>
using namespace std;
namespace ue2 {
/** Hard limit on the maximum number of edges we'll clone before we throw up
* our hands and report 'Pattern too large.' */
static const size_t MAX_ASSERT_EDGES = 300000;
/** Flags representing the word-boundary assertions, \\b or \\B. */
static const int WORDBOUNDARY_FLAGS = POS_FLAG_ASSERT_WORD_TO_WORD
| POS_FLAG_ASSERT_WORD_TO_NONWORD
| POS_FLAG_ASSERT_NONWORD_TO_WORD
| POS_FLAG_ASSERT_NONWORD_TO_NONWORD
| POS_FLAG_ASSERT_WORD_TO_WORD_UCP
| POS_FLAG_ASSERT_WORD_TO_NONWORD_UCP
| POS_FLAG_ASSERT_NONWORD_TO_WORD_UCP
| POS_FLAG_ASSERT_NONWORD_TO_NONWORD_UCP;
#define OPEN_EDGE 0U
#define DEAD_EDGE (~0U)
static
u32 disjunct(u32 flags1, u32 flags2) {
/* from two asserts in parallel */
DEBUG_PRINTF("disjunct %x %x\n", flags1, flags2);
u32 rv;
if (flags1 == DEAD_EDGE) {
rv = flags2;
} else if (flags2 == DEAD_EDGE) {
rv = flags1;
} else if (flags1 == OPEN_EDGE || flags2 == OPEN_EDGE) {
rv = OPEN_EDGE;
} else {
rv = flags1 | flags2;
}
DEBUG_PRINTF("--> %x\n", rv);
return rv;
}
static
u32 conjunct(u32 flags1, u32 flags2) {
/* from two asserts in series */
DEBUG_PRINTF("conjunct %x %x\n", flags1, flags2);
u32 rv;
if (flags1 == OPEN_EDGE) {
rv = flags2;
} else if (flags2 == OPEN_EDGE) {
rv = flags1;
} else if (flags1 & flags2) {
rv = flags1 & flags2;
} else {
rv = DEAD_EDGE; /* the conjunction of two different word boundary
* assertion is impassable */
}
DEBUG_PRINTF("--> %x\n", rv);
return rv;
}
typedef map<pair<NFAVertex, NFAVertex>, NFAEdge> edge_cache_t;
static
void replaceAssertVertex(NGHolder &g, NFAVertex t, const ExpressionInfo &expr,
edge_cache_t &edge_cache, u32 &assert_edge_count) {
DEBUG_PRINTF("replacing assert vertex %zu\n", g[t].index);
const u32 flags = g[t].assert_flags;
DEBUG_PRINTF("consider assert vertex %zu with flags %u\n", g[t].index,
flags);
// Wire up all the predecessors to all the successors.
for (const auto &inEdge : in_edges_range(t, g)) {
NFAVertex u = source(inEdge, g);
if (u == t) {
continue; // ignore self-loops
}
const u32 flags_inc_in = conjunct(g[inEdge].assert_flags,
flags);
if (flags_inc_in == DEAD_EDGE) {
DEBUG_PRINTF("fail, in-edge has bad flags %d\n",
g[inEdge].assert_flags);
continue;
}
for (const auto &outEdge : out_edges_range(t, g)) {
NFAVertex v = target(outEdge, g);
DEBUG_PRINTF("consider path [%zu,%zu,%zu]\n", g[u].index,
g[t].index, g[v].index);
if (v == t) {
continue; // ignore self-loops
}
const u32 flags_final = conjunct(g[outEdge].assert_flags,
flags_inc_in);
if (flags_final == DEAD_EDGE) {
DEBUG_PRINTF("fail, out-edge has bad flags %d\n",
g[outEdge].assert_flags);
continue;
}
if ((g[u].assert_flags & POS_FLAG_MULTILINE_START)
&& v == g.acceptEod) {
DEBUG_PRINTF("fail, (?m)^ does not match \\n at eod\n");
continue;
}
/* Replace path (u,t,v) with direct edge (u,v), unless the edge
* already exists, in which case we just need to edit its
* properties.
*
* Use edge_cache to prevent us going O(N).
*/
auto cache_key = make_pair(u, v);
auto ecit = edge_cache.find(cache_key);
if (ecit == edge_cache.end()) {
DEBUG_PRINTF("adding edge %zu %zu\n", g[u].index, g[v].index);
NFAEdge e = add_edge(u, v, g);
edge_cache.emplace(cache_key, e);
g[e].assert_flags = flags;
if (++assert_edge_count > MAX_ASSERT_EDGES) {
throw CompileError(expr.index, "Pattern is too large.");
}
} else {
NFAEdge e = ecit->second;
DEBUG_PRINTF("updating edge %zu %zu [a %zu]\n", g[u].index,
g[v].index, g[t].index);
// Edge already exists.
u32 &e_flags = g[e].assert_flags;
e_flags = disjunct(e_flags, flags_final);
assert(e_flags != DEAD_EDGE);
}
}
}
// Clear vertex t to remove all the old edges.
/* no need to clear the cache, as we will never look up its edge as it is
* unreachable */
clear_vertex(t, g);
}
static
void setReportId(ReportManager &rm, NGHolder &g, const ExpressionInfo &expr,
NFAVertex v, s32 adj) {
// Don't try and set the report ID of a special vertex.
assert(!is_special(v, g));
// There should be no reports set already.
assert(g[v].reports.empty());
Report r = rm.getBasicInternalReport(expr, adj);
g[v].reports.insert(rm.getInternalId(r));
DEBUG_PRINTF("set report id for vertex %zu, adj %d\n", g[v].index, adj);
}
static
void checkForMultilineStart(ReportManager &rm, NGHolder &g,
const ExpressionInfo &expr) {
vector<NFAEdge> dead;
for (auto v : adjacent_vertices_range(g.start, g)) {
if (!(g[v].assert_flags & POS_FLAG_MULTILINE_START)) {
continue;
}
DEBUG_PRINTF("mls %zu %08x\n", g[v].index, g[v].assert_flags);
/* we have found a multi-line start (maybe more than one) */
/* we need to interpose a dummy dot vertex between v and accept if
* required so that ^ doesn't match trailing \n */
for (const auto &e : out_edges_range(v, g)) {
if (target(e, g) == g.accept) {
dead.emplace_back(e);
}
}
/* assert has been resolved; clear flag */
g[v].assert_flags &= ~POS_FLAG_MULTILINE_START;
}
for (const auto &e : dead) {
NFAVertex dummy = add_vertex(g);
g[dummy].char_reach.setall();
setReportId(rm, g, expr, dummy, -1);
add_edge(source(e, g), dummy, g[e], g);
add_edge(dummy, g.accept, g);
}
remove_edges(dead, g);
}
static
bool hasAssertVertices(const NGHolder &g) {
for (auto v : vertices_range(g)) {
int flags = g[v].assert_flags;
if (flags & WORDBOUNDARY_FLAGS) {
return true;
}
}
return false;
}
/** \brief Convert temporary assert vertices (from construction method) to
* edge-based flags.
*
* Remove the horrors that are the temporary assert vertices which arise from
* our construction method. Allows the rest of our code base to live in
* blissful ignorance of their existence. */
void removeAssertVertices(ReportManager &rm, NGHolder &g,
const ExpressionInfo &expr) {
size_t num = 0;
DEBUG_PRINTF("before: graph has %zu vertices\n", num_vertices(g));
// Sweep over the graph and ascertain that we do actually have vertices
// with assertion flags set. Otherwise, we're done.
if (!hasAssertVertices(g)) {
DEBUG_PRINTF("no assert vertices, done\n");
return;
}
u32 assert_edge_count = 0;
// Build a cache of (u, v) vertex pairs to edge descriptors.
edge_cache_t edge_cache;
for (const auto &e : edges_range(g)) {
edge_cache[make_pair(source(e, g), target(e, g))] = e;
}
for (auto v : vertices_range(g)) {
if (g[v].assert_flags & WORDBOUNDARY_FLAGS) {
replaceAssertVertex(g, v, expr, edge_cache, assert_edge_count);
num++;
}
}
checkForMultilineStart(rm, g, expr);
if (num) {
DEBUG_PRINTF("resolved %zu assert vertices\n", num);
pruneUseless(g);
pruneEmptyVertices(g);
renumber_vertices(g);
renumber_edges(g);
}
DEBUG_PRINTF("after: graph has %zu vertices\n", num_vertices(g));
assert(!hasAssertVertices(g));
}
} // namespace ue2

53
src/compiler/asserts.h
View File

@ -1,53 +0,0 @@
/*
* Copyright (c) 2015-2017, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/** \file
* \brief Convert temporary assert vertices (from construction method) to
* edge-based flags.
*/
#ifndef ASSERTS_H
#define ASSERTS_H
namespace ue2 {
class ExpressionInfo;
class ReportManager;
class NGHolder;
/** \brief Convert temporary assert vertices (from construction method) to
* edge-based flags.
*
* Remove the horrors that are the temporary assert vertices which arise from
* our construction method. Allows the rest of our code base to live in
* blissful ignorance of their existence. */
void removeAssertVertices(ReportManager &rm, NGHolder &g,
const ExpressionInfo &expr);
} // namespace ue2
#endif // ASSERTS_H

635
src/compiler/compiler.cpp
View File

@ -1,635 +0,0 @@
/*
* Copyright (c) 2015-2020, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/** \file
* \brief Compiler front-end interface.
*/
#include "allocator.h"
#include "asserts.h"
#include "compiler.h"
#include "crc32.h"
#include "database.h"
#include "grey.h"
#include "hs_internal.h"
#include "hs_runtime.h"
#include "ue2common.h"
#include "nfagraph/ng_builder.h"
#include "nfagraph/ng_dump.h"
#include "nfagraph/ng.h"
#include "nfagraph/ng_util.h"
#include "parser/buildstate.h"
#include "parser/dump.h"
#include "parser/Component.h"
#include "parser/logical_combination.h"
#include "parser/parse_error.h"
#include "parser/Parser.h" // for flags
#include "parser/position.h"
#include "parser/position_dump.h"
#include "parser/position_info.h"
#include "parser/prefilter.h"
#include "parser/shortcut_literal.h"
#include "parser/unsupported.h"
#include "parser/utf8_validate.h"
#include "rose/rose_build.h"
#include "rose/rose_internal.h"
#include "som/slot_manager_dump.h"
#include "util/bytecode_ptr.h"
#include "util/compile_error.h"
#include "util/target_info.h"
#include "util/verify_types.h"
#include "util/ue2string.h"
#include <algorithm>
#include <cassert>
#include <cstdlib>
#include <cstring>
#include <fstream>
#include <memory>
#include <sstream>
using namespace std;
namespace ue2 {
static
void validateExt(const hs_expr_ext &ext) {
static const unsigned long long ALL_EXT_FLAGS = HS_EXT_FLAG_MIN_OFFSET |
HS_EXT_FLAG_MAX_OFFSET |
HS_EXT_FLAG_MIN_LENGTH |
HS_EXT_FLAG_EDIT_DISTANCE |
HS_EXT_FLAG_HAMMING_DISTANCE;
if (ext.flags & ~ALL_EXT_FLAGS) {
throw CompileError("Invalid hs_expr_ext flag set.");
}
if ((ext.flags & HS_EXT_FLAG_MIN_OFFSET) &&
(ext.flags & HS_EXT_FLAG_MAX_OFFSET) &&
(ext.min_offset > ext.max_offset)) {
throw CompileError("In hs_expr_ext, min_offset must be less than or "
"equal to max_offset.");
}
if ((ext.flags & HS_EXT_FLAG_MIN_LENGTH) &&
(ext.flags & HS_EXT_FLAG_MAX_OFFSET) &&
(ext.min_length > ext.max_offset)) {
throw CompileError("In hs_expr_ext, min_length must be less than or "
"equal to max_offset.");
}
if ((ext.flags & HS_EXT_FLAG_EDIT_DISTANCE) &&
(ext.flags & HS_EXT_FLAG_HAMMING_DISTANCE)) {
throw CompileError("In hs_expr_ext, cannot have both edit distance and "
"Hamming distance.");
}
}
void ParsedLitExpression::parseLiteral(const char *expression, size_t len,
bool nocase) {
const char *c = expression;
for (size_t i = 0; i < len; i++) {
lit.push_back(*c, nocase);
c++;
}
}
ParsedLitExpression::ParsedLitExpression(unsigned index_in,
const char *expression,
size_t expLength, unsigned flags,
ReportID report)
: expr(index_in, false, flags & HS_FLAG_SINGLEMATCH, false, false,
SOM_NONE, report, 0, MAX_OFFSET, 0, 0, 0, false) {
// For pure literal expression, below 'HS_FLAG_'s are unuseful:
// DOTALL/ALLOWEMPTY/UTF8/UCP/PREFILTER/COMBINATION/QUIET/MULTILINE
if (flags & ~HS_FLAG_ALL) {
DEBUG_PRINTF("Unrecognised flag, flags=%u.\n", flags);
throw CompileError("Unrecognised flag.");
}
// FIXME: we disallow highlander + SOM, see UE-1850.
if ((flags & HS_FLAG_SINGLEMATCH) && (flags & HS_FLAG_SOM_LEFTMOST)) {
throw CompileError("HS_FLAG_SINGLEMATCH is not supported in "
"combination with HS_FLAG_SOM_LEFTMOST.");
}
// Set SOM type.
if (flags & HS_FLAG_SOM_LEFTMOST) {
expr.som = SOM_LEFT;
}
// Transfer expression text into ue2_literal.
bool nocase = flags & HS_FLAG_CASELESS ? true : false;
parseLiteral(expression, expLength, nocase);
}
ParsedExpression::ParsedExpression(unsigned index_in, const char *expression,
unsigned flags, ReportID report,
const hs_expr_ext *ext)
: expr(index_in, flags & HS_FLAG_ALLOWEMPTY, flags & HS_FLAG_SINGLEMATCH,
false, flags & HS_FLAG_PREFILTER, SOM_NONE, report, 0, MAX_OFFSET,
0, 0, 0, flags & HS_FLAG_QUIET) {
// We disallow SOM + Quiet.
if ((flags & HS_FLAG_QUIET) && (flags & HS_FLAG_SOM_LEFTMOST)) {
throw CompileError("HS_FLAG_QUIET is not supported in "
"combination with HS_FLAG_SOM_LEFTMOST.");
}
flags &= ~HS_FLAG_QUIET;
ParseMode mode(flags);
component = parse(expression, mode);
expr.utf8 = mode.utf8; /* utf8 may be set by parse() */
const size_t len = strlen(expression);
if (expr.utf8 && !isValidUtf8(expression, len)) {
throw ParseError("Expression is not valid UTF-8.");
}
if (!component) {
assert(0); // parse() should have thrown a ParseError.
throw ParseError("Parse error.");
}
if (flags & ~HS_FLAG_ALL) {
DEBUG_PRINTF("Unrecognised flag, flags=%u.\n", flags);
throw CompileError("Unrecognised flag.");
}
// FIXME: we disallow highlander + SOM, see UE-1850.
if ((flags & HS_FLAG_SINGLEMATCH) && (flags & HS_FLAG_SOM_LEFTMOST)) {
throw CompileError("HS_FLAG_SINGLEMATCH is not supported in "
"combination with HS_FLAG_SOM_LEFTMOST.");
}
// FIXME: we disallow prefilter + SOM, see UE-1899.
if ((flags & HS_FLAG_PREFILTER) && (flags & HS_FLAG_SOM_LEFTMOST)) {
throw CompileError("HS_FLAG_PREFILTER is not supported in "
"combination with HS_FLAG_SOM_LEFTMOST.");
}
// Set SOM type.
if (flags & HS_FLAG_SOM_LEFTMOST) {
expr.som = SOM_LEFT;
}
// Set extended parameters, if we have them.
if (ext) {
// Ensure that the given parameters make sense.
validateExt(*ext);
if (ext->flags & HS_EXT_FLAG_MIN_OFFSET) {
expr.min_offset = ext->min_offset;
}
if (ext->flags & HS_EXT_FLAG_MAX_OFFSET) {
expr.max_offset = ext->max_offset;
}
if (ext->flags & HS_EXT_FLAG_MIN_LENGTH) {
expr.min_length = ext->min_length;
}
if (ext->flags & HS_EXT_FLAG_EDIT_DISTANCE) {
expr.edit_distance = ext->edit_distance;
}
if (ext->flags & HS_EXT_FLAG_HAMMING_DISTANCE) {
expr.hamm_distance = ext->hamming_distance;
}
}
// These are validated in validateExt, so an error will already have been
// thrown if these conditions don't hold.
assert(expr.max_offset >= expr.min_offset);
assert(expr.max_offset >= expr.min_length);
// Since prefiltering and SOM aren't supported together, we must squash any
// min_length constraint as well.
if (flags & HS_FLAG_PREFILTER && expr.min_length) {
DEBUG_PRINTF("prefiltering mode: squashing min_length constraint\n");
expr.min_length = 0;
}
}
#if defined(DUMP_SUPPORT) || defined(DEBUG)
/**
* \brief Dumps the parse tree to screen in debug mode and to disk in dump
* mode.
*/
void dumpExpression(UNUSED const ParsedExpression &pe,
UNUSED const char *stage, UNUSED const Grey &grey) {
#if defined(DEBUG)
DEBUG_PRINTF("===== Rule ID: %u (expression index: %u) =====\n",
pe.expr.report, pe.expr.index);
ostringstream debug_tree;
dumpTree(debug_tree, pe.component.get());
printf("%s\n", debug_tree.str().c_str());
#endif // DEBUG
#if defined(DUMP_SUPPORT)
if (grey.dumpFlags & Grey::DUMP_PARSE) {
stringstream ss;
ss << grey.dumpPath << "Expr_" << pe.expr.index << "_componenttree_"
<< stage << ".txt";
ofstream out(ss.str().c_str());
out << "Component Tree for " << pe.expr.report << endl;
dumpTree(out, pe.component.get());
if (pe.expr.utf8) {
out << "UTF8 mode" << endl;
}
}
#endif // DEBUG
}
#endif
/** \brief Run Component tree optimisations on \a expr. */
static
void optimise(ParsedExpression &pe) {
if (pe.expr.min_length || pe.expr.som) {
return;
}
DEBUG_PRINTF("optimising\n");
pe.component->optimise(true /* root is connected to sds */);
}
void addExpression(NG &ng, unsigned index, const char *expression,
unsigned flags, const hs_expr_ext *ext, ReportID id) {
assert(expression);
const CompileContext &cc = ng.cc;
DEBUG_PRINTF("index=%u, id=%u, flags=%u, expr='%s'\n", index, id, flags,
expression);
if (flags & HS_FLAG_COMBINATION) {
if (flags & ~(HS_FLAG_COMBINATION | HS_FLAG_QUIET |
HS_FLAG_SINGLEMATCH)) {
throw CompileError("only HS_FLAG_QUIET and HS_FLAG_SINGLEMATCH "
"are supported in combination "
"with HS_FLAG_COMBINATION.");
}
if (flags & HS_FLAG_QUIET) {
DEBUG_PRINTF("skip QUIET logical combination expression %u\n", id);
} else {
u32 ekey = INVALID_EKEY;
u64a min_offset = 0;
u64a max_offset = MAX_OFFSET;
if (flags & HS_FLAG_SINGLEMATCH) {
ekey = ng.rm.getExhaustibleKey(id);
}
if (ext) {
validateExt(*ext);
if (ext->flags & ~(HS_EXT_FLAG_MIN_OFFSET |
HS_EXT_FLAG_MAX_OFFSET)) {
throw CompileError("only HS_EXT_FLAG_MIN_OFFSET and "
"HS_EXT_FLAG_MAX_OFFSET extra flags "
"are supported in combination "
"with HS_FLAG_COMBINATION.");
}
if (ext->flags & HS_EXT_FLAG_MIN_OFFSET) {
min_offset = ext->min_offset;
}
if (ext->flags & HS_EXT_FLAG_MAX_OFFSET) {
max_offset = ext->max_offset;
}
}
ng.rm.pl.parseLogicalCombination(id, expression, ekey, min_offset,
max_offset);
DEBUG_PRINTF("parsed logical combination expression %u\n", id);
}
return;
}
// Ensure that our pattern isn't too long (in characters).
if (strlen(expression) > cc.grey.limitPatternLength) {
throw CompileError("Pattern length exceeds limit.");
}
// Do per-expression processing: errors here will result in an exception
// being thrown up to our caller
ParsedExpression pe(index, expression, flags, id, ext);
dumpExpression(pe, "orig", cc.grey);
// Apply prefiltering transformations if desired.
if (pe.expr.prefilter) {
prefilterTree(pe.component, ParseMode(flags));
dumpExpression(pe, "prefiltered", cc.grey);
}
// Expressions containing zero-width assertions and other extended pcre
// types aren't supported yet. This call will throw a ParseError exception
// if the component tree contains such a construct.
checkUnsupported(*pe.component);
pe.component->checkEmbeddedStartAnchor(true);
pe.component->checkEmbeddedEndAnchor(true);
if (cc.grey.optimiseComponentTree) {
optimise(pe);
dumpExpression(pe, "opt", cc.grey);
}
DEBUG_PRINTF("component=%p, nfaId=%u, reportId=%u\n",
pe.component.get(), pe.expr.index, pe.expr.report);
// You can only use the SOM flags if you've also specified an SOM
// precision mode.
if (pe.expr.som != SOM_NONE && cc.streaming && !ng.ssm.somPrecision()) {
throw CompileError("To use a SOM expression flag in streaming mode, "
"an SOM precision mode (e.g. "
"HS_MODE_SOM_HORIZON_LARGE) must be specified.");
}
// If this expression is a literal, we can feed it directly to Rose rather
// than building the NFA graph.
if (shortcutLiteral(ng, pe)) {
DEBUG_PRINTF("took literal short cut\n");
return;
}
auto built_expr = buildGraph(ng.rm, cc, pe);
if (!built_expr.g) {
DEBUG_PRINTF("NFA build failed on ID %u, but no exception was "
"thrown.\n", pe.expr.report);
throw CompileError("Internal error.");
}
if (!pe.expr.allow_vacuous && matches_everywhere(*built_expr.g)) {
throw CompileError("Pattern matches empty buffer; use "
"HS_FLAG_ALLOWEMPTY to enable support.");
}
if (!ng.addGraph(built_expr.expr, std::move(built_expr.g))) {
DEBUG_PRINTF("NFA addGraph failed on ID %u.\n", pe.expr.report);
throw CompileError("Error compiling expression.");
}
}
void addLitExpression(NG &ng, unsigned index, const char *expression,
unsigned flags, const hs_expr_ext *ext, ReportID id,
size_t expLength) {
assert(expression);
const CompileContext &cc = ng.cc;
DEBUG_PRINTF("index=%u, id=%u, flags=%u, expr='%s', len='%zu'\n", index,
id, flags, expression, expLength);
// Extended parameters are not supported for pure literal patterns.
if (ext && ext->flags != 0LLU) {
throw CompileError("Extended parameters are not supported for pure "
"literal matching API.");
}
// Ensure that our pattern isn't too long (in characters).
if (expLength > cc.grey.limitPatternLength) {
throw CompileError("Pattern length exceeds limit.");
}
// filter out flags not supported by pure literal API.
u64a not_supported = HS_FLAG_DOTALL | HS_FLAG_ALLOWEMPTY | HS_FLAG_UTF8 |
HS_FLAG_UCP | HS_FLAG_PREFILTER | HS_FLAG_COMBINATION |
HS_FLAG_QUIET | HS_FLAG_MULTILINE;
if (flags & not_supported) {
throw CompileError("Only HS_FLAG_CASELESS, HS_FLAG_SINGLEMATCH and "
"HS_FLAG_SOM_LEFTMOST are supported in literal API.");
}
// This expression must be a pure literal, we can build ue2_literal
// directly based on expression text.
ParsedLitExpression ple(index, expression, expLength, flags, id);
// Feed the ue2_literal into Rose.
const auto &expr = ple.expr;
if (ng.addLiteral(ple.lit, expr.index, expr.report, expr.highlander,
expr.som, expr.quiet)) {
DEBUG_PRINTF("took pure literal\n");
return;
}
}
static
bytecode_ptr<RoseEngine> generateRoseEngine(NG &ng) {
const u32 minWidth =
ng.minWidth.is_finite() ? verify_u32(ng.minWidth) : ROSE_BOUND_INF;
auto rose = ng.rose->buildRose(minWidth);
if (!rose) {
DEBUG_PRINTF("error building rose\n");
assert(0);
return nullptr;
}
dumpReportManager(ng.rm, ng.cc.grey);
dumpSomSlotManager(ng.ssm, ng.cc.grey);
dumpSmallWrite(rose.get(), ng.cc.grey);
return rose;
}
platform_t target_to_platform(const target_t &target_info) {
platform_t p;
p = 0;
if (!target_info.has_avx2()) {
p |= HS_PLATFORM_NOAVX2;
}
if (!target_info.has_avx512()) {
p |= HS_PLATFORM_NOAVX512;
}
if (!target_info.has_avx512vbmi()) {
p |= HS_PLATFORM_NOAVX512VBMI;
}
return p;
}
/** \brief Encapsulate the given bytecode (RoseEngine) in a newly-allocated
* \ref hs_database, ensuring that it is padded correctly to give cacheline
* alignment. */
static
hs_database_t *dbCreate(const char *in_bytecode, size_t len, u64a platform) {
size_t db_len = sizeof(struct hs_database) + len;
DEBUG_PRINTF("db size %zu\n", db_len);
DEBUG_PRINTF("db platform %llx\n", platform);
struct hs_database *db = (struct hs_database *)hs_database_alloc(db_len);
if (hs_check_alloc(db) != HS_SUCCESS) {
hs_database_free(db);
return nullptr;
}
// So that none of our database is uninitialized
memset(db, 0, db_len);
// we need to align things manually
size_t shift = (uintptr_t)db->bytes & 0x3f;
DEBUG_PRINTF("shift is %zu\n", shift);
db->bytecode = offsetof(struct hs_database, bytes) - shift;
char *bytecode = (char *)db + db->bytecode;
assert(ISALIGNED_CL(bytecode));
db->magic = HS_DB_MAGIC;
db->version = HS_DB_VERSION;
db->length = len;
db->platform = platform;
// Copy bytecode
memcpy(bytecode, in_bytecode, len);
db->crc32 = Crc32c_ComputeBuf(0, bytecode, db->length);
return db;
}
struct hs_database *build(NG &ng, unsigned int *length, u8 pureFlag) {
assert(length);
auto rose = generateRoseEngine(ng);
struct RoseEngine *roseHead = rose.get();
roseHead->pureLiteral = pureFlag;
if (!rose) {
throw CompileError("Unable to generate bytecode.");
}
*length = rose.size();
if (!*length) {
DEBUG_PRINTF("RoseEngine has zero length\n");
assert(0);
throw CompileError("Internal error.");
}
const char *bytecode = (const char *)(rose.get());
const platform_t p = target_to_platform(ng.cc.target_info);
struct hs_database *db = dbCreate(bytecode, *length, p);
if (!db) {
throw CompileError("Could not allocate memory for bytecode.");
}
return db;
}
static
void stripFromPositions(vector<PositionInfo> &v, Position pos) {
auto removed = remove(v.begin(), v.end(), PositionInfo(pos));
v.erase(removed, v.end());
}
static
void connectInitialStates(GlushkovBuildState &bs,
const ParsedExpression &expr) {
vector<PositionInfo> initials = expr.component->first();
const NFABuilder &builder = bs.getBuilder();
const Position startState = builder.getStart();
const Position startDotStarState = builder.getStartDotStar();
DEBUG_PRINTF("wiring initials = %s\n",
dumpPositions(initials.begin(), initials.end()).c_str());
vector<PositionInfo> starts = {startState, startDotStarState};
// strip start and startDs, which can be present due to boundaries
stripFromPositions(initials, startState);
stripFromPositions(initials, startDotStarState);
// replace epsilons with accepts
for (const auto &s : initials) {
if (s.pos != GlushkovBuildState::POS_EPSILON) {
continue;
}
assert(starts.size() == 2); /* start, startds */
vector<PositionInfo> starts_temp = starts;
starts_temp[0].flags = s.flags;
starts_temp[1].flags = s.flags;
bs.connectAccepts(starts_temp);
}
if (!initials.empty()) {
bs.connectRegions(starts, initials);
}
}
static
void connectFinalStates(GlushkovBuildState &bs, const ParsedExpression &expr) {
vector<PositionInfo> finals = expr.component->last();
DEBUG_PRINTF("wiring finals = %s\n",
dumpPositions(finals.begin(), finals.end()).c_str());
bs.connectAccepts(finals);
}
#ifndef NDEBUG
static
bool isSupported(const Component &c) {
try {
checkUnsupported(c);
return true;
}
catch (ParseError &) {
return false;
}
}
#endif
BuiltExpression buildGraph(ReportManager &rm, const CompileContext &cc,
const ParsedExpression &pe) {
assert(isSupported(*pe.component));
const auto builder = makeNFABuilder(rm, cc, pe);
assert(builder);
// Set up START and ACCEPT states; retrieve the special states
const auto bs = makeGlushkovBuildState(*builder, pe.expr.prefilter);
// Map position IDs to characters/components
pe.component->notePositions(*bs);
// Wire the start dotstar state to the firsts
connectInitialStates(*bs, pe);
DEBUG_PRINTF("wire up body of expr\n");
// Build the rest of the FOLLOW set
vector<PositionInfo> initials = {builder->getStartDotStar(),
builder->getStart()};
pe.component->buildFollowSet(*bs, initials);
// Wire the lasts to the accept state
connectFinalStates(*bs, pe);
// Create our edges
bs->buildEdges();
BuiltExpression built_expr = builder->getGraph();
assert(built_expr.g);
dumpDotWrapper(*built_expr.g, built_expr.expr, "00_before_asserts",
cc.grey);
removeAssertVertices(rm, *built_expr.g, built_expr.expr);
return built_expr;
}
} // namespace ue2

172
src/compiler/compiler.h
View File

@ -1,172 +0,0 @@
/*
* Copyright (c) 2015-2019, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/** \file
* \brief Compiler front-end interface
*/
#ifndef COMPILER_H
#define COMPILER_H
#include "ue2common.h"
#include "database.h"
#include "compiler/expression_info.h"
#include "parser/Component.h"
#include "util/noncopyable.h"
#include "util/ue2string.h"
#include <memory>
struct hs_database;
struct hs_expr_ext;
namespace ue2 {
struct CompileContext;
struct Grey;
struct target_t;
class NG;
class NGHolder;
class ReportManager;
/** \brief Class gathering together the pieces of a parsed expression. */
class ParsedExpression : noncopyable {
public:
ParsedExpression(unsigned index, const char *expression, unsigned flags,
ReportID report, const hs_expr_ext *ext = nullptr);
/** \brief Expression information (from flags, extparam etc) */
ExpressionInfo expr;
/** \brief Root node of parsed component tree. */
std::unique_ptr<Component> component;
};
/** \brief Class gathering together the pieces of a parsed lit-expression. */
class ParsedLitExpression : noncopyable {
public:
ParsedLitExpression(unsigned index, const char *expression,
size_t expLength, unsigned flags, ReportID report);
void parseLiteral(const char *expression, size_t len, bool nocase);
/** \brief Expression information (from flags, extparam etc) */
ExpressionInfo expr;
/** \brief Format the lit-expression text into Hyperscan literal type. */
ue2_literal lit;
};
/**
* \brief Class gathering together the pieces of an expression that has been
* built into an NFA graph.
*/
struct BuiltExpression {
/** \brief Expression information (from flags, extparam etc) */
ExpressionInfo expr;
/** \brief Built Glushkov NFA graph. */
std::unique_ptr<NGHolder> g;
};
/**
* Add an expression to the compiler.
*
* @param ng
* The global NG object.
* @param index
* The index of the expression (used for errors)
* @param expression
* NULL-terminated PCRE expression
* @param flags
* The full set of Hyperscan flags associated with this rule.
* @param ext
* Struct containing extra parameters for this expression, or NULL if
* none.
* @param report
* The identifier to associate with the expression; returned by engine on
* match.
*/
void addExpression(NG &ng, unsigned index, const char *expression,
unsigned flags, const hs_expr_ext *ext, ReportID report);
void addLitExpression(NG &ng, unsigned index, const char *expression,
unsigned flags, const hs_expr_ext *ext, ReportID id,
size_t expLength);
/**
* Build a Hyperscan database out of the expressions we've been given. A
* fatal error will result in an exception being thrown.
*
* @param ng
* The global NG object.
* @param[out] length
* The number of bytes occupied by the compiled structure.
* @param pureFlag
* The flag indicating invocation from literal API or not.
* @return
* The compiled structure. Should be deallocated with the
* hs_database_free() function.
*/
struct hs_database *build(NG &ng, unsigned int *length, u8 pureFlag);
/**
* Constructs an NFA graph from the given expression tree.
*
* @param rm
* Global ReportManager for this compile.
* @param cc
* Global compile context for this compile.
* @param expr
* ParsedExpression object.
* @return
* nullptr on error.
*/
BuiltExpression buildGraph(ReportManager &rm, const CompileContext &cc,
const ParsedExpression &expr);
/**
* Build a platform_t out of a target_t.
*/
platform_t target_to_platform(const target_t &target_info);
#if defined(DUMP_SUPPORT) || defined(DEBUG)
void dumpExpression(const ParsedExpression &expr, const char *stage,
const Grey &grey);
#else
static really_inline
void dumpExpression(UNUSED const ParsedExpression &expr,
UNUSED const char *stage, UNUSED const Grey &grey) {
}
#endif
} // namespace
#endif // COMPILER_H

110
src/compiler/error.cpp
View File

@ -1,110 +0,0 @@
/*
* Copyright (c) 2015-2016, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/** \file
* \brief Compile-time error utils.
*/
#include "allocator.h"
#include "error.h"
#include "ue2common.h"
#include "hs_compile.h"
#include "util/compile_error.h"
#include <cstring>
#include <string>
using std::string;
static const char failureNoMemory[] = "Unable to allocate memory.";
static const char failureInternal[] = "Internal error.";
static const char failureBadAlloc[] = "Allocator returned misaligned memory.";
extern const hs_compile_error_t hs_enomem = {
const_cast<char *>(failureNoMemory), 0
};
extern const hs_compile_error_t hs_einternal = {
const_cast<char *>(failureInternal), 0
};
extern const hs_compile_error_t hs_badalloc = {
const_cast<char *>(failureBadAlloc), 0
};
namespace ue2 {
hs_compile_error_t *generateCompileError(const string &err, int expression) {
hs_compile_error_t *ret =
(struct hs_compile_error *)hs_misc_alloc(sizeof(hs_compile_error_t));
if (ret) {
hs_error_t e = hs_check_alloc(ret);
if (e != HS_SUCCESS) {
hs_misc_free(ret);
return const_cast<hs_compile_error_t *>(&hs_badalloc);
}
char *msg = (char *)hs_misc_alloc(err.size() + 1);
if (msg) {
e = hs_check_alloc(msg);
if (e != HS_SUCCESS) {
hs_misc_free(msg);
return const_cast<hs_compile_error_t *>(&hs_badalloc);
}
memcpy(msg, err.c_str(), err.size() + 1);
ret->message = msg;
} else {
hs_misc_free(ret);
ret = nullptr;
}
}
if (!ret || !ret->message) {
return const_cast<hs_compile_error_t *>(&hs_enomem);
}
ret->expression = expression;
return ret;
}
hs_compile_error_t *generateCompileError(const CompileError &e) {
return generateCompileError(e.reason, e.hasIndex ? (int)e.index : -1);
}
void freeCompileError(hs_compile_error_t *error) {
if (!error) {
return;
}
if (error == &hs_enomem || error == &hs_einternal ||
error == &hs_badalloc) {
// These are not allocated.
return;
}
hs_misc_free(error->message);
hs_misc_free(error);
}
} // namespace ue2

55
src/compiler/error.h
View File

@ -1,55 +0,0 @@
/*
* Copyright (c) 2015, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/** \file
* \brief Compile-time error utils.
*/
#ifndef COMPILE_ERROR_H
#define COMPILE_ERROR_H
#include <string>
struct hs_compile_error;
// Special errors that aren't allocated with hs_alloc/hs_free.
extern const hs_compile_error hs_enomem;
extern const hs_compile_error hs_einternal;
namespace ue2 {
class CompileError;
hs_compile_error *generateCompileError(const std::string &err, int expression);
hs_compile_error *generateCompileError(const CompileError &e);
void freeCompileError(hs_compile_error *error);
} // namespace ue2
#endif

108
src/compiler/expression_info.h
View File

@ -1,108 +0,0 @@
/*
* Copyright (c) 2017-2018, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/**
* \file
* \brief ExpressionInfo class for storing the properties of an expression.
*/
#ifndef COMPILER_EXPRESSION_INFO_H
#define COMPILER_EXPRESSION_INFO_H
#include "ue2common.h"
#include "som/som.h"
namespace ue2 {
/** \brief Properties of an expression. */
class ExpressionInfo {
public:
ExpressionInfo(unsigned int index_in, bool allow_vacuous_in,
bool highlander_in, bool utf8_in, bool prefilter_in,
som_type som_in, ReportID report_in, u64a min_offset_in,
u64a max_offset_in, u64a min_length_in, u32 edit_distance_in,
u32 hamm_distance_in, bool quiet_in)
: index(index_in), report(report_in), allow_vacuous(allow_vacuous_in),
highlander(highlander_in), utf8(utf8_in), prefilter(prefilter_in),
som(som_in), min_offset(min_offset_in), max_offset(max_offset_in),
min_length(min_length_in), edit_distance(edit_distance_in),
hamm_distance(hamm_distance_in), quiet(quiet_in) {}
/**
* \brief Index of the expression represented by this graph.
*
* Used:
* - down the track in error handling;
* - for identifying parts of an expression in highlander mode.
*/
unsigned int index;
/** \brief Report ID specified by the user. */
ReportID report;
/** \brief Vacuous pattern is allowed. (HS_FLAG_ALLOWEMPTY) */
bool allow_vacuous;
/** \brief "Highlander" (single match) pattern. (HS_FLAG_SINGLEMATCH) */
bool highlander;
/** \brief UTF-8 pattern. (HS_FLAG_UTF8) */
bool utf8;
/** \brief Prefiltering pattern. (HS_FLAG_PREFILTER) */
bool prefilter;
/** \brief Start-of-match type requested, or SOM_NONE. */
som_type som;
/** \brief Minimum match offset extended parameter. 0 if not used. */
u64a min_offset;
/**
* \brief Maximum match offset extended parameter.
* MAX_OFFSET if not used.
*/
u64a max_offset;
/** \brief Minimum match length extended parameter. 0 if not used. */
u64a min_length;
/**
* \brief Approximate matching edit distance extended parameter.
* 0 if not used.
*/
u32 edit_distance;
u32 hamm_distance;
/** \brief Quiet on match. */
bool quiet;
};
}
#endif // COMPILER_EXPRESSION_INFO_H

601
src/crc32.c
View File

@ -1,601 +0,0 @@
/*
* Copyright (c) 2015-2017, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#include "crc32.h"
#include "config.h"
#include "ue2common.h"
#include "util/arch.h"
#if !defined(HAVE_SSE42)
/***
*** What follows is derived from Intel's Slicing-by-8 CRC32 impl, which is BSD
*** licensed and available from http://sourceforge.net/projects/slicing-by-8/
***/
/*
* Copyright (c) 2004-2006 Intel Corporation - All Rights Reserved
*
*
* This software program is licensed subject to the BSD License,
* available at http://www.opensource.org/licenses/bsd-license.html.
*
* Abstract:
*
* Tables for software CRC generation
*/
/*
* The following CRC lookup table was generated automagically
* using the following model parameters:
*
* Generator Polynomial = ................. 0x1EDC6F41
* Generator Polynomial Length = .......... 32 bits
* Reflected Bits = ....................... TRUE
* Table Generation Offset = .............. 32 bits
* Number of Slices = ..................... 8 slices
* Slice Lengths = ........................ 8 8 8 8 8 8 8 8
* Directory Name = ....................... .\
* File Name = ............................ 8x256_tables.c
*/
static
u32 crc_tableil8_o32[256] =
{
0x00000000, 0xF26B8303, 0xE13B70F7, 0x1350F3F4, 0xC79A971F, 0x35F1141C, 0x26A1E7E8, 0xD4CA64EB,
0x8AD958CF, 0x78B2DBCC, 0x6BE22838, 0x9989AB3B, 0x4D43CFD0, 0xBF284CD3, 0xAC78BF27, 0x5E133C24,
0x105EC76F, 0xE235446C, 0xF165B798, 0x030E349B, 0xD7C45070, 0x25AFD373, 0x36FF2087, 0xC494A384,
0x9A879FA0, 0x68EC1CA3, 0x7BBCEF57, 0x89D76C54, 0x5D1D08BF, 0xAF768BBC, 0xBC267848, 0x4E4DFB4B,
0x20BD8EDE, 0xD2D60DDD, 0xC186FE29, 0x33ED7D2A, 0xE72719C1, 0x154C9AC2, 0x061C6936, 0xF477EA35,
0xAA64D611, 0x580F5512, 0x4B5FA6E6, 0xB93425E5, 0x6DFE410E, 0x9F95C20D, 0x8CC531F9, 0x7EAEB2FA,
0x30E349B1, 0xC288CAB2, 0xD1D83946, 0x23B3BA45, 0xF779DEAE, 0x05125DAD, 0x1642AE59, 0xE4292D5A,
0xBA3A117E, 0x4851927D, 0x5B016189, 0xA96AE28A, 0x7DA08661, 0x8FCB0562, 0x9C9BF696, 0x6EF07595,
0x417B1DBC, 0xB3109EBF, 0xA0406D4B, 0x522BEE48, 0x86E18AA3, 0x748A09A0, 0x67DAFA54, 0x95B17957,
0xCBA24573, 0x39C9C670, 0x2A993584, 0xD8F2B687, 0x0C38D26C, 0xFE53516F, 0xED03A29B, 0x1F682198,
0x5125DAD3, 0xA34E59D0, 0xB01EAA24, 0x42752927, 0x96BF4DCC, 0x64D4CECF, 0x77843D3B, 0x85EFBE38,
0xDBFC821C, 0x2997011F, 0x3AC7F2EB, 0xC8AC71E8, 0x1C661503, 0xEE0D9600, 0xFD5D65F4, 0x0F36E6F7,
0x61C69362, 0x93AD1061, 0x80FDE395, 0x72966096, 0xA65C047D, 0x5437877E, 0x4767748A, 0xB50CF789,
0xEB1FCBAD, 0x197448AE, 0x0A24BB5A, 0xF84F3859, 0x2C855CB2, 0xDEEEDFB1, 0xCDBE2C45, 0x3FD5AF46,
0x7198540D, 0x83F3D70E, 0x90A324FA, 0x62C8A7F9, 0xB602C312, 0x44694011, 0x5739B3E5, 0xA55230E6,
0xFB410CC2, 0x092A8FC1, 0x1A7A7C35, 0xE811FF36, 0x3CDB9BDD, 0xCEB018DE, 0xDDE0EB2A, 0x2F8B6829,
0x82F63B78, 0x709DB87B, 0x63CD4B8F, 0x91A6C88C, 0x456CAC67, 0xB7072F64, 0xA457DC90, 0x563C5F93,
0x082F63B7, 0xFA44E0B4, 0xE9141340, 0x1B7F9043, 0xCFB5F4A8, 0x3DDE77AB, 0x2E8E845F, 0xDCE5075C,
0x92A8FC17, 0x60C37F14, 0x73938CE0, 0x81F80FE3, 0x55326B08, 0xA759E80B, 0xB4091BFF, 0x466298FC,
0x1871A4D8, 0xEA1A27DB, 0xF94AD42F, 0x0B21572C, 0xDFEB33C7, 0x2D80B0C4, 0x3ED04330, 0xCCBBC033,
0xA24BB5A6, 0x502036A5, 0x4370C551, 0xB11B4652, 0x65D122B9, 0x97BAA1BA, 0x84EA524E, 0x7681D14D,
0x2892ED69, 0xDAF96E6A, 0xC9A99D9E, 0x3BC21E9D, 0xEF087A76, 0x1D63F975, 0x0E330A81, 0xFC588982,
0xB21572C9, 0x407EF1CA, 0x532E023E, 0xA145813D, 0x758FE5D6, 0x87E466D5, 0x94B49521, 0x66DF1622,
0x38CC2A06, 0xCAA7A905, 0xD9F75AF1, 0x2B9CD9F2, 0xFF56BD19, 0x0D3D3E1A, 0x1E6DCDEE, 0xEC064EED,
0xC38D26C4, 0x31E6A5C7, 0x22B65633, 0xD0DDD530, 0x0417B1DB, 0xF67C32D8, 0xE52CC12C, 0x1747422F,
0x49547E0B, 0xBB3FFD08, 0xA86F0EFC, 0x5A048DFF, 0x8ECEE914, 0x7CA56A17, 0x6FF599E3, 0x9D9E1AE0,
0xD3D3E1AB, 0x21B862A8, 0x32E8915C, 0xC083125F, 0x144976B4, 0xE622F5B7, 0xF5720643, 0x07198540,
0x590AB964, 0xAB613A67, 0xB831C993, 0x4A5A4A90, 0x9E902E7B, 0x6CFBAD78, 0x7FAB5E8C, 0x8DC0DD8F,
0xE330A81A, 0x115B2B19, 0x020BD8ED, 0xF0605BEE, 0x24AA3F05, 0xD6C1BC06, 0xC5914FF2, 0x37FACCF1,
0x69E9F0D5, 0x9B8273D6, 0x88D28022, 0x7AB90321, 0xAE7367CA, 0x5C18E4C9, 0x4F48173D, 0xBD23943E,
0xF36E6F75, 0x0105EC76, 0x12551F82, 0xE03E9C81, 0x34F4F86A, 0xC69F7B69, 0xD5CF889D, 0x27A40B9E,
0x79B737BA, 0x8BDCB4B9, 0x988C474D, 0x6AE7C44E, 0xBE2DA0A5, 0x4C4623A6, 0x5F16D052, 0xAD7D5351
};
/*
* end of the CRC lookup table crc_tableil8_o32
*/
/*
* The following CRC lookup table was generated automagically
* using the following model parameters:
*
* Generator Polynomial = ................. 0x1EDC6F41
* Generator Polynomial Length = .......... 32 bits
* Reflected Bits = ....................... TRUE
* Table Generation Offset = .............. 32 bits
* Number of Slices = ..................... 8 slices
* Slice Lengths = ........................ 8 8 8 8 8 8 8 8
* Directory Name = ....................... .\
* File Name = ............................ 8x256_tables.c
*/
static
u32 crc_tableil8_o40[256] =
{
0x00000000, 0x13A29877, 0x274530EE, 0x34E7A899, 0x4E8A61DC, 0x5D28F9AB, 0x69CF5132, 0x7A6DC945,
0x9D14C3B8, 0x8EB65BCF, 0xBA51F356, 0xA9F36B21, 0xD39EA264, 0xC03C3A13, 0xF4DB928A, 0xE7790AFD,
0x3FC5F181, 0x2C6769F6, 0x1880C16F, 0x0B225918, 0x714F905D, 0x62ED082A, 0x560AA0B3, 0x45A838C4,
0xA2D13239, 0xB173AA4E, 0x859402D7, 0x96369AA0, 0xEC5B53E5, 0xFFF9CB92, 0xCB1E630B, 0xD8BCFB7C,
0x7F8BE302, 0x6C297B75, 0x58CED3EC, 0x4B6C4B9B, 0x310182DE, 0x22A31AA9, 0x1644B230, 0x05E62A47,
0xE29F20BA, 0xF13DB8CD, 0xC5DA1054, 0xD6788823, 0xAC154166, 0xBFB7D911, 0x8B507188, 0x98F2E9FF,
0x404E1283, 0x53EC8AF4, 0x670B226D, 0x74A9BA1A, 0x0EC4735F, 0x1D66EB28, 0x298143B1, 0x3A23DBC6,
0xDD5AD13B, 0xCEF8494C, 0xFA1FE1D5, 0xE9BD79A2, 0x93D0B0E7, 0x80722890, 0xB4958009, 0xA737187E,
0xFF17C604, 0xECB55E73, 0xD852F6EA, 0xCBF06E9D, 0xB19DA7D8, 0xA23F3FAF, 0x96D89736, 0x857A0F41,
0x620305BC, 0x71A19DCB, 0x45463552, 0x56E4AD25, 0x2C896460, 0x3F2BFC17, 0x0BCC548E, 0x186ECCF9,
0xC0D23785, 0xD370AFF2, 0xE797076B, 0xF4359F1C, 0x8E585659, 0x9DFACE2E, 0xA91D66B7, 0xBABFFEC0,
0x5DC6F43D, 0x4E646C4A, 0x7A83C4D3, 0x69215CA4, 0x134C95E1, 0x00EE0D96, 0x3409A50F, 0x27AB3D78,
0x809C2506, 0x933EBD71, 0xA7D915E8, 0xB47B8D9F, 0xCE1644DA, 0xDDB4DCAD, 0xE9537434, 0xFAF1EC43,
0x1D88E6BE, 0x0E2A7EC9, 0x3ACDD650, 0x296F4E27, 0x53028762, 0x40A01F15, 0x7447B78C, 0x67E52FFB,
0xBF59D487, 0xACFB4CF0, 0x981CE469, 0x8BBE7C1E, 0xF1D3B55B, 0xE2712D2C, 0xD69685B5, 0xC5341DC2,
0x224D173F, 0x31EF8F48, 0x050827D1, 0x16AABFA6, 0x6CC776E3, 0x7F65EE94, 0x4B82460D, 0x5820DE7A,
0xFBC3FAF9, 0xE861628E, 0xDC86CA17, 0xCF245260, 0xB5499B25, 0xA6EB0352, 0x920CABCB, 0x81AE33BC,
0x66D73941, 0x7575A136, 0x419209AF, 0x523091D8, 0x285D589D, 0x3BFFC0EA, 0x0F186873, 0x1CBAF004,
0xC4060B78, 0xD7A4930F, 0xE3433B96, 0xF0E1A3E1, 0x8A8C6AA4, 0x992EF2D3, 0xADC95A4A, 0xBE6BC23D,
0x5912C8C0, 0x4AB050B7, 0x7E57F82E, 0x6DF56059, 0x1798A91C, 0x043A316B, 0x30DD99F2, 0x237F0185,
0x844819FB, 0x97EA818C, 0xA30D2915, 0xB0AFB162, 0xCAC27827, 0xD960E050, 0xED8748C9, 0xFE25D0BE,
0x195CDA43, 0x0AFE4234, 0x3E19EAAD, 0x2DBB72DA, 0x57D6BB9F, 0x447423E8, 0x70938B71, 0x63311306,
0xBB8DE87A, 0xA82F700D, 0x9CC8D894, 0x8F6A40E3, 0xF50789A6, 0xE6A511D1, 0xD242B948, 0xC1E0213F,
0x26992BC2, 0x353BB3B5, 0x01DC1B2C, 0x127E835B, 0x68134A1E, 0x7BB1D269, 0x4F567AF0, 0x5CF4E287,
0x04D43CFD, 0x1776A48A, 0x23910C13, 0x30339464, 0x4A5E5D21, 0x59FCC556, 0x6D1B6DCF, 0x7EB9F5B8,
0x99C0FF45, 0x8A626732, 0xBE85CFAB, 0xAD2757DC, 0xD74A9E99, 0xC4E806EE, 0xF00FAE77, 0xE3AD3600,
0x3B11CD7C, 0x28B3550B, 0x1C54FD92, 0x0FF665E5, 0x759BACA0, 0x663934D7, 0x52DE9C4E, 0x417C0439,
0xA6050EC4, 0xB5A796B3, 0x81403E2A, 0x92E2A65D, 0xE88F6F18, 0xFB2DF76F, 0xCFCA5FF6, 0xDC68C781,
0x7B5FDFFF, 0x68FD4788, 0x5C1AEF11, 0x4FB87766, 0x35D5BE23, 0x26772654, 0x12908ECD, 0x013216BA,
0xE64B1C47, 0xF5E98430, 0xC10E2CA9, 0xD2ACB4DE, 0xA8C17D9B, 0xBB63E5EC, 0x8F844D75, 0x9C26D502,
0x449A2E7E, 0x5738B609, 0x63DF1E90, 0x707D86E7, 0x0A104FA2, 0x19B2D7D5, 0x2D557F4C, 0x3EF7E73B,
0xD98EEDC6, 0xCA2C75B1, 0xFECBDD28, 0xED69455F, 0x97048C1A, 0x84A6146D, 0xB041BCF4, 0xA3E32483
};
/*
* end of the CRC lookup table crc_tableil8_o40
*/
/*
* The following CRC lookup table was generated automagically
* using the following model parameters:
*
* Generator Polynomial = ................. 0x1EDC6F41
* Generator Polynomial Length = .......... 32 bits
* Reflected Bits = ....................... TRUE
* Table Generation Offset = .............. 32 bits
* Number of Slices = ..................... 8 slices
* Slice Lengths = ........................ 8 8 8 8 8 8 8 8
* Directory Name = ....................... .\
* File Name = ............................ 8x256_tables.c
*/
static
u32 crc_tableil8_o48[256] =
{
0x00000000, 0xA541927E, 0x4F6F520D, 0xEA2EC073, 0x9EDEA41A, 0x3B9F3664, 0xD1B1F617, 0x74F06469,
0x38513EC5, 0x9D10ACBB, 0x773E6CC8, 0xD27FFEB6, 0xA68F9ADF, 0x03CE08A1, 0xE9E0C8D2, 0x4CA15AAC,
0x70A27D8A, 0xD5E3EFF4, 0x3FCD2F87, 0x9A8CBDF9, 0xEE7CD990, 0x4B3D4BEE, 0xA1138B9D, 0x045219E3,
0x48F3434F, 0xEDB2D131, 0x079C1142, 0xA2DD833C, 0xD62DE755, 0x736C752B, 0x9942B558, 0x3C032726,
0xE144FB14, 0x4405696A, 0xAE2BA919, 0x0B6A3B67, 0x7F9A5F0E, 0xDADBCD70, 0x30F50D03, 0x95B49F7D,
0xD915C5D1, 0x7C5457AF, 0x967A97DC, 0x333B05A2, 0x47CB61CB, 0xE28AF3B5, 0x08A433C6, 0xADE5A1B8,
0x91E6869E, 0x34A714E0, 0xDE89D493, 0x7BC846ED, 0x0F382284, 0xAA79B0FA, 0x40577089, 0xE516E2F7,
0xA9B7B85B, 0x0CF62A25, 0xE6D8EA56, 0x43997828, 0x37691C41, 0x92288E3F, 0x78064E4C, 0xDD47DC32,
0xC76580D9, 0x622412A7, 0x880AD2D4, 0x2D4B40AA, 0x59BB24C3, 0xFCFAB6BD, 0x16D476CE, 0xB395E4B0,
0xFF34BE1C, 0x5A752C62, 0xB05BEC11, 0x151A7E6F, 0x61EA1A06, 0xC4AB8878, 0x2E85480B, 0x8BC4DA75,
0xB7C7FD53, 0x12866F2D, 0xF8A8AF5E, 0x5DE93D20, 0x29195949, 0x8C58CB37, 0x66760B44, 0xC337993A,
0x8F96C396, 0x2AD751E8, 0xC0F9919B, 0x65B803E5, 0x1148678C, 0xB409F5F2, 0x5E273581, 0xFB66A7FF,
0x26217BCD, 0x8360E9B3, 0x694E29C0, 0xCC0FBBBE, 0xB8FFDFD7, 0x1DBE4DA9, 0xF7908DDA, 0x52D11FA4,
0x1E704508, 0xBB31D776, 0x511F1705, 0xF45E857B, 0x80AEE112, 0x25EF736C, 0xCFC1B31F, 0x6A802161,
0x56830647, 0xF3C29439, 0x19EC544A, 0xBCADC634, 0xC85DA25D, 0x6D1C3023, 0x8732F050, 0x2273622E,
0x6ED23882, 0xCB93AAFC, 0x21BD6A8F, 0x84FCF8F1, 0xF00C9C98, 0x554D0EE6, 0xBF63CE95, 0x1A225CEB,
0x8B277743, 0x2E66E53D, 0xC448254E, 0x6109B730, 0x15F9D359, 0xB0B84127, 0x5A968154, 0xFFD7132A,
0xB3764986, 0x1637DBF8, 0xFC191B8B, 0x595889F5, 0x2DA8ED9C, 0x88E97FE2, 0x62C7BF91, 0xC7862DEF,
0xFB850AC9, 0x5EC498B7, 0xB4EA58C4, 0x11ABCABA, 0x655BAED3, 0xC01A3CAD, 0x2A34FCDE, 0x8F756EA0,
0xC3D4340C, 0x6695A672, 0x8CBB6601, 0x29FAF47F, 0x5D0A9016, 0xF84B0268, 0x1265C21B, 0xB7245065,
0x6A638C57, 0xCF221E29, 0x250CDE5A, 0x804D4C24, 0xF4BD284D, 0x51FCBA33, 0xBBD27A40, 0x1E93E83E,
0x5232B292, 0xF77320EC, 0x1D5DE09F, 0xB81C72E1, 0xCCEC1688, 0x69AD84F6, 0x83834485, 0x26C2D6FB,
0x1AC1F1DD, 0xBF8063A3, 0x55AEA3D0, 0xF0EF31AE, 0x841F55C7, 0x215EC7B9, 0xCB7007CA, 0x6E3195B4,
0x2290CF18, 0x87D15D66, 0x6DFF9D15, 0xC8BE0F6B, 0xBC4E6B02, 0x190FF97C, 0xF321390F, 0x5660AB71,
0x4C42F79A, 0xE90365E4, 0x032DA597, 0xA66C37E9, 0xD29C5380, 0x77DDC1FE, 0x9DF3018D, 0x38B293F3,
0x7413C95F, 0xD1525B21, 0x3B7C9B52, 0x9E3D092C, 0xEACD6D45, 0x4F8CFF3B, 0xA5A23F48, 0x00E3AD36,
0x3CE08A10, 0x99A1186E, 0x738FD81D, 0xD6CE4A63, 0xA23E2E0A, 0x077FBC74, 0xED517C07, 0x4810EE79,
0x04B1B4D5, 0xA1F026AB, 0x4BDEE6D8, 0xEE9F74A6, 0x9A6F10CF, 0x3F2E82B1, 0xD50042C2, 0x7041D0BC,
0xAD060C8E, 0x08479EF0, 0xE2695E83, 0x4728CCFD, 0x33D8A894, 0x96993AEA, 0x7CB7FA99, 0xD9F668E7,
0x9557324B, 0x3016A035, 0xDA386046, 0x7F79F238, 0x0B899651, 0xAEC8042F, 0x44E6C45C, 0xE1A75622,
0xDDA47104, 0x78E5E37A, 0x92CB2309, 0x378AB177, 0x437AD51E, 0xE63B4760, 0x0C158713, 0xA954156D,
0xE5F54FC1, 0x40B4DDBF, 0xAA9A1DCC, 0x0FDB8FB2, 0x7B2BEBDB, 0xDE6A79A5, 0x3444B9D6, 0x91052BA8
};
/*
* end of the CRC lookup table crc_tableil8_o48
*/
/*
* The following CRC lookup table was generated automagically
* using the following model parameters:
*
* Generator Polynomial = ................. 0x1EDC6F41
* Generator Polynomial Length = .......... 32 bits
* Reflected Bits = ....................... TRUE
* Table Generation Offset = .............. 32 bits
* Number of Slices = ..................... 8 slices
* Slice Lengths = ........................ 8 8 8 8 8 8 8 8
* Directory Name = ....................... .\
* File Name = ............................ 8x256_tables.c
*/
static
u32 crc_tableil8_o56[256] =
{
0x00000000, 0xDD45AAB8, 0xBF672381, 0x62228939, 0x7B2231F3, 0xA6679B4B, 0xC4451272, 0x1900B8CA,
0xF64463E6, 0x2B01C95E, 0x49234067, 0x9466EADF, 0x8D665215, 0x5023F8AD, 0x32017194, 0xEF44DB2C,
0xE964B13D, 0x34211B85, 0x560392BC, 0x8B463804, 0x924680CE, 0x4F032A76, 0x2D21A34F, 0xF06409F7,
0x1F20D2DB, 0xC2657863, 0xA047F15A, 0x7D025BE2, 0x6402E328, 0xB9474990, 0xDB65C0A9, 0x06206A11,
0xD725148B, 0x0A60BE33, 0x6842370A, 0xB5079DB2, 0xAC072578, 0x71428FC0, 0x136006F9, 0xCE25AC41,
0x2161776D, 0xFC24DDD5, 0x9E0654EC, 0x4343FE54, 0x5A43469E, 0x8706EC26, 0xE524651F, 0x3861CFA7,
0x3E41A5B6, 0xE3040F0E, 0x81268637, 0x5C632C8F, 0x45639445, 0x98263EFD, 0xFA04B7C4, 0x27411D7C,
0xC805C650, 0x15406CE8, 0x7762E5D1, 0xAA274F69, 0xB327F7A3, 0x6E625D1B, 0x0C40D422, 0xD1057E9A,
0xABA65FE7, 0x76E3F55F, 0x14C17C66, 0xC984D6DE, 0xD0846E14, 0x0DC1C4AC, 0x6FE34D95, 0xB2A6E72D,
0x5DE23C01, 0x80A796B9, 0xE2851F80, 0x3FC0B538, 0x26C00DF2, 0xFB85A74A, 0x99A72E73, 0x44E284CB,
0x42C2EEDA, 0x9F874462, 0xFDA5CD5B, 0x20E067E3, 0x39E0DF29, 0xE4A57591, 0x8687FCA8, 0x5BC25610,
0xB4868D3C, 0x69C32784, 0x0BE1AEBD, 0xD6A40405, 0xCFA4BCCF, 0x12E11677, 0x70C39F4E, 0xAD8635F6,
0x7C834B6C, 0xA1C6E1D4, 0xC3E468ED, 0x1EA1C255, 0x07A17A9F, 0xDAE4D027, 0xB8C6591E, 0x6583F3A6,
0x8AC7288A, 0x57828232, 0x35A00B0B, 0xE8E5A1B3, 0xF1E51979, 0x2CA0B3C1, 0x4E823AF8, 0x93C79040,
0x95E7FA51, 0x48A250E9, 0x2A80D9D0, 0xF7C57368, 0xEEC5CBA2, 0x3380611A, 0x51A2E823, 0x8CE7429B,
0x63A399B7, 0xBEE6330F, 0xDCC4BA36, 0x0181108E, 0x1881A844, 0xC5C402FC, 0xA7E68BC5, 0x7AA3217D,
0x52A0C93F, 0x8FE56387, 0xEDC7EABE, 0x30824006, 0x2982F8CC, 0xF4C75274, 0x96E5DB4D, 0x4BA071F5,
0xA4E4AAD9, 0x79A10061, 0x1B838958, 0xC6C623E0, 0xDFC69B2A, 0x02833192, 0x60A1B8AB, 0xBDE41213,
0xBBC47802, 0x6681D2BA, 0x04A35B83, 0xD9E6F13B, 0xC0E649F1, 0x1DA3E349, 0x7F816A70, 0xA2C4C0C8,
0x4D801BE4, 0x90C5B15C, 0xF2E73865, 0x2FA292DD, 0x36A22A17, 0xEBE780AF, 0x89C50996, 0x5480A32E,
0x8585DDB4, 0x58C0770C, 0x3AE2FE35, 0xE7A7548D, 0xFEA7EC47, 0x23E246FF, 0x41C0CFC6, 0x9C85657E,
0x73C1BE52, 0xAE8414EA, 0xCCA69DD3, 0x11E3376B, 0x08E38FA1, 0xD5A62519, 0xB784AC20, 0x6AC10698,
0x6CE16C89, 0xB1A4C631, 0xD3864F08, 0x0EC3E5B0, 0x17C35D7A, 0xCA86F7C2, 0xA8A47EFB, 0x75E1D443,
0x9AA50F6F, 0x47E0A5D7, 0x25C22CEE, 0xF8878656, 0xE1873E9C, 0x3CC29424, 0x5EE01D1D, 0x83A5B7A5,
0xF90696D8, 0x24433C60, 0x4661B559, 0x9B241FE1, 0x8224A72B, 0x5F610D93, 0x3D4384AA, 0xE0062E12,
0x0F42F53E, 0xD2075F86, 0xB025D6BF, 0x6D607C07, 0x7460C4CD, 0xA9256E75, 0xCB07E74C, 0x16424DF4,
0x106227E5, 0xCD278D5D, 0xAF050464, 0x7240AEDC, 0x6B401616, 0xB605BCAE, 0xD4273597, 0x09629F2F,
0xE6264403, 0x3B63EEBB, 0x59416782, 0x8404CD3A, 0x9D0475F0, 0x4041DF48, 0x22635671, 0xFF26FCC9,
0x2E238253, 0xF36628EB, 0x9144A1D2, 0x4C010B6A, 0x5501B3A0, 0x88441918, 0xEA669021, 0x37233A99,
0xD867E1B5, 0x05224B0D, 0x6700C234, 0xBA45688C, 0xA345D046, 0x7E007AFE, 0x1C22F3C7, 0xC167597F,
0xC747336E, 0x1A0299D6, 0x782010EF, 0xA565BA57, 0xBC65029D, 0x6120A825, 0x0302211C, 0xDE478BA4,
0x31035088, 0xEC46FA30, 0x8E647309, 0x5321D9B1, 0x4A21617B, 0x9764CBC3, 0xF54642FA, 0x2803E842
};
/*
* end of the CRC lookup table crc_tableil8_o56
*/
/*
* The following CRC lookup table was generated automagically
* using the following model parameters:
*
* Generator Polynomial = ................. 0x1EDC6F41
* Generator Polynomial Length = .......... 32 bits
* Reflected Bits = ....................... TRUE
* Table Generation Offset = .............. 32 bits
* Number of Slices = ..................... 8 slices
* Slice Lengths = ........................ 8 8 8 8 8 8 8 8
* Directory Name = ....................... .\
* File Name = ............................ 8x256_tables.c
*/
static
u32 crc_tableil8_o64[256] =
{
0x00000000, 0x38116FAC, 0x7022DF58, 0x4833B0F4, 0xE045BEB0, 0xD854D11C, 0x906761E8, 0xA8760E44,
0xC5670B91, 0xFD76643D, 0xB545D4C9, 0x8D54BB65, 0x2522B521, 0x1D33DA8D, 0x55006A79, 0x6D1105D5,
0x8F2261D3, 0xB7330E7F, 0xFF00BE8B, 0xC711D127, 0x6F67DF63, 0x5776B0CF, 0x1F45003B, 0x27546F97,
0x4A456A42, 0x725405EE, 0x3A67B51A, 0x0276DAB6, 0xAA00D4F2, 0x9211BB5E, 0xDA220BAA, 0xE2336406,
0x1BA8B557, 0x23B9DAFB, 0x6B8A6A0F, 0x539B05A3, 0xFBED0BE7, 0xC3FC644B, 0x8BCFD4BF, 0xB3DEBB13,
0xDECFBEC6, 0xE6DED16A, 0xAEED619E, 0x96FC0E32, 0x3E8A0076, 0x069B6FDA, 0x4EA8DF2E, 0x76B9B082,
0x948AD484, 0xAC9BBB28, 0xE4A80BDC, 0xDCB96470, 0x74CF6A34, 0x4CDE0598, 0x04EDB56C, 0x3CFCDAC0,
0x51EDDF15, 0x69FCB0B9, 0x21CF004D, 0x19DE6FE1, 0xB1A861A5, 0x89B90E09, 0xC18ABEFD, 0xF99BD151,
0x37516AAE, 0x0F400502, 0x4773B5F6, 0x7F62DA5A, 0xD714D41E, 0xEF05BBB2, 0xA7360B46, 0x9F2764EA,
0xF236613F, 0xCA270E93, 0x8214BE67, 0xBA05D1CB, 0x1273DF8F, 0x2A62B023, 0x625100D7, 0x5A406F7B,
0xB8730B7D, 0x806264D1, 0xC851D425, 0xF040BB89, 0x5836B5CD, 0x6027DA61, 0x28146A95, 0x10050539,
0x7D1400EC, 0x45056F40, 0x0D36DFB4, 0x3527B018, 0x9D51BE5C, 0xA540D1F0, 0xED736104, 0xD5620EA8,
0x2CF9DFF9, 0x14E8B055, 0x5CDB00A1, 0x64CA6F0D, 0xCCBC6149, 0xF4AD0EE5, 0xBC9EBE11, 0x848FD1BD,
0xE99ED468, 0xD18FBBC4, 0x99BC0B30, 0xA1AD649C, 0x09DB6AD8, 0x31CA0574, 0x79F9B580, 0x41E8DA2C,
0xA3DBBE2A, 0x9BCAD186, 0xD3F96172, 0xEBE80EDE, 0x439E009A, 0x7B8F6F36, 0x33BCDFC2, 0x0BADB06E,
0x66BCB5BB, 0x5EADDA17, 0x169E6AE3, 0x2E8F054F, 0x86F90B0B, 0xBEE864A7, 0xF6DBD453, 0xCECABBFF,
0x6EA2D55C, 0x56B3BAF0, 0x1E800A04, 0x269165A8, 0x8EE76BEC, 0xB6F60440, 0xFEC5B4B4, 0xC6D4DB18,
0xABC5DECD, 0x93D4B161, 0xDBE70195, 0xE3F66E39, 0x4B80607D, 0x73910FD1, 0x3BA2BF25, 0x03B3D089,
0xE180B48F, 0xD991DB23, 0x91A26BD7, 0xA9B3047B, 0x01C50A3F, 0x39D46593, 0x71E7D567, 0x49F6BACB,
0x24E7BF1E, 0x1CF6D0B2, 0x54C56046, 0x6CD40FEA, 0xC4A201AE, 0xFCB36E02, 0xB480DEF6, 0x8C91B15A,
0x750A600B, 0x4D1B0FA7, 0x0528BF53, 0x3D39D0FF, 0x954FDEBB, 0xAD5EB117, 0xE56D01E3, 0xDD7C6E4F,
0xB06D6B9A, 0x887C0436, 0xC04FB4C2, 0xF85EDB6E, 0x5028D52A, 0x6839BA86, 0x200A0A72, 0x181B65DE,
0xFA2801D8, 0xC2396E74, 0x8A0ADE80, 0xB21BB12C, 0x1A6DBF68, 0x227CD0C4, 0x6A4F6030, 0x525E0F9C,
0x3F4F0A49, 0x075E65E5, 0x4F6DD511, 0x777CBABD, 0xDF0AB4F9, 0xE71BDB55, 0xAF286BA1, 0x9739040D,
0x59F3BFF2, 0x61E2D05E, 0x29D160AA, 0x11C00F06, 0xB9B60142, 0x81A76EEE, 0xC994DE1A, 0xF185B1B6,
0x9C94B463, 0xA485DBCF, 0xECB66B3B, 0xD4A70497, 0x7CD10AD3, 0x44C0657F, 0x0CF3D58B, 0x34E2BA27,
0xD6D1DE21, 0xEEC0B18D, 0xA6F30179, 0x9EE26ED5, 0x36946091, 0x0E850F3D, 0x46B6BFC9, 0x7EA7D065,
0x13B6D5B0, 0x2BA7BA1C, 0x63940AE8, 0x5B856544, 0xF3F36B00, 0xCBE204AC, 0x83D1B458, 0xBBC0DBF4,
0x425B0AA5, 0x7A4A6509, 0x3279D5FD, 0x0A68BA51, 0xA21EB415, 0x9A0FDBB9, 0xD23C6B4D, 0xEA2D04E1,
0x873C0134, 0xBF2D6E98, 0xF71EDE6C, 0xCF0FB1C0, 0x6779BF84, 0x5F68D028, 0x175B60DC, 0x2F4A0F70,
0xCD796B76, 0xF56804DA, 0xBD5BB42E, 0x854ADB82, 0x2D3CD5C6, 0x152DBA6A, 0x5D1E0A9E, 0x650F6532,
0x081E60E7, 0x300F0F4B, 0x783CBFBF, 0x402DD013, 0xE85BDE57, 0xD04AB1FB, 0x9879010F, 0xA0686EA3
};
/*
* end of the CRC lookup table crc_tableil8_o64
*/
/*
* The following CRC lookup table was generated automagically
* using the following model parameters:
*
* Generator Polynomial = ................. 0x1EDC6F41
* Generator Polynomial Length = .......... 32 bits
* Reflected Bits = ....................... TRUE
* Table Generation Offset = .............. 32 bits
* Number of Slices = ..................... 8 slices
* Slice Lengths = ........................ 8 8 8 8 8 8 8 8
* Directory Name = ....................... .\
* File Name = ............................ 8x256_tables.c
*/
static
u32 crc_tableil8_o72[256] =
{
0x00000000, 0xEF306B19, 0xDB8CA0C3, 0x34BCCBDA, 0xB2F53777, 0x5DC55C6E, 0x697997B4, 0x8649FCAD,
0x6006181F, 0x8F367306, 0xBB8AB8DC, 0x54BAD3C5, 0xD2F32F68, 0x3DC34471, 0x097F8FAB, 0xE64FE4B2,
0xC00C303E, 0x2F3C5B27, 0x1B8090FD, 0xF4B0FBE4, 0x72F90749, 0x9DC96C50, 0xA975A78A, 0x4645CC93,
0xA00A2821, 0x4F3A4338, 0x7B8688E2, 0x94B6E3FB, 0x12FF1F56, 0xFDCF744F, 0xC973BF95, 0x2643D48C,
0x85F4168D, 0x6AC47D94, 0x5E78B64E, 0xB148DD57, 0x370121FA, 0xD8314AE3, 0xEC8D8139, 0x03BDEA20,
0xE5F20E92, 0x0AC2658B, 0x3E7EAE51, 0xD14EC548, 0x570739E5, 0xB83752FC, 0x8C8B9926, 0x63BBF23F,
0x45F826B3, 0xAAC84DAA, 0x9E748670, 0x7144ED69, 0xF70D11C4, 0x183D7ADD, 0x2C81B107, 0xC3B1DA1E,
0x25FE3EAC, 0xCACE55B5, 0xFE729E6F, 0x1142F576, 0x970B09DB, 0x783B62C2, 0x4C87A918, 0xA3B7C201,
0x0E045BEB, 0xE13430F2, 0xD588FB28, 0x3AB89031, 0xBCF16C9C, 0x53C10785, 0x677DCC5F, 0x884DA746,
0x6E0243F4, 0x813228ED, 0xB58EE337, 0x5ABE882E, 0xDCF77483, 0x33C71F9A, 0x077BD440, 0xE84BBF59,
0xCE086BD5, 0x213800CC, 0x1584CB16, 0xFAB4A00F, 0x7CFD5CA2, 0x93CD37BB, 0xA771FC61, 0x48419778,
0xAE0E73CA, 0x413E18D3, 0x7582D309, 0x9AB2B810, 0x1CFB44BD, 0xF3CB2FA4, 0xC777E47E, 0x28478F67,
0x8BF04D66, 0x64C0267F, 0x507CEDA5, 0xBF4C86BC, 0x39057A11, 0xD6351108, 0xE289DAD2, 0x0DB9B1CB,
0xEBF65579, 0x04C63E60, 0x307AF5BA, 0xDF4A9EA3, 0x5903620E, 0xB6330917, 0x828FC2CD, 0x6DBFA9D4,
0x4BFC7D58, 0xA4CC1641, 0x9070DD9B, 0x7F40B682, 0xF9094A2F, 0x16392136, 0x2285EAEC, 0xCDB581F5,
0x2BFA6547, 0xC4CA0E5E, 0xF076C584, 0x1F46AE9D, 0x990F5230, 0x763F3929, 0x4283F2F3, 0xADB399EA,
0x1C08B7D6, 0xF338DCCF, 0xC7841715, 0x28B47C0C, 0xAEFD80A1, 0x41CDEBB8, 0x75712062, 0x9A414B7B,
0x7C0EAFC9, 0x933EC4D0, 0xA7820F0A, 0x48B26413, 0xCEFB98BE, 0x21CBF3A7, 0x1577387D, 0xFA475364,
0xDC0487E8, 0x3334ECF1, 0x0788272B, 0xE8B84C32, 0x6EF1B09F, 0x81C1DB86, 0xB57D105C, 0x5A4D7B45,
0xBC029FF7, 0x5332F4EE, 0x678E3F34, 0x88BE542D, 0x0EF7A880, 0xE1C7C399, 0xD57B0843, 0x3A4B635A,
0x99FCA15B, 0x76CCCA42, 0x42700198, 0xAD406A81, 0x2B09962C, 0xC439FD35, 0xF08536EF, 0x1FB55DF6,
0xF9FAB944, 0x16CAD25D, 0x22761987, 0xCD46729E, 0x4B0F8E33, 0xA43FE52A, 0x90832EF0, 0x7FB345E9,
0x59F09165, 0xB6C0FA7C, 0x827C31A6, 0x6D4C5ABF, 0xEB05A612, 0x0435CD0B, 0x308906D1, 0xDFB96DC8,
0x39F6897A, 0xD6C6E263, 0xE27A29B9, 0x0D4A42A0, 0x8B03BE0D, 0x6433D514, 0x508F1ECE, 0xBFBF75D7,
0x120CEC3D, 0xFD3C8724, 0xC9804CFE, 0x26B027E7, 0xA0F9DB4A, 0x4FC9B053, 0x7B757B89, 0x94451090,
0x720AF422, 0x9D3A9F3B, 0xA98654E1, 0x46B63FF8, 0xC0FFC355, 0x2FCFA84C, 0x1B736396, 0xF443088F,
0xD200DC03, 0x3D30B71A, 0x098C7CC0, 0xE6BC17D9, 0x60F5EB74, 0x8FC5806D, 0xBB794BB7, 0x544920AE,
0xB206C41C, 0x5D36AF05, 0x698A64DF, 0x86BA0FC6, 0x00F3F36B, 0xEFC39872, 0xDB7F53A8, 0x344F38B1,
0x97F8FAB0, 0x78C891A9, 0x4C745A73, 0xA344316A, 0x250DCDC7, 0xCA3DA6DE, 0xFE816D04, 0x11B1061D,
0xF7FEE2AF, 0x18CE89B6, 0x2C72426C, 0xC3422975, 0x450BD5D8, 0xAA3BBEC1, 0x9E87751B, 0x71B71E02,
0x57F4CA8E, 0xB8C4A197, 0x8C786A4D, 0x63480154, 0xE501FDF9, 0x0A3196E0, 0x3E8D5D3A, 0xD1BD3623,
0x37F2D291, 0xD8C2B988, 0xEC7E7252, 0x034E194B, 0x8507E5E6, 0x6A378EFF, 0x5E8B4525, 0xB1BB2E3C
};
/*
* end of the CRC lookup table crc_tableil8_o72
*/
/*
* The following CRC lookup table was generated automagically
* using the following model parameters:
*
* Generator Polynomial = ................. 0x1EDC6F41
* Generator Polynomial Length = .......... 32 bits
* Reflected Bits = ....................... TRUE
* Table Generation Offset = .............. 32 bits
* Number of Slices = ..................... 8 slices
* Slice Lengths = ........................ 8 8 8 8 8 8 8 8
* Directory Name = ....................... .\
* File Name = ............................ 8x256_tables.c
*/
static
u32 crc_tableil8_o80[256] =
{
0x00000000, 0x68032CC8, 0xD0065990, 0xB8057558, 0xA5E0C5D1, 0xCDE3E919, 0x75E69C41, 0x1DE5B089,
0x4E2DFD53, 0x262ED19B, 0x9E2BA4C3, 0xF628880B, 0xEBCD3882, 0x83CE144A, 0x3BCB6112, 0x53C84DDA,
0x9C5BFAA6, 0xF458D66E, 0x4C5DA336, 0x245E8FFE, 0x39BB3F77, 0x51B813BF, 0xE9BD66E7, 0x81BE4A2F,
0xD27607F5, 0xBA752B3D, 0x02705E65, 0x6A7372AD, 0x7796C224, 0x1F95EEEC, 0xA7909BB4, 0xCF93B77C,
0x3D5B83BD, 0x5558AF75, 0xED5DDA2D, 0x855EF6E5, 0x98BB466C, 0xF0B86AA4, 0x48BD1FFC, 0x20BE3334,
0x73767EEE, 0x1B755226, 0xA370277E, 0xCB730BB6, 0xD696BB3F, 0xBE9597F7, 0x0690E2AF, 0x6E93CE67,
0xA100791B, 0xC90355D3, 0x7106208B, 0x19050C43, 0x04E0BCCA, 0x6CE39002, 0xD4E6E55A, 0xBCE5C992,
0xEF2D8448, 0x872EA880, 0x3F2BDDD8, 0x5728F110, 0x4ACD4199, 0x22CE6D51, 0x9ACB1809, 0xF2C834C1,
0x7AB7077A, 0x12B42BB2, 0xAAB15EEA, 0xC2B27222, 0xDF57C2AB, 0xB754EE63, 0x0F519B3B, 0x6752B7F3,
0x349AFA29, 0x5C99D6E1, 0xE49CA3B9, 0x8C9F8F71, 0x917A3FF8, 0xF9791330, 0x417C6668, 0x297F4AA0,
0xE6ECFDDC, 0x8EEFD114, 0x36EAA44C, 0x5EE98884, 0x430C380D, 0x2B0F14C5, 0x930A619D, 0xFB094D55,
0xA8C1008F, 0xC0C22C47, 0x78C7591F, 0x10C475D7, 0x0D21C55E, 0x6522E996, 0xDD279CCE, 0xB524B006,
0x47EC84C7, 0x2FEFA80F, 0x97EADD57, 0xFFE9F19F, 0xE20C4116, 0x8A0F6DDE, 0x320A1886, 0x5A09344E,
0x09C17994, 0x61C2555C, 0xD9C72004, 0xB1C40CCC, 0xAC21BC45, 0xC422908D, 0x7C27E5D5, 0x1424C91D,
0xDBB77E61, 0xB3B452A9, 0x0BB127F1, 0x63B20B39, 0x7E57BBB0, 0x16549778, 0xAE51E220, 0xC652CEE8,
0x959A8332, 0xFD99AFFA, 0x459CDAA2, 0x2D9FF66A, 0x307A46E3, 0x58796A2B, 0xE07C1F73, 0x887F33BB,
0xF56E0EF4, 0x9D6D223C, 0x25685764, 0x4D6B7BAC, 0x508ECB25, 0x388DE7ED, 0x808892B5, 0xE88BBE7D,
0xBB43F3A7, 0xD340DF6F, 0x6B45AA37, 0x034686FF, 0x1EA33676, 0x76A01ABE, 0xCEA56FE6, 0xA6A6432E,
0x6935F452, 0x0136D89A, 0xB933ADC2, 0xD130810A, 0xCCD53183, 0xA4D61D4B, 0x1CD36813, 0x74D044DB,
0x27180901, 0x4F1B25C9, 0xF71E5091, 0x9F1D7C59, 0x82F8CCD0, 0xEAFBE018, 0x52FE9540, 0x3AFDB988,
0xC8358D49, 0xA036A181, 0x1833D4D9, 0x7030F811, 0x6DD54898, 0x05D66450, 0xBDD31108, 0xD5D03DC0,
0x8618701A, 0xEE1B5CD2, 0x561E298A, 0x3E1D0542, 0x23F8B5CB, 0x4BFB9903, 0xF3FEEC5B, 0x9BFDC093,
0x546E77EF, 0x3C6D5B27, 0x84682E7F, 0xEC6B02B7, 0xF18EB23E, 0x998D9EF6, 0x2188EBAE, 0x498BC766,
0x1A438ABC, 0x7240A674, 0xCA45D32C, 0xA246FFE4, 0xBFA34F6D, 0xD7A063A5, 0x6FA516FD, 0x07A63A35,
0x8FD9098E, 0xE7DA2546, 0x5FDF501E, 0x37DC7CD6, 0x2A39CC5F, 0x423AE097, 0xFA3F95CF, 0x923CB907,
0xC1F4F4DD, 0xA9F7D815, 0x11F2AD4D, 0x79F18185, 0x6414310C, 0x0C171DC4, 0xB412689C, 0xDC114454,
0x1382F328, 0x7B81DFE0, 0xC384AAB8, 0xAB878670, 0xB66236F9, 0xDE611A31, 0x66646F69, 0x0E6743A1,
0x5DAF0E7B, 0x35AC22B3, 0x8DA957EB, 0xE5AA7B23, 0xF84FCBAA, 0x904CE762, 0x2849923A, 0x404ABEF2,
0xB2828A33, 0xDA81A6FB, 0x6284D3A3, 0x0A87FF6B, 0x17624FE2, 0x7F61632A, 0xC7641672, 0xAF673ABA,
0xFCAF7760, 0x94AC5BA8, 0x2CA92EF0, 0x44AA0238, 0x594FB2B1, 0x314C9E79, 0x8949EB21, 0xE14AC7E9,
0x2ED97095, 0x46DA5C5D, 0xFEDF2905, 0x96DC05CD, 0x8B39B544, 0xE33A998C, 0x5B3FECD4, 0x333CC01C,
0x60F48DC6, 0x08F7A10E, 0xB0F2D456, 0xD8F1F89E, 0xC5144817, 0xAD1764DF, 0x15121187, 0x7D113D4F
};
/*
* end of the CRC lookup table crc_tableil8_o80
*/
/*
* The following CRC lookup table was generated automagically
* using the following model parameters:
*
* Generator Polynomial = ................. 0x1EDC6F41
* Generator Polynomial Length = .......... 32 bits
* Reflected Bits = ....................... TRUE
* Table Generation Offset = .............. 32 bits
* Number of Slices = ..................... 8 slices
* Slice Lengths = ........................ 8 8 8 8 8 8 8 8
* Directory Name = ....................... .\
* File Name = ............................ 8x256_tables.c
*/
static
u32 crc_tableil8_o88[256] =
{
0x00000000, 0x493C7D27, 0x9278FA4E, 0xDB448769, 0x211D826D, 0x6821FF4A, 0xB3657823, 0xFA590504,
0x423B04DA, 0x0B0779FD, 0xD043FE94, 0x997F83B3, 0x632686B7, 0x2A1AFB90, 0xF15E7CF9, 0xB86201DE,
0x847609B4, 0xCD4A7493, 0x160EF3FA, 0x5F328EDD, 0xA56B8BD9, 0xEC57F6FE, 0x37137197, 0x7E2F0CB0,
0xC64D0D6E, 0x8F717049, 0x5435F720, 0x1D098A07, 0xE7508F03, 0xAE6CF224, 0x7528754D, 0x3C14086A,
0x0D006599, 0x443C18BE, 0x9F789FD7, 0xD644E2F0, 0x2C1DE7F4, 0x65219AD3, 0xBE651DBA, 0xF759609D,
0x4F3B6143, 0x06071C64, 0xDD439B0D, 0x947FE62A, 0x6E26E32E, 0x271A9E09, 0xFC5E1960, 0xB5626447,
0x89766C2D, 0xC04A110A, 0x1B0E9663, 0x5232EB44, 0xA86BEE40, 0xE1579367, 0x3A13140E, 0x732F6929,
0xCB4D68F7, 0x827115D0, 0x593592B9, 0x1009EF9E, 0xEA50EA9A, 0xA36C97BD, 0x782810D4, 0x31146DF3,
0x1A00CB32, 0x533CB615, 0x8878317C, 0xC1444C5B, 0x3B1D495F, 0x72213478, 0xA965B311, 0xE059CE36,
0x583BCFE8, 0x1107B2CF, 0xCA4335A6, 0x837F4881, 0x79264D85, 0x301A30A2, 0xEB5EB7CB, 0xA262CAEC,
0x9E76C286, 0xD74ABFA1, 0x0C0E38C8, 0x453245EF, 0xBF6B40EB, 0xF6573DCC, 0x2D13BAA5, 0x642FC782,
0xDC4DC65C, 0x9571BB7B, 0x4E353C12, 0x07094135, 0xFD504431, 0xB46C3916, 0x6F28BE7F, 0x2614C358,
0x1700AEAB, 0x5E3CD38C, 0x857854E5, 0xCC4429C2, 0x361D2CC6, 0x7F2151E1, 0xA465D688, 0xED59ABAF,
0x553BAA71, 0x1C07D756, 0xC743503F, 0x8E7F2D18, 0x7426281C, 0x3D1A553B, 0xE65ED252, 0xAF62AF75,
0x9376A71F, 0xDA4ADA38, 0x010E5D51, 0x48322076, 0xB26B2572, 0xFB575855, 0x2013DF3C, 0x692FA21B,
0xD14DA3C5, 0x9871DEE2, 0x4335598B, 0x0A0924AC, 0xF05021A8, 0xB96C5C8F, 0x6228DBE6, 0x2B14A6C1,
0x34019664, 0x7D3DEB43, 0xA6796C2A, 0xEF45110D, 0x151C1409, 0x5C20692E, 0x8764EE47, 0xCE589360,
0x763A92BE, 0x3F06EF99, 0xE44268F0, 0xAD7E15D7, 0x572710D3, 0x1E1B6DF4, 0xC55FEA9D, 0x8C6397BA,
0xB0779FD0, 0xF94BE2F7, 0x220F659E, 0x6B3318B9, 0x916A1DBD, 0xD856609A, 0x0312E7F3, 0x4A2E9AD4,
0xF24C9B0A, 0xBB70E62D, 0x60346144, 0x29081C63, 0xD3511967, 0x9A6D6440, 0x4129E329, 0x08159E0E,
0x3901F3FD, 0x703D8EDA, 0xAB7909B3, 0xE2457494, 0x181C7190, 0x51200CB7, 0x8A648BDE, 0xC358F6F9,
0x7B3AF727, 0x32068A00, 0xE9420D69, 0xA07E704E, 0x5A27754A, 0x131B086D, 0xC85F8F04, 0x8163F223,
0xBD77FA49, 0xF44B876E, 0x2F0F0007, 0x66337D20, 0x9C6A7824, 0xD5560503, 0x0E12826A, 0x472EFF4D,
0xFF4CFE93, 0xB67083B4, 0x6D3404DD, 0x240879FA, 0xDE517CFE, 0x976D01D9, 0x4C2986B0, 0x0515FB97,
0x2E015D56, 0x673D2071, 0xBC79A718, 0xF545DA3F, 0x0F1CDF3B, 0x4620A21C, 0x9D642575, 0xD4585852,
0x6C3A598C, 0x250624AB, 0xFE42A3C2, 0xB77EDEE5, 0x4D27DBE1, 0x041BA6C6, 0xDF5F21AF, 0x96635C88,
0xAA7754E2, 0xE34B29C5, 0x380FAEAC, 0x7133D38B, 0x8B6AD68F, 0xC256ABA8, 0x19122CC1, 0x502E51E6,
0xE84C5038, 0xA1702D1F, 0x7A34AA76, 0x3308D751, 0xC951D255, 0x806DAF72, 0x5B29281B, 0x1215553C,
0x230138CF, 0x6A3D45E8, 0xB179C281, 0xF845BFA6, 0x021CBAA2, 0x4B20C785, 0x906440EC, 0xD9583DCB,
0x613A3C15, 0x28064132, 0xF342C65B, 0xBA7EBB7C, 0x4027BE78, 0x091BC35F, 0xD25F4436, 0x9B633911,
0xA777317B, 0xEE4B4C5C, 0x350FCB35, 0x7C33B612, 0x866AB316, 0xCF56CE31, 0x14124958, 0x5D2E347F,
0xE54C35A1, 0xAC704886, 0x7734CFEF, 0x3E08B2C8, 0xC451B7CC, 0x8D6DCAEB, 0x56294D82, 0x1F1530A5
};
/*
* end of the CRC lookup table crc_tableil8_o88
*/
//#define VERIFY_ASSERTION
#ifdef VERIFY_ASSERTION
// Trivial byte-by-byte version: you can switch on the assertion in the
// Crc32_ComputeBuf function (by defining VERIFY_ASSERTION) to check this
// against the slicing variant.
static really_inline
u32 crc32c(u32 running_crc, const unsigned char* p_buf, size_t length) {
u32 crc = running_crc;
while (length--) {
crc = crc_tableil8_o32[(crc ^ *p_buf++) & 0x000000FF] ^ (crc >> 8);
}
return crc;
}
#endif // VERIFY_ASSERTION
// Slicing-by-8 approach, which is much faster. Derived from Intel's
// BSD-licensed code, with additions to handled aligned case automatically.
static really_inline
u32 crc32c_sb8_64_bit(u32 running_crc, const unsigned char* p_buf,
const size_t length) {
u32 crc = running_crc;
// Process byte-by-byte until p_buf is aligned
const unsigned char *aligned_buf = ROUNDUP_PTR(p_buf, 4);
size_t init_bytes = aligned_buf - p_buf;
size_t running_length = ((length - init_bytes)/8)*8;
size_t end_bytes = length - init_bytes - running_length;
while (p_buf < aligned_buf) {
crc = crc_tableil8_o32[(crc ^ *p_buf++) & 0x000000FF] ^ (crc >> 8);
}
// Main aligned loop, processes eight bytes at a time.
u32 term1, term2;
for (size_t li = 0; li < running_length/8; li++) {
u32 block = *(const u32 *)p_buf;
crc ^= block;
p_buf += 4;
term1 = crc_tableil8_o88[crc & 0x000000FF] ^
crc_tableil8_o80[(crc >> 8) & 0x000000FF];
term2 = crc >> 16;
crc = term1 ^
crc_tableil8_o72[term2 & 0x000000FF] ^
crc_tableil8_o64[(term2 >> 8) & 0x000000FF];
block = *(const u32 *)p_buf;
term1 = crc_tableil8_o56[block & 0x000000FF] ^
crc_tableil8_o48[(block >> 8) & 0x000000FF];
term2 = block >> 16;
crc = crc ^
term1 ^
crc_tableil8_o40[term2 & 0x000000FF] ^
crc_tableil8_o32[(term2 >> 8) & 0x000000FF];
p_buf += 4;
}
// Remaining bytes
for(size_t li = 0; li < end_bytes; li++) {
crc = crc_tableil8_o32[(crc ^ *p_buf++) & 0x000000FF] ^ (crc >> 8);
}
return crc;
}
#else // HAVE_SSE42
#include "util/arch/x86/crc32.h"
#endif
#ifdef VERIFY_ASSERTION
#include <assert.h>
#endif
// Externally visible function
u32 Crc32c_ComputeBuf(u32 inCrc32, const void *buf, size_t bufLen) {
#if defined(HAVE_SSE42)
u32 crc = crc32c_sse42(inCrc32, (const unsigned char *)buf, bufLen);
#else
u32 crc = crc32c_sb8_64_bit(inCrc32, (const unsigned char *)buf, bufLen);
#endif
#ifdef VERIFY_ASSERTION
assert(crc == crc32c(inCrc32, (const unsigned char *)buf, bufLen));
#endif
return crc;
}

46
src/crc32.h
View File

@ -1,46 +0,0 @@
/*
* Copyright (c) 2015, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#ifndef CRC32_H_36A5015B5840C1
#define CRC32_H_36A5015B5840C1
#include "ue2common.h"
#ifdef __cplusplus
extern "C"
{
#endif
u32 Crc32c_ComputeBuf(u32 inCrc32, const void *buf, size_t bufLen);
#ifdef __cplusplus
}
#endif
#endif /* CRC32_H_36A5015B5840C1 */

454
src/database.c
View File

@ -1,454 +0,0 @@
/*
* Copyright (c) 2015-2020, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/** \file
* \brief Runtime code for hs_database manipulation.
*/
#include <stdio.h>
#include <string.h>
#include "allocator.h"
#include "hs_common.h"
#include "hs_internal.h"
#include "hs_version.h"
#include "ue2common.h"
#include "database.h"
#include "crc32.h"
#include "rose/rose_internal.h"
#include "util/unaligned.h"
static really_inline
int db_correctly_aligned(const void *db) {
return ISALIGNED_N(db, alignof(unsigned long long));
}
HS_PUBLIC_API
hs_error_t HS_CDECL hs_free_database(hs_database_t *db) {
if (db && db->magic != HS_DB_MAGIC) {
return HS_INVALID;
}
hs_database_free(db);
return HS_SUCCESS;
}
HS_PUBLIC_API
hs_error_t HS_CDECL hs_serialize_database(const hs_database_t *db, char **bytes,
size_t *serialized_length) {
if (!db || !bytes || !serialized_length) {
return HS_INVALID;
}
if (!db_correctly_aligned(db)) {
return HS_BAD_ALIGN;
}
hs_error_t ret = validDatabase(db);
if (ret != HS_SUCCESS) {
return ret;
}
size_t length = sizeof(struct hs_database) + db->length;
char *out = hs_misc_alloc(length);
ret = hs_check_alloc(out);
if (ret != HS_SUCCESS) {
hs_misc_free(out);
return ret;
}
memset(out, 0, length);
u32 *buf = (u32 *)out;
*buf = db->magic;
buf++;
*buf = db->version;
buf++;
*buf = db->length;
buf++;
memcpy(buf, &db->platform, sizeof(u64a));
buf += 2;
*buf = db->crc32;
buf++;
*buf = db->reserved0;
buf++;
*buf = db->reserved1;
buf++;
const char *bytecode = hs_get_bytecode(db);
memcpy(buf, bytecode, db->length);
*bytes = out;
*serialized_length = length;
return HS_SUCCESS;
}
// check that the database header's platform is compatible with the current
// runtime platform.
static
hs_error_t db_check_platform(const u64a p) {
if (p != hs_current_platform
&& p != (hs_current_platform | hs_current_platform_no_avx2)
&& p != (hs_current_platform | hs_current_platform_no_avx512)
&& p != (hs_current_platform | hs_current_platform_no_avx512vbmi)) {
return HS_DB_PLATFORM_ERROR;
}
// passed all checks
return HS_SUCCESS;
}
// Decode and check the database header, returning appropriate errors or
// HS_SUCCESS if it's OK. The header should be allocated on the stack
// and later copied into the deserialized database.
static
hs_error_t db_decode_header(const char **bytes, const size_t length,
struct hs_database *header) {
if (!*bytes) {
return HS_INVALID;
}
if (length < sizeof(struct hs_database)) {
return HS_INVALID;
}
// There's no requirement, really, that the serialized stream of bytes
// we've been given is 4-byte aligned, so we use unaligned loads here.
const u32 *buf = (const u32 *)*bytes;
// Zero header so that none of it (e.g. its padding) is uninitialized.
memset(header, 0, sizeof(struct hs_database));
header->magic = unaligned_load_u32(buf++);
if (header->magic != HS_DB_MAGIC) {
return HS_INVALID;
}
header->version = unaligned_load_u32(buf++);
if (header->version != HS_DB_VERSION) {
return HS_DB_VERSION_ERROR;
}
header->length = unaligned_load_u32(buf++);
if (length != sizeof(struct hs_database) + header->length) {
DEBUG_PRINTF("bad length %zu, expecting %zu\n", length,
sizeof(struct hs_database) + header->length);
return HS_INVALID;
}
header->platform = unaligned_load_u64a(buf);
buf += 2;
header->crc32 = unaligned_load_u32(buf++);
header->reserved0 = unaligned_load_u32(buf++);
header->reserved1 = unaligned_load_u32(buf++);
*bytes = (const char *)buf;
return HS_SUCCESS; // Header checks out
}
// Check the CRC on a database
static
hs_error_t db_check_crc(const hs_database_t *db) {
const char *bytecode = hs_get_bytecode(db);
u32 crc = Crc32c_ComputeBuf(0, bytecode, db->length);
if (crc != db->crc32) {
DEBUG_PRINTF("crc mismatch! 0x%x != 0x%x\n", crc, db->crc32);
return HS_INVALID;
}
return HS_SUCCESS;
}
static
void db_copy_bytecode(const char *serialized, hs_database_t *db) {
// we need to align things manually
uintptr_t shift = (uintptr_t)db->bytes & 0x3f;
db->bytecode = offsetof(struct hs_database, bytes) - shift;
char *bytecode = (char *)db + db->bytecode;
// Copy the bytecode into place
memcpy(bytecode, serialized, db->length);
}
HS_PUBLIC_API
hs_error_t HS_CDECL hs_deserialize_database_at(const char *bytes,
const size_t length,
hs_database_t *db) {
if (!bytes || !db) {
return HS_INVALID;
}
// We require the user to deserialize into an 8-byte aligned region.
if (!ISALIGNED_N(db, 8)) {
return HS_BAD_ALIGN;
}
// Decode the header
hs_database_t header;
hs_error_t ret = db_decode_header(&bytes, length, &header);
if (ret != HS_SUCCESS) {
return ret;
}
// Make sure the serialized database is for our platform
ret = db_check_platform(header.platform);
if (ret != HS_SUCCESS) {
return ret;
}
// Zero new space for safety
size_t dblength = sizeof(struct hs_database) + header.length;
memset(db, 0, dblength);
// Copy the decoded header into place
memcpy(db, &header, sizeof(header));
// Copy the bytecode into the correctly-aligned location, set offsets
db_copy_bytecode(bytes, db);
if (db_check_crc(db) != HS_SUCCESS) {
return HS_INVALID;
}
return HS_SUCCESS;
}
HS_PUBLIC_API
hs_error_t HS_CDECL hs_deserialize_database(const char *bytes,
const size_t length,
hs_database_t **db) {
if (!bytes || !db) {
return HS_INVALID;
}
*db = NULL;
// Decode and check the header
hs_database_t header;
hs_error_t ret = db_decode_header(&bytes, length, &header);
if (ret != HS_SUCCESS) {
return ret;
}
// Make sure the serialized database is for our platform
ret = db_check_platform(header.platform);
if (ret != HS_SUCCESS) {
return ret;
}
// Allocate space for new database
size_t dblength = sizeof(struct hs_database) + header.length;
struct hs_database *tempdb = hs_database_alloc(dblength);
ret = hs_check_alloc(tempdb);
if (ret != HS_SUCCESS) {
hs_database_free(tempdb);
return ret;
}
// Zero new space for safety
memset(tempdb, 0, dblength);
// Copy the decoded header into place
memcpy(tempdb, &header, sizeof(header));
// Copy the bytecode into the correctly-aligned location, set offsets
db_copy_bytecode(bytes, tempdb);
if (db_check_crc(tempdb) != HS_SUCCESS) {
hs_database_free(tempdb);
return HS_INVALID;
}
*db = tempdb;
return HS_SUCCESS;
}
HS_PUBLIC_API
hs_error_t HS_CDECL hs_database_size(const hs_database_t *db, size_t *size) {
if (!size) {
return HS_INVALID;
}
hs_error_t ret = validDatabase(db);
if (unlikely(ret != HS_SUCCESS)) {
return ret;
}
*size = sizeof(struct hs_database) + db->length;
return HS_SUCCESS;
}
HS_PUBLIC_API
hs_error_t HS_CDECL hs_serialized_database_size(const char *bytes,
const size_t length,
size_t *size) {
// Decode and check the header
hs_database_t header;
hs_error_t ret = db_decode_header(&bytes, length, &header);
if (ret != HS_SUCCESS) {
return ret;
}
if (!size) {
return HS_INVALID;
}
*size = sizeof(struct hs_database) + header.length;
return HS_SUCCESS;
}
hs_error_t dbIsValid(const hs_database_t *db) {
if (db->magic != HS_DB_MAGIC) {
DEBUG_PRINTF("bad magic\n");
return HS_INVALID;
}
if (db->version != HS_DB_VERSION) {
DEBUG_PRINTF("bad version\n");
return HS_DB_VERSION_ERROR;
}
if (db_check_platform(db->platform) != HS_SUCCESS) {
DEBUG_PRINTF("bad platform\n");
return HS_DB_PLATFORM_ERROR;
}
if (!ISALIGNED_16(hs_get_bytecode(db))) {
DEBUG_PRINTF("bad alignment\n");
return HS_INVALID;
}
hs_error_t rv = db_check_crc(db);
if (rv != HS_SUCCESS) {
DEBUG_PRINTF("bad crc\n");
return rv;
}
return HS_SUCCESS;
}
/** Allocate a buffer and prints the database info into it. Returns an
* appropriate error code on failure, or HS_SUCCESS on success. */
static
hs_error_t print_database_string(char **s, u32 version, const platform_t plat,
u32 raw_mode) {
assert(s);
*s = NULL;
u8 release = (version >> 8) & 0xff;
u8 minor = (version >> 16) & 0xff;
u8 major = (version >> 24) & 0xff;
const char *features = (plat & HS_PLATFORM_NOAVX512VBMI)
? (plat & HS_PLATFORM_NOAVX512)
? (plat & HS_PLATFORM_NOAVX2) ? "" : "AVX2"
: "AVX512"
: "AVX512VBMI";
const char *mode = NULL;
if (raw_mode == HS_MODE_STREAM) {
mode = "STREAM";
} else if (raw_mode == HS_MODE_VECTORED) {
mode = "VECTORED";
} else {
assert(raw_mode == HS_MODE_BLOCK);
mode = "BLOCK";
}
// Initial allocation size, which should be large enough to print our info.
// If it isn't, snprintf will tell us and we can resize appropriately.
size_t len = 256;
while (1) {
char *buf = hs_misc_alloc(len);
hs_error_t ret = hs_check_alloc(buf);
if (ret != HS_SUCCESS) {
hs_misc_free(buf);
return ret;
}
int p_len = snprintf(
buf, len, "Version: %u.%u.%u Features: %s Mode: %s",
major, minor, release, features, mode);
if (p_len < 0) {
DEBUG_PRINTF("snprintf output error, returned %d\n", p_len);
hs_misc_free(buf);
break;
} else if ((size_t)p_len < len) { // output fit within buffer.
assert(buf[p_len] == '\0');
*s = buf;
return HS_SUCCESS;
} else { // output didn't fit: resize and reallocate.
len = (size_t)p_len + 1; // must add one for null terminator.
hs_misc_free(buf);
}
}
return HS_NOMEM;
}
HS_PUBLIC_API
hs_error_t HS_CDECL hs_serialized_database_info(const char *bytes,
size_t length, char **info) {
if (!info) {
return HS_INVALID;
}
*info = NULL;
// Decode and check the header
hs_database_t header;
hs_error_t ret = db_decode_header(&bytes, length, &header);
if (ret != HS_SUCCESS) {
return ret;
}
u32 mode = unaligned_load_u32(bytes + offsetof(struct RoseEngine, mode));
return print_database_string(info, header.version, header.platform, mode);
}
HS_PUBLIC_API
hs_error_t HS_CDECL hs_database_info(const hs_database_t *db, char **info) {
if (!info) {
return HS_INVALID;
}
*info = NULL;
if (!db || !db_correctly_aligned(db) || db->magic != HS_DB_MAGIC) {
return HS_INVALID;
}
platform_t plat;
plat = db->platform;
const struct RoseEngine *rose = hs_get_bytecode(db);
return print_database_string(info, db->version, plat, rose->mode);
}

143
src/database.h
View File

@ -1,143 +0,0 @@
/*
* Copyright (c) 2015-2020, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/** \file
* \brief Runtime code for hs_database manipulation.
*/
#ifndef DATABASE_H_D467FD6F343DDE
#define DATABASE_H_D467FD6F343DDE
#ifdef __cplusplus
extern "C"
{
#endif
#include "hs_compile.h" // for HS_MODE_ flags
#include "hs_version.h"
#include "ue2common.h"
#include "util/arch.h"
#define HS_DB_VERSION HS_VERSION_32BIT
#define HS_DB_MAGIC (0xdbdbdbdbU)
// Values in here cannot (easily) change - add new ones!
// CPU type is the low 6 bits (we can't need more than 64, surely!)
#define HS_PLATFORM_INTEL 1
#define HS_PLATFORM_ARM 2
#define HS_PLATFORM_CPU_MASK 0x3F
#define HS_PLATFORM_NOAVX2 (4<<13)
#define HS_PLATFORM_NOAVX512 (8<<13)
#define HS_PLATFORM_NOAVX512VBMI (0x10<<13)
/** \brief Platform features bitmask. */
typedef u64a platform_t;
static UNUSED
const platform_t hs_current_platform = {
#if !defined(HAVE_AVX2)
HS_PLATFORM_NOAVX2 |
#endif
#if !defined(HAVE_AVX512)
HS_PLATFORM_NOAVX512 |
#endif
#if !defined(HAVE_AVX512VBMI)
HS_PLATFORM_NOAVX512VBMI |
#endif
0,
};
static UNUSED
const platform_t hs_current_platform_no_avx2 = {
HS_PLATFORM_NOAVX2 |
HS_PLATFORM_NOAVX512 |
HS_PLATFORM_NOAVX512VBMI |
0,
};
static UNUSED
const platform_t hs_current_platform_no_avx512 = {
HS_PLATFORM_NOAVX512 |
HS_PLATFORM_NOAVX512VBMI |
0,
};
static UNUSED
const platform_t hs_current_platform_no_avx512vbmi = {
HS_PLATFORM_NOAVX512VBMI |
0,
};
/*
* a header to enclose the actual bytecode - useful for keeping info about the
* compiled data.
*/
struct hs_database {
u32 magic;
u32 version;
u32 length;
u64a platform;
u32 crc32;
u32 reserved0;
u32 reserved1;
u32 bytecode; // offset relative to db start
u32 padding[16];
char bytes[];
};
static really_inline
const void *hs_get_bytecode(const struct hs_database *db) {
return ((const char *)db + db->bytecode);
}
/**
* Cheap database sanity checks used in block mode scan calls and streaming
* mode open calls.
*/
static really_inline
hs_error_t validDatabase(const hs_database_t *db) {
if (!db || db->magic != HS_DB_MAGIC) {
return HS_INVALID;
}
if (db->version != HS_DB_VERSION) {
return HS_DB_VERSION_ERROR;
}
return HS_SUCCESS;
}
hs_error_t dbIsValid(const struct hs_database *db);
#ifdef __cplusplus
} /* extern "C" */
#endif
#endif /* DATABASE_H_D467FD6F343DDE */

153
src/dispatcher.c
View File

@ -1,153 +0,0 @@
/*
* Copyright (c) 2016-2020, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#include "config.h"
#include "hs_common.h"
#include "hs_runtime.h"
#include "ue2common.h"
#if defined(ARCH_IA32) || defined(ARCH_X86_64)
#include "util/arch/x86/cpuid_inline.h"
#endif
#include "util/join.h"
#if defined(DISABLE_AVX512_DISPATCH)
#define avx512_ disabled_
#define check_avx512() (0)
#endif
#if defined(DISABLE_AVX512VBMI_DISPATCH)
#define avx512vbmi_ disabled_
#define check_avx512vbmi() (0)
#endif
#define CREATE_DISPATCH(RTYPE, NAME, ...) \
/* create defns */ \
RTYPE JOIN(avx512vbmi_, NAME)(__VA_ARGS__); \
RTYPE JOIN(avx512_, NAME)(__VA_ARGS__); \
RTYPE JOIN(avx2_, NAME)(__VA_ARGS__); \
RTYPE JOIN(corei7_, NAME)(__VA_ARGS__); \
RTYPE JOIN(core2_, NAME)(__VA_ARGS__); \
\
/* error func */ \
static inline RTYPE JOIN(error_, NAME)(__VA_ARGS__) { \
return (RTYPE)HS_ARCH_ERROR; \
} \
\
/* resolver */ \
static RTYPE (*JOIN(resolve_, NAME)(void))(__VA_ARGS__) { \
if (check_avx512vbmi()) { \
return JOIN(avx512vbmi_, NAME); \
} \
if (check_avx512()) { \
return JOIN(avx512_, NAME); \
} \
if (check_avx2()) { \
return JOIN(avx2_, NAME); \
} \
if (check_sse42() && check_popcnt()) { \
return JOIN(corei7_, NAME); \
} \
if (check_ssse3()) { \
return JOIN(core2_, NAME); \
} \
/* anything else is fail */ \
return JOIN(error_, NAME); \
} \
\
/* function */ \
HS_PUBLIC_API \
RTYPE NAME(__VA_ARGS__) __attribute__((ifunc("resolve_" #NAME)))
CREATE_DISPATCH(hs_error_t, hs_scan, const hs_database_t *db, const char *data,
unsigned length, unsigned flags, hs_scratch_t *scratch,
match_event_handler onEvent, void *userCtx);
CREATE_DISPATCH(hs_error_t, hs_stream_size, const hs_database_t *database,
size_t *stream_size);
CREATE_DISPATCH(hs_error_t, hs_database_size, const hs_database_t *db,
size_t *size);
CREATE_DISPATCH(hs_error_t, dbIsValid, const hs_database_t *db);
CREATE_DISPATCH(hs_error_t, hs_free_database, hs_database_t *db);
CREATE_DISPATCH(hs_error_t, hs_open_stream, const hs_database_t *db,
unsigned int flags, hs_stream_t **stream);
CREATE_DISPATCH(hs_error_t, hs_scan_stream, hs_stream_t *id, const char *data,
unsigned int length, unsigned int flags, hs_scratch_t *scratch,
match_event_handler onEvent, void *ctxt);
CREATE_DISPATCH(hs_error_t, hs_close_stream, hs_stream_t *id,
hs_scratch_t *scratch, match_event_handler onEvent, void *ctxt);
CREATE_DISPATCH(hs_error_t, hs_scan_vector, const hs_database_t *db,
const char *const *data, const unsigned int *length,
unsigned int count, unsigned int flags, hs_scratch_t *scratch,
match_event_handler onevent, void *context);
CREATE_DISPATCH(hs_error_t, hs_database_info, const hs_database_t *db, char **info);
CREATE_DISPATCH(hs_error_t, hs_copy_stream, hs_stream_t **to_id,
const hs_stream_t *from_id);
CREATE_DISPATCH(hs_error_t, hs_reset_stream, hs_stream_t *id,
unsigned int flags, hs_scratch_t *scratch,
match_event_handler onEvent, void *context);
CREATE_DISPATCH(hs_error_t, hs_reset_and_copy_stream, hs_stream_t *to_id,
const hs_stream_t *from_id, hs_scratch_t *scratch,
match_event_handler onEvent, void *context);
CREATE_DISPATCH(hs_error_t, hs_serialize_database, const hs_database_t *db,
char **bytes, size_t *length);
CREATE_DISPATCH(hs_error_t, hs_deserialize_database, const char *bytes,
const size_t length, hs_database_t **db);
CREATE_DISPATCH(hs_error_t, hs_deserialize_database_at, const char *bytes,
const size_t length, hs_database_t *db);
CREATE_DISPATCH(hs_error_t, hs_serialized_database_info, const char *bytes,
size_t length, char **info);
CREATE_DISPATCH(hs_error_t, hs_serialized_database_size, const char *bytes,
const size_t length, size_t *deserialized_size);
CREATE_DISPATCH(hs_error_t, hs_compress_stream, const hs_stream_t *stream,
char *buf, size_t buf_space, size_t *used_space);
CREATE_DISPATCH(hs_error_t, hs_expand_stream, const hs_database_t *db,
hs_stream_t **stream, const char *buf,size_t buf_size);
CREATE_DISPATCH(hs_error_t, hs_reset_and_expand_stream, hs_stream_t *to_stream,
const char *buf, size_t buf_size, hs_scratch_t *scratch,
match_event_handler onEvent, void *context);
/** INTERNALS **/
CREATE_DISPATCH(u32, Crc32c_ComputeBuf, u32 inCrc32, const void *buf, size_t bufLen);

49
src/fdr/engine_description.cpp
View File

@ -1,49 +0,0 @@
/*
* Copyright (c) 2015, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#include "engine_description.h"
#include "hs_compile.h" // for hs_platform_info
#include "util/target_info.h"
namespace ue2 {
EngineDescription::~EngineDescription() {}
bool EngineDescription::isValidOnTarget(const target_t &target_in) const {
return target_in.can_run_on_code_built_for(code_target);
}
target_t targetByArchFeatures(u64a cpu_features) {
hs_platform_info p;
p.tune = HS_TUNE_FAMILY_GENERIC;
p.cpu_features = cpu_features;
return target_t(p);
}
} // namespace ue2

61
src/fdr/engine_description.h
View File

@ -1,61 +0,0 @@
/*
* Copyright (c) 2015-2017, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#ifndef ENGINE_DESCRIPTION_H
#define ENGINE_DESCRIPTION_H
#include "ue2common.h"
#include "util/target_info.h"
namespace ue2 {
class EngineDescription {
u32 id;
target_t code_target; // the target that we built this code for
u32 numBuckets;
public:
EngineDescription(u32 id_in, const target_t &code_target_in,
u32 numBuckets_in)
: id(id_in), code_target(code_target_in), numBuckets(numBuckets_in) {}
virtual ~EngineDescription();
u32 getID() const { return id; }
u32 getNumBuckets() const { return numBuckets; }
bool isValidOnTarget(const target_t &target_in) const;
virtual u32 getDefaultFloodSuffixLength() const = 0;
};
/** Returns a target given a CPU feature set value. */
target_t targetByArchFeatures(u64a cpu_features);
} // namespace ue2
#endif

855
src/fdr/fdr.c
View File

@ -1,855 +0,0 @@
/*
* Copyright (c) 2015-2017, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#include "fdr.h"
#include "fdr_confirm.h"
#include "fdr_confirm_runtime.h"
#include "fdr_internal.h"
#include "fdr_loadval.h"
#include "flood_runtime.h"
#include "scratch.h"
#include "teddy.h"
#include "teddy_internal.h"
#include "util/arch.h"
#include "util/bitutils.h"
#include "util/simd_utils.h"
#include "util/uniform_ops.h"
/** \brief number of bytes processed in each iteration */
#define ITER_BYTES 16
/** \brief total zone buffer size */
#define ZONE_TOTAL_SIZE 64
/** \brief maximum number of allowed zones */
#define ZONE_MAX 3
/** \brief zone information.
*
* Zone represents a region of data to scan in FDR.
*
* The incoming buffer is to split in multiple zones to ensure two properties:
* 1: that we can read 8? bytes behind to generate a hash safely
* 2: that we can read the 3 byte after the current byte (domain > 8)
*/
struct zone {
/** \brief copied buffer, used only when it is a boundary zone. */
u8 ALIGN_CL_DIRECTIVE buf[ZONE_TOTAL_SIZE];
/** \brief shift amount for fdr state to avoid unwanted match. */
u8 shift;
/** \brief if boundary zone, start points into the zone buffer after the
* pre-padding. Otherwise, points to the main buffer, appropriately. */
const u8 *start;
/** \brief if boundary zone, end points to the end of zone. Otherwise,
* pointer to the main buffer, appropriately. */
const u8 *end;
/** \brief the amount to adjust to go from a pointer in the zones region
* (between start and end) to a pointer in the original data buffer. */
ptrdiff_t zone_pointer_adjust;
/** \brief firstFloodDetect from FDR_Runtime_Args for non-boundary zones,
* otherwise end of the zone buf. floodPtr always points inside the same
* buffer as the start pointe. */
const u8 *floodPtr;
};
static
const ALIGN_CL_DIRECTIVE u8 zone_or_mask[ITER_BYTES+1][ITER_BYTES] = {
{ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },
{ 0xff, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },
{ 0xff, 0xff, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },
{ 0xff, 0xff, 0xff, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },
{ 0xff, 0xff, 0xff, 0xff, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },
{ 0xff, 0xff, 0xff, 0xff, 0xff, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },
{ 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },
{ 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },
{ 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },
{ 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
0xff, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },
{ 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
0xff, 0xff, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },
{ 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
0xff, 0xff, 0xff, 0x00, 0x00, 0x00, 0x00, 0x00 },
{ 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
0xff, 0xff, 0xff, 0xff, 0x00, 0x00, 0x00, 0x00 },
{ 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
0xff, 0xff, 0xff, 0xff, 0xff, 0x00, 0x00, 0x00 },
{ 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0x00, 0x00 },
{ 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0x00 },
{ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 }
};
/* generates an initial state mask based on the last byte-ish of history rather
* than being all accepting. If there is no history to consider, the state is
* generated based on the minimum length of each bucket in order to prevent
* confirms.
*/
static really_inline
m128 getInitState(const struct FDR *fdr, u8 len_history, const u64a *ft,
const struct zone *z) {
m128 s;
if (len_history) {
/* +1: the zones ensure that we can read the byte at z->end */
u32 tmp = lv_u16(z->start + z->shift - 1, z->buf, z->end + 1);
tmp &= fdr->domainMask;
s = load_m128_from_u64a(ft + tmp);
s = rshiftbyte_m128(s, 1);
} else {
s = fdr->start;
}
return s;
}
static really_inline
void get_conf_stride_1(const u8 *itPtr, UNUSED const u8 *start_ptr,
UNUSED const u8 *end_ptr, u32 domain_mask_flipped,
const u64a *ft, u64a *conf0, u64a *conf8, m128 *s) {
/* +1: the zones ensure that we can read the byte at z->end */
assert(itPtr >= start_ptr && itPtr + ITER_BYTES <= end_ptr);
u64a domain_mask = ~domain_mask_flipped;
u64a it_hi = *(const u64a *)itPtr;
u64a it_lo = *(const u64a *)(itPtr + 8);
u64a reach0 = domain_mask & it_hi;
u64a reach1 = domain_mask & (it_hi >> 8);
u64a reach2 = domain_mask & (it_hi >> 16);
u64a reach3 = domain_mask & (it_hi >> 24);
u64a reach4 = domain_mask & (it_hi >> 32);
u64a reach5 = domain_mask & (it_hi >> 40);
u64a reach6 = domain_mask & (it_hi >> 48);
u64a reach7 = domain_mask & ((it_hi >> 56) | (it_lo << 8));
u64a reach8 = domain_mask & it_lo;
u64a reach9 = domain_mask & (it_lo >> 8);
u64a reach10 = domain_mask & (it_lo >> 16);
u64a reach11 = domain_mask & (it_lo >> 24);
u64a reach12 = domain_mask & (it_lo >> 32);
u64a reach13 = domain_mask & (it_lo >> 40);
u64a reach14 = domain_mask & (it_lo >> 48);
u64a reach15 = domain_mask & unaligned_load_u32(itPtr + 15);
m128 st0 = load_m128_from_u64a(ft + reach0);
m128 st1 = lshiftbyte_m128(load_m128_from_u64a(ft + reach1), 1);
m128 st2 = lshiftbyte_m128(load_m128_from_u64a(ft + reach2), 2);
m128 st3 = lshiftbyte_m128(load_m128_from_u64a(ft + reach3), 3);
m128 st4 = lshiftbyte_m128(load_m128_from_u64a(ft + reach4), 4);
m128 st5 = lshiftbyte_m128(load_m128_from_u64a(ft + reach5), 5);
m128 st6 = lshiftbyte_m128(load_m128_from_u64a(ft + reach6), 6);
m128 st7 = lshiftbyte_m128(load_m128_from_u64a(ft + reach7), 7);
m128 st8 = load_m128_from_u64a(ft + reach8);
m128 st9 = lshiftbyte_m128(load_m128_from_u64a(ft + reach9), 1);
m128 st10 = lshiftbyte_m128(load_m128_from_u64a(ft + reach10), 2);
m128 st11 = lshiftbyte_m128(load_m128_from_u64a(ft + reach11), 3);
m128 st12 = lshiftbyte_m128(load_m128_from_u64a(ft + reach12), 4);
m128 st13 = lshiftbyte_m128(load_m128_from_u64a(ft + reach13), 5);
m128 st14 = lshiftbyte_m128(load_m128_from_u64a(ft + reach14), 6);
m128 st15 = lshiftbyte_m128(load_m128_from_u64a(ft + reach15), 7);
st0 = or128(st0, st1);
st2 = or128(st2, st3);
st4 = or128(st4, st5);
st6 = or128(st6, st7);
st0 = or128(st0, st2);
st4 = or128(st4, st6);
st0 = or128(st0, st4);
st8 = or128(st8, st9);
st10 = or128(st10, st11);
st12 = or128(st12, st13);
st14 = or128(st14, st15);
st8 = or128(st8, st10);
st12 = or128(st12, st14);
st8 = or128(st8, st12);
m128 st = or128(*s, st0);
*conf0 = movq(st) ^ ~0ULL;
st = rshiftbyte_m128(st, 8);
st = or128(st, st8);
*conf8 = movq(st) ^ ~0ULL;
*s = rshiftbyte_m128(st, 8);
}
static really_inline
void get_conf_stride_2(const u8 *itPtr, UNUSED const u8 *start_ptr,
UNUSED const u8 *end_ptr, u32 domain_mask_flipped,
const u64a *ft, u64a *conf0, u64a *conf8, m128 *s) {
assert(itPtr >= start_ptr && itPtr + ITER_BYTES <= end_ptr);
u64a reach0 = andn(domain_mask_flipped, itPtr);
u64a reach2 = andn(domain_mask_flipped, itPtr + 2);
u64a reach4 = andn(domain_mask_flipped, itPtr + 4);
u64a reach6 = andn(domain_mask_flipped, itPtr + 6);
m128 st0 = load_m128_from_u64a(ft + reach0);
m128 st2 = load_m128_from_u64a(ft + reach2);
m128 st4 = load_m128_from_u64a(ft + reach4);
m128 st6 = load_m128_from_u64a(ft + reach6);
u64a reach8 = andn(domain_mask_flipped, itPtr + 8);
u64a reach10 = andn(domain_mask_flipped, itPtr + 10);
u64a reach12 = andn(domain_mask_flipped, itPtr + 12);
u64a reach14 = andn(domain_mask_flipped, itPtr + 14);
m128 st8 = load_m128_from_u64a(ft + reach8);
m128 st10 = load_m128_from_u64a(ft + reach10);
m128 st12 = load_m128_from_u64a(ft + reach12);
m128 st14 = load_m128_from_u64a(ft + reach14);
st2 = lshiftbyte_m128(st2, 2);
st4 = lshiftbyte_m128(st4, 4);
st6 = lshiftbyte_m128(st6, 6);
*s = or128(*s, st0);
*s = or128(*s, st2);
*s = or128(*s, st4);
*s = or128(*s, st6);
*conf0 = movq(*s);
*s = rshiftbyte_m128(*s, 8);
*conf0 ^= ~0ULL;
st10 = lshiftbyte_m128(st10, 2);
st12 = lshiftbyte_m128(st12, 4);
st14 = lshiftbyte_m128(st14, 6);
*s = or128(*s, st8);
*s = or128(*s, st10);
*s = or128(*s, st12);
*s = or128(*s, st14);
*conf8 = movq(*s);
*s = rshiftbyte_m128(*s, 8);
*conf8 ^= ~0ULL;
}
static really_inline
void get_conf_stride_4(const u8 *itPtr, UNUSED const u8 *start_ptr,
UNUSED const u8 *end_ptr, u32 domain_mask_flipped,
const u64a *ft, u64a *conf0, u64a *conf8, m128 *s) {
assert(itPtr >= start_ptr && itPtr + ITER_BYTES <= end_ptr);
u64a reach0 = andn(domain_mask_flipped, itPtr);
u64a reach4 = andn(domain_mask_flipped, itPtr + 4);
u64a reach8 = andn(domain_mask_flipped, itPtr + 8);
u64a reach12 = andn(domain_mask_flipped, itPtr + 12);
m128 st0 = load_m128_from_u64a(ft + reach0);
m128 st4 = load_m128_from_u64a(ft + reach4);
m128 st8 = load_m128_from_u64a(ft + reach8);
m128 st12 = load_m128_from_u64a(ft + reach12);
st4 = lshiftbyte_m128(st4, 4);
st12 = lshiftbyte_m128(st12, 4);
*s = or128(*s, st0);
*s = or128(*s, st4);
*conf0 = movq(*s);
*s = rshiftbyte_m128(*s, 8);
*conf0 ^= ~0ULL;
*s = or128(*s, st8);
*s = or128(*s, st12);
*conf8 = movq(*s);
*s = rshiftbyte_m128(*s, 8);
*conf8 ^= ~0ULL;
}
static really_inline
void do_confirm_fdr(u64a *conf, u8 offset, hwlmcb_rv_t *control,
const u32 *confBase, const struct FDR_Runtime_Args *a,
const u8 *ptr, u32 *last_match_id, struct zone *z) {
const u8 bucket = 8;
if (likely(!*conf)) {
return;
}
/* ptr is currently referring to a location in the zone's buffer, we also
* need a pointer in the original, main buffer for the final string compare.
*/
const u8 *ptr_main = (const u8 *)((uintptr_t)ptr + z->zone_pointer_adjust);
const u8 *confLoc = ptr;
do {
u32 bit = findAndClearLSB_64(conf);
u32 byte = bit / bucket + offset;
u32 bitRem = bit % bucket;
u32 idx = bitRem;
u32 cf = confBase[idx];
if (!cf) {
continue;
}
const struct FDRConfirm *fdrc = (const struct FDRConfirm *)
((const u8 *)confBase + cf);
if (!(fdrc->groups & *control)) {
continue;
}
u64a confVal = unaligned_load_u64a(confLoc + byte - sizeof(u64a) + 1);
confWithBit(fdrc, a, ptr_main - a->buf + byte, control,
last_match_id, confVal, conf, bit);
} while (unlikely(!!*conf));
}
static really_inline
void dumpZoneInfo(UNUSED struct zone *z, UNUSED size_t zone_id) {
#ifdef DEBUG
DEBUG_PRINTF("zone: zone=%zu, bufPtr=%p\n", zone_id, z->buf);
DEBUG_PRINTF("zone: startPtr=%p, endPtr=%p, shift=%u\n",
z->start, z->end, z->shift);
DEBUG_PRINTF("zone: zone_pointer_adjust=%zd, floodPtr=%p\n",
z->zone_pointer_adjust, z->floodPtr);
DEBUG_PRINTF("zone buf:");
for (size_t i = 0; i < ZONE_TOTAL_SIZE; i++) {
if (i % 8 == 0) {
printf("_");
}
if (z->buf[i]) {
printf("%02x", z->buf[i]);
} else {
printf("..");
}
}
printf("\n");
#endif
};
/**
* \brief Updates attributes for non-boundary region zone.
*/
static really_inline
void createMainZone(const u8 *flood, const u8 *begin, const u8 *end,
struct zone *z) {
z->zone_pointer_adjust = 0; /* zone buffer is the main buffer */
z->start = begin;
z->end = end;
z->floodPtr = flood;
z->shift = 0;
}
/**
* \brief Create zone for short cases (<= ITER_BYTES).
*
* For this case we need to copy everything into the zone's internal buffer.
*
* We need to ensure that we run over real data if it exists (in history or
* before zone begin). We also need to ensure 8 bytes before any data being
* matched can be read (to perform a conf hash).
*
* We also need to ensure that the data at z->end can be read.
*
* Hence, the zone consists of:
* 16 bytes of history,
* 1 - 24 bytes of data form the buffer (ending at end),
* 1 byte of final padding
*/
static really_inline
void createShortZone(const u8 *buf, const u8 *hend, const u8 *begin,
const u8 *end, struct zone *z) {
/* the floodPtr for BOUNDARY zones are maximum of end of zone buf to avoid
* the checks in boundary zone. */
z->floodPtr = z->buf + ZONE_TOTAL_SIZE;
ptrdiff_t z_len = end - begin;
assert(z_len > 0);
assert(z_len <= ITER_BYTES);
z->shift = ITER_BYTES - z_len; /* ignore bytes outside region specified */
static const size_t ZONE_SHORT_DATA_OFFSET = 16; /* after history */
/* we are guaranteed to always have 16 initialised bytes at the end of
* the history buffer (they may be garbage coming from the stream state
* preceding hbuf, but bytes that don't correspond to actual history
* shouldn't affect computations). */
*(m128 *)z->buf = loadu128(hend - sizeof(m128));
/* The amount of data we have to copy from main buffer. */
size_t copy_len = MIN((size_t)(end - buf),
ITER_BYTES + sizeof(CONF_TYPE));
u8 *zone_data = z->buf + ZONE_SHORT_DATA_OFFSET;
switch (copy_len) {
case 1:
*zone_data = *(end - 1);
break;
case 2:
*(u16 *)zone_data = unaligned_load_u16(end - 2);
break;
case 3:
*(u16 *)zone_data = unaligned_load_u16(end - 3);
*(zone_data + 2) = *(end - 1);
break;
case 4:
*(u32 *)zone_data = unaligned_load_u32(end - 4);
break;
case 5:
case 6:
case 7:
/* perform copy with 2 overlapping 4-byte chunks from buf. */
*(u32 *)zone_data = unaligned_load_u32(end - copy_len);
unaligned_store_u32(zone_data + copy_len - sizeof(u32),
unaligned_load_u32(end - sizeof(u32)));
break;
case 8:
*(u64a *)zone_data = unaligned_load_u64a(end - 8);
break;
case 9:
case 10:
case 11:
case 12:
case 13:
case 14:
case 15:
/* perform copy with 2 overlapping 8-byte chunks from buf. */
*(u64a *)zone_data = unaligned_load_u64a(end - copy_len);
unaligned_store_u64a(zone_data + copy_len - sizeof(u64a),
unaligned_load_u64a(end - sizeof(u64a)));
break;
case 16:
/* copy 16-bytes from buf. */
*(m128 *)zone_data = loadu128(end - 16);
break;
default:
assert(copy_len <= sizeof(m128) + sizeof(u64a));
/* perform copy with (potentially overlapping) 8-byte and 16-byte chunks.
*/
*(u64a *)zone_data = unaligned_load_u64a(end - copy_len);
storeu128(zone_data + copy_len - sizeof(m128),
loadu128(end - sizeof(m128)));
break;
}
/* set the start and end location of the zone buf
* to be scanned */
u8 *z_end = z->buf + ZONE_SHORT_DATA_OFFSET + copy_len;
assert(ZONE_SHORT_DATA_OFFSET + copy_len >= ITER_BYTES);
/* copy the post-padding byte; this is required for domain > 8 due to
* overhang */
assert(ZONE_SHORT_DATA_OFFSET + copy_len + 3 < 64);
*z_end = 0;
z->end = z_end;
z->start = z_end - ITER_BYTES;
z->zone_pointer_adjust = (ptrdiff_t)((uintptr_t)end - (uintptr_t)z_end);
assert(z->start + z->shift == z_end - z_len);
}
/**
* \brief Create a zone for the start region.
*
* This function requires that there is > ITER_BYTES of data in the buffer to
* scan. The start zone itself is always responsible for scanning exactly
* ITER_BYTES of data - there are no warmup/junk bytes scanned.
*
* This zone ensures that the byte at z->end can be read and corresponds to
* the next byte of data.
*
* 8 bytes of history data are provided before z->start to allow proper hash
* generation in streaming mode. If buf != begin, upto 8 bytes of data
* prior to begin is also provided.
*
* Although we are not interested in bare literals which start before begin
* if buf != begin, lookarounds associated with the literal may require
* the data prior to begin for hash purposes.
*/
static really_inline
void createStartZone(const u8 *buf, const u8 *hend, const u8 *begin,
struct zone *z) {
assert(ITER_BYTES == sizeof(m128));
assert(sizeof(CONF_TYPE) == 8);
static const size_t ZONE_START_BEGIN = sizeof(CONF_TYPE);
const u8 *end = begin + ITER_BYTES;
/* set floodPtr to the end of zone buf to avoid checks in start zone */
z->floodPtr = z->buf + ZONE_TOTAL_SIZE;
z->shift = 0; /* we are processing ITER_BYTES of real data */
/* we are guaranteed to always have 16 initialised bytes at the end of the
* history buffer (they may be garbage coming from the stream state
* preceding hbuf, but bytes that don't correspond to actual history
* shouldn't affect computations). However, for start zones, history is only
* required for conf hash purposes so we only need 8 bytes */
unaligned_store_u64a(z->buf, unaligned_load_u64a(hend - sizeof(u64a)));
/* The amount of data we have to copy from main buffer. */
size_t copy_len = MIN((size_t)(end - buf),
ITER_BYTES + sizeof(CONF_TYPE));
assert(copy_len >= 16);
/* copy the post-padding byte; this is required for domain > 8 due to
* overhang. The start requires that there is data after the zone so it
* it safe to dereference end */
z->buf[ZONE_START_BEGIN + copy_len] = *end;
/* set the start and end location of the zone buf to be scanned */
u8 *z_end = z->buf + ZONE_START_BEGIN + copy_len;
z->end = z_end;
z->start = z_end - ITER_BYTES;
/* copy the first 8 bytes of the valid region */
unaligned_store_u64a(z->buf + ZONE_START_BEGIN,
unaligned_load_u64a(end - copy_len));
/* copy the last 16 bytes, may overlap with the previous 8 byte write */
storeu128(z_end - sizeof(m128), loadu128(end - sizeof(m128)));
z->zone_pointer_adjust = (ptrdiff_t)((uintptr_t)end - (uintptr_t)z_end);
assert(ZONE_START_BEGIN + copy_len + 3 < 64);
}
/**
* \brief Create a zone for the end region.
*
* This function requires that there is > ITER_BYTES of data in the buffer to
* scan. The end zone is responsible for a scanning the <= ITER_BYTES rump of
* data and optional ITER_BYTES. The main zone cannot handle the last 3 bytes
* of the buffer. The end zone is required to handle an optional full
* ITER_BYTES from main zone when there are less than 3 bytes to scan. The
* main zone size is reduced by ITER_BYTES in this case.
*
* This zone ensures that the byte at z->end can be read by filling it with a
* padding character.
*
* Upto 8 bytes of data prior to begin is also provided for the purposes of
* generating hashes. History is not copied, as all locations which require
* history for generating a hash are the responsiblity of the start zone.
*/
static really_inline
void createEndZone(const u8 *buf, const u8 *begin, const u8 *end,
struct zone *z) {
/* the floodPtr for BOUNDARY zones are maximum of end of zone buf to avoid
* the checks in boundary zone. */
z->floodPtr = z->buf + ZONE_TOTAL_SIZE;
ptrdiff_t z_len = end - begin;
assert(z_len > 0);
size_t iter_bytes_second = 0;
size_t z_len_first = z_len;
if (z_len > ITER_BYTES) {
z_len_first = z_len - ITER_BYTES;
iter_bytes_second = ITER_BYTES;
}
z->shift = ITER_BYTES - z_len_first;
const u8 *end_first = end - iter_bytes_second;
/* The amount of data we have to copy from main buffer for the
* first iteration. */
size_t copy_len_first = MIN((size_t)(end_first - buf),
ITER_BYTES + sizeof(CONF_TYPE));
assert(copy_len_first >= 16);
size_t total_copy_len = copy_len_first + iter_bytes_second;
assert(total_copy_len + 3 < 64);
/* copy the post-padding byte; this is required for domain > 8 due to
* overhang */
z->buf[total_copy_len] = 0;
/* set the start and end location of the zone buf
* to be scanned */
u8 *z_end = z->buf + total_copy_len;
z->end = z_end;
z->start = z_end - ITER_BYTES - iter_bytes_second;
assert(z->start + z->shift == z_end - z_len);
u8 *z_end_first = z_end - iter_bytes_second;
/* copy the first 8 bytes of the valid region */
unaligned_store_u64a(z->buf,
unaligned_load_u64a(end_first - copy_len_first));
/* copy the last 16 bytes, may overlap with the previous 8 byte write */
storeu128(z_end_first - sizeof(m128), loadu128(end_first - sizeof(m128)));
if (iter_bytes_second) {
storeu128(z_end - sizeof(m128), loadu128(end - sizeof(m128)));
}
z->zone_pointer_adjust = (ptrdiff_t)((uintptr_t)end - (uintptr_t)z_end);
}
/**
* \brief Prepare zones.
*
* This function prepares zones with actual buffer and some padded bytes.
* The actual ITER_BYTES bytes in zone is preceded by main buf and/or
* history buf and succeeded by padded bytes possibly from main buf,
* if available.
*/
static really_inline
size_t prepareZones(const u8 *buf, size_t len, const u8 *hend,
size_t start, const u8 *flood, struct zone *zoneArr) {
const u8 *ptr = buf + start;
size_t remaining = len - start;
if (remaining <= ITER_BYTES) {
/* enough bytes to make only one zone */
createShortZone(buf, hend, ptr, buf + len, &zoneArr[0]);
return 1;
}
/* enough bytes to make more than one zone */
size_t numZone = 0;
createStartZone(buf, hend, ptr, &zoneArr[numZone++]);
ptr += ITER_BYTES;
assert(ptr < buf + len);
/* find maximum buffer location that the main zone can scan
* - must be a multiple of ITER_BYTES, and
* - cannot contain the last 3 bytes (due to 3 bytes read behind the
end of buffer in FDR main loop)
*/
const u8 *main_end = buf + start + ROUNDDOWN_N(len - start - 3, ITER_BYTES);
/* create a zone if multiple of ITER_BYTES are found */
if (main_end > ptr) {
createMainZone(flood, ptr, main_end, &zoneArr[numZone++]);
ptr = main_end;
}
/* create a zone with rest of the data from the main buffer */
createEndZone(buf, ptr, buf + len, &zoneArr[numZone++]);
return numZone;
}
#define INVALID_MATCH_ID (~0U)
#define FDR_MAIN_LOOP(zz, s, get_conf_fn) \
do { \
const u8 *tryFloodDetect = zz->floodPtr; \
const u8 *start_ptr = zz->start; \
const u8 *end_ptr = zz->end; \
for (const u8 *itPtr = ROUNDDOWN_PTR(start_ptr, 64); itPtr + 4*ITER_BYTES <= end_ptr; \
itPtr += 4*ITER_BYTES) { \
__builtin_prefetch(itPtr); \
} \
\
for (const u8 *itPtr = start_ptr; itPtr + ITER_BYTES <= end_ptr; \
itPtr += ITER_BYTES) { \
if (unlikely(itPtr > tryFloodDetect)) { \
tryFloodDetect = floodDetect(fdr, a, &itPtr, tryFloodDetect,\
&floodBackoff, &control, \
ITER_BYTES); \
if (unlikely(control == HWLM_TERMINATE_MATCHING)) { \
return HWLM_TERMINATED; \
} \
} \
__builtin_prefetch(itPtr + ITER_BYTES); \
u64a conf0; \
u64a conf8; \
get_conf_fn(itPtr, start_ptr, end_ptr, domain_mask_flipped, \
ft, &conf0, &conf8, &s); \
do_confirm_fdr(&conf0, 0, &control, confBase, a, itPtr, \
&last_match_id, zz); \
do_confirm_fdr(&conf8, 8, &control, confBase, a, itPtr, \
&last_match_id, zz); \
if (unlikely(control == HWLM_TERMINATE_MATCHING)) { \
return HWLM_TERMINATED; \
} \
} /* end for loop */ \
} while (0) \
static never_inline
hwlm_error_t fdr_engine_exec(const struct FDR *fdr,
const struct FDR_Runtime_Args *a,
hwlm_group_t control) {
assert(ISALIGNED_CL(fdr));
u32 floodBackoff = FLOOD_BACKOFF_START;
u32 last_match_id = INVALID_MATCH_ID;
u32 domain_mask_flipped = ~fdr->domainMask;
u8 stride = fdr->stride;
const u64a *ft =
(const u64a *)((const u8 *)fdr + ROUNDUP_CL(sizeof(struct FDR)));
assert(ISALIGNED_CL(ft));
const u32 *confBase = (const u32 *)((const u8 *)fdr + fdr->confOffset);
assert(ISALIGNED_CL(confBase));
struct zone zones[ZONE_MAX];
assert(fdr->domain > 8 && fdr->domain < 16);
memset(zones, 0, sizeof(zones));
size_t numZone = prepareZones(a->buf, a->len,
a->buf_history + a->len_history,
a->start_offset, a->firstFloodDetect, zones);
assert(numZone <= ZONE_MAX);
m128 state = getInitState(fdr, a->len_history, ft, &zones[0]);
for (size_t curZone = 0; curZone < numZone; curZone++) {
struct zone *z = &zones[curZone];
dumpZoneInfo(z, curZone);
/* When a zone contains less data than is processed in an iteration
* of FDR_MAIN_LOOP(), we need to scan over some extra data.
*
* We have chosen to scan this extra data at the start of the
* iteration. The extra data is either data we have already scanned or
* garbage (if it is earlier than offset 0),
*
* As a result we need to shift the incoming state back so that it will
* properly line up with the data being scanned.
*
* We also need to forbid reporting any matches in the data being
* rescanned as they have already been reported (or are over garbage but
* later stages should also provide that safety guarantee).
*/
u8 shift = z->shift;
state = variable_byte_shift_m128(state, shift);
state = or128(state, load128(zone_or_mask[shift]));
switch (stride) {
case 1:
FDR_MAIN_LOOP(z, state, get_conf_stride_1);
break;
case 2:
FDR_MAIN_LOOP(z, state, get_conf_stride_2);
break;
case 4:
FDR_MAIN_LOOP(z, state, get_conf_stride_4);
break;
default:
break;
}
}
return HWLM_SUCCESS;
}
#if defined(HAVE_AVX2)
#define ONLY_AVX2(func) func
#else
#define ONLY_AVX2(func) NULL
#endif
typedef hwlm_error_t (*FDRFUNCTYPE)(const struct FDR *fdr,
const struct FDR_Runtime_Args *a,
hwlm_group_t control);
static const FDRFUNCTYPE funcs[] = {
fdr_engine_exec,
NULL, /* old: fast teddy */
NULL, /* old: fast teddy */
ONLY_AVX2(fdr_exec_fat_teddy_msks1),
ONLY_AVX2(fdr_exec_fat_teddy_msks1_pck),
ONLY_AVX2(fdr_exec_fat_teddy_msks2),
ONLY_AVX2(fdr_exec_fat_teddy_msks2_pck),
ONLY_AVX2(fdr_exec_fat_teddy_msks3),
ONLY_AVX2(fdr_exec_fat_teddy_msks3_pck),
ONLY_AVX2(fdr_exec_fat_teddy_msks4),
ONLY_AVX2(fdr_exec_fat_teddy_msks4_pck),
fdr_exec_teddy_msks1,
fdr_exec_teddy_msks1_pck,
fdr_exec_teddy_msks2,
fdr_exec_teddy_msks2_pck,
fdr_exec_teddy_msks3,
fdr_exec_teddy_msks3_pck,
fdr_exec_teddy_msks4,
fdr_exec_teddy_msks4_pck,
};
#define FAKE_HISTORY_SIZE 16
static const u8 fake_history[FAKE_HISTORY_SIZE];
hwlm_error_t fdrExec(const struct FDR *fdr, const u8 *buf, size_t len,
size_t start, HWLMCallback cb,
struct hs_scratch *scratch, hwlm_group_t groups) {
// We guarantee (for safezone construction) that it is safe to read 16
// bytes before the end of the history buffer.
const u8 *hbuf = fake_history + FAKE_HISTORY_SIZE;
const struct FDR_Runtime_Args a = {
buf,
len,
hbuf,
0,
start,
cb,
scratch,
nextFloodDetect(buf, len, FLOOD_BACKOFF_START),
0
};
if (unlikely(a.start_offset >= a.len)) {
return HWLM_SUCCESS;
} else {
assert(funcs[fdr->engineID]);
return funcs[fdr->engineID](fdr, &a, groups);
}
}
hwlm_error_t fdrExecStreaming(const struct FDR *fdr, const u8 *hbuf,
size_t hlen, const u8 *buf, size_t len,
size_t start, HWLMCallback cb,
struct hs_scratch *scratch,
hwlm_group_t groups) {
struct FDR_Runtime_Args a = {
buf,
len,
hbuf,
hlen,
start,
cb,
scratch,
nextFloodDetect(buf, len, FLOOD_BACKOFF_START),
/* we are guaranteed to always have 16 initialised bytes at the end of
* the history buffer (they may be garbage). */
hbuf ? unaligned_load_u64a(hbuf + hlen - sizeof(u64a)) : (u64a)0
};
hwlm_error_t ret;
if (unlikely(a.start_offset >= a.len)) {
ret = HWLM_SUCCESS;
} else {
assert(funcs[fdr->engineID]);
ret = funcs[fdr->engineID](fdr, &a, groups);
}
return ret;
}

85
src/fdr/fdr.h
View File

@ -1,85 +0,0 @@
/*
* Copyright (c) 2015-2017, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/** \file
* \brief FDR literal matcher: runtime API.
*/
#ifndef FDR_H
#define FDR_H
#include "ue2common.h"
#include "hwlm/hwlm.h"
// C linkage in the API
#ifdef __cplusplus
extern "C" {
#endif
struct FDR;
struct hs_scratch;
/**
* \brief Block-mode scan.
*
* \param fdr FDR matcher engine.
* \param buf Buffer to scan.
* \param len Length of buffer to scan.
* \param start First offset in buf at which a match may start.
* \param cb Callback to call when a match is found.
* \param scratch Scratch supplied to callback on match.
* \param groups Initial groups mask.
*/
hwlm_error_t fdrExec(const struct FDR *fdr, const u8 *buf, size_t len,
size_t start, HWLMCallback cb, struct hs_scratch *scratch,
hwlm_group_t groups);
/**
* \brief Streaming-mode scan.
*
* \param fdr FDR matcher engine.
* \param hbuf History buffer.
* \param hlen Length of history buffer (hbuf).
* \param buf Buffer to scan.
* \param len Length of buffer to scan (buf).
* \param start First offset in buf at which a match may start.
* \param cb Callback to call when a match is found.
* \param scratch Scratch supplied to callback on match.
* \param groups Initial groups mask.
*/
hwlm_error_t fdrExecStreaming(const struct FDR *fdr, const u8 *hbuf,
size_t hlen, const u8 *buf, size_t len,
size_t start, HWLMCallback cb,
struct hs_scratch *scratch,
hwlm_group_t groups);
#ifdef __cplusplus
}
#endif // __cplusplus
#endif // FDR_H

917
src/fdr/fdr_compile.cpp
View File

@ -1,917 +0,0 @@
/*
* Copyright (c) 2015-2019, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/** \file
* \brief FDR literal matcher: build API.
*/
#include "fdr_compile.h"
#include "fdr_internal.h"
#include "fdr_confirm.h"
#include "fdr_compile_internal.h"
#include "fdr_engine_description.h"
#include "teddy_compile.h"
#include "teddy_engine_description.h"
#include "grey.h"
#include "ue2common.h"
#include "hwlm/hwlm_build.h"
#include "util/compare.h"
#include "util/container.h"
#include "util/dump_mask.h"
#include "util/math.h"
#include "util/noncopyable.h"
#include "util/target_info.h"
#include "util/ue2string.h"
#include "util/verify_types.h"
#include <algorithm>
#include <array>
#include <cassert>
#include <cctype>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <limits>
#include <map>
#include <memory>
#include <numeric>
#include <set>
#include <string>
#include <unordered_map>
#include <unordered_set>
#include <vector>
#include <boost/multi_array.hpp>
using namespace std;
namespace ue2 {
namespace {
class FDRCompiler : noncopyable {
private:
const FDREngineDescription &eng;
const Grey &grey;
vector<u8> tab;
vector<hwlmLiteral> lits;
map<BucketIndex, std::vector<LiteralIndex> > bucketToLits;
bool make_small;
u8 *tabIndexToMask(u32 indexInTable);
#ifdef DEBUG
void dumpMasks(const u8 *defaultMask);
#endif
void setupTab();
bytecode_ptr<FDR> setupFDR();
void createInitialState(FDR *fdr);
public:
FDRCompiler(vector<hwlmLiteral> lits_in,
map<BucketIndex, std::vector<LiteralIndex>> bucketToLits_in,
const FDREngineDescription &eng_in,
bool make_small_in, const Grey &grey_in)
: eng(eng_in), grey(grey_in), tab(eng_in.getTabSizeBytes()),
lits(move(lits_in)), bucketToLits(move(bucketToLits_in)),
make_small(make_small_in) {}
bytecode_ptr<FDR> build();
};
u8 *FDRCompiler::tabIndexToMask(u32 indexInTable) {
assert(indexInTable < tab.size());
return &tab[0] + (indexInTable * (eng.getSchemeWidth() / 8));
}
static
void setbit(u8 *msk, u32 bit) {
msk[bit / 8] |= 1U << (bit % 8);
}
static
void clearbit(u8 *msk, u32 bit) {
msk[bit / 8] &= ~(1U << (bit % 8));
}
static
void andMask(u8 *dest, const u8 *a, const u8 *b, u32 num_bytes) {
for (u32 i = 0; i < num_bytes; i++) {
dest[i] = a[i] & b[i];
}
}
void FDRCompiler::createInitialState(FDR *fdr) {
u8 *start = (u8 *)&fdr->start;
/* initial state should to be 1 in each slot in the bucket up to bucket
* minlen - 1, and 0 thereafter */
for (BucketIndex b = 0; b < eng.getNumBuckets(); b++) {
// Find the minimum length for the literals in this bucket.
const vector<LiteralIndex> &bucket_lits = bucketToLits[b];
u32 min_len = ~0U;
for (const LiteralIndex &lit_idx : bucket_lits) {
min_len = min(min_len, verify_u32(lits[lit_idx].s.length()));
}
DEBUG_PRINTF("bucket %u has min_len=%u\n", b, min_len);
assert(min_len);
for (PositionInBucket i = 0; i < eng.getBucketWidth(b); i++) {
if (i < min_len - 1) {
setbit(start, eng.getSchemeBit(b, i));
}
}
}
}
/**
* \brief Lay out FDR structures in bytecode.
*
* Note that each major structure (header, table, confirm, flood control) is
* cacheline-aligned.
*/
bytecode_ptr<FDR> FDRCompiler::setupFDR() {
auto floodTable = setupFDRFloodControl(lits, eng, grey);
auto confirmTable = setupFullConfs(lits, eng, bucketToLits, make_small);
size_t headerSize = sizeof(FDR);
size_t tabSize = eng.getTabSizeBytes();
// Note: we place each major structure here on a cacheline boundary.
size_t size = ROUNDUP_CL(headerSize) + ROUNDUP_CL(tabSize) +
ROUNDUP_CL(confirmTable.size()) + floodTable.size();
DEBUG_PRINTF("sizes base=%zu tabSize=%zu confirm=%zu floodControl=%zu "
"total=%zu\n",
headerSize, tabSize, confirmTable.size(), floodTable.size(),
size);
auto fdr = make_zeroed_bytecode_ptr<FDR>(size, 64);
assert(fdr); // otherwise would have thrown std::bad_alloc
u8 *fdr_base = (u8 *)fdr.get();
// Write header.
fdr->size = size;
fdr->engineID = eng.getID();
fdr->maxStringLen = verify_u32(maxLen(lits));
fdr->numStrings = verify_u32(lits.size());
assert(eng.bits > 8 && eng.bits < 16); // we allow domains 9 to 15 only
fdr->domain = eng.bits;
fdr->domainMask = (1 << eng.bits) - 1;
fdr->tabSize = tabSize;
fdr->stride = eng.stride;
createInitialState(fdr.get());
// Write table.
u8 *ptr = fdr_base + ROUNDUP_CL(sizeof(FDR));
assert(ISALIGNED_CL(ptr));
copy(tab.begin(), tab.end(), ptr);
ptr += ROUNDUP_CL(tabSize);
// Write confirm structures.
assert(ISALIGNED_CL(ptr));
fdr->confOffset = verify_u32(ptr - fdr_base);
memcpy(ptr, confirmTable.get(), confirmTable.size());
ptr += ROUNDUP_CL(confirmTable.size());
// Write flood control structures.
assert(ISALIGNED_CL(ptr));
fdr->floodOffset = verify_u32(ptr - fdr_base);
memcpy(ptr, floodTable.get(), floodTable.size());
ptr += floodTable.size(); // last write, no need to round up
return fdr;
}
//#define DEBUG_ASSIGNMENT
/**
* Utility class for computing:
*
* score(count, len) = pow(count, 1.05) * pow(len, -3)
*
* Calling pow() is expensive. This is mitigated by using pre-computed LUTs for
* small inputs and a cache for larger ones.
*/
class Scorer {
unordered_map<u32, double> count_factor_cache;
// LUT: pow(count, 1.05) for small values of count.
static const array<double, 100> count_lut;
double count_factor(u32 count) {
if (count < count_lut.size()) {
return count_lut[count];
}
auto it = count_factor_cache.find(count);
if (it != count_factor_cache.end()) {
return it->second;
}
double r = our_pow(count, 1.05);
count_factor_cache.emplace(count, r);
return r;
}
// LUT: pow(len, -3) for len in range [0,8].
static const array<double, 9> len_lut;
double len_factor(u32 len) {
assert(len <= len_lut.size());
return len_lut[len];
}
public:
double operator()(u32 len, u32 count) {
if (len == 0) {
return numeric_limits<double>::max();
}
return count_factor(count) * len_factor(len);
}
};
const array<double, 100> Scorer::count_lut{{
pow(0, 1.05), pow(1, 1.05), pow(2, 1.05), pow(3, 1.05), pow(4, 1.05),
pow(5, 1.05), pow(6, 1.05), pow(7, 1.05), pow(8, 1.05), pow(9, 1.05),
pow(10, 1.05), pow(11, 1.05), pow(12, 1.05), pow(13, 1.05), pow(14, 1.05),
pow(15, 1.05), pow(16, 1.05), pow(17, 1.05), pow(18, 1.05), pow(19, 1.05),
pow(20, 1.05), pow(21, 1.05), pow(22, 1.05), pow(23, 1.05), pow(24, 1.05),
pow(25, 1.05), pow(26, 1.05), pow(27, 1.05), pow(28, 1.05), pow(29, 1.05),
pow(30, 1.05), pow(31, 1.05), pow(32, 1.05), pow(33, 1.05), pow(34, 1.05),
pow(35, 1.05), pow(36, 1.05), pow(37, 1.05), pow(38, 1.05), pow(39, 1.05),
pow(40, 1.05), pow(41, 1.05), pow(42, 1.05), pow(43, 1.05), pow(44, 1.05),
pow(45, 1.05), pow(46, 1.05), pow(47, 1.05), pow(48, 1.05), pow(49, 1.05),
pow(50, 1.05), pow(51, 1.05), pow(52, 1.05), pow(53, 1.05), pow(54, 1.05),
pow(55, 1.05), pow(56, 1.05), pow(57, 1.05), pow(58, 1.05), pow(59, 1.05),
pow(60, 1.05), pow(61, 1.05), pow(62, 1.05), pow(63, 1.05), pow(64, 1.05),
pow(65, 1.05), pow(66, 1.05), pow(67, 1.05), pow(68, 1.05), pow(69, 1.05),
pow(70, 1.05), pow(71, 1.05), pow(72, 1.05), pow(73, 1.05), pow(74, 1.05),
pow(75, 1.05), pow(76, 1.05), pow(77, 1.05), pow(78, 1.05), pow(79, 1.05),
pow(80, 1.05), pow(81, 1.05), pow(82, 1.05), pow(83, 1.05), pow(84, 1.05),
pow(85, 1.05), pow(86, 1.05), pow(87, 1.05), pow(88, 1.05), pow(89, 1.05),
pow(90, 1.05), pow(91, 1.05), pow(92, 1.05), pow(93, 1.05), pow(94, 1.05),
pow(95, 1.05), pow(96, 1.05), pow(97, 1.05), pow(98, 1.05), pow(99, 1.05),
}};
const array<double, 9> Scorer::len_lut{{
0, pow(1, -3.0), pow(2, -3.0), pow(3, -3.0), pow(4, -3.0),
pow(5, -3.0), pow(6, -3.0), pow(7, -3.0), pow(8, -3.0)}};
/**
* Returns true if the two given literals should be placed in the same chunk as
* they are identical except for a difference in caselessness.
*/
static
bool isEquivLit(const hwlmLiteral &a, const hwlmLiteral &b,
const hwlmLiteral *last_nocase_lit) {
const size_t a_len = a.s.size();
const size_t b_len = b.s.size();
if (a_len != b_len) {
return false;
}
bool nocase = last_nocase_lit && a_len == last_nocase_lit->s.size() &&
!cmp(a.s.c_str(), last_nocase_lit->s.c_str(), a_len, true);
return !cmp(a.s.c_str(), b.s.c_str(), a.s.size(), nocase);
}
struct Chunk {
Chunk(u32 first_id_in, u32 count_in, u32 length_in)
: first_id(first_id_in), count(count_in), length(length_in) {}
u32 first_id; //!< first id in this chunk
u32 count; //!< how many are in this chunk
u32 length; //!< how long things in the chunk are
};
static
vector<Chunk> assignChunks(const vector<hwlmLiteral> &lits,
const map<u32, u32> &lenCounts) {
const u32 CHUNK_MAX = 512;
const u32 MAX_CONSIDERED_LENGTH = 16;
// TODO: detailed early stage literal analysis for v. small cases (actually
// look at lits) yes - after we factor this out and merge in the Teddy
// style of building we can look at this, although the teddy merge
// modelling is quite different. It's still probably adaptable to some
// extent for this class of problem.
vector<Chunk> chunks;
chunks.reserve(CHUNK_MAX);
const u32 maxPerChunk = lits.size() /
(CHUNK_MAX - MIN(MAX_CONSIDERED_LENGTH, lenCounts.size())) + 1;
u32 currentSize = 0;
u32 chunkStartID = 0;
const hwlmLiteral *last_nocase_lit = nullptr;
for (u32 i = 0; i < lits.size() && chunks.size() < CHUNK_MAX - 1; i++) {
const auto &lit = lits[i];
DEBUG_PRINTF("i=%u, lit=%s%s\n", i, escapeString(lit.s).c_str(),
lit.nocase ? " (nocase)" : "");
// If this literal is identical to the last one (aside from differences
// in caselessness), keep going even if we will "overfill" a chunk; we
// don't want to split identical literals into different buckets.
if (i != 0 && isEquivLit(lit, lits[i - 1], last_nocase_lit)) {
DEBUG_PRINTF("identical lit\n");
goto next_literal;
}
if ((currentSize < MAX_CONSIDERED_LENGTH &&
(lit.s.size() != currentSize)) ||
(currentSize != 1 && ((i - chunkStartID) >= maxPerChunk))) {
currentSize = lit.s.size();
if (!chunks.empty()) {
chunks.back().count = i - chunkStartID;
}
chunkStartID = i;
chunks.emplace_back(i, 0, currentSize);
}
next_literal:
if (lit.nocase) {
last_nocase_lit = &lit;
}
}
assert(!chunks.empty());
chunks.back().count = lits.size() - chunkStartID;
// close off chunks with an empty row
chunks.emplace_back(lits.size(), 0, 0);
#ifdef DEBUG_ASSIGNMENT
for (size_t j = 0; j < chunks.size(); j++) {
const auto &chunk = chunks[j];
printf("chunk %zu first_id=%u count=%u length=%u\n", j, chunk.first_id,
chunk.count, chunk.length);
}
#endif
DEBUG_PRINTF("built %zu chunks (%zu lits)\n", chunks.size(), lits.size());
assert(chunks.size() <= CHUNK_MAX);
return chunks;
}
static
map<BucketIndex, vector<LiteralIndex>> assignStringsToBuckets(
vector<hwlmLiteral> &lits,
const FDREngineDescription &eng) {
const double MAX_SCORE = numeric_limits<double>::max();
assert(!lits.empty()); // Shouldn't be called with no literals.
// Count the number of literals for each length.
map<u32, u32> lenCounts;
for (const auto &lit : lits) {
lenCounts[lit.s.size()]++;
}
#ifdef DEBUG_ASSIGNMENT
for (const auto &m : lenCounts) {
printf("l<%u>:%u ", m.first, m.second);
}
printf("\n");
#endif
// Sort literals by literal length. If tied on length, use lexicographic
// ordering (of the reversed literals).
stable_sort(lits.begin(), lits.end(),
[](const hwlmLiteral &a, const hwlmLiteral &b) {
if (a.s.size() != b.s.size()) {
return a.s.size() < b.s.size();
}
auto p = mismatch(a.s.rbegin(), a.s.rend(), b.s.rbegin());
if (p.first != a.s.rend()) {
return *p.first < *p.second;
}
// Sort caseless variants first.
return a.nocase > b.nocase;
});
vector<Chunk> chunks = assignChunks(lits, lenCounts);
const u32 numChunks = chunks.size();
const u32 numBuckets = eng.getNumBuckets();
// 2D array of (score, chunk index) pairs, indexed by
// [chunk_index][bucket_index].
boost::multi_array<pair<double, u32>, 2> t(
boost::extents[numChunks][numBuckets]);
Scorer scorer;
for (u32 j = 0; j < numChunks; j++) {
u32 cnt = 0;
for (u32 k = j; k < numChunks; ++k) {
cnt += chunks[k].count;
}
t[j][0] = {scorer(chunks[j].length, cnt), 0};
}
for (u32 i = 1; i < numBuckets; i++) {
for (u32 j = 0; j < numChunks - 1; j++) { // don't do last, empty row
pair<double, u32> best = {MAX_SCORE, 0};
u32 cnt = chunks[j].count;
for (u32 k = j + 1; k < numChunks - 1; k++) {
auto score = scorer(chunks[j].length, cnt);
if (score > best.first) {
break; // now worse locally than our best score, give up
}
score += t[k][i-1].first;
if (score < best.first) {
best = {score, k};
}
cnt += chunks[k].count;
}
t[j][i] = best;
}
t[numChunks - 1][i] = {0,0}; // fill in empty final row for next iter
}
#ifdef DEBUG_ASSIGNMENT
for (u32 j = 0; j < numChunks; j++) {
printf("%03u: ", j);
for (u32 i = 0; i < numBuckets; i++) {
const auto &v = t[j][i];
printf("<%0.3f,%3d> ", v.first, v.second);
}
printf("\n");
}
#endif
// our best score is in t[0][N_BUCKETS-1] and we can follow the links
// to find where our buckets should start and what goes into them
vector<vector<LiteralIndex>> buckets;
for (u32 i = 0, n = numBuckets; n && (i != numChunks - 1); n--) {
u32 j = t[i][n - 1].second;
if (j == 0) {
j = numChunks - 1;
}
// put chunks between i - j into bucket (numBuckets - n).
u32 first_id = chunks[i].first_id;
u32 last_id = chunks[j].first_id;
assert(first_id < last_id);
UNUSED const auto &first_lit = lits[first_id];
UNUSED const auto &last_lit = lits[last_id - 1];
DEBUG_PRINTF("placing [%u-%u) in one bucket (%u lits, len %zu-%zu, "
"score %0.4f)\n",
first_id, last_id, last_id - first_id,
first_lit.s.length(), last_lit.s.length(),
scorer(first_lit.s.length(), last_id - first_id));
vector<LiteralIndex> litIds;
u32 cnt = last_id - first_id;
// long literals first for included literals checking
for (u32 k = 0; k < cnt; k++) {
litIds.emplace_back(last_id - k - 1);
}
i = j;
buckets.emplace_back(litIds);
}
// reverse bucket id, longer literals come first
map<BucketIndex, vector<LiteralIndex>> bucketToLits;
size_t bucketCnt = buckets.size();
for (size_t i = 0; i < bucketCnt; i++) {
bucketToLits.emplace(bucketCnt - i - 1, move(buckets[i]));
}
return bucketToLits;
}
#ifdef DEBUG
void FDRCompiler::dumpMasks(const u8 *defaultMask) {
const size_t width = eng.getSchemeWidth();
printf("default mask: %s\n", dumpMask(defaultMask, width).c_str());
for (u32 i = 0; i < eng.getNumTableEntries(); i++) {
u8 *m = tabIndexToMask(i);
if (memcmp(m, defaultMask, width / 8)) {
printf("tab %04x: %s\n", i, dumpMask(m, width).c_str());
}
}
}
#endif
static
bool getMultiEntriesAtPosition(const FDREngineDescription &eng,
const vector<LiteralIndex> &vl,
const vector<hwlmLiteral> &lits,
SuffixPositionInString pos,
map<u32, unordered_set<u32>> &m2) {
assert(eng.bits < 32);
u32 distance = 0;
if (eng.bits <= 8) {
distance = 1;
} else if (eng.bits <= 16) {
distance = 2;
} else {
distance = 4;
}
for (auto i = vl.begin(), e = vl.end(); i != e; ++i) {
if (e - i > 5) {
__builtin_prefetch(&lits[*(i + 5)]);
}
const hwlmLiteral &lit = lits[*i];
const size_t sz = lit.s.size();
u32 mask = 0;
u32 dontCares = 0;
for (u32 cnt = 0; cnt < distance; cnt++) {
int newPos = pos - cnt;
u8 dontCareByte = 0x0;
u8 maskByte = 0x0;
if (newPos < 0 || ((u32)newPos >= sz)) {
dontCareByte = 0xff;
} else {
u8 c = lit.s[sz - newPos - 1];
maskByte = c;
u32 remainder = eng.bits - cnt * 8;
assert(remainder != 0);
if (remainder < 8) {
u8 cmask = (1U << remainder) - 1;
maskByte &= cmask;
dontCareByte |= ~cmask;
}
if (lit.nocase && ourisalpha(c)) {
maskByte &= 0xdf;
dontCareByte |= 0x20;
}
}
u32 loc = cnt * 8;
mask |= maskByte << loc;
dontCares |= dontCareByte << loc;
}
// truncate m and dc down to nBits
mask &= (1U << eng.bits) - 1;
dontCares &= (1U << eng.bits) - 1;
if (dontCares == ((1U << eng.bits) - 1)) {
return true;
}
m2[dontCares].insert(mask);
}
return false;
}
void FDRCompiler::setupTab() {
const size_t mask_size = eng.getSchemeWidth() / 8;
assert(mask_size);
vector<u8> defaultMask(mask_size, 0xff);
for (u32 i = 0; i < eng.getNumTableEntries(); i++) {
memcpy(tabIndexToMask(i), &defaultMask[0], mask_size);
}
for (BucketIndex b = 0; b < eng.getNumBuckets(); b++) {
const vector<LiteralIndex> &vl = bucketToLits[b];
SuffixPositionInString pLimit = eng.getBucketWidth(b);
for (SuffixPositionInString pos = 0; pos < pLimit; pos++) {
u32 bit = eng.getSchemeBit(b, pos);
map<u32, unordered_set<u32>> m2;
bool done = getMultiEntriesAtPosition(eng, vl, lits, pos, m2);
if (done) {
clearbit(&defaultMask[0], bit);
continue;
}
for (const auto &elem : m2) {
u32 dc = elem.first;
const unordered_set<u32> &mskSet = elem.second;
u32 v = ~dc;
do {
u32 b2 = v & dc;
for (const u32 &mskVal : mskSet) {
u32 val = (mskVal & ~dc) | b2;
clearbit(tabIndexToMask(val), bit);
}
v = (v + (dc & -dc)) | ~dc;
} while (v != ~dc);
}
}
}
for (u32 i = 0; i < eng.getNumTableEntries(); i++) {
u8 *m = tabIndexToMask(i);
andMask(m, m, &defaultMask[0], mask_size);
}
#ifdef DEBUG
dumpMasks(&defaultMask[0]);
#endif
}
bytecode_ptr<FDR> FDRCompiler::build() {
setupTab();
return setupFDR();
}
static
bool isSuffix(const hwlmLiteral &lit1, const hwlmLiteral &lit2) {
const auto &s1 = lit1.s;
const auto &s2 = lit2.s;
size_t len1 = s1.length();
size_t len2 = s2.length();
assert(len1 >= len2);
if (lit1.nocase || lit2.nocase) {
return equal(s2.begin(), s2.end(), s1.begin() + len1 - len2,
[](char a, char b) { return mytoupper(a) == mytoupper(b); });
} else {
return equal(s2.begin(), s2.end(), s1.begin() + len1 - len2);
}
}
/*
* if lit2 is a suffix of lit1 but the case sensitivity, groups or mask info
* of lit2 is a subset of lit1, then lit1 can't squash lit2 and lit2 can
* possibly match when lit1 matches. In this case, we can't do bucket
* squashing. e.g. AAA(no case) in bucket 0, AA(no case) and aa in bucket 1,
* we can't squash bucket 1 if we have input like "aaa" as aa can also match.
*/
static
bool includedCheck(const hwlmLiteral &lit1, const hwlmLiteral &lit2) {
/* lit1 is caseless and lit2 is case sensitive */
if ((lit1.nocase && !lit2.nocase)) {
return true;
}
/* lit2's group is a subset of lit1 */
if (lit1.groups != lit2.groups &&
(lit2.groups == (lit1.groups & lit2.groups))) {
return true;
}
/* TODO: narrow down cases for mask check */
if (lit1.cmp != lit2.cmp || lit1.msk != lit2.msk) {
return true;
}
return false;
}
/*
* if lit2 is an included literal of both lit0 and lit1, then lit0 and lit1
* shouldn't match at the same offset, otherwise we give up squashing for lit1.
* e.g. lit0:AAA(no case), lit1:aa, lit2:A(no case). We can have duplicate
* matches for input "aaa" if lit0 and lit1 both squash lit2.
*/
static
bool checkParentLit(
const vector<hwlmLiteral> &lits, u32 pos1,
const unordered_set<u32> &parent_map,
const unordered_map<u32, unordered_set<u32>> &exception_map) {
assert(pos1 < lits.size());
const auto &lit1 = lits[pos1];
for (const auto pos2 : parent_map) {
if (contains(exception_map, pos2)) {
const auto &exception_pos = exception_map.at(pos2);
if (contains(exception_pos, pos1)) {
return false;
}
}
/* if lit1 isn't an exception of lit2, then we have to do further
* exclusive check.
* TODO: More mask checks. Note if two literals are group exclusive,
* it is possible that they match at the same offset. */
assert(pos2 < lits.size());
const auto &lit2 = lits[pos2];
if (isSuffix(lit2, lit1)) {
return false;
}
}
return true;
}
static
void buildSquashMask(vector<hwlmLiteral> &lits, u32 id1, u32 bucket1,
size_t start, const vector<pair<u32, u32>> &group,
unordered_map<u32, unordered_set<u32>> &parent_map,
unordered_map<u32, unordered_set<u32>> &exception_map) {
auto &lit1 = lits[id1];
DEBUG_PRINTF("b:%u len:%zu\n", bucket1, lit1.s.length());
size_t cnt = group.size();
bool included = false;
bool exception = false;
u32 child_id = ~0U;
for (size_t i = start; i < cnt; i++) {
u32 bucket2 = group[i].first;
assert(bucket2 >= bucket1);
u32 id2 = group[i].second;
auto &lit2 = lits[id2];
// check if lit2 is a suffix of lit1
if (isSuffix(lit1, lit2)) {
/* if we have a included literal in the same bucket,
* quit and let the included literal to do possible squashing */
if (bucket1 == bucket2) {
DEBUG_PRINTF("same bucket\n");
return;
}
/* if lit2 is a suffix but doesn't pass included checks for
* extra info, we give up sqaushing */
if (includedCheck(lit1, lit2)) {
DEBUG_PRINTF("find exceptional suffix %u\n", lit2.id);
exception_map[id1].insert(id2);
exception = true;
} else if (checkParentLit(lits, id1, parent_map[id2],
exception_map)) {
if (lit1.included_id == INVALID_LIT_ID) {
DEBUG_PRINTF("find suffix lit1 %u lit2 %u\n",
lit1.id, lit2.id);
lit1.included_id = lit2.id;
} else {
/* if we have multiple included literals in one bucket,
* give up squashing. */
DEBUG_PRINTF("multiple included literals\n");
lit1.included_id = INVALID_LIT_ID;
return;
}
child_id = id2;
included = true;
}
}
size_t next = i + 1;
u32 nextBucket = next < cnt ? group[next].first : ~0U;
if (bucket2 != nextBucket) {
if (included) {
if (exception) {
/* give up if we have exception literals
* in the same bucket as the included literal. */
lit1.included_id = INVALID_LIT_ID;
} else {
parent_map[child_id].insert(id1);
lit1.squash |= 1U << bucket2;
DEBUG_PRINTF("build squash mask %2x for %u\n",
lit1.squash, lit1.id);
}
return;
}
exception = false;
}
}
}
static constexpr u32 INCLUDED_LIMIT = 1000;
static
void findIncludedLits(vector<hwlmLiteral> &lits,
const vector<vector<pair<u32, u32>>> &lastCharMap) {
/* Map for finding the positions of literal which includes a literal
* in FDR hwlm literal vector. */
unordered_map<u32, unordered_set<u32>> parent_map;
/* Map for finding the positions of exception literals which could
* sometimes match if a literal matches in FDR hwlm literal vector. */
unordered_map<u32, unordered_set<u32>> exception_map;
for (const auto &group : lastCharMap) {
size_t cnt = group.size();
if (cnt > INCLUDED_LIMIT) {
continue;
}
for (size_t i = 0; i < cnt; i++) {
u32 bucket1 = group[i].first;
u32 id1 = group[i].second;
buildSquashMask(lits, id1, bucket1, i + 1, group, parent_map,
exception_map);
}
}
}
static
void addIncludedInfo(
vector<hwlmLiteral> &lits, u32 nBuckets,
map<BucketIndex, vector<LiteralIndex>> &bucketToLits) {
vector<vector<pair<u32, u32>>> lastCharMap(256);
for (BucketIndex b = 0; b < nBuckets; b++) {
if (!bucketToLits[b].empty()) {
for (const LiteralIndex &lit_idx : bucketToLits[b]) {
const auto &lit = lits[lit_idx];
u8 c = mytoupper(lit.s.back());
lastCharMap[c].emplace_back(b, lit_idx);
}
}
}
findIncludedLits(lits, lastCharMap);
}
} // namespace
static
unique_ptr<HWLMProto> fdrBuildProtoInternal(u8 engType,
vector<hwlmLiteral> &lits,
bool make_small,
const target_t &target,
const Grey &grey, u32 hint) {
DEBUG_PRINTF("cpu has %s\n", target.has_avx2() ? "avx2" : "no-avx2");
if (grey.fdrAllowTeddy) {
auto proto = teddyBuildProtoHinted(engType, lits, make_small, hint,
target);
if (proto) {
DEBUG_PRINTF("build with teddy succeeded\n");
return proto;
} else {
DEBUG_PRINTF("build with teddy failed, will try with FDR\n");
}
}
auto des = (hint == HINT_INVALID) ? chooseEngine(target, lits, make_small)
: getFdrDescription(hint);
if (!des) {
return nullptr;
}
// temporary hack for unit testing
if (hint != HINT_INVALID) {
des->bits = 9;
des->stride = 1;
}
auto bucketToLits = assignStringsToBuckets(lits, *des);
addIncludedInfo(lits, des->getNumBuckets(), bucketToLits);
auto proto =
std::make_unique<HWLMProto>(engType, move(des), lits, bucketToLits,
make_small);
return proto;
}
unique_ptr<HWLMProto> fdrBuildProto(u8 engType, vector<hwlmLiteral> lits,
bool make_small, const target_t &target,
const Grey &grey) {
return fdrBuildProtoInternal(engType, lits, make_small, target, grey,
HINT_INVALID);
}
static
bytecode_ptr<FDR> fdrBuildTableInternal(const HWLMProto &proto,
const Grey &grey) {
if (proto.teddyEng) {
return teddyBuildTable(proto, grey);
}
FDRCompiler fc(proto.lits, proto.bucketToLits, *(proto.fdrEng),
proto.make_small, grey);
return fc.build();
}
bytecode_ptr<FDR> fdrBuildTable(const HWLMProto &proto, const Grey &grey) {
return fdrBuildTableInternal(proto, grey);
}
#if !defined(RELEASE_BUILD)
unique_ptr<HWLMProto> fdrBuildProtoHinted(u8 engType,
vector<hwlmLiteral> lits,
bool make_small, u32 hint,
const target_t &target,
const Grey &grey) {
return fdrBuildProtoInternal(engType, lits, make_small, target, grey,
hint);
}
#endif
size_t fdrSize(const FDR *fdr) {
assert(fdr);
return fdr->size;
}
} // namespace ue2

72
src/fdr/fdr_compile.h
View File

@ -1,72 +0,0 @@
/*
* Copyright (c) 2015-2017, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/** \file
* \brief FDR literal matcher: build API.
*/
#ifndef FDR_COMPILE_H
#define FDR_COMPILE_H
#include "ue2common.h"
#include "hwlm/hwlm_build.h"
#include "util/bytecode_ptr.h"
#include <vector>
struct FDR;
namespace ue2 {
struct hwlmLiteral;
struct Grey;
struct target_t;
bytecode_ptr<FDR> fdrBuildTable(const HWLMProto &proto, const Grey &grey);
#if !defined(RELEASE_BUILD)
std::unique_ptr<HWLMProto> fdrBuildProtoHinted(
u8 engType,
std::vector<hwlmLiteral> lits,
bool make_small, u32 hint,
const target_t &target,
const Grey &grey);
#endif
std::unique_ptr<HWLMProto> fdrBuildProto(
u8 engType,
std::vector<hwlmLiteral> lits,
bool make_small, const target_t &target,
const Grey &grey);
/** \brief Returns size in bytes of the given FDR engine. */
size_t fdrSize(const struct FDR *fdr);
} // namespace ue2
#endif

87
src/fdr/fdr_compile_internal.h
View File

@ -1,87 +0,0 @@
/*
* Copyright (c) 2015-2017, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#ifndef FDR_COMPILE_INTERNAL_H
#define FDR_COMPILE_INTERNAL_H
#include "ue2common.h"
#include "hwlm/hwlm_literal.h"
#include "util/bytecode_ptr.h"
#include <map>
#include <utility>
#include <vector>
struct FDRConfirm;
struct LitInfo;
namespace ue2 {
// a pile of decorative typedefs
// good for documentation purposes more than anything else
typedef u32 LiteralIndex;
typedef u32 SuffixPositionInString; // zero is last byte, counting back
// into the string
typedef u32 BucketIndex;
typedef u32 SchemeBitIndex;
typedef u32 PositionInBucket; // zero is 'we are matching right now!",
// counting towards future matches
class EngineDescription;
class FDREngineDescription;
struct hwlmStreamingControl;
struct Grey;
bytecode_ptr<u8> setupFullConfs(
const std::vector<hwlmLiteral> &lits,
const EngineDescription &eng,
const std::map<BucketIndex, std::vector<LiteralIndex>> &bucketToLits,
bool make_small);
// all suffixes include an implicit max_bucket_width suffix to ensure that
// we always read a full-scale flood "behind" us in terms of what's in our
// state; if we don't have a flood that's long enough we won't be in the
// right state yet to allow blindly advancing
bytecode_ptr<u8> setupFDRFloodControl(const std::vector<hwlmLiteral> &lits,
const EngineDescription &eng,
const Grey &grey);
bytecode_ptr<u8>
fdrBuildTableStreaming(const std::vector<hwlmLiteral> &lits,
hwlmStreamingControl &stream_control);
static constexpr u32 HINT_INVALID = 0xffffffff;
// fdr_compile_util.cpp utilities
size_t maxLen(const std::vector<hwlmLiteral> &lits);
size_t minLenCount(const std::vector<hwlmLiteral> &lits, size_t *count);
u32 absdiff(u32 i, u32 j);
} // namespace ue2
#endif

65
src/fdr/fdr_compile_util.cpp
View File

@ -1,65 +0,0 @@
/*
* Copyright (c) 2015, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#include "fdr_compile_internal.h"
#include "hwlm/hwlm_literal.h"
#include <algorithm>
#include <vector>
using namespace std;
namespace ue2 {
size_t maxLen(const vector<hwlmLiteral> &lits) {
size_t rv = 0;
for (const auto &lit : lits) {
rv = max(rv, lit.s.size());
}
return rv;
}
size_t minLenCount(const vector<hwlmLiteral> &lits, size_t *count) {
size_t rv = (size_t)-1;
*count = 0;
for (const auto &lit : lits) {
if (lit.s.size() < rv) {
rv = lit.s.size();
*count = 1;
} else if (lit.s.size() == rv) {
(*count)++;
}
}
return rv;
}
u32 absdiff(u32 i, u32 j) {
return (i > j) ? (i - j) : (j - i);
}
} // namespace ue2

94
src/fdr/fdr_confirm.h
View File

@ -1,94 +0,0 @@
/*
* Copyright (c) 2015-2019, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#ifndef FDR_CONFIRM_H
#define FDR_CONFIRM_H
#include "ue2common.h"
#include "hwlm/hwlm.h"
static really_inline
u32 mul_hash_64(u64a lv, u64a andmsk, u64a mult, u32 nBits) {
return ((lv & andmsk) * mult) >> (sizeof(u64a)*8 - nBits);
}
// data structures
// TODO: fix this hard-coding
#define CONF_TYPE u64a
#define CONF_HASH_CALL mul_hash_64
/**
* \brief Flag indicating this literal doesn't need to be delivered more than
* once, used in LitInfo::flags.
*/
#define FDR_LIT_FLAG_NOREPEAT 1
/**
* \brief Structure describing a literal, linked to by FDRConfirm.
*
* This structure is followed in memory by a variable-sized string prefix, for
* strings that are longer than CONF_TYPE.
*/
struct LitInfo {
CONF_TYPE v;
CONF_TYPE msk;
hwlm_group_t groups;
u32 id; // literal ID as passed in
u8 size;
u8 flags; //!< bitfield of flags from FDR_LIT_FLAG_* above.
u8 next;
};
#define FDRC_FLAG_NO_CONFIRM 1
#define FDRC_FLAG_NOREPEAT 2
/**
* \brief FDR confirm header.
*
* This structure is followed in memory by:
*
* -# lit index mapping (array of u32)
* -# list of LitInfo structures
*/
struct FDRConfirm {
CONF_TYPE andmsk;
CONF_TYPE mult;
u32 nBits;
hwlm_group_t groups;
};
static really_inline
const u32 *getConfirmLitIndex(const struct FDRConfirm *fdrc) {
const u8 *base = (const u8 *)fdrc;
const u32 *litIndex =
(const u32 *)(base + ROUNDUP_N(sizeof(*fdrc), alignof(u32)));
assert(ISALIGNED(litIndex));
return litIndex;
}
#endif // FDR_CONFIRM_H

340
src/fdr/fdr_confirm_compile.cpp
View File

@ -1,340 +0,0 @@
/*
* Copyright (c) 2015-2019, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#include "fdr_internal.h"
#include "fdr_compile_internal.h"
#include "fdr_confirm.h"
#include "engine_description.h"
#include "teddy_engine_description.h"
#include "ue2common.h"
#include "util/alloc.h"
#include "util/bitutils.h"
#include "util/compare.h"
#include "util/container.h"
#include "util/verify_types.h"
#include <algorithm>
#include <cstring>
#include <set>
using namespace std;
namespace ue2 {
using BC2CONF = map<BucketIndex, bytecode_ptr<FDRConfirm>>;
static
u64a make_u64a_mask(const vector<u8> &v) {
assert(v.size() <= sizeof(u64a));
if (v.size() > sizeof(u64a)) {
throw std::exception();
}
u64a mask = 0;
size_t vlen = v.size();
size_t len = std::min(vlen, sizeof(mask));
unsigned char *m = (unsigned char *)&mask;
memcpy(m + sizeof(mask) - len, &v[vlen - len], len);
return mask;
}
/**
* Build a temporary vector of LitInfo structures (without the corresponding
* pointers to the actual strings; these cannot be laid out yet). These
* stay in 1:1 correspondence with the lits[] vector as that's the only
* place we have to obtain our full strings.
*/
static
void fillLitInfo(const vector<hwlmLiteral> &lits, vector<LitInfo> &tmpLitInfo,
CONF_TYPE &andmsk) {
const CONF_TYPE all_ones = ~(u64a)0;
andmsk = all_ones; // fill in with 'and' of all literal masks
for (LiteralIndex i = 0; i < lits.size(); i++) {
const hwlmLiteral &lit = lits[i];
LitInfo &info = tmpLitInfo[i];
memset(&info, 0, sizeof(info));
info.id = lit.id;
u8 flags = 0;
if (lit.noruns) {
flags |= FDR_LIT_FLAG_NOREPEAT;
}
info.flags = flags;
info.size = verify_u8(max(lit.msk.size(), lit.s.size()));
info.groups = lit.groups;
// these are built up assuming a LE machine
CONF_TYPE msk = all_ones;
CONF_TYPE val = 0;
for (u32 j = 0; j < sizeof(CONF_TYPE); j++) {
u32 shiftLoc = (sizeof(CONF_TYPE) - j - 1) * 8;
if (j >= lit.s.size()) {
msk &= ~((CONF_TYPE)0xff << shiftLoc);
} else {
u8 c = lit.s[lit.s.size() - j - 1];
if (lit.nocase && ourisalpha(c)) {
msk &= ~((CONF_TYPE)CASE_BIT << shiftLoc);
val |= (CONF_TYPE)(c & CASE_CLEAR) << shiftLoc;
} else {
val |= (CONF_TYPE)c << shiftLoc;
}
}
}
info.v = val;
info.msk = msk;
if (!lit.msk.empty()) {
u64a l_msk = make_u64a_mask(lit.msk);
u64a l_cmp = make_u64a_mask(lit.cmp);
// test for consistency - if there's intersection, then v and msk
// values must line up
UNUSED u64a intersection = l_msk & info.msk;
assert((info.v & intersection) == (l_cmp & intersection));
// incorporate lit.msk, lit.cmp into v and msk
info.msk |= l_msk;
info.v |= l_cmp;
}
andmsk &= info.msk;
}
}
//#define FDR_CONFIRM_DUMP 1
static
bytecode_ptr<FDRConfirm> getFDRConfirm(const vector<hwlmLiteral> &lits,
bool make_small) {
// Every literal must fit within CONF_TYPE.
assert(all_of_in(lits, [](const hwlmLiteral &lit) {
return lit.s.size() <= sizeof(CONF_TYPE);
}));
vector<LitInfo> tmpLitInfo(lits.size());
CONF_TYPE andmsk;
fillLitInfo(lits, tmpLitInfo, andmsk);
#ifdef FDR_CONFIRM_DUMP
printf("-------------------\n");
#endif
// just magic numbers and crude measures for now
u32 nBits;
if (make_small) {
nBits = min(10U, lg2(lits.size()) + 1);
} else {
nBits = lg2(lits.size()) + 4;
}
CONF_TYPE mult = (CONF_TYPE)0x0b4e0ef37bc32127ULL;
// we can walk the vector and assign elements from the vectors to a
// map by hash value
map<u32, vector<LiteralIndex> > res2lits;
hwlm_group_t gm = 0;
for (LiteralIndex i = 0; i < lits.size(); i++) {
LitInfo & li = tmpLitInfo[i];
u32 hash = CONF_HASH_CALL(li.v, andmsk, mult, nBits);
DEBUG_PRINTF("%016llx --> %u\n", li.v, hash);
res2lits[hash].emplace_back(i);
gm |= li.groups;
}
#ifdef FDR_CONFIRM_DUMP
// print out the literals reversed - makes it easier to line up analyses
// that are end-offset based
for (const auto &m : res2lits) {
const u32 &hash = m.first;
const vector<LiteralIndex> &vlidx = m.second;
if (vlidx.size() <= 1) {
continue;
}
printf("%x -> %zu literals\n", hash, vlidx.size());
size_t min_len = lits[vlidx.front()].s.size();
vector<set<u8>> vsl; // contains the set of chars at each location
// reversed from the end
for (const auto &litIdx : vlidx) {
const auto &lit = lits[litIdx];
if (lit.s.size() > vsl.size()) {
vsl.resize(lit.s.size());
}
for (size_t j = lit.s.size(); j != 0; j--) {
vsl[lit.s.size() - j].insert(lit.s[j - 1]);
}
min_len = min(min_len, lit.s.size());
}
printf("common ");
for (size_t j = 0; j < min_len; j++) {
if (vsl[j].size() == 1) {
printf("%02x", *vsl[j].begin());
} else {
printf("__");
}
}
printf("\n");
for (const auto &litIdx : vlidx) {
const auto &lit = lits[litIdx];
printf("%8x %c", lit.id, lit.nocase ? '!' : ' ');
for (size_t j = lit.s.size(); j != 0; j--) {
size_t dist_from_end = lit.s.size() - j;
if (dist_from_end < min_len && vsl[dist_from_end].size() == 1) {
printf("__");
} else {
printf("%02x", lit.s[j - 1]);
}
}
printf("\n");
}
size_t total_compares = 0;
for (const auto &v : vsl) {
total_compares += v.size();
}
size_t total_string_size = 0;
for (const auto &litIdx : vlidx) {
const auto &lit = lits[litIdx];
total_string_size += lit.s.size();
}
printf("Total compare load: %zu Total string size: %zu\n\n",
total_compares, total_string_size);
}
#endif
const size_t bitsToLitIndexSize = (1U << nBits) * sizeof(u32);
// this size can now be a worst-case as we can always be a bit smaller
size_t size = ROUNDUP_N(sizeof(FDRConfirm), alignof(u32)) +
ROUNDUP_N(bitsToLitIndexSize, alignof(LitInfo)) +
sizeof(LitInfo) * lits.size();
size = ROUNDUP_N(size, alignof(FDRConfirm));
auto fdrc = make_zeroed_bytecode_ptr<FDRConfirm>(size);
assert(fdrc); // otherwise would have thrown std::bad_alloc
fdrc->andmsk = andmsk;
fdrc->mult = mult;
fdrc->nBits = nBits;
fdrc->groups = gm;
// After the FDRConfirm, we have the lit index array.
u8 *fdrc_base = (u8 *)fdrc.get();
u8 *ptr = fdrc_base + sizeof(*fdrc);
ptr = ROUNDUP_PTR(ptr, alignof(u32));
u32 *bitsToLitIndex = (u32 *)ptr;
ptr += bitsToLitIndexSize;
// After the lit index array, we have the LitInfo structures themselves,
// which vary in size (as each may have a variable-length string after it).
ptr = ROUNDUP_PTR(ptr, alignof(LitInfo));
// Walk the map by hash value assigning indexes and laying out the
// elements (and their associated string confirm material) in memory.
for (const auto &m : res2lits) {
const u32 hash = m.first;
const vector<LiteralIndex> &vlidx = m.second;
bitsToLitIndex[hash] = verify_u32(ptr - fdrc_base);
for (auto i = vlidx.begin(), e = vlidx.end(); i != e; ++i) {
LiteralIndex litIdx = *i;
// Write LitInfo header.
LitInfo &finalLI = *(LitInfo *)ptr;
finalLI = tmpLitInfo[litIdx];
ptr += sizeof(LitInfo); // String starts directly after LitInfo.
assert(lits[litIdx].s.size() <= sizeof(CONF_TYPE));
if (next(i) == e) {
finalLI.next = 0;
} else {
finalLI.next = 1;
}
}
assert((size_t)(ptr - fdrc_base) <= size);
}
// Return actual used size, not worst-case size. Must be rounded up to
// FDRConfirm alignment so that the caller can lay out a sequence of these.
size_t actual_size = ROUNDUP_N((size_t)(ptr - fdrc_base),
alignof(FDRConfirm));
assert(actual_size <= size);
fdrc.shrink(actual_size);
return fdrc;
}
bytecode_ptr<u8>
setupFullConfs(const vector<hwlmLiteral> &lits,
const EngineDescription &eng,
const map<BucketIndex, vector<LiteralIndex>> &bucketToLits,
bool make_small) {
unique_ptr<TeddyEngineDescription> teddyDescr =
getTeddyDescription(eng.getID());
BC2CONF bc2Conf;
u32 totalConfirmSize = 0;
for (BucketIndex b = 0; b < eng.getNumBuckets(); b++) {
if (contains(bucketToLits, b)) {
vector<hwlmLiteral> vl;
for (const LiteralIndex &lit_idx : bucketToLits.at(b)) {
vl.emplace_back(lits[lit_idx]);
}
DEBUG_PRINTF("b %d sz %zu\n", b, vl.size());
auto fc = getFDRConfirm(vl, make_small);
totalConfirmSize += fc.size();
bc2Conf.emplace(b, move(fc));
}
}
u32 nBuckets = eng.getNumBuckets();
u32 totalConfSwitchSize = ROUNDUP_CL(nBuckets * sizeof(u32));
u32 totalSize = totalConfSwitchSize + totalConfirmSize;
auto buf = make_zeroed_bytecode_ptr<u8>(totalSize, 64);
assert(buf); // otherwise would have thrown std::bad_alloc
u32 *confBase = (u32 *)buf.get();
u8 *ptr = buf.get() + totalConfSwitchSize;
assert(ISALIGNED_CL(ptr));
for (const auto &m : bc2Conf) {
const BucketIndex &idx = m.first;
const bytecode_ptr<FDRConfirm> &p = m.second;
// confirm offset is relative to the base of this structure, now
u32 confirm_offset = verify_u32(ptr - buf.get());
memcpy(ptr, p.get(), p.size());
ptr += p.size();
confBase[idx] = confirm_offset;
}
return buf;
}
} // namespace ue2

104
src/fdr/fdr_confirm_runtime.h
View File

@ -1,104 +0,0 @@
/*
* Copyright (c) 2015-2019, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#ifndef FDR_CONFIRM_RUNTIME_H
#define FDR_CONFIRM_RUNTIME_H
#include "scratch.h"
#include "fdr_internal.h"
#include "fdr_loadval.h"
#include "hwlm/hwlm.h"
#include "ue2common.h"
#include "util/bitutils.h"
#include "util/compare.h"
// this is ordinary confirmation function which runs through
// the whole confirmation procedure
static really_inline
void confWithBit(const struct FDRConfirm *fdrc, const struct FDR_Runtime_Args *a,
size_t i, hwlmcb_rv_t *control, u32 *last_match,
u64a conf_key, u64a *conf, u8 bit) {
assert(i < a->len);
assert(i >= a->start_offset);
assert(ISALIGNED(fdrc));
const u8 * buf = a->buf;
u32 c = CONF_HASH_CALL(conf_key, fdrc->andmsk, fdrc->mult,
fdrc->nBits);
u32 start = getConfirmLitIndex(fdrc)[c];
if (likely(!start)) {
return;
}
const struct LitInfo *li
= (const struct LitInfo *)((const u8 *)fdrc + start);
struct hs_scratch *scratch = a->scratch;
assert(!scratch->fdr_conf);
scratch->fdr_conf = conf;
scratch->fdr_conf_offset = bit;
u8 oldNext; // initialized in loop
do {
assert(ISALIGNED(li));
if (unlikely((conf_key & li->msk) != li->v)) {
goto out;
}
if ((*last_match == li->id) && (li->flags & FDR_LIT_FLAG_NOREPEAT)) {
goto out;
}
const u8 *loc = buf + i - li->size + 1;
if (loc < buf) {
u32 full_overhang = buf - loc;
size_t len_history = a->len_history;
// can't do a vectored confirm either if we don't have
// the bytes
if (full_overhang > len_history) {
goto out;
}
}
assert(li->size <= sizeof(CONF_TYPE));
if (unlikely(!(li->groups & *control))) {
goto out;
}
*last_match = li->id;
*control = a->cb(i, li->id, scratch);
out:
oldNext = li->next; // oldNext is either 0 or an 'adjust' value
li++;
} while (oldNext);
scratch->fdr_conf = NULL;
}
#endif

210
src/fdr/fdr_dump.cpp
View File

@ -1,210 +0,0 @@
/*
* Copyright (c) 2015-2020, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#include "config.h"
#include "fdr_compile.h"
#include "fdr_compile_internal.h"
#include "fdr_confirm.h"
#include "fdr_dump.h"
#include "fdr_engine_description.h"
#include "fdr_internal.h"
#include "teddy_engine_description.h"
#include "teddy_internal.h"
#include "ue2common.h"
#include <cstdio>
#include <memory>
#ifndef DUMP_SUPPORT
#error No dump support!
#endif
using namespace std;
namespace ue2 {
static
bool fdrIsTeddy(const FDR *fdr) {
assert(fdr);
u32 engine = fdr->engineID;
/* teddys don't have an fdr engine description (which is why the dump code
* is so broken). */
return !getFdrDescription(engine);
}
static
void dumpLitIndex(const FDRConfirm *fdrc, FILE *f) {
const u32 *lit_index = getConfirmLitIndex(fdrc);
u32 num_lits = 1U << fdrc->nBits;
u32 lits_used = count_if(lit_index, lit_index + num_lits,
[](u32 idx) { return idx != 0; });
fprintf(f, " load %u/%u (%0.2f%%)\n", lits_used, num_lits,
(double)lits_used / (double)(num_lits)*100);
}
static
void dumpConfirms(const void *fdr_base, u32 conf_offset, u32 num_confirms,
FILE *f) {
const u32 *conf = (const u32 *)((const char *)fdr_base + conf_offset);
for (u32 i = 0; i < num_confirms; i++) {
const auto *fdrc = (const FDRConfirm *)((const char *)conf + conf[i]);
fprintf(f, " confirm %u\n", i);
fprintf(f, " andmsk 0x%016llx\n", fdrc->andmsk);
fprintf(f, " mult 0x%016llx\n", fdrc->mult);
fprintf(f, " nbits %u\n", fdrc->nBits);
fprintf(f, " groups 0x%016llx\n", fdrc->groups);
dumpLitIndex(fdrc, f);
}
}
static
void dumpTeddyReinforced(const u8 *rmsk, const u32 num_tables, FILE *f) {
// dump reinforcement masks
for (u32 b = 0; b < num_tables; b++) {
fprintf(f, " reinforcement table for bucket %u..%u:\n",
b * 8, b * 8 + 7);
for (u32 i = 0; i <= N_CHARS; i++) {
fprintf(f, " 0x%02x: ", i);
for (u32 j = 0; j < 8; j++) {
u8 val = rmsk[b * ((N_CHARS + 1) * 8) + i * 8 + j];
for (u32 k = 0; k < 8; k++) {
fprintf(f, "%s", ((val >> k) & 0x1) ? "1" : "0");
}
fprintf(f, " ");
}
fprintf(f, "\n");
}
fprintf(f, "\n");
}
}
static
void dumpTeddyDupMasks(const u8 *dmsk, u32 numMasks, FILE *f) {
// dump nibble masks
u32 maskWidth = 2;
fprintf(f, " dup nibble masks:\n");
for (u32 i = 0; i < numMasks * 2; i++) {
fprintf(f, " -%d%s: ", 1 + i / 2, (i % 2) ? "hi" : "lo");
for (u32 j = 0; j < 16 * maskWidth * 2; j++) {
u8 val = dmsk[i * 16 * maskWidth * 2 + j];
for (u32 k = 0; k < 8; k++) {
fprintf(f, "%s", ((val >> k) & 0x1) ? "1" : "0");
}
fprintf(f, " ");
}
fprintf(f, "\n");
}
fprintf(f, "\n");
}
static
void dumpTeddyMasks(const u8 *baseMsk, u32 numMasks, u32 maskWidth, FILE *f) {
// dump nibble masks
fprintf(f, " nibble masks:\n");
for (u32 i = 0; i < numMasks * 2; i++) {
fprintf(f, " -%d%s: ", 1 + i / 2, (i % 2) ? "hi" : "lo");
for (u32 j = 0; j < 16 * maskWidth; j++) {
u8 val = baseMsk[i * 16 * maskWidth + j];
for (u32 k = 0; k < 8; k++) {
fprintf(f, "%s", ((val >> k) & 0x1) ? "1" : "0");
}
fprintf(f, " ");
}
fprintf(f, "\n");
}
fprintf(f, "\n");
}
static
void dumpTeddy(const Teddy *teddy, FILE *f) {
fprintf(f, "TEDDY: %u\n", teddy->engineID);
auto des = getTeddyDescription(teddy->engineID);
if (!des) {
fprintf(f, " <unknown engine>\n");
return;
}
fprintf(f, " masks %u\n", des->numMasks);
fprintf(f, " buckets %u\n", des->getNumBuckets());
fprintf(f, " packed %s\n", des->packed ? "true" : "false");
fprintf(f, " strings %u\n", teddy->numStrings);
fprintf(f, " size %zu bytes\n", fdrSize((const FDR *)teddy));
fprintf(f, " max length %u\n", teddy->maxStringLen);
fprintf(f, " floodoff %u (%x)\n", teddy->floodOffset,
teddy->floodOffset);
fprintf(f, "\n");
u32 maskWidth = des->getNumBuckets() / 8;
size_t headerSize = sizeof(Teddy);
const u8 *teddy_base = (const u8 *)teddy;
const u8 *baseMsk = teddy_base + ROUNDUP_CL(headerSize);
dumpTeddyMasks(baseMsk, des->numMasks, maskWidth, f);
size_t maskLen = des->numMasks * 16 * 2 * maskWidth;
const u8 *rdmsk = baseMsk + ROUNDUP_CL(maskLen);
if (maskWidth == 1) { // reinforcement table in Teddy
dumpTeddyReinforced(rdmsk, maskWidth, f);
} else { // dup nibble mask table in Fat Teddy
assert(maskWidth == 2);
dumpTeddyDupMasks(rdmsk, des->numMasks, f);
}
dumpConfirms(teddy, teddy->confOffset, des->getNumBuckets(), f);
}
static
void dumpFDR(const FDR *fdr, FILE *f) {
fprintf(f, "FDR: %u\n", fdr->engineID);
auto des = getFdrDescription(fdr->engineID);
if (!des) {
fprintf(f, " <unknown engine>\n");
return;
}
fprintf(f, " domain %u\n", fdr->domain);
fprintf(f, " stride %u\n", fdr->stride);
fprintf(f, " strings %u\n", fdr->numStrings);
fprintf(f, " size %zu bytes\n", fdrSize(fdr));
fprintf(f, " max length %u\n", fdr->maxStringLen);
fprintf(f, " floodoff %u (%x)\n", fdr->floodOffset, fdr->floodOffset);
fprintf(f, "\n");
dumpConfirms(fdr, fdr->confOffset, des->getNumBuckets(), f);
}
void fdrPrintStats(const FDR *fdr, FILE *f) {
if (fdrIsTeddy(fdr)) {
dumpTeddy((const Teddy *)fdr, f);
} else {
dumpFDR(fdr, f);
}
}
} // namespace ue2

49
src/fdr/fdr_dump.h
View File

@ -1,49 +0,0 @@
/*
* Copyright (c) 2015, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/** \file
* \brief FDR literal matcher: dump API.
*/
#ifndef FDR_DUMP_H
#define FDR_DUMP_H
#if defined(DUMP_SUPPORT)
#include <cstdio>
struct FDR;
namespace ue2 {
void fdrPrintStats(const struct FDR *fdr, FILE *f);
} // namespace ue2
#endif // DUMP_SUPPORT
#endif // FDR_DUMP_H

227
src/fdr/fdr_engine_description.cpp
View File

@ -1,227 +0,0 @@
/*
* Copyright (c) 2015-2017, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#include "fdr_compile_internal.h"
#include "fdr_engine_description.h"
#include "hs_compile.h"
#include "util/target_info.h"
#include "util/compare.h" // for ourisalpha()
#include <cassert>
#include <cstdlib>
#include <map>
#include <string>
using namespace std;
namespace ue2 {
FDREngineDescription::FDREngineDescription(const FDREngineDef &def)
: EngineDescription(def.id, targetByArchFeatures(def.cpu_features),
def.numBuckets),
schemeWidth(def.schemeWidth), stride(0), bits(0) {}
u32 FDREngineDescription::getDefaultFloodSuffixLength() const {
// rounding up, so that scheme width 32 and 6 buckets is 6 not 5!
// the +1 avoids pain due to various reach choices
return ((getSchemeWidth() + getNumBuckets() - 1) / getNumBuckets()) + 1;
}
void getFdrDescriptions(vector<FDREngineDescription> *out) {
static const FDREngineDef def = {0, 64, 8, 0};
out->clear();
out->emplace_back(def);
}
static
u32 findDesiredStride(size_t num_lits, size_t min_len, size_t min_len_count) {
u32 desiredStride = 1; // always our safe fallback
if (min_len > 1) {
if (num_lits < 250) {
// small cases we just go for it
desiredStride = min_len;
} else if (num_lits < 800) {
// intermediate cases
desiredStride = min_len - 1;
} else if (num_lits < 5000) {
// for larger but not huge sizes, go to stride 2 only if we have at
// least minlen 3
desiredStride = MIN(min_len - 1, 2);
}
}
// patch if count is quite large - a ton of length 2 literals can
// break things
#ifdef TRY_THIS_LATER
if ((min_len == 2) && (desiredStride == 2) && (min_len_count > 20)) {
desiredStride = 1;
}
#endif
// patch stuff just for the stride 4 case; don't let min_len=4,
// desiredStride=4 through as even a few length 4 literals can break things
// (far more fragile)
if ((min_len == 4) && (desiredStride == 4) && (min_len_count > 2)) {
desiredStride = 2;
}
return desiredStride;
}
unique_ptr<FDREngineDescription> chooseEngine(const target_t &target,
const vector<hwlmLiteral> &vl,
bool make_small) {
vector<FDREngineDescription> allDescs;
getFdrDescriptions(&allDescs);
// find desired stride
size_t count;
size_t msl = minLenCount(vl, &count);
u32 desiredStride = findDesiredStride(vl.size(), msl, count);
DEBUG_PRINTF("%zu lits, msl=%zu, desiredStride=%u\n", vl.size(), msl,
desiredStride);
FDREngineDescription *best = nullptr;
u32 best_score = 0;
FDREngineDescription &eng = allDescs[0];
for (u32 domain = 9; domain <= 15; domain++) {
for (size_t stride = 1; stride <= 4; stride *= 2) {
// to make sure that domains >=14 have stride 1 according to origin
if (domain > 13 && stride > 1) {
continue;
}
if (!eng.isValidOnTarget(target)) {
continue;
}
if (msl < stride) {
continue;
}
u32 score = 100;
score -= absdiff(desiredStride, stride);
if (stride <= desiredStride) {
score += stride;
}
u32 effLits = vl.size(); /* * desiredStride;*/
u32 ideal;
if (effLits < eng.getNumBuckets()) {
if (stride == 1) {
ideal = 8;
} else {
ideal = 10;
}
} else if (effLits < 20) {
ideal = 10;
} else if (effLits < 100) {
ideal = 11;
} else if (effLits < 1000) {
ideal = 12;
} else if (effLits < 10000) {
ideal = 13;
} else {
ideal = 15;
}
if (ideal != 8 && eng.schemeWidth == 32) {
ideal += 1;
}
if (make_small) {
ideal -= 2;
}
if (stride > 1) {
ideal++;
}
DEBUG_PRINTF("effLits %u\n", effLits);
if (target.is_atom_class() && !make_small && effLits < 4000) {
/* Unless it is a very heavy case, we want to build smaller
* tables on lightweight machines due to their small caches. */
ideal -= 2;
}
score -= absdiff(ideal, domain);
DEBUG_PRINTF("fdr %u: width=%u, domain=%u, buckets=%u, stride=%zu "
"-> score=%u\n",
eng.getID(), eng.schemeWidth, domain,
eng.getNumBuckets(), stride, score);
if (!best || score > best_score) {
eng.bits = domain;
eng.stride = stride;
best = &eng;
best_score = score;
}
}
}
if (!best) {
DEBUG_PRINTF("failed to find engine\n");
return nullptr;
}
DEBUG_PRINTF("using engine %u\n", best->getID());
return std::make_unique<FDREngineDescription>(*best);
}
SchemeBitIndex FDREngineDescription::getSchemeBit(BucketIndex b,
PositionInBucket p) const {
assert(p < getBucketWidth(b));
SchemeBitIndex sbi = p * getNumBuckets() + b;
assert(sbi < getSchemeWidth());
return sbi;
}
u32 FDREngineDescription::getBucketWidth(BucketIndex) const {
u32 sw = getSchemeWidth();
u32 nm = getNumBuckets();
assert(sw % nm == 0);
return sw/nm;
}
unique_ptr<FDREngineDescription> getFdrDescription(u32 engineID) {
vector<FDREngineDescription> allDescs;
getFdrDescriptions(&allDescs);
if (engineID >= allDescs.size()) {
return nullptr;
}
return std::make_unique<FDREngineDescription>(allDescs[engineID]);
}
} // namespace ue2

73
src/fdr/fdr_engine_description.h
View File

@ -1,73 +0,0 @@
/*
* Copyright (c) 2015-2017, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#ifndef FDR_ENGINE_DESCRIPTION_H
#define FDR_ENGINE_DESCRIPTION_H
#include "engine_description.h"
#include <map>
#include <memory>
#include <vector>
namespace ue2 {
struct FDREngineDef {
u32 id;
u32 schemeWidth;
u32 numBuckets;
u64a cpu_features;
};
class FDREngineDescription : public EngineDescription {
public:
u32 schemeWidth;
u32 stride;
u32 bits;
u32 getSchemeWidth() const { return schemeWidth; }
u32 getBucketWidth(BucketIndex b) const;
SchemeBitIndex getSchemeBit(BucketIndex b, PositionInBucket p) const;
u32 getNumTableEntries() const { return 1 << bits; }
u32 getTabSizeBytes() const {
return schemeWidth / 8 * getNumTableEntries();
}
explicit FDREngineDescription(const FDREngineDef &def);
u32 getDefaultFloodSuffixLength() const override;
};
std::unique_ptr<FDREngineDescription>
chooseEngine(const target_t &target, const std::vector<hwlmLiteral> &vl,
bool make_small);
std::unique_ptr<FDREngineDescription> getFdrDescription(u32 engineID);
void getFdrDescriptions(std::vector<FDREngineDescription> *out);
} // namespace ue2
#endif

105
src/fdr/fdr_internal.h
View File

@ -1,105 +0,0 @@
/*
* Copyright (c) 2015-2017, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/** \file
* \brief FDR literal matcher: data structures.
*/
#ifndef FDR_INTERNAL_H
#define FDR_INTERNAL_H
#include "ue2common.h"
#include "hwlm/hwlm.h" // for hwlm_group_t, HWLMCallback
struct hs_scratch;
typedef enum {
NOT_CAUTIOUS, //!< not near a boundary (quantify?)
VECTORING //!< potentially vectoring
} CautionReason;
/** \brief number of different ids that can be triggered by floods of any given
* character. */
#define FDR_FLOOD_MAX_IDS 16
struct FDRFlood {
hwlm_group_t allGroups; //!< all the groups or'd together
u32 suffix;
/** \brief 0 to FDR_FLOOD_MAX_IDS-1 ids that are generated once per char on
* a flood.
* If larger we won't handle this through the flood path at all. */
u16 idCount;
u32 ids[FDR_FLOOD_MAX_IDS]; //!< the ids
hwlm_group_t groups[FDR_FLOOD_MAX_IDS]; //!< group ids to go with string ids
};
/** \brief FDR structure.
*
* 1. struct as-is
* 2. primary matching table
* 3. confirm stuff
*/
struct FDR {
u32 engineID;
u32 size;
u32 maxStringLen;
u32 numStrings;
u32 confOffset;
u32 floodOffset;
u8 stride; /* stride - how frequently the data is consulted by the first
* stage matcher */
u8 domain; /* number of bits used to index into main FDR table. This value
* is used only of debugging/asserts. */
u16 domainMask; /* pre-computed domain mask */
u32 tabSize; /* pre-computed hashtable size in bytes */
m128 start; /* initial start state to use at offset 0. The state has been
* set up based on the min length of buckets to reduce the need
* for pointless confirms. */
};
/** \brief FDR runtime arguments.
*
* This structure handles read-only things that are passed extensively around
* the FDR run-time functions. They are set by the API, passed by value into
* the main function, then a pointer is passed around to all the various
* sub-functions (confirm & flood). */
struct FDR_Runtime_Args {
const u8 *buf;
size_t len;
const u8 *buf_history;
size_t len_history;
size_t start_offset;
HWLMCallback cb;
struct hs_scratch *scratch;
const u8 *firstFloodDetect;
const u64a histBytes;
};
#endif

71
src/fdr/fdr_loadval.h
View File

@ -1,71 +0,0 @@
/*
* Copyright (c) 2015-2016, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#ifndef FDR_LOADVAL_H
#define FDR_LOADVAL_H
#include "ue2common.h"
#include "util/unaligned.h"
#define MAKE_LOADVAL(type, name) \
static really_inline \
type name(const u8 *ptr, UNUSED const u8 *lo, UNUSED const u8 *hi)
#define NORMAL_SAFE(type) \
do { \
assert(ptr >= lo); \
assert(ptr + sizeof(type) - 1 < hi); \
} while(0)
#define MAKE_LOOP_CE(TYPE) \
TYPE v = 0; \
for (TYPE i = 0; i < sizeof(TYPE); i++) { \
if ((lo <= ptr + i) && (ptr + i < hi)) { \
v += (TYPE)ptr[i] << (i*8); \
} \
} \
return v;
// no suffix = normal (unaligned)
// _ce = cautious everywhere (in both directions); test against hi and lo
MAKE_LOADVAL(u16, lv_u16) {
NORMAL_SAFE(u16);
return unaligned_load_u16(ptr);
}
MAKE_LOADVAL(u64a, lv_u64a) {
NORMAL_SAFE(u32);
return unaligned_load_u64a(ptr);
}
MAKE_LOADVAL(u16, lv_u16_ce) { MAKE_LOOP_CE(u16); }
MAKE_LOADVAL(u64a, lv_u64a_ce) { MAKE_LOOP_CE(u64a); }
#endif

231
src/fdr/flood_compile.cpp
View File

@ -1,231 +0,0 @@
/*
* Copyright (c) 2015-2017, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#include "fdr_internal.h"
#include "fdr_confirm.h"
#include "fdr_compile_internal.h"
#include "fdr_engine_description.h"
#include "grey.h"
#include "ue2common.h"
#include "util/alloc.h"
#include "util/bitutils.h"
#include "util/charreach.h"
#include "util/compare.h"
#include "util/ue2string.h"
#include "util/verify_types.h"
#include <cstring>
#include <map>
#include <memory>
#include <string>
#include <vector>
using namespace std;
namespace ue2 {
namespace {
struct FloodComparator {
bool operator()(const FDRFlood &f1, const FDRFlood &f2) const {
return std::memcmp(&f1, &f2, sizeof(f1)) < 0;
}
};
}
static
bool isDifferent(u8 oldC, u8 c, bool caseless) {
if (caseless) {
return mytolower(oldC) != mytolower(c);
} else {
return oldC != c;
}
}
static
void updateFloodSuffix(vector<FDRFlood> &tmpFlood, u8 c, u32 suffix) {
FDRFlood &fl = tmpFlood[c];
fl.suffix = MAX(fl.suffix, suffix + 1);
DEBUG_PRINTF("Updated Flood Suffix for char 0x%02x to %u\n", c, fl.suffix);
}
static
void addFlood(vector<FDRFlood> &tmpFlood, u8 c, const hwlmLiteral &lit,
u32 suffix) {
FDRFlood &fl = tmpFlood[c];
fl.suffix = MAX(fl.suffix, suffix + 1);
if (fl.idCount < FDR_FLOOD_MAX_IDS) {
fl.ids[fl.idCount] = lit.id;
fl.allGroups |= lit.groups;
fl.groups[fl.idCount] = lit.groups;
// when idCount gets to max_ids this flood no longer happens
// only incremented one more time to avoid arithmetic overflow
DEBUG_PRINTF("Added Flood for char '%c' suffix=%u len[%hu]=%u\n",
c, fl.suffix, fl.idCount, suffix);
fl.idCount++;
}
}
bytecode_ptr<u8> setupFDRFloodControl(const vector<hwlmLiteral> &lits,
const EngineDescription &eng,
const Grey &grey) {
vector<FDRFlood> tmpFlood(N_CHARS);
u32 default_suffix = eng.getDefaultFloodSuffixLength();
// zero everything to avoid spurious distinctions in the compares
memset(&tmpFlood[0], 0, N_CHARS * sizeof(FDRFlood));
for (u32 c = 0; c < N_CHARS; c++) {
tmpFlood[c].suffix = default_suffix;
}
for (const auto &lit : lits) {
DEBUG_PRINTF("lit: '%s'%s\n", escapeString(lit.s).c_str(),
lit.nocase ? " (nocase)" : "");
u32 litSize = verify_u32(lit.s.size());
u32 maskSize = (u32)lit.msk.size();
u8 c = lit.s[litSize - 1];
bool nocase = ourisalpha(c) ? lit.nocase : false;
if (nocase && maskSize && (lit.msk[maskSize - 1] & CASE_BIT)) {
c = (lit.cmp[maskSize - 1] & CASE_BIT) ? mytolower(c) : mytoupper(c);
nocase = false;
}
u32 iEnd = MAX(litSize, maskSize);
u32 upSuffix = iEnd; // upSuffix is used as an upper case suffix length
// for case-less, or as a suffix length for case-sensitive;
u32 loSuffix = iEnd; // loSuffix used only for case-less as a lower case suffix
// length;
for (u32 i = 0; i < iEnd; i++) {
if (i < litSize) {
if (isDifferent(c, lit.s[litSize - i - 1], lit.nocase)) {
DEBUG_PRINTF("non-flood char in literal[%u]: "
"0x%02x != 0x%02x\n",
i, c, lit.s[litSize - i - 1]);
upSuffix = MIN(upSuffix, i);
loSuffix = MIN(loSuffix, i); // makes sense only for case-less
break;
}
}
if (i < maskSize) {
u8 m = lit.msk[maskSize - i - 1];
u8 cm = lit.cmp[maskSize - i - 1] & m;
if(nocase) {
if ((mytoupper(c) & m) != cm) {
DEBUG_PRINTF("non-flood char in mask[%u] %c != %c\n",
i, mytoupper(c), cm);
upSuffix = MIN(upSuffix, i);
}
if ((mytolower(c) & m) != cm) {
DEBUG_PRINTF("non-flood char in mask[%u] %c != %c\n",
i, mytolower(c), cm);
loSuffix = MIN(loSuffix, i);
}
if (loSuffix != iEnd && upSuffix != iEnd) {
break;
}
} else if ((c & m) != cm) {
DEBUG_PRINTF("non-flood char in mask[%u] %c != %c\n", i, c, cm);
upSuffix = MIN(upSuffix, i);
break;
}
}
}
if(upSuffix != iEnd) {
updateFloodSuffix(tmpFlood, nocase ? mytoupper(c) : c, upSuffix);
} else {
addFlood(tmpFlood, nocase ? mytoupper(c) : c, lit, upSuffix);
}
if (nocase) {
if(loSuffix != iEnd) {
updateFloodSuffix(tmpFlood, mytolower(c), loSuffix);
} else {
addFlood(tmpFlood, mytolower(c), lit, loSuffix);
}
}
}
#ifdef DEBUG
for (u32 i = 0; i < N_CHARS; i++) {
FDRFlood &fl = tmpFlood[i];
if (!fl.idCount) {
continue;
}
printf("i is %02x fl->idCount is %hd fl->suffix is %d fl->allGroups is "
"%016llx\n", i, fl.idCount, fl.suffix, fl.allGroups);
for (u32 j = 0; j < fl.idCount; j++) {
printf("j is %d fl.groups[j] %016llx\n", j, fl.groups[j]);
}
}
#endif
// If flood detection has been switched off in the grey box, we comply by
// setting idCount too high for all floods.
if (!grey.fdrAllowFlood) {
for (auto &fl : tmpFlood) {
fl.idCount = FDR_FLOOD_MAX_IDS;
}
}
map<FDRFlood, CharReach, FloodComparator> flood2chars;
for (u32 i = 0; i < N_CHARS; i++) {
FDRFlood fl = tmpFlood[i];
flood2chars[fl].set(i);
}
u32 nDistinctFloods = flood2chars.size();
size_t floodHeaderSize = sizeof(u32) * N_CHARS;
size_t floodStructSize = sizeof(FDRFlood) * nDistinctFloods;
size_t totalSize = ROUNDUP_16(floodHeaderSize + floodStructSize);
auto buf = make_zeroed_bytecode_ptr<u8>(totalSize, 16);
assert(buf); // otherwise would have thrown std::bad_alloc
u32 *floodHeader = (u32 *)buf.get();
FDRFlood *layoutFlood = (FDRFlood *)(buf.get() + floodHeaderSize);
u32 currentFloodIndex = 0;
for (const auto &m : flood2chars) {
const FDRFlood &fl = m.first;
const CharReach &cr = m.second;
layoutFlood[currentFloodIndex] = fl;
for (size_t c = cr.find_first(); c != cr.npos; c = cr.find_next(c)) {
floodHeader[c] = currentFloodIndex;
}
currentFloodIndex++;
}
DEBUG_PRINTF("made a flood structure with %zu + %zu = %zu\n",
floodHeaderSize, floodStructSize, totalSize);
return buf;
}
} // namespace ue2

337
src/fdr/flood_runtime.h
View File

@ -1,337 +0,0 @@
/*
* Copyright (c) 2015-2017, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#ifndef FLOOD_RUNTIME
#define FLOOD_RUNTIME
#if defined(ARCH_64_BIT)
#define FLOOD_64
#else
#define FLOOD_32
#endif
#define FLOOD_MINIMUM_SIZE 256
#define FLOOD_BACKOFF_START 32
static really_inline
const u8 * nextFloodDetect(const u8 * buf, size_t len, u32 floodBackoff) {
// if we don't have a flood at either the start or end,
// or have a very small buffer, don't bother with flood detection
if (len < FLOOD_MINIMUM_SIZE) {
return buf + len;
}
/* entry points in runtime.c prefetch relevant data */
#ifndef FLOOD_32
u64a x11 = *(const u64a *)ROUNDUP_PTR(buf, 8);
u64a x12 = *(const u64a *)ROUNDUP_PTR(buf+8, 8);
if (x11 == x12) {
return buf + floodBackoff;
}
u64a x21 = *(const u64a *)ROUNDUP_PTR(buf + len/2, 8);
u64a x22 = *(const u64a *)ROUNDUP_PTR(buf + len/2 + 8, 8);
if (x21 == x22) {
return buf + floodBackoff;
}
u64a x31 = *(const u64a *)ROUNDUP_PTR(buf + len - 24, 8);
u64a x32 = *(const u64a *)ROUNDUP_PTR(buf + len - 16, 8);
if (x31 == x32) {
return buf + floodBackoff;
}
#else
u32 x11 = *(const u32 *)ROUNDUP_PTR(buf, 4);
u32 x12 = *(const u32 *)ROUNDUP_PTR(buf+4, 4);
if (x11 == x12) {
return buf + floodBackoff;
}
u32 x21 = *(const u32 *)ROUNDUP_PTR(buf + len/2, 4);
u32 x22 = *(const u32 *)ROUNDUP_PTR(buf + len/2 + 4, 4);
if (x21 == x22) {
return buf + floodBackoff;
}
u32 x31 = *(const u32 *)ROUNDUP_PTR(buf + len - 12, 4);
u32 x32 = *(const u32 *)ROUNDUP_PTR(buf + len - 8, 4);
if (x31 == x32) {
return buf + floodBackoff;
}
#endif
return buf + len;
}
static really_inline
const u8 * floodDetect(const struct FDR * fdr,
const struct FDR_Runtime_Args * a,
const u8 ** ptrPtr,
const u8 * tryFloodDetect,
u32 * floodBackoffPtr,
hwlmcb_rv_t * control,
u32 iterBytes) {
DEBUG_PRINTF("attempting flood detection at %p\n", tryFloodDetect);
const u8 * buf = a->buf;
const size_t len = a->len;
HWLMCallback cb = a->cb;
struct hs_scratch *scratch = a->scratch;
const u8 * ptr = *ptrPtr;
// tryFloodDetect is never put in places where unconditional
// reads a short distance forward or backward here
// TODO: rationale for this line needs to be rediscovered!!
size_t mainLoopLen = len > 2 * iterBytes ? len - 2 * iterBytes : 0;
const u32 i = ptr - buf;
u32 j = i;
// go from c to our FDRFlood structure
u8 c = buf[i];
const u8 * fBase = ((const u8 *)fdr) + fdr->floodOffset;
u32 fIdx = ((const u32 *)fBase)[c];
const struct FDRFlood * fsb = (const struct FDRFlood *)(fBase + sizeof(u32) * 256);
const struct FDRFlood * fl = &fsb[fIdx];
#ifndef FLOOD_32
u64a cmpVal = c;
cmpVal |= cmpVal << 8;
cmpVal |= cmpVal << 16;
cmpVal |= cmpVal << 32;
u64a probe = *(const u64a *)ROUNDUP_PTR(buf+i, 8);
#else
u32 cmpVal = c;
cmpVal |= cmpVal << 8;
cmpVal |= cmpVal << 16;
u32 probe = *(const u32 *)ROUNDUP_PTR(buf+i, 4);
#endif
if ((probe != cmpVal) || (fl->idCount >= FDR_FLOOD_MAX_IDS)) {
*floodBackoffPtr *= 2;
goto floodout;
}
if (i < fl->suffix + 7) {
*floodBackoffPtr *= 2;
goto floodout;
}
j = i - fl->suffix;
#ifndef FLOOD_32
j -= (u32)((uintptr_t)buf + j) & 0x7; // push j back to yield 8-aligned addrs
for (; j + 32 < mainLoopLen; j += 32) {
u64a v = *(const u64a *)(buf + j);
u64a v2 = *(const u64a *)(buf + j + 8);
u64a v3 = *(const u64a *)(buf + j + 16);
u64a v4 = *(const u64a *)(buf + j + 24);
if ((v4 != cmpVal) || (v3 != cmpVal) || (v2 != cmpVal) || (v != cmpVal)) {
break;
}
}
for (; j + 8 < mainLoopLen; j += 8) {
u64a v = *(const u64a *)(buf + j);
if (v != cmpVal) {
break;
}
}
#else
j -= (u32)((size_t)buf + j) & 0x3; // push j back to yield 4-aligned addrs
for (; j + 16 < mainLoopLen; j += 16) {
u32 v = *(const u32 *)(buf + j);
u32 v2 = *(const u32 *)(buf + j + 4);
u32 v3 = *(const u32 *)(buf + j + 8);
u32 v4 = *(const u32 *)(buf + j + 12);
if ((v4 != cmpVal) || (v3 != cmpVal) || (v2 != cmpVal) || (v != cmpVal)) {
break;
}
}
for (; j + 4 < mainLoopLen; j += 4) {
u32 v = *(const u32 *)(buf + j);
if (v != cmpVal) {
break;
}
}
#endif
for (; j < mainLoopLen; j++) {
u8 v = *(const u8 *)(buf + j);
if (v != c) {
break;
}
}
if (j > i ) {
j--; // needed for some reaches
u32 itersAhead = (j-i)/iterBytes;
u32 floodSize = itersAhead*iterBytes;
DEBUG_PRINTF("flooding %u size j %u i %u fl->idCount %hu "
"*control %016llx fl->allGroups %016llx\n",
floodSize, j, i, fl->idCount, *control, fl->allGroups);
DEBUG_PRINTF("mainloopLen %zu mainStart ??? mainEnd ??? len %zu\n",
mainLoopLen, len);
if (fl->idCount && (*control & fl->allGroups)) {
switch (fl->idCount) {
#if !defined(FLOOD_DEBUG)
// Carefully unrolled code
case 1:
for (u32 t = 0; t < floodSize && (*control & fl->allGroups);
t += 4) {
DEBUG_PRINTF("aaa %u %llx\n", t, fl->groups[0]);
if (*control & fl->groups[0]) {
*control = cb(i + t + 0, fl->ids[0], scratch);
}
if (*control & fl->groups[0]) {
*control = cb(i + t + 1, fl->ids[0], scratch);
}
if (*control & fl->groups[0]) {
*control = cb(i + t + 2, fl->ids[0], scratch);
}
if (*control & fl->groups[0]) {
*control = cb(i + t + 3, fl->ids[0], scratch);
}
}
break;
case 2:
for (u32 t = 0; t < floodSize && (*control & fl->allGroups); t += 4) {
if (*control & fl->groups[0]) {
*control = cb(i + t, fl->ids[0], scratch);
}
if (*control & fl->groups[1]) {
*control = cb(i + t, fl->ids[1], scratch);
}
if (*control & fl->groups[0]) {
*control =
cb(i + t + 1, fl->ids[0], scratch);
}
if (*control & fl->groups[1]) {
*control = cb(i + t + 1, fl->ids[1], scratch);
}
if (*control & fl->groups[0]) {
*control = cb(i + t + 2, fl->ids[0], scratch);
}
if (*control & fl->groups[1]) {
*control = cb(i + t + 2, fl->ids[1], scratch);
}
if (*control & fl->groups[0]) {
*control = cb(i + t + 3, fl->ids[0], scratch);
}
if (*control & fl->groups[1]) {
*control = cb(i + t + 3, fl->ids[1], scratch);
}
}
break;
case 3:
for (u32 t = 0; t < floodSize && (*control & fl->allGroups); t += 2) {
if (*control & fl->groups[0]) {
*control = cb(i + t, fl->ids[0], scratch);
}
if (*control & fl->groups[1]) {
*control = cb(i + t, fl->ids[1], scratch);
}
if (*control & fl->groups[2]) {
*control = cb(i + t, fl->ids[2], scratch);
}
if (*control & fl->groups[0]) {
*control = cb(i + t + 1, fl->ids[0], scratch);
}
if (*control & fl->groups[1]) {
*control = cb(i + t + 1, fl->ids[1], scratch);
}
if (*control & fl->groups[2]) {
*control = cb(i + t + 1, fl->ids[2], scratch);
}
}
break;
default:
// slow generalized loop
for (u32 t = 0; t < floodSize && (*control & fl->allGroups); t += 2) {
if (*control & fl->groups[0]) {
*control = cb(i + t, fl->ids[0], scratch);
}
if (*control & fl->groups[1]) {
*control = cb(i + t, fl->ids[1], scratch);
}
if (*control & fl->groups[2]) {
*control = cb(i + t, fl->ids[2], scratch);
}
if (*control & fl->groups[3]) {
*control = cb(i + t, fl->ids[3], scratch);
}
for (u32 t2 = 4; t2 < fl->idCount; t2++) {
if (*control & fl->groups[t2]) {
*control = cb(i + t, fl->ids[t2], scratch);
}
}
if (*control & fl->groups[0]) {
*control = cb(i + t + 1, fl->ids[0], scratch);
}
if (*control & fl->groups[1]) {
*control = cb(i + t + 1, fl->ids[1], scratch);
}
if (*control & fl->groups[2]) {
*control = cb(i + t + 1, fl->ids[2], scratch);
}
if (*control & fl->groups[3]) {
*control = cb(i + t + 1, fl->ids[3], scratch);
}
for (u32 t2 = 4; t2 < fl->idCount; t2++) {
if (*control & fl->groups[t2]) {
*control = cb(i + t + 1, fl->ids[t2], scratch);
}
}
}
break;
#else
// Fallback for debugging
default:
for (u32 t = 0; t < floodSize && (*control & fl->allGroups); t++) {
for (u32 t2 = 0; t2 < fl->idCount; t2++) {
if (*control & fl->groups[t2]) {
*control = cb(i + t, fl->ids[t2], scratch);
}
}
}
#endif
}
}
ptr += floodSize;
} else {
*floodBackoffPtr *= 2;
}
floodout:
if (j + *floodBackoffPtr < mainLoopLen - 128) {
tryFloodDetect = buf + MAX(i,j) + *floodBackoffPtr;
} else {
tryFloodDetect = buf + mainLoopLen; // set so we never do another flood detect
}
*ptrPtr = ptr;
DEBUG_PRINTF("finished flood detection at %p (next check %p)\n",
ptr, tryFloodDetect);
return tryFloodDetect;
}
#endif

1116
src/fdr/teddy.c
View File

File diff suppressed because it is too large Load Diff

110
src/fdr/teddy.h
View File

@ -1,110 +0,0 @@
/*
* Copyright (c) 2016-2017, Intel Corporation
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Intel Corporation nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/** \file
* \brief Teddy literal matcher: function declarations.
*/
#ifndef TEDDY_H_
#define TEDDY_H_
#include "hwlm/hwlm.h" // for hwlm_group_t
#include "util/arch.h"
struct FDR; // forward declaration from fdr_internal.h
struct FDR_Runtime_Args;
hwlm_error_t fdr_exec_teddy_msks1(const struct FDR *fdr,
const struct FDR_Runtime_Args *a,
hwlm_group_t control);
hwlm_error_t fdr_exec_teddy_msks1_pck(const struct FDR *fdr,
const struct FDR_Runtime_Args *a,
hwlm_group_t control);
hwlm_error_t fdr_exec_teddy_msks2(const struct FDR *fdr,
const struct FDR_Runtime_Args *a,
hwlm_group_t control);
hwlm_error_t fdr_exec_teddy_msks2_pck(const struct FDR *fdr,
const struct FDR_Runtime_Args *a,
hwlm_group_t control);
hwlm_error_t fdr_exec_teddy_msks3(const struct FDR *fdr,
const struct FDR_Runtime_Args *a,
hwlm_group_t control);
hwlm_error_t fdr_exec_teddy_msks3_pck(const struct FDR *fdr,
const struct FDR_Runtime_Args *a,
hwlm_group_t control);
hwlm_error_t fdr_exec_teddy_msks4(const struct FDR *fdr,
const struct FDR_Runtime_Args *a,
hwlm_group_t control);
hwlm_error_t fdr_exec_teddy_msks4_pck(const struct FDR *fdr,
const struct FDR_Runtime_Args *a,
hwlm_group_t control);
#if defined(HAVE_AVX2)
hwlm_error_t fdr_exec_fat_teddy_msks1(const struct FDR *fdr,
const struct FDR_Runtime_Args *a,
hwlm_group_t control);
hwlm_error_t fdr_exec_fat_teddy_msks1_pck(const struct FDR *fdr,
const struct FDR_Runtime_Args *a,
hwlm_group_t control);
hwlm_error_t fdr_exec_fat_teddy_msks2(const struct FDR *fdr,
const struct FDR_Runtime_Args *a,
hwlm_group_t control);
hwlm_error_t fdr_exec_fat_teddy_msks2_pck(const struct FDR *fdr,
const struct FDR_Runtime_Args *a,
hwlm_group_t control);
hwlm_error_t fdr_exec_fat_teddy_msks3(const struct FDR *fdr,
const struct FDR_Runtime_Args *a,
hwlm_group_t control);
hwlm_error_t fdr_exec_fat_teddy_msks3_pck(const struct FDR *fdr,
const struct FDR_Runtime_Args *a,
hwlm_group_t control);
hwlm_error_t fdr_exec_fat_teddy_msks4(const struct FDR *fdr,
const struct FDR_Runtime_Args *a,
hwlm_group_t control);
hwlm_error_t fdr_exec_fat_teddy_msks4_pck(const struct FDR *fdr,
const struct FDR_Runtime_Args *a,
hwlm_group_t control);
#endif /* HAVE_AVX2 */
#endif /* TEDDY_H_ */

Some files were not shown because too many files have changed in this diff Show More