8f26c5e65f
After upgrading our (ClickHouse's) libcxx from 15 to 16, the compiler
started to complain about usage of an incomplete type "RoseInstruction"
in this (header) function:
void RoseProgram::replace(Iter it, std::unique_ptr<RoseInstruction> ri) {
...
The reason is that libcxx 16 is the first version which implements C++23
constexpr std::unique_ptr (P2273R3, see (*)). RoseProgram::replace()
happens to be be const-evaluatable and the compiler tries to run
std::unique_ptr's ctor + dtor. This fails because at this point
RoseInstruction isn't defined yet.
There are two ways of fixing this:
1. Include rose_build_instruction.h (which contains RoseInstruction)
into rose_build_program.h. Disadvantage: The new include will
propagate transitively into all callers.
2. Move the function implementation into the source file which sees
RoseInstruction's definition already. Disadvantage: Template
instantiation is no longer automatic, instead there must be either a)
explicit template instantiation (e.g. in rose_build_program.cpp) or
b) all callers which instantiate the function must live in the same
source file and do the instantiations by themselves. Fortunately, the
latter is the case here, but potential future code outside
rose_build_program.cpp will require ugly explicit instantiation.
(*) https://en.cppreference.com/w/cpp/23
Vectorscan?
A fork of Intel's Hyperscan, modified to run on more platforms. Currently ARM NEON/ASIMD is 100% functional, and Power VSX are in development. ARM SVE2 will be implemented when harwdare becomes accessible to the developers. More platforms will follow in the future, on demand/request.
Vectorscan will follow Intel's API and internal algorithms where possible, but will not hesitate to make code changes where it is thought of giving better performance or better portability. In addition, the code will be gradually simplified and made more uniform and all architecture specific -currently Intel- #ifdefs will be removed and abstracted away.
Why the fork?
Originally, the ARM porting was supposed to be merged into Intel's own Hyperscan, and 2 Pull Requests had been made to the project for this reason (1, 2). Unfortunately, the PRs were rejected for now and the forseeable future, thus we have created Vectorscan for our own multi-architectural and opensource collaborative needs.
What is Hyperscan?
Hyperscan is a high-performance multiple regex matching library. It follows the regular expression syntax of the commonly-used libpcre library, but is a standalone library with its own C API.
Hyperscan uses hybrid automata techniques to allow simultaneous matching of large numbers (up to tens of thousands) of regular expressions and for the matching of regular expressions across streams of data.
Vectorscan is typically used in a DPI library stack, just like Hyperscan.
Cross Compiling for AArch64
- To cross compile for AArch64, first adjust the variables set in cmake/setenv-arm64-cross.sh.
export CROSS=<arm-cross-compiler-dir>/bin/aarch64-linux-gnu-export CROSS_SYS=<arm-cross-compiler-system-dir>export BOOST_PATH=<boost-source-dir>
- Set the environment variables:
source cmake/setenv-arm64-cross.sh
- Configure Vectorscan:
mkdir <build-dir-name>cd <build-dir>cmake -DCROSS_COMPILE_AARCH64=1 <hyperscan-source-dir> -DCMAKE_TOOLCHAIN_FILE=<hyperscan-source-dir>/cmake/arm64-cross.cmake
- Build Vectorscan:
make -jTwhere T is the number of threads used to compile.cmake --build . -- -j Tcan also be used instead of make.
Compiling for SVE
The following cmake variables can be set in order to target Arm's Scalable Vector Extension. They are listed in ascending order of strength, with cmake detecting whether the feature is available in the compiler and falling back to a weaker version if not. Only one of these variables needs to be set as weaker variables will be implied as set.
BUILD_SVEBUILD_SVE2BUILD_SVE2_BITPERM
Documentation
Information on building the Hyperscan library and using its API is available in the Developer Reference Guide.
License
Vectorscan, like Hyperscan is licensed under the BSD License. See the LICENSE file in the project repository.
Versioning
The master branch on Github will always contain the most recent release of
Hyperscan. Each version released to master goes through QA and testing before
it is released; if you're a user, rather than a developer, this is the version
you should be using.
Further development towards the next release takes place on the develop
branch.
Get Involved
The official homepage for Vectorscan is at www.github.com/VectorCamp/vectorscan.
Original Hyperscan links
The official homepage for Hyperscan is at www.hyperscan.io.
If you have questions or comments, we encourage you to join the mailing list. Bugs can be filed by sending email to the list, or by creating an issue on Github.
If you wish to contact the Hyperscan team at Intel directly, without posting publicly to the mailing list, send email to hyperscan@intel.com.