53 Commits

Author SHA1 Message Date
Justin Viiret
7b54856642 Rose: allow block-mode merge of small prefixes
Previously, we disallowed the merging of all Rose prefixes in block mode
where the literal sets are not identical.

This change allows merging if the prefix graphs to be merged are very
small, as a small performance improvement for cases with lots of tiny
prefixes.

This check is deliberately conservative: graphs must have some common
vertices, and the result of the merge must not give up any
accelerability.
2016-03-01 11:34:26 +11:00
Justin Viiret
42d34f19d1 Dump: don't call dumpNfaNotes for SOM reverse NFAs
These NFAs have no queue index.
2016-03-01 11:34:26 +11:00
Justin Viiret
961e303ff3 SET_GROUPS instr: don't generate more than one 2016-03-01 11:34:26 +11:00
Justin Viiret
cd133f77ee DEDUPE instr: generate only when necessary 2016-03-01 11:34:11 +11:00
Justin Viiret
09bf568d95 Rose: clean up use of scratch, RoseContext 2016-03-01 11:32:11 +11:00
Justin Viiret
9e9bb6a960 Rose: pack global state bits into one u8
Eliminate the RoseRuntimeState structure in favour of a single status
byte that is stored in scratch and copied to/from stream state.
2016-03-01 11:32:01 +11:00
Justin Viiret
28f379d738 Rose: remove alignment req for anchored DFA state 2016-03-01 11:32:01 +11:00
Justin Viiret
060defe6c4 Rose: move more report handling work into program
Move report preconditions (bounds, exhaustion, etc) into program
instructions and use a more direct path to the user match callback than
the adaptor functions.

Report handling has been moved to new file src/report.h. Reporting from
EOD now uses the same instructions as normal report handling, rather
than its own.

Jump target tracking in rose_build_bytecode.cpp has been cleaned up.
2016-03-01 11:32:01 +11:00
Justin Viiret
9eb328b455 RoseRuntimeState no longer needs to be packed
This structure only contains u8 values now. In the future we may wish to
eliminate it entirely and store the few bits we need more directly.
2016-03-01 11:29:04 +11:00
Justin Viiret
435b08b984 Docs for Rose callback types 2016-03-01 11:29:04 +11:00
Justin Viiret
4feabf7bd6 Make Rose callback types explicitly take scratch 2016-03-01 11:29:04 +11:00
Justin Viiret
70620327cc Remove RoseContext::userCtx
All Rose callbacks receive scratch as their context.
2016-03-01 11:29:04 +11:00
Justin Viiret
b36197df26 roseEodRunMatcher: correct early return value 2016-03-01 11:28:56 +11:00
Justin Viiret
621dfbebb7 nfaCheckFinalState: define return value
Make nfaCheckFinalState return MO_HALT_MATCHING when the user instructs
us (via the callback return value) to halt matching. In the caller,
check this value and stop matching if told.
2016-03-01 11:28:03 +11:00
Justin Viiret
de61b32e98 Use fatbit for anch log, delay slots in scratch
Since these structures are in scratch, they do not have to be as small
as possible and we can use fatbit instead of multibit to improve
performance.
2016-03-01 11:24:17 +11:00
Justin Viiret
1c2fca8840 rose_build_anchored: take ref, not pointer 2016-03-01 11:24:17 +11:00
Justin Viiret
69682ed263 Account for multi-dfa case with ANCHORED_DELAY
Specifically, we must set build_context::floatingMinLiteralMatchOffset
to 1 when thew anchored table contains multiple DFAs, as they can
produce unordered matches.

This check was already been done, but too late to affect the generation
of ANCHORED_DELAY instructions.
2016-03-01 11:24:13 +11:00
Justin Viiret
d7c8ffc7fd Use correct type for anchored matcher build 2016-03-01 11:24:09 +11:00
Justin Viiret
8783750c72 Remove dupe engine, state ptrs from RoseContext
Remove the RoseEngine and stream state pointers frose RoseContext, as
they are also present in core_info.

Unify stream state handing in Rose to always use a char * (we were often
a u8 * for no particularly good reason) and tidy up.
2016-03-01 11:23:56 +11:00
Justin Viiret
10cda4cc33 Rose: Move all literal operations into program
Replace the RoseLiteral structure with more program instructions; now,
instead of each literal ID leading to a RoseLiteral, it simply has a
program to run (and a delay rebuild program).

This commit also makes some other improvements:

 * CHECK_STATE instruction, for use instead of a sparse iterator over a
   single element.
 * Elide some checks (CHECK_LIT_EARLY, ANCHORED_DELAY, etc) when not
   needed.
 * Flatten PUSH_DELAYED behaviour to one instruction per delayed
   literal, rather than the mask/index-list approach used before.
 * Simple program cache at compile time for deduplication.
2016-03-01 11:23:56 +11:00
Matthew Barr
fe475cc069 alignof() should operate on a type-id 2016-03-01 11:23:48 +11:00
Justin Viiret
fafcc83520 Delete unused build_context::depths 2016-03-01 11:23:11 +11:00
Justin Viiret
48c9d7c381 Remove use of depth from Rose entirely 2016-03-01 11:23:11 +11:00
Justin Viiret
14f18bd6e8 Don't use depth for in-flight check 2016-03-01 11:23:11 +11:00
Justin Viiret
3d87e382fa Remove CHECK_DEPTH instruction 2016-03-01 11:23:11 +11:00
Justin Viiret
e051077a26 Remove "dot" entries from leftfix lookarounds
Note that we have to be careful to leave the first lookaround entry in
place, if it's a dot. This should eventually be done with a program
instruction.
2016-03-01 11:23:06 +11:00
Justin Viiret
5fc4289dbe roseRunProgram: iter state on stack 2016-03-01 11:20:36 +11:00
Justin Viiret
2abc038f1c roseCatchUpLeftfixes: iter state on stack 2016-03-01 11:20:36 +11:00
Justin Viiret
dd692c5d2b roseBlockHasEodWork: iter state on stack 2016-03-01 11:20:36 +11:00
Justin Viiret
09319940bf roseFlushLastByteHistory: iter state on stack 2016-03-01 11:20:36 +11:00
Justin Viiret
b2a76e6e2b roseCheckNfaEod: use sparse iterator for EOD
Rather than checking all active outfix/suffix engines, use a sparse
iterator to check only those engines that accept at EOD.
2016-03-01 11:20:26 +11:00
Justin Viiret
b6508811c0 writeEodProgram: avoid make_move_iterator warning
Avoid an ambiguity between std:: and boost::make_move_iterator on builds
against libc++.
2016-03-01 11:18:17 +11:00
Justin Viiret
b2ebdac642 rose: Extend program to handle literals, iterators
- cleanups
- add sparse iter instructions
- merge "root" and "sparse iter" programs together
- move program execution to new file program_runtime.h
- simplify EOD execution
2016-03-01 11:17:31 +11:00
Justin Viiret
db4176c13e convertAnchPrefixToBounds: check size of delay_adj
Avoid subtracting delay_adj from a smaller max bound.
2016-03-01 11:16:29 +11:00
Justin Viiret
86a52971ca Remove dead code: EdgeSourceStateCompare 2016-03-01 11:16:13 +11:00
Justin Viiret
d67c7583ea rose: Extend the interpreter to handle more work
- Use program for EOD sparse iterator
- Use program for literal sparse iterator
- Eliminate RoseRole, RosePred, RoseVertexProps::role
- Small performance optimizations
2016-03-01 11:16:02 +11:00
Justin Viiret
9cb2233589 rose: Use an interpreter for role runtime
Replace much of the RoseRole structure with an interpreted program,
simplifying the Rose runtime and making it much more flexible.
2016-03-01 11:16:02 +11:00
Alex Coyte
a7d8dafb71 detach the sidecar 2016-03-01 11:13:23 +11:00
Alex Coyte
b9c5d65f0e Rework literal overlap checks for merging engines
Also increase the size of chunks we consider merging for castles.
2016-03-01 11:10:24 +11:00
Alex Coyte
5e0d10d805 Allow lag on castle infixes to be reduced
Reducing lag allows for castles to be merged more effectively
2016-03-01 11:10:13 +11:00
Justin Viiret
8c09d054c9 Add per-top findMinWidth etc for NFA graphs 2015-12-07 09:38:32 +11:00
Justin Viiret
748d46c124 CastleProto: track next top explicitly
Repeats may be removed (e.g. by pruning in role aliasing passes)
leaving "holes" in the top map. Track the next top to use explicitly,
rather than using repeats.size().
2015-12-07 09:38:32 +11:00
Justin Viiret
da23e8306a assignDkeys: use flat_set<ReportID>, not set 2015-12-07 09:38:32 +11:00
Justin Viiret
8dac64d1dc findMinWidth, findMaxWidth: width for a given top
Currently only implemented for Castle suffixes.
2015-12-07 09:38:32 +11:00
Justin Viiret
03953f34b1 RoseDedupeAuxImpl: collect unique suffixes first 2015-12-07 09:38:32 +11:00
Justin Viiret
1267922ca7 role aliasing: simplify hashRightRoleProperties
Using the full report set for a suffix as an input to this hash was very
slow at scale.
2015-12-07 09:38:32 +11:00
Justin Viiret
abbd548899 ng_execute: update interface to use flat_set
This changes all the execute_graph() interfaces so that instead of
mutating a std::set of vertices, they accept an initial flat_set of
states and return a resultant flat_set of states after execution.

(Note that internally execute_graph() still uses bitsets)

This is both faster and more flexible.
2015-11-18 15:27:17 +11:00
Justin Viiret
2a2576e907 Introduce copy_bytes for writing into bytecode
Protects memcpy from nullptr sources, which triggers failures in GCC's
UB sanitizer.
2015-11-18 15:26:16 +11:00
Alex Coyte
1507b3fd36 move oversize graph check out of Automaton_holder ctor 2015-11-10 14:36:14 +11:00
Justin Viiret
c7bebf8836 RoseBuildImpl: init base_id members
These are set late in the Rose build process, when final IDs are
allocated.
2015-11-10 14:25:04 +11:00