1448 Commits

Author SHA1 Message Date
Justin Viiret
8783750c72 Remove dupe engine, state ptrs from RoseContext
Remove the RoseEngine and stream state pointers frose RoseContext, as
they are also present in core_info.

Unify stream state handing in Rose to always use a char * (we were often
a u8 * for no particularly good reason) and tidy up.
2016-03-01 11:23:56 +11:00
Matthew Barr
39886a0968 Coverity: Restore output stream format 2016-03-01 11:23:56 +11:00
Justin Viiret
10cda4cc33 Rose: Move all literal operations into program
Replace the RoseLiteral structure with more program instructions; now,
instead of each literal ID leading to a RoseLiteral, it simply has a
program to run (and a delay rebuild program).

This commit also makes some other improvements:

 * CHECK_STATE instruction, for use instead of a sparse iterator over a
   single element.
 * Elide some checks (CHECK_LIT_EARLY, ANCHORED_DELAY, etc) when not
   needed.
 * Flatten PUSH_DELAYED behaviour to one instruction per delayed
   literal, rather than the mask/index-list approach used before.
 * Simple program cache at compile time for deduplication.
2016-03-01 11:23:56 +11:00
Alex Coyte
255d84a83a squashing: prevent generation of pairs of squash states 2016-03-01 11:23:56 +11:00
Matthew Barr
fe475cc069 alignof() should operate on a type-id 2016-03-01 11:23:48 +11:00
Justin Viiret
fafcc83520 Delete unused build_context::depths 2016-03-01 11:23:11 +11:00
Justin Viiret
48c9d7c381 Remove use of depth from Rose entirely 2016-03-01 11:23:11 +11:00
Justin Viiret
14f18bd6e8 Don't use depth for in-flight check 2016-03-01 11:23:11 +11:00
Justin Viiret
3d87e382fa Remove CHECK_DEPTH instruction 2016-03-01 11:23:11 +11:00
Justin Viiret
e051077a26 Remove "dot" entries from leftfix lookarounds
Note that we have to be careful to leave the first lookaround entry in
place, if it's a dot. This should eventually be done with a program
instruction.
2016-03-01 11:23:06 +11:00
Justin Viiret
e92a20e5fa ComponentRepeat: remove firsts_cache, precalc code
Firsts are easy to compute in ComponentRepeat::first() now.
2016-03-01 11:22:45 +11:00
Justin Viiret
3d049d6de3 ComponentRepeat: wire X{0,N} and (X?){N} the same 2016-03-01 11:22:45 +11:00
Justin Viiret
997c0c9efd ComponentRepeat: wire R{0,N} as (R{1,N})?
Change the way that we wire up the edges in a bounded repeat to avoid
large fan-out from predecessors.
2016-03-01 11:22:45 +11:00
Justin Viiret
98eff64edf ng_prefilter: turn large max bound into inf
During prefilter region replacement, turn regions with very large max
bounds into repeats with inf max bound. This improves compile time and
the likelihood that we will actually be able to build an implementation
for such patterns.
2016-03-01 11:22:45 +11:00
Anatoly Burakov
87424713a7 Multibyte acceleration compile side 2016-03-01 11:21:39 +11:00
Anatoly Burakov
081b3ef369 Multibyte truffle runtime 2016-03-01 11:21:39 +11:00
Anatoly Burakov
47b17ade27 Multibyte shufti runtime 2016-03-01 11:21:39 +11:00
Anatoly Burakov
dd2ec6bdac Multibyte vermicelli runtime 2016-03-01 11:21:39 +11:00
Anatoly Burakov
77ff826bbf Adding bitmatchers 2016-03-01 11:21:39 +11:00
Anatoly Burakov
68f6849687 Adding AVX2 version of truffle 2016-03-01 11:21:39 +11:00
Justin Viiret
abb5a82057 scratch: remove sparse iter state (now unused) 2016-03-01 11:21:23 +11:00
Justin Viiret
5fc4289dbe roseRunProgram: iter state on stack 2016-03-01 11:20:36 +11:00
Justin Viiret
2abc038f1c roseCatchUpLeftfixes: iter state on stack 2016-03-01 11:20:36 +11:00
Justin Viiret
dd692c5d2b roseBlockHasEodWork: iter state on stack 2016-03-01 11:20:36 +11:00
Justin Viiret
09319940bf roseFlushLastByteHistory: iter state on stack 2016-03-01 11:20:36 +11:00
Justin Viiret
b2a76e6e2b roseCheckNfaEod: use sparse iterator for EOD
Rather than checking all active outfix/suffix engines, use a sparse
iterator to check only those engines that accept at EOD.
2016-03-01 11:20:26 +11:00
Justin Viiret
04dfed2602 runtime: hoist broken check in streaming mode 2016-03-01 11:20:22 +11:00
Justin Viiret
b6508811c0 writeEodProgram: avoid make_move_iterator warning
Avoid an ambiguity between std:: and boost::make_move_iterator on builds
against libc++.
2016-03-01 11:18:17 +11:00
Justin Viiret
b2ebdac642 rose: Extend program to handle literals, iterators
- cleanups
- add sparse iter instructions
- merge "root" and "sparse iter" programs together
- move program execution to new file program_runtime.h
- simplify EOD execution
2016-03-01 11:17:31 +11:00
Justin Viiret
8069e99bee make_disjoint: Remove dead code 2016-03-01 11:17:28 +11:00
Justin Viiret
db4176c13e convertAnchPrefixToBounds: check size of delay_adj
Avoid subtracting delay_adj from a smaller max bound.
2016-03-01 11:16:29 +11:00
Justin Viiret
326abeb3ee Perform an early removeRedundancy call on graph
This allows sibling character classes to be merged together before graph
component splitting is done by calcComponents().

In particular, this transforms (A|a)(B|b)(C|c) into [Aa][Bb][Cc]
earlier.
2016-03-01 11:16:17 +11:00
Justin Viiret
86a52971ca Remove dead code: EdgeSourceStateCompare 2016-03-01 11:16:13 +11:00
Justin Viiret
d67c7583ea rose: Extend the interpreter to handle more work
- Use program for EOD sparse iterator
- Use program for literal sparse iterator
- Eliminate RoseRole, RosePred, RoseVertexProps::role
- Small performance optimizations
2016-03-01 11:16:02 +11:00
Justin Viiret
9cb2233589 rose: Use an interpreter for role runtime
Replace much of the RoseRole structure with an interpreted program,
simplifying the Rose runtime and making it much more flexible.
2016-03-01 11:16:02 +11:00
Alex Coyte
a7d8dafb71 detach the sidecar 2016-03-01 11:13:23 +11:00
Alex Coyte
e065c4d60b make nfaExecCastle0_QR() more efficent
1. Reverse scan for the last escape and only process later events.
2. Only scheck subcastles which may expire for staleness
2016-03-01 11:13:22 +11:00
Alex Coyte
b9c5d65f0e Rework literal overlap checks for merging engines
Also increase the size of chunks we consider merging for castles.
2016-03-01 11:10:24 +11:00
Alex Coyte
05beadf52f Introduce REPEAT_ALWAYS model for {0,} castle repeats
As Castle guards the repeats, no more state is needed for these repeats
2016-03-01 11:10:20 +11:00
Alex Coyte
5e0d10d805 Allow lag on castle infixes to be reduced
Reducing lag allows for castles to be merged more effectively
2016-03-01 11:10:13 +11:00
Alex Coyte
e58786e192 Use add_edge_if_not_present in somMayGoBackwards()
As somMayGoBackwards() operates on a copy of the graph where virtual
starts have been collapsed on to startDs, we need to be careful not to
create parallel edges.
2016-03-01 11:09:49 +11:00
Xiang Wang
7bcd2b07c9 simplify max clique analysis 2015-12-07 09:38:33 +11:00
Justin Viiret
8c09d054c9 Add per-top findMinWidth etc for NFA graphs 2015-12-07 09:38:32 +11:00
Justin Viiret
748d46c124 CastleProto: track next top explicitly
Repeats may be removed (e.g. by pruning in role aliasing passes)
leaving "holes" in the top map. Track the next top to use explicitly,
rather than using repeats.size().
2015-12-07 09:38:32 +11:00
Justin Viiret
8427d83780 CastleProto: track mapping of reports to tops
This allows us to speed up report-based queries, like dedupe checking.
2015-12-07 09:38:32 +11:00
Justin Viiret
da23e8306a assignDkeys: use flat_set<ReportID>, not set 2015-12-07 09:38:32 +11:00
Justin Viiret
8dac64d1dc findMinWidth, findMaxWidth: width for a given top
Currently only implemented for Castle suffixes.
2015-12-07 09:38:32 +11:00
Justin Viiret
03953f34b1 RoseDedupeAuxImpl: collect unique suffixes first 2015-12-07 09:38:32 +11:00
Justin Viiret
1267922ca7 role aliasing: simplify hashRightRoleProperties
Using the full report set for a suffix as an input to this hash was very
slow at scale.
2015-12-07 09:38:32 +11:00
Justin Viiret
b87590ce9d castle: simplify find_next_top
Tops are no longer sparse in CastleProto, so the linear scan for holes
isn't necessary.
2015-12-07 09:38:32 +11:00