prometheus support locally managed (#369)

Co-authored-by: Daniel Eisenberg <danielei@checkpoint.com>

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,36 @@
+---
+name: "Bug Report"
+about: "Report a bug with open-appsec"
+labels: [bug]
+---
+
+**Checklist**
+- Have you checked the open-appsec troubleshooting guides - https://docs.openappsec.io/troubleshooting/troubleshooting
+  - Yes / No
+- Have you checked the existing issues and discussions in github for the same issue 
+  - Yes / No
+- Have you checked the knwon limitations same issue - https://docs.openappsec.io/release-notes#limitations
+  - Yes / No
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Run '...'
+3. See error '...'
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots or Logs**
+If applicable, add screenshots or logs to help explain the issue.
+
+**Environment (please complete the following information):**
+- open-appsec version: 
+- Deployment type (Docker, Kubernetes, etc.): 
+- OS:
+
+**Additional context**
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,8 @@
+blank_issues_enabled: false
+contact_links:
+  - name: "Documentation & Troubleshooting"
+    url: "https://docs.openappsec.io/"
+    about: "Check the documentation before submitting an issue."
+  - name: "Feature Requests & Discussions"
+    url: "https://github.com/openappsec/openappsec/discussions"
+    about: "Please open a discussion for feature requests."

.github/ISSUE_TEMPLATE/nginx_version_support.md

@@ -0,0 +1,17 @@
+---
+name: "Nginx Version Support Request"
+about: "Request for a specific Nginx version to be supported"
+---
+
+**Nginx & OS Version:**
+Which Nginx and OS version are you using? 
+
+**Output of nginx -V**
+Share the output of nginx -v
+
+**Expected Behavior:**
+What do you expect to happen with this version?
+
+**Checklist**
+- Have you considered a docker based deployment - find more information here https://docs.openappsec.io/getting-started/start-with-docker?
+  - Yes / No

README.md

@@ -6,7 +6,7 @@
 [![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/6629/badge)](https://bestpractices.coreinfrastructure.org/projects/6629)
 
 # About
-[open-appsec](https://www.openappsec.io) (openappsec.io) builds on machine learning to provide preemptive web app & API threat protection against OWASP-Top-10 and zero-day attacks. It can be deployed as an add-on to Kubernetes Ingress, NGINX, Envoy (soon), and API Gateways.
+[open-appsec](https://www.openappsec.io) (openappsec.io) builds on machine learning to provide preemptive web app & API threat protection against OWASP-Top-10 and zero-day attacks. It can be deployed as an add-on to Linux, Docker or K8s deployments, on NGINX, Kong, APISIX, or Envoy.
 
 The open-appsec engine learns how users normally interact with your web application. It then uses this information to automatically detect requests that fall outside of normal operations, and conducts further analysis to decide whether the request is malicious or not.
 
@@ -39,13 +39,13 @@ open-appsec can be managed using multiple methods:
 * [Using SaaS Web Management](https://docs.openappsec.io/getting-started/using-the-web-ui-saas)
 
 open-appsec Web UI:
-![image](https://github.com/openappsec/openappsec/assets/114033741/22d99379-df52-45c8-984f-1b820635f3b9)
+<img width="1854" height="775" alt="image" src="https://github.com/user-attachments/assets/4c6f7b0a-14f3-4f02-9ab0-ddadc9979b8d" />
+
 
 
 ## Deployment Playgrounds (Virtual labs)
 You can experiment with open-appsec using [Playgrounds](https://www.openappsec.io/playground)
-
-![image](https://github.com/openappsec/openappsec/assets/114033741/14d35d69-4577-48fc-ae87-ea344888e94d)
+<img width="781" height="878" alt="image" src="https://github.com/user-attachments/assets/0ddee216-5cdf-4288-8c41-cc28cfbf3297" />
 
 # Resources
 * [Project Website](https://openappsec.io)
@@ -54,27 +54,21 @@ You can experiment with open-appsec using [Playgrounds](https://www.openappsec.i
 
 # Installation
 
-For Kubernetes (NGINX Ingress) using the installer:
+For Kubernetes (NGINX /Kong / APISIX / Istio) using Helm: follow [documentation](https://docs.openappsec.io/getting-started/start-with-kubernetes)
 
-```bash
-$ wget https://downloads.openappsec.io/open-appsec-k8s-install && chmod +x open-appsec-k8s-install
-$ ./open-appsec-k8s-install
-```
-
-For Kubernetes (NGINX or Kong) using Helm: follow [documentation](https://docs.openappsec.io/getting-started/start-with-kubernetes/install-using-helm-ingress-nginx-and-kong) – use this method if you’ve built your own containers. 
-
-For Linux (NGINX or Kong) using the installer (list of supported/pre-compiled NGINX attachments is available [here](https://downloads.openappsec.io/packages/supported-nginx.txt)):
+For Linux (NGINX / Kong / APISIX) using the installer (list of supported/pre-compiled NGINX attachments is available [here](https://downloads.openappsec.io/packages/supported-nginx.txt)):
 
 ```bash
 $ wget https://downloads.openappsec.io/open-appsec-install && chmod +x open-appsec-install
 $ ./open-appsec-install --auto
 ```
+For kong Lua Based plug in follow [documentation](https://docs.openappsec.io/getting-started/start-with-linux)
 
 For Linux, if you’ve built your own package use the following commands:
 
 ```bash
 $ install-cp-nano-agent.sh --install --hybrid_mode
-$ install-cp-nano-service-http-transaction-handler.sh –install
+$ install-cp-nano-service-http-transaction-handler.sh --install
 $ install-cp-nano-attachment-registration-manager.sh --install
 ```
 You can add the ```--token <token>``` and ```--email <email address>``` options to the first command, to get a token follow [documentation](https://docs.openappsec.io/getting-started/using-the-web-ui-saas/connect-deployed-agents-to-saas-management-k8s-and-linux). 
@@ -177,7 +171,7 @@ open-appsec code was audited by an independent third party in September-October
 See the [full report](https://github.com/openappsec/openappsec/blob/main/LEXFO-CHP20221014-Report-Code_audit-OPEN-APPSEC-v1.2.pdf).
 
 ### Reporting security vulnerabilities
-If you've found a vulnerability or a potential vulnerability in open-appsec please let us know at securityalert@openappsec.io. We'll send a confirmation email to acknowledge your report within 24 hours, and we'll send an additional email when we've identified the issue positively or negatively.
+If you've found a vulnerability or a potential vulnerability in open-appsec please let us know at security-alert@openappsec.io. We'll send a confirmation email to acknowledge your report within 24 hours, and we'll send an additional email when we've identified the issue positively or negatively.
 
 
 # License

attachments/nginx/nginx_attachment_util/nginx_attachment_util.cc

@@ -95,6 +95,18 @@ getFailOpenHoldTimeout()
     return conf_data.getNumericalValue("fail_open_hold_timeout");
 }
 
+unsigned int
+getHoldVerdictPollingTime()
+{
+    return conf_data.getNumericalValue("hold_verdict_polling_time");
+}
+
+unsigned int
+getHoldVerdictRetries()
+{
+    return conf_data.getNumericalValue("hold_verdict_retries");
+}
+
 unsigned int
 getMaxSessionsPerMinute()
 {
@@ -173,6 +185,12 @@ getReqBodySizeTrigger()
     return conf_data.getNumericalValue("body_size_trigger");
 }
 
+unsigned int
+getRemoveResServerHeader()
+{
+    return conf_data.getNumericalValue("remove_server_header");
+}
+
 int
 isIPAddress(c_str ip_str)
 {

attachments/nginx/nginx_attachment_util/nginx_attachment_util_ut/nginx_attachment_util_ut.cc

@@ -66,35 +66,41 @@ TEST_F(HttpAttachmentUtilTest, GetValidAttachmentConfiguration)
             "\"static_resources_path\": \"" + static_resources_path + "\",\n"
             "\"min_retries_for_verdict\": 1,\n"
             "\"max_retries_for_verdict\": 3,\n"
-            "\"body_size_trigger\": 777\n"
+            "\"hold_verdict_retries\": 3,\n"
+            "\"hold_verdict_polling_time\": 1,\n"
+            "\"body_size_trigger\": 777,\n"
+            "\"remove_server_header\": 1\n"
         "}\n";
     ofstream valid_configuration_file(attachment_configuration_file_name);
     valid_configuration_file << valid_configuration;
     valid_configuration_file.close();
 
     EXPECT_EQ(initAttachmentConfig(attachment_configuration_file_name.c_str()), 1);
-    EXPECT_EQ(getDbgLevel(), 2);
+    EXPECT_EQ(getDbgLevel(), 2u);
     EXPECT_EQ(getStaticResourcesPath(), static_resources_path);
     EXPECT_EQ(isFailOpenMode(), 0);
-    EXPECT_EQ(getFailOpenTimeout(), 1234);
+    EXPECT_EQ(getFailOpenTimeout(), 1234u);
     EXPECT_EQ(isFailOpenHoldMode(), 1);
-    EXPECT_EQ(getFailOpenHoldTimeout(), 4321);
+    EXPECT_EQ(getFailOpenHoldTimeout(), 4321u);
     EXPECT_EQ(isFailOpenOnSessionLimit(), 1);
-    EXPECT_EQ(getMaxSessionsPerMinute(), 0);
-    EXPECT_EQ(getNumOfNginxIpcElements(), 200);
-    EXPECT_EQ(getKeepAliveIntervalMsec(), 10000);
-    EXPECT_EQ(getResProccessingTimeout(), 420);
-    EXPECT_EQ(getReqProccessingTimeout(), 42);
-    EXPECT_EQ(getRegistrationThreadTimeout(), 101);
-    EXPECT_EQ(getReqHeaderThreadTimeout(), 10);
-    EXPECT_EQ(getReqBodyThreadTimeout(), 155);
-    EXPECT_EQ(getResHeaderThreadTimeout(), 1);
-    EXPECT_EQ(getResBodyThreadTimeout(), 0);
-    EXPECT_EQ(getMinRetriesForVerdict(), 1);
-    EXPECT_EQ(getMaxRetriesForVerdict(), 3);
-    EXPECT_EQ(getReqBodySizeTrigger(), 777);
-    EXPECT_EQ(getWaitingForVerdictThreadTimeout(), 75);
+    EXPECT_EQ(getMaxSessionsPerMinute(), 0u);
+    EXPECT_EQ(getNumOfNginxIpcElements(), 200u);
+    EXPECT_EQ(getKeepAliveIntervalMsec(), 10000u);
+    EXPECT_EQ(getResProccessingTimeout(), 420u);
+    EXPECT_EQ(getReqProccessingTimeout(), 42u);
+    EXPECT_EQ(getRegistrationThreadTimeout(), 101u);
+    EXPECT_EQ(getReqHeaderThreadTimeout(), 10u);
+    EXPECT_EQ(getReqBodyThreadTimeout(), 155u);
+    EXPECT_EQ(getResHeaderThreadTimeout(), 1u);
+    EXPECT_EQ(getResBodyThreadTimeout(), 0u);
+    EXPECT_EQ(getMinRetriesForVerdict(), 1u);
+    EXPECT_EQ(getMaxRetriesForVerdict(), 3u);
+    EXPECT_EQ(getReqBodySizeTrigger(), 777u);
+    EXPECT_EQ(getWaitingForVerdictThreadTimeout(), 75u);
     EXPECT_EQ(getInspectionMode(), ngx_http_inspection_mode::BLOCKING_THREAD);
+    EXPECT_EQ(getRemoveResServerHeader(), 1u);
+    EXPECT_EQ(getHoldVerdictRetries(), 3u);
+    EXPECT_EQ(getHoldVerdictPollingTime(), 1u);
 
     EXPECT_EQ(isDebugContext("1.2.3.4", "5.6.7.8", 80, "GET", "test", "/abc"), 1);
     EXPECT_EQ(isDebugContext("1.2.3.9", "5.6.7.8", 80, "GET", "test", "/abc"), 0);

build_system/docker/CMakeLists.txt

@@ -1,4 +1,4 @@
-install(FILES Dockerfile entry.sh install-cp-agent-intelligence-service.sh install-cp-crowdsec-aux.sh DESTINATION .)
+install(FILES Dockerfile entry.sh install-cp-agent-intelligence-service.sh install-cp-crowdsec-aux.sh self_managed_openappsec_manifest.json DESTINATION .)
 
 add_custom_command(
     OUTPUT ${CMAKE_INSTALL_PREFIX}/agent-docker.img

build_system/docker/Dockerfile

@@ -1,5 +1,7 @@
 FROM alpine
 
+ENV OPENAPPSEC_NANO_AGENT=TRUE
+
 RUN apk add --no-cache -u busybox
 RUN apk add --no-cache -u zlib
 RUN apk add --no-cache bash
@@ -11,8 +13,12 @@ RUN apk add --no-cache libunwind
 RUN apk add --no-cache gdb
 RUN apk add --no-cache libxml2
 RUN apk add --no-cache pcre2
+RUN apk add --no-cache ca-certificates
 RUN apk add --update coreutils
 
+
+COPY self_managed_openappsec_manifest.json /tmp/self_managed_openappsec_manifest.json
+
 COPY install*.sh /nano-service-installers/
 COPY entry.sh /entry.sh
 

build_system/docker/entry.sh

@@ -6,6 +6,8 @@ HTTP_TRANSACTION_HANDLER_SERVICE="install-cp-nano-service-http-transaction-handl
 ATTACHMENT_REGISTRATION_SERVICE="install-cp-nano-attachment-registration-manager.sh"
 ORCHESTRATION_INSTALLATION_SCRIPT="install-cp-nano-agent.sh"
 CACHE_INSTALLATION_SCRIPT="install-cp-nano-agent-cache.sh"
+PROMETHEUS_INSTALLATION_SCRIPT="install-cp-nano-service-prometheus.sh"
+NGINX_CENTRAL_MANAGER_INSTALLATION_SCRIPT="install-cp-nano-central-nginx-manager.sh"
 
 var_fog_address=
 var_proxy=
@@ -13,6 +15,21 @@ var_mode=
 var_token=
 var_ignore=
 init=
+active_watchdog_pid=
+
+cleanup() {
+    local signal="$1"
+    echo "[$(date '+%Y-%m-%d %H:%M:%S')] Signal ${signal} was received, exiting gracefully..." >&2
+    if [ -n "${active_watchdog_pid}" ] && ps -p ${active_watchdog_pid} > /dev/null 2>&1; then
+        kill -TERM ${active_watchdog_pid} 2>/dev/null || true
+        wait ${active_watchdog_pid} 2>/dev/null || true
+    fi
+    echo "Cleanup completed. Exiting now." >&2
+    exit 0
+}
+
+trap 'cleanup SIGTERM' SIGTERM
+trap 'cleanup SIGINT' SIGINT
 
 if [ ! -f /nano-service-installers/$ORCHESTRATION_INSTALLATION_SCRIPT ]; then
     echo "Error: agent installation package doesn't exist."
@@ -44,8 +61,11 @@ while true; do
 done
 
 if [ -z $var_token ] && [ $var_mode != "--hybrid_mode" ]; then
-    echo "Error: Token was not provided as input argument."
-    exit 1
+    var_token=$(env | grep 'AGENT_TOKEN=' | cut -d'=' -f2-)
+    if  [ -z $var_token ]; then
+        echo "Error: Token was not provided as input argument."
+        exit 1
+    fi
 fi
 
 orchestration_service_installation_flags="--container_mode --skip_registration"
@@ -78,6 +98,14 @@ fi
 /nano-service-installers/$CACHE_INSTALLATION_SCRIPT --install
 /nano-service-installers/$HTTP_TRANSACTION_HANDLER_SERVICE --install
 
+if [ "$PROMETHEUS" == "true" ]; then
+    /nano-service-installers/$PROMETHEUS_INSTALLATION_SCRIPT --install
+fi
+
+if [ "$CENTRAL_NGINX_MANAGER" == "true" ]; then
+    /nano-service-installers/$NGINX_CENTRAL_MANAGER_INSTALLATION_SCRIPT --install
+fi
+
 if [ "$CROWDSEC_ENABLED" == "true" ]; then
     /nano-service-installers/$INTELLIGENCE_INSTALLATION_SCRIPT --install
     /nano-service-installers/$CROWDSEC_INSTALLATION_SCRIPT --install
@@ -90,25 +118,16 @@ if [ -f "$FILE" ]; then
 fi
 
 touch /etc/cp/watchdog/wd.startup
+/etc/cp/watchdog/cp-nano-watchdog >/dev/null 2>&1 &
+active_watchdog_pid=$!
 while true; do
-    if [ -z "$init" ]; then
-        init=true
-        /etc/cp/watchdog/cp-nano-watchdog >/dev/null 2>&1 &
-        sleep 5
-        active_watchdog_pid=$(pgrep -f -x -o "/bin/bash /etc/cp/watchdog/cp-nano-watchdog")
-    fi
-
-    current_watchdog_pid=$(pgrep -f -x -o "/bin/bash /etc/cp/watchdog/cp-nano-watchdog")
-    if [ ! -f /tmp/restart_watchdog ] && [ "$current_watchdog_pid" != "$active_watchdog_pid" ]; then
-        echo "Error: Watchdog exited abnormally"
-        exit 1
-    elif [ -f /tmp/restart_watchdog ]; then
+    if [ -f /tmp/restart_watchdog ]; then
         rm -f /tmp/restart_watchdog
-        kill -9 "$(pgrep -f -x -o "/bin/bash /etc/cp/watchdog/cp-nano-watchdog")"
-        /etc/cp/watchdog/cp-nano-watchdog >/dev/null 2>&1 &
-        sleep 5
-        active_watchdog_pid=$(pgrep -f -x -o "/bin/bash /etc/cp/watchdog/cp-nano-watchdog")
+        kill -9 ${active_watchdog_pid}
+    fi
+    if [ ! "$(ps -f | grep cp-nano-watchdog | grep ${active_watchdog_pid})" ]; then
+        /etc/cp/watchdog/cp-nano-watchdog >/dev/null 2>&1 &
+        active_watchdog_pid=$!
     fi
-
     sleep 5
 done

components/CMakeLists.txt

@@ -1,4 +1,3 @@
-add_subdirectory(report_messaging)
 add_subdirectory(http_manager)
 add_subdirectory(signal_handler)
 add_subdirectory(gradual_deployment)
@@ -8,3 +7,4 @@ add_subdirectory(pending_key)
 add_subdirectory(utils)
 add_subdirectory(attachment-intakers)
 add_subdirectory(security_apps)
+add_subdirectory(nginx_message_reader)

components/attachment-intakers/attachment_registrator/attachment_registrator.cc

@@ -39,6 +39,8 @@ USE_DEBUG_FLAG(D_ATTACHMENT_REGISTRATION);
 
 using namespace std;
 
+static const AlertInfo alert(AlertTeam::CORE, "attachment registrator");
+
 class AttachmentRegistrator::Impl
 {
 public:
@@ -163,7 +165,7 @@ private:
                 break;
             }
             default:
-                dbgAssert(false) << "Unsupported Attachment " << static_cast<int>(type);
+                dbgAssert(false) << alert << "Unsupported Attachment " << static_cast<int>(type);
         }
 
         if (!family_id.empty()) handler_path << family_id << "_";
@@ -175,7 +177,9 @@ private:
     string
     genRegCommand(const string &family_id, const uint num_of_members, const AttachmentType type) const
     {
-        dbgAssert(num_of_members > 0) << "Failed to generate a registration command for an empty group of attachments";
+        dbgAssert(num_of_members > 0)
+            << alert
+            << "Failed to generate a registration command for an empty group of attachments";
 
         static const string registration_format = "/etc/cp/watchdog/cp-nano-watchdog --register ";
         stringstream registration_command;
@@ -187,7 +191,7 @@ private:
                 break;
             }
             default:
-                dbgAssert(false) << "Unsupported Attachment " << static_cast<int>(type);
+                dbgAssert(false) << alert << "Unsupported Attachment " << static_cast<int>(type);
         }
 
         if (!family_id.empty()) registration_command << " --family " << family_id;
@@ -265,7 +269,7 @@ private:
             return -1;
         }
 
-        dbgAssert(new_socket.unpack() > 0) << "Generated socket is OK yet negative";
+        dbgAssert(new_socket.unpack() > 0) << alert << "Generated socket is OK yet negative";
         return new_socket.unpack();
     }
 
@@ -281,7 +285,7 @@ private:
         }
 
         I_Socket::socketFd client_socket = accepted_socket.unpack();
-        dbgAssert(client_socket > 0) << "Generated client socket is OK yet negative";
+        dbgAssert(client_socket > 0) << alert << "Generated client socket is OK yet negative";
         auto close_socket_on_exit = make_scope_exit([&]() { i_socket->closeSocket(client_socket); });
 
         Maybe<uint8_t> attachment_id = readNumericParam(client_socket);
@@ -375,7 +379,7 @@ private:
         }
 
         I_Socket::socketFd client_socket = accepted_socket.unpack();
-        dbgAssert(client_socket > 0) << "Generated client socket is OK yet negative";
+        dbgAssert(client_socket > 0) << alert << "Generated client socket is OK yet negative";
         auto close_socket_on_exit = make_scope_exit([&]() { i_socket->closeSocket(client_socket); });
 
         Maybe<AttachmentType> attachment_type = readAttachmentType(client_socket);

components/attachment-intakers/nginx_attachment/nginx_attachment.cc

@@ -31,10 +31,12 @@
 #include <stdarg.h>
 
 #include <boost/range/iterator_range.hpp>
+#include <boost/algorithm/string.hpp>
 #include <boost/regex.hpp>
 
 #include "nginx_attachment_config.h"
 #include "nginx_attachment_opaque.h"
+#include "generic_rulebase/evaluators/trigger_eval.h"
 #include "nginx_parser.h"
 #include "i_instance_awareness.h"
 #include "common.h"
@@ -76,6 +78,7 @@ using namespace std;
 using ChunkType = ngx_http_chunk_type_e;
 
 static const uint32_t corrupted_session_id = CORRUPTED_SESSION_ID;
+static const AlertInfo alert(AlertTeam::CORE, "nginx attachment");
 
 class FailopenModeListener : public Listener<FailopenModeEvent>
 {
@@ -128,6 +131,7 @@ class NginxAttachment::Impl
     Singleton::Provide<I_StaticResourcesHandler>::From<NginxAttachment>
 {
     static constexpr auto INSPECT = ngx_http_cp_verdict_e::TRAFFIC_VERDICT_INSPECT;
+    static constexpr auto LIMIT_RESPONSE_HEADERS = ngx_http_cp_verdict_e::LIMIT_RESPONSE_HEADERS;
     static constexpr auto ACCEPT = ngx_http_cp_verdict_e::TRAFFIC_VERDICT_ACCEPT;
     static constexpr auto DROP = ngx_http_cp_verdict_e::TRAFFIC_VERDICT_DROP;
     static constexpr auto INJECT = ngx_http_cp_verdict_e::TRAFFIC_VERDICT_INJECT;
@@ -259,6 +263,22 @@ public:
             );
         }
 
+        const char* ignored_headers_env = getenv("SAAS_IGNORED_UPSTREAM_HEADERS");
+        if (ignored_headers_env) {
+            string ignored_headers_str = ignored_headers_env;
+            ignored_headers_str = NGEN::Strings::trim(ignored_headers_str);
+
+            if (!ignored_headers_str.empty()) {
+                dbgInfo(D_HTTP_MANAGER)
+                    << "Ignoring SAAS_IGNORED_UPSTREAM_HEADERS environment variable: "
+                    << ignored_headers_str;
+
+                vector<string> ignored_headers_vec;
+                boost::split(ignored_headers_vec, ignored_headers_str, boost::is_any_of(";"));
+                for (const string &header : ignored_headers_vec) ignored_headers.insert(header);
+            }
+        }
+
         dbgInfo(D_NGINX_ATTACHMENT) << "Successfully initialized NGINX Attachment";
     }
 
@@ -410,7 +430,10 @@ private:
     bool
     registerAttachmentProcess(uint32_t nginx_user_id, uint32_t nginx_group_id, I_Socket::socketFd new_socket)
     {
-        dbgAssert(server_sock > 0) << "Registration attempt occurred while registration socket is uninitialized";
+        dbgAssert(server_sock > 0)
+        << alert
+        << "Registration attempt occurred while registration socket is uninitialized";
+
 #ifdef FAILURE_TEST
         bool did_fail_on_purpose = false;
 #endif
@@ -802,10 +825,10 @@ private:
         case ChunkType::HOLD_DATA:
             return "HOLD_DATA";
         case ChunkType::COUNT:
-            dbgAssert(false) << "Invalid 'COUNT' ChunkType";
+            dbgAssert(false) << alert << "Invalid 'COUNT' ChunkType";
             return "";
         }
-        dbgAssert(false) << "ChunkType was not handled by the switch case";
+        dbgAssert(false) << alert << "ChunkType was not handled by the switch case";
         return "";
     }
 
@@ -1030,7 +1053,11 @@ private:
             case ChunkType::REQUEST_START:
                 return handleStartTransaction(data, opaque);
             case ChunkType::REQUEST_HEADER:
-                return handleMultiModifiableChunks(NginxParser::parseRequestHeaders(data), "request header", true);
+                return handleMultiModifiableChunks(
+                    NginxParser::parseRequestHeaders(data, ignored_headers),
+                    "request header",
+                    true
+                );
             case ChunkType::REQUEST_BODY:
                 return handleModifiableChunk(NginxParser::parseRequestBody(data), "request body", true);
             case ChunkType::REQUEST_END: {
@@ -1121,17 +1148,29 @@ private:
     handleCustomWebResponse(
         SharedMemoryIPC *ipc,
         vector<const char *> &verdict_data,
-        vector<uint16_t> &verdict_data_sizes)
+        vector<uint16_t> &verdict_data_sizes,
+        string web_user_response_id)
     {
         ngx_http_cp_web_response_data_t web_response_data;
-
+        ScopedContext ctx;
+        if (web_user_response_id != "") {
+            dbgTrace(D_NGINX_ATTACHMENT)
+            << "web user response ID registered in contex: "
+            << web_user_response_id;
+            set<string> triggers_set{web_user_response_id};
+            ctx.registerValue<set<GenericConfigId>>(TriggerMatcher::ctx_key, triggers_set);
+        }
         WebTriggerConf web_trigger_conf = getConfigurationWithDefault<WebTriggerConf>(
             WebTriggerConf::default_trigger_conf,
             "rulebase",
             "webUserResponse"
         );
 
+        bool remove_event_id_param =
+            getProfileAgentSettingWithDefault<string>("false", "nginxAttachment.removeRedirectEventId") == "true";
+
         string uuid;
+        string redirectUrl;
         if (i_transaction_table->hasState<NginxAttachmentOpaque>()) {
             NginxAttachmentOpaque &opaque = i_transaction_table->getState<NginxAttachmentOpaque>();
             uuid = opaque.getSessionUUID();
@@ -1141,7 +1180,12 @@ private:
         if (web_trigger_conf.getDetailsLevel() == "Redirect") {
             web_response_data.response_data.redirect_data.redirect_location_size =
                 web_trigger_conf.getRedirectURL().size();
-            web_response_data.response_data.redirect_data.add_event_id = web_trigger_conf.getAddEventId() ? 1 : 0;
+            bool add_event = web_trigger_conf.getAddEventId();
+            if (add_event && !remove_event_id_param) {
+                web_response_data.response_data.redirect_data.redirect_location_size +=
+                    strlen("?event_id=") + uuid.size();
+            }
+            web_response_data.response_data.redirect_data.add_event_id = add_event ? 1 : 0;
             web_response_data.web_repsonse_type = static_cast<uint8_t>(ngx_web_response_type_e::REDIRECT_WEB_RESPONSE);
         } else {
             web_response_data.response_data.custom_response_data.title_size =
@@ -1155,8 +1199,13 @@ private:
         verdict_data_sizes.push_back(sizeof(ngx_http_cp_web_response_data_t));
 
         if (web_trigger_conf.getDetailsLevel() == "Redirect") {
-            verdict_data.push_back(reinterpret_cast<const char *>(web_trigger_conf.getRedirectURL().data()));
-            verdict_data_sizes.push_back(web_trigger_conf.getRedirectURL().size());
+            redirectUrl = web_trigger_conf.getRedirectURL();
+            if (!remove_event_id_param && web_trigger_conf.getAddEventId()) {
+                redirectUrl += "?event-id=" + uuid;
+            }
+
+            verdict_data.push_back(reinterpret_cast<const char *>(redirectUrl.data()));
+            verdict_data_sizes.push_back(redirectUrl.size());
         } else {
             verdict_data.push_back(reinterpret_cast<const char *>(web_trigger_conf.getResponseTitle().data()));
             verdict_data_sizes.push_back(web_trigger_conf.getResponseTitle().size());
@@ -1232,7 +1281,7 @@ private:
         if (verdict.getVerdict() == DROP) {
             nginx_attachment_event.addTrafficVerdictCounter(nginxAttachmentEvent::trafficVerdict::DROP);
             verdict_to_send.modification_count = 1;
-            return handleCustomWebResponse(ipc, verdict_fragments, fragments_sizes);
+            return handleCustomWebResponse(ipc, verdict_fragments, fragments_sizes, verdict.getWebUserResponseID());
         }
 
         if (verdict.getVerdict() == ACCEPT) {
@@ -1458,11 +1507,17 @@ private:
         opaque.activateContext();
 
         FilterVerdict verdict = handleChunkedData(*chunked_data_type, inspection_data, opaque);
-
         bool is_header =
             *chunked_data_type == ChunkType::REQUEST_HEADER  ||
             *chunked_data_type == ChunkType::RESPONSE_HEADER ||
             *chunked_data_type == ChunkType::CONTENT_LENGTH;
+
+        if (verdict.getVerdict() == LIMIT_RESPONSE_HEADERS) {
+            handleVerdictResponse(verdict, attachment_ipc, transaction_data->session_id, is_header);
+            popData(attachment_ipc);
+            verdict = FilterVerdict(INSPECT);
+        }
+
         handleVerdictResponse(verdict, attachment_ipc, transaction_data->session_id, is_header);
 
         bool is_final_verdict = verdict.getVerdict() == ACCEPT ||
@@ -1575,6 +1630,8 @@ private:
                 return "INJECT";
             case INSPECT:
                 return "INSPECT";
+            case LIMIT_RESPONSE_HEADERS:
+                return "LIMIT_RESPONSE_HEADERS";
             case IRRELEVANT:
                 return "IRRELEVANT";
             case RECONF:
@@ -1582,7 +1639,7 @@ private:
             case WAIT:
                 return "WAIT";
         }
-        dbgAssert(false) << "Invalid EventVerdict enum: " << static_cast<int>(verdict.getVerdict());
+        dbgAssert(false) << alert << "Invalid EventVerdict enum: " << static_cast<int>(verdict.getVerdict());
         return string();
     }
 
@@ -1633,13 +1690,14 @@ private:
             return false;
         }
 
-        dbgAssert(sock.unpack() > 0) << "The generated server socket is OK, yet negative";
+        dbgAssert(sock.unpack() > 0) << alert << "The generated server socket is OK, yet negative";
         server_sock = sock.unpack();
 
         I_MainLoop::Routine accept_attachment_routine =
             [this] ()
             {
                 dbgAssert(inst_awareness->getUniqueID().ok())
+                    << alert
                     << "NGINX attachment Initialized without Instance Awareness";
 
                 bool did_fail_on_purpose = false;
@@ -1652,7 +1710,7 @@ private:
                     << (did_fail_on_purpose ? "Intentional Failure" : new_sock.getErr());
                     return;
                 }
-                dbgAssert(new_sock.unpack() > 0) << "The generated client socket is OK, yet negative";
+                dbgAssert(new_sock.unpack() > 0) << alert << "The generated client socket is OK, yet negative";
                 I_Socket::socketFd new_attachment_socket = new_sock.unpack();
 
                 Maybe<string> uid = getUidFromSocket(new_attachment_socket);
@@ -1711,7 +1769,9 @@ private:
     Maybe<string>
     getUidFromSocket(I_Socket::socketFd new_attachment_socket)
     {
-        dbgAssert(server_sock > 0) << "Registration attempt occurred while registration socket is uninitialized";
+        dbgAssert(server_sock > 0)
+            << alert
+            << "Registration attempt occurred while registration socket is uninitialized";
 
         bool did_fail_on_purpose = false;
         DELAY_IF_NEEDED(IntentionalFailureHandler::FailureType::ReceiveDataFromSocket);
@@ -1793,6 +1853,7 @@ private:
     HttpAttachmentConfig attachment_config;
     I_MainLoop::RoutineID attachment_routine_id = 0;
     bool traffic_indicator = false;
+    unordered_set<string> ignored_headers;
 
     // Interfaces
     I_Socket *i_socket                              = nullptr;

components/attachment-intakers/nginx_attachment/nginx_attachment_config.cc

@@ -203,6 +203,13 @@ HttpAttachmentConfig::setFailOpenTimeout()
         "NGINX wait thread timeout msec"
     ));
 
+    conf_data.setNumericalValue("remove_server_header", getAttachmentConf<uint>(
+        0,
+        "agent.removeServerHeader.nginxModule",
+        "HTTP manager",
+        "Response server header removal"
+    ));
+
     uint inspection_mode = getAttachmentConf<uint>(
         static_cast<uint>(ngx_http_inspection_mode_e::NON_BLOCKING_THREAD),
         "agent.inspectionMode.nginxModule",
@@ -233,6 +240,21 @@ HttpAttachmentConfig::setRetriesForVerdict()
         "Max retries for verdict"
     ));
 
+    conf_data.setNumericalValue("hold_verdict_retries", getAttachmentConf<uint>(
+        3,
+        "agent.retriesForHoldVerdict.nginxModule",
+        "HTTP manager",
+        "Retries for hold verdict"
+    ));
+
+    conf_data.setNumericalValue("hold_verdict_polling_time", getAttachmentConf<uint>(
+        1,
+        "agent.holdVerdictPollingInterval.nginxModule",
+        "HTTP manager",
+        "Hold verdict polling interval seconds"
+    ));
+
+
     conf_data.setNumericalValue("body_size_trigger", getAttachmentConf<uint>(
         200000,
         "agent.reqBodySizeTrigger.nginxModule",

components/attachment-intakers/nginx_attachment/nginx_attachment_opaque.cc

@@ -19,12 +19,15 @@
 
 #include "config.h"
 #include "virtual_modifiers.h"
+#include "agent_core_utilities.h"
 
 using namespace std;
 using namespace boost::uuids;
 
 USE_DEBUG_FLAG(D_HTTP_MANAGER);
 
+extern bool is_keep_alive_ctx;
+
 NginxAttachmentOpaque::NginxAttachmentOpaque(HttpTransactionData _transaction_data)
         :
     TableOpaqueSerialize<NginxAttachmentOpaque>(this),
@@ -67,6 +70,12 @@ NginxAttachmentOpaque::NginxAttachmentOpaque(HttpTransactionData _transaction_da
         ctx.registerValue(HttpTransactionData::uri_query_decoded, decoded_url.substr(question_mark_location + 1));
     }
     ctx.registerValue(HttpTransactionData::uri_path_decoded, decoded_url);
+
+    // Register waf_tag from transaction data if available
+    const std::string& waf_tag = transaction_data.getWafTag();
+    if (!waf_tag.empty()) {
+        ctx.registerValue(HttpTransactionData::waf_tag_ctx, waf_tag);
+    }
 }
 
 NginxAttachmentOpaque::~NginxAttachmentOpaque()
@@ -119,3 +128,47 @@ NginxAttachmentOpaque::setSavedData(const string &name, const string &data, EnvK
     saved_data[name] = data;
     ctx.registerValue(name, data, log_ctx);
 }
+
+bool
+NginxAttachmentOpaque::setKeepAliveCtx(const string &hdr_key, const string &hdr_val)
+{
+    if (!is_keep_alive_ctx) return false;
+
+    static pair<string, string> keep_alive_hdr;
+    static bool keep_alive_hdr_initialized = false;
+
+    if (keep_alive_hdr_initialized) {
+        if (!keep_alive_hdr.first.empty() && hdr_key == keep_alive_hdr.first && hdr_val == keep_alive_hdr.second) {
+            dbgTrace(D_HTTP_MANAGER) << "Registering keep alive context";
+            ctx.registerValue("keep_alive_request_ctx", true);
+            return true;
+        }
+        return false;
+    }
+
+    const char* saas_keep_alive_hdr_name_env = getenv("SAAS_KEEP_ALIVE_HDR_NAME");
+    if (saas_keep_alive_hdr_name_env) {
+        keep_alive_hdr.first = NGEN::Strings::trim(saas_keep_alive_hdr_name_env);
+        dbgInfo(D_HTTP_MANAGER) << "Using SAAS_KEEP_ALIVE_HDR_NAME environment variable: " << keep_alive_hdr.first;
+    }
+
+    if (!keep_alive_hdr.first.empty()) {
+        const char* saas_keep_alive_hdr_value_env = getenv("SAAS_KEEP_ALIVE_HDR_VALUE");
+        if (saas_keep_alive_hdr_value_env) {
+            keep_alive_hdr.second = NGEN::Strings::trim(saas_keep_alive_hdr_value_env);
+            dbgInfo(D_HTTP_MANAGER)
+                << "Using SAAS_KEEP_ALIVE_HDR_VALUE environment variable: "
+                << keep_alive_hdr.second;
+        }
+
+        if (!keep_alive_hdr.second.empty() && (hdr_key == keep_alive_hdr.first && hdr_val == keep_alive_hdr.second)) {
+            dbgTrace(D_HTTP_MANAGER) << "Registering keep alive context";
+            ctx.registerValue("keep_alive_request_ctx", true);
+            keep_alive_hdr_initialized = true;
+            return true;
+        }
+    }
+
+    keep_alive_hdr_initialized = true;
+    return false;
+}

components/attachment-intakers/nginx_attachment/nginx_attachment_opaque.h

@@ -85,6 +85,7 @@ public:
         EnvKeyAttr::LogSection log_ctx = EnvKeyAttr::LogSection::NONE
     );
     void setApplicationState(const ApplicationState &app_state) { application_state = app_state; }
+    bool setKeepAliveCtx(const std::string &hdr_key, const std::string &hdr_val);
 
 private:
     CompressionStream       *response_compression_stream;

components/attachment-intakers/nginx_attachment/nginx_parser.cc

@@ -29,6 +29,7 @@ USE_DEBUG_FLAG(D_NGINX_ATTACHMENT_PARSER);
 Buffer NginxParser::tenant_header_key = Buffer();
 static const Buffer proxy_ip_header_key("X-Forwarded-For", 15, Buffer::MemoryType::STATIC);
 static const Buffer source_ip("sourceip", 8, Buffer::MemoryType::STATIC);
+bool is_keep_alive_ctx = getenv("SAAS_KEEP_ALIVE_HDR_NAME") != nullptr;
 
 map<Buffer, CompressionType> NginxParser::content_encodings = {
     {Buffer("identity"), CompressionType::NO_COMPRESSION},
@@ -177,37 +178,70 @@ getActivetenantAndProfile(const string &str, const string &deli = ",")
 }
 
 Maybe<vector<HttpHeader>>
-NginxParser::parseRequestHeaders(const Buffer &data)
+NginxParser::parseRequestHeaders(const Buffer &data, const unordered_set<string> &ignored_headers)
 {
-    auto parsed_headers = genHeaders(data);
-    if (!parsed_headers.ok()) return parsed_headers.passErr();
+    auto maybe_parsed_headers = genHeaders(data);
+    if (!maybe_parsed_headers.ok()) return maybe_parsed_headers.passErr();
 
     auto i_transaction_table = Singleton::Consume<I_TableSpecific<SessionID>>::by<NginxAttachment>();
+    auto parsed_headers = maybe_parsed_headers.unpack();
+    NginxAttachmentOpaque &opaque = i_transaction_table->getState<NginxAttachmentOpaque>();
 
-    for (const HttpHeader &header : *parsed_headers) {
+    if (is_keep_alive_ctx || !ignored_headers.empty()) {
+        bool is_last_header_removed = false;
+        parsed_headers.erase(
+            remove_if(
+                parsed_headers.begin(),
+                parsed_headers.end(),
+                [&opaque, &is_last_header_removed, &ignored_headers](const HttpHeader &header)
+                {
+                    string hdr_key = static_cast<string>(header.getKey());
+                    string hdr_val = static_cast<string>(header.getValue());
+                    if (
+                        opaque.setKeepAliveCtx(hdr_key, hdr_val)
+                        || ignored_headers.find(hdr_key) != ignored_headers.end()
+                    ) {
+                        dbgTrace(D_NGINX_ATTACHMENT_PARSER) << "Header was removed from headers list: " << hdr_key;
+                        if (header.isLastHeader()) {
+                            dbgTrace(D_NGINX_ATTACHMENT_PARSER) << "Last header was removed from headers list";
+                            is_last_header_removed = true;
+                        }
+                        return true;
+                    }
+                    return false;
+                }
+            ),
+            parsed_headers.end()
+        );
+        if (is_last_header_removed) {
+            dbgTrace(D_NGINX_ATTACHMENT_PARSER) << "Adjusting last header flag";
+            if (!parsed_headers.empty()) parsed_headers.back().setIsLastHeader();
+        }
+    }
+
+    for (const HttpHeader &header : parsed_headers) {
         auto source_identifiers = getConfigurationWithDefault<UsersAllIdentifiersConfig>(
             UsersAllIdentifiersConfig(),
             "rulebase",
             "usersIdentifiers"
         );
         source_identifiers.parseRequestHeaders(header);
-
-        NginxAttachmentOpaque &opaque = i_transaction_table->getState<NginxAttachmentOpaque>();
         opaque.addToSavedData(
             HttpTransactionData::req_headers,
             static_cast<string>(header.getKey()) + ": " + static_cast<string>(header.getValue()) + "\r\n"
         );
 
-        if (NginxParser::tenant_header_key == header.getKey()) {
+        const auto &header_key = header.getKey();
+        if (NginxParser::tenant_header_key == header_key) {
             dbgDebug(D_NGINX_ATTACHMENT_PARSER)
                 << "Identified active tenant header. Key: "
-                << dumpHex(header.getKey())
+                << dumpHex(header_key)
                 << ", Value: "
                 << dumpHex(header.getValue());
 
             auto active_tenant_and_profile = getActivetenantAndProfile(header.getValue());
             opaque.setSessionTenantAndProfile(active_tenant_and_profile[0], active_tenant_and_profile[1]);
-        } else if (proxy_ip_header_key == header.getKey()) {
+        } else if (proxy_ip_header_key == header_key) {
             source_identifiers.setXFFValuesToOpaqueCtx(header, UsersAllIdentifiersConfig::ExtractType::PROXYIP);
         }
     }
@@ -345,12 +379,15 @@ NginxParser::parseResponseBody(const Buffer &raw_response_body, CompressionStrea
 Maybe<CompressionType>
 NginxParser::parseContentEncoding(const vector<HttpHeader> &headers)
 {
-    static const Buffer content_encoding_header_key("Content-Encoding");
+    dbgFlow(D_NGINX_ATTACHMENT_PARSER) << "Parsing \"Content-Encoding\" header";
+    static const Buffer content_encoding_header_key("content-encoding");
 
     auto it = find_if(
         headers.begin(),
         headers.end(),
-        [&] (const HttpHeader &http_header) { return http_header.getKey() == content_encoding_header_key; }
+        [&] (const HttpHeader &http_header) {
+            return http_header.getKey().isEqualLowerCase(content_encoding_header_key);
+        }
     );
     if (it == headers.end()) {
         dbgTrace(D_NGINX_ATTACHMENT_PARSER)

components/attachment-intakers/nginx_attachment/nginx_parser.h

@@ -28,7 +28,10 @@ public:
     static Maybe<HttpTransactionData> parseStartTrasaction(const Buffer &data);
     static Maybe<ResponseCode> parseResponseCode(const Buffer &data);
     static Maybe<uint64_t> parseContentLength(const Buffer &data);
-    static Maybe<std::vector<HttpHeader>> parseRequestHeaders(const Buffer &data);
+    static Maybe<std::vector<HttpHeader>> parseRequestHeaders(
+        const Buffer &data,
+        const std::unordered_set<std::string> &ignored_headers
+    );
     static Maybe<std::vector<HttpHeader>> parseResponseHeaders(const Buffer &data);
     static Maybe<HttpBody> parseRequestBody(const Buffer &data);
     static Maybe<HttpBody> parseResponseBody(const Buffer &raw_response_body, CompressionStream *compression_stream);

components/attachment-intakers/nginx_attachment/user_identifiers_config.cc

@@ -282,21 +282,39 @@ isIpTrusted(const string &value, const vector<CIDRSData> &cidr_values)
 }
 
 Maybe<string>
-UsersAllIdentifiersConfig::parseXForwardedFor(const string &str) const
+UsersAllIdentifiersConfig::parseXForwardedFor(const string &str, ExtractType type) const
 {
     vector<string> header_values = split(str);
-
     if (header_values.empty()) return genError("No IP found in the xff header list");
 
     vector<string> xff_values = getHeaderValuesFromConfig("x-forwarded-for");
     vector<CIDRSData> cidr_values(xff_values.begin(), xff_values.end());
+    string last_valid_ip;
 
-    for (const string &value : header_values) {
-        if (!IPAddr::createIPAddr(value).ok()) {
-            dbgWarning(D_NGINX_ATTACHMENT_PARSER) << "Invalid IP address found in the xff header IPs list: " << value;
-            return genError("Invalid IP address");
+    for (auto it = header_values.rbegin(); it != header_values.rend() - 1; ++it) {
+        if (!IPAddr::createIPAddr(*it).ok()) {
+            dbgWarning(D_NGINX_ATTACHMENT_PARSER) << "Invalid IP address found in the xff header IPs list: " << *it;
+            if (last_valid_ip.empty()) {
+                return genError("Invalid IP address");
+            }
+            return last_valid_ip;
         }
-        if (!isIpTrusted(value, cidr_values)) return genError("Untrusted Ip found");
+        last_valid_ip = *it;
+        if (type == ExtractType::PROXYIP) continue;
+        if (!isIpTrusted(*it, cidr_values)) {
+            dbgDebug(D_NGINX_ATTACHMENT_PARSER) << "Found untrusted IP in the xff header IPs list: " << *it;
+            return *it;
+        }
+    }
+
+    if (!IPAddr::createIPAddr(header_values[0]).ok()) {
+        dbgWarning(D_NGINX_ATTACHMENT_PARSER)
+            << "Invalid IP address found in the xff header IPs list: "
+            << header_values[0];
+        if (last_valid_ip.empty()) {
+            return genError("No Valid Ip address was found");
+        }
+        return last_valid_ip;
     }
 
     return header_values[0];
@@ -312,9 +330,7 @@ UsersAllIdentifiersConfig::setXFFValuesToOpaqueCtx(const HttpHeader &header, Ext
         return;
     }
     NginxAttachmentOpaque &opaque = i_transaction_table->getState<NginxAttachmentOpaque>();
-    opaque.setSavedData(HttpTransactionData::xff_vals_ctx, header.getValue());
-    dbgTrace(D_NGINX_ATTACHMENT_PARSER) << "xff found, value from header: " << static_cast<string>(header.getValue());
-    auto value = parseXForwardedFor(header.getValue());
+    auto value = parseXForwardedFor(header.getValue(), type);
     if (!value.ok()) {
         dbgTrace(D_NGINX_ATTACHMENT_PARSER) << "Could not extract source identifier from X-Forwarded-For header";
         return;
@@ -323,8 +339,13 @@ UsersAllIdentifiersConfig::setXFFValuesToOpaqueCtx(const HttpHeader &header, Ext
     if (type == ExtractType::SOURCEIDENTIFIER) {
         opaque.setSourceIdentifier(header.getKey(), value.unpack());
         dbgDebug(D_NGINX_ATTACHMENT_PARSER)
-            << "Added source identifir to XFF "
+            << "Added source identifier from XFF header"
             <<  value.unpack();
+        opaque.setSavedData(HttpTransactionData::xff_vals_ctx, header.getValue());
+        opaque.setSavedData(HttpTransactionData::source_identifier, value.unpack());
+        dbgTrace(D_NGINX_ATTACHMENT_PARSER)
+            << "XFF found, set ctx with value from header: "
+            << static_cast<string>(header.getValue());
     } else {
         opaque.setSavedData(HttpTransactionData::proxy_ip_ctx, value.unpack());
     }
@@ -345,6 +366,24 @@ UsersAllIdentifiersConfig::setCustomHeaderToOpaqueCtx(const HttpHeader &header)
     return;
 }
 
+void
+UsersAllIdentifiersConfig::setWafTagValuesToOpaqueCtx(const HttpHeader &header) const
+{
+    auto i_transaction_table = Singleton::Consume<I_TableSpecific<SessionID>>::by<NginxAttachment>();
+    if (!i_transaction_table || !i_transaction_table->hasState<NginxAttachmentOpaque>()) {
+        dbgDebug(D_NGINX_ATTACHMENT_PARSER) << "Can't get the transaction table";
+        return;
+    }
+
+    NginxAttachmentOpaque &opaque = i_transaction_table->getState<NginxAttachmentOpaque>();
+    opaque.setSavedData(HttpTransactionData::waf_tag_ctx, static_cast<string>(header.getValue()));
+
+    dbgDebug(D_NGINX_ATTACHMENT_PARSER)
+        << "Added waf tag to context: "
+        <<  static_cast<string>(header.getValue());
+    return;
+}
+
 Maybe<string>
 UsersAllIdentifiersConfig::parseCookieElement(
     const string::const_iterator &start,

components/gradual_deployment/gradual_deployment.cc

@@ -128,7 +128,7 @@ private:
                 break;
             }
             default:
-                dbgAssert(false) << "Unsupported IP type";
+                dbgAssert(false) << AlertInfo(AlertTeam::CORE, "gradual deployment") << "Unsupported IP type";
         }
         return address;
     }
@@ -142,7 +142,7 @@ private:
         if (temp_params_list.size() == 1) {
             Maybe<IPAddr> maybe_ip = IPAddr::createIPAddr(temp_params_list[0]);
             if (!maybe_ip.ok()) return genError("Could not create IP address, " + maybe_ip.getErr());
-            IpAddress addr = move(ConvertToIpAddress(maybe_ip.unpackMove()));
+            IpAddress addr = ConvertToIpAddress(maybe_ip.unpackMove());
 
             return move(IPRange{.start = addr, .end = addr});
         }
@@ -157,11 +157,11 @@ private:
             IPAddr max_addr = maybe_ip_max.unpackMove();
             if (min_addr > max_addr) return genError("Could not create ip range - start greater then end");
 
-            IpAddress addr_min = move(ConvertToIpAddress(move(min_addr)));
-            IpAddress addr_max = move(ConvertToIpAddress(move(max_addr)));
+            IpAddress addr_min = ConvertToIpAddress(move(min_addr));
+            IpAddress addr_max = ConvertToIpAddress(move(max_addr));
             if (addr_max.ip_type != addr_min.ip_type) return genError("Range IP's type does not match");
 
-            return move(IPRange{.start = move(addr_min), .end = move(addr_max)});
+            return IPRange{.start = move(addr_min), .end = move(addr_max)};
         }
 
         return genError("Illegal range received: " + range);

components/http_manager/http_manager.cc

@@ -15,19 +15,18 @@
 
 #include <string>
 #include <map>
-#include <sys/stat.h>
-#include <climits>
 #include <unordered_map>
-#include <boost/range/iterator_range.hpp>
+#include <unordered_set>
+#include <boost/algorithm/string.hpp>
 #include <fstream>
 #include <algorithm>
 
 #include "common.h"
 #include "config.h"
-#include "table_opaque.h"
 #include "http_manager_opaque.h"
 #include "log_generator.h"
 #include "http_inspection_events.h"
+#include "agent_core_utilities.h"
 
 USE_DEBUG_FLAG(D_HTTP_MANAGER);
 
@@ -38,6 +37,7 @@ operator<<(ostream &os, const EventVerdict &event)
 {
     switch (event.getVerdict()) {
         case ngx_http_cp_verdict_e::TRAFFIC_VERDICT_INSPECT: return os << "Inspect";
+        case ngx_http_cp_verdict_e::LIMIT_RESPONSE_HEADERS: return os << "Limit Response Headers";
         case ngx_http_cp_verdict_e::TRAFFIC_VERDICT_ACCEPT: return os << "Accept";
         case ngx_http_cp_verdict_e::TRAFFIC_VERDICT_DROP: return os << "Drop";
         case ngx_http_cp_verdict_e::TRAFFIC_VERDICT_INJECT: return os << "Inject";
@@ -46,7 +46,10 @@ operator<<(ostream &os, const EventVerdict &event)
         case ngx_http_cp_verdict_e::TRAFFIC_VERDICT_WAIT: return os << "Wait";
     }
 
-    dbgAssert(false) << "Illegal Event Verdict value: " << static_cast<uint>(event.getVerdict());
+    dbgAssert(false)
+        << AlertInfo(AlertTeam::CORE, "http manager")
+        << "Illegal Event Verdict value: "
+        << static_cast<uint>(event.getVerdict());
     return os;
 }
 
@@ -91,12 +94,14 @@ public:
         ctx.registerValue(app_sec_marker_key, i_transaction_table->keyToString(), EnvKeyAttr::LogSection::MARKER);
 
         HttpManagerOpaque &state = i_transaction_table->getState<HttpManagerOpaque>();
-        string event_key = static_cast<string>(event.getKey());
-        if (event_key == getProfileAgentSettingWithDefault<string>("", "agent.customHeaderValueLogging")) {
+        
+        const auto &custom_header = getProfileAgentSettingWithDefault<string>("", "agent.customHeaderValueLogging");
+        
+        if (event.getKey().isEqualLowerCase(custom_header)) {
             string event_value = static_cast<string>(event.getValue());
             dbgTrace(D_HTTP_MANAGER)
                 << "Found header key and value - ("
-                << event_key
+                << custom_header
                 << ": "
                 << event_value
                 << ") that matched agent settings";
@@ -192,7 +197,6 @@ public:
         if (state.getUserDefinedValue().ok()) {
             ctx.registerValue("UserDefined", state.getUserDefinedValue().unpack(), EnvKeyAttr::LogSection::DATA);
         }
-
         return handleEvent(EndRequestEvent().performNamedQuery());
     }
 
@@ -320,9 +324,13 @@ private:
                 << respond.second.getVerdict();
 
             state.setApplicationVerdict(respond.first, respond.second.getVerdict());
+            state.setApplicationWebResponse(respond.first, respond.second.getWebUserResponseByPractice());
         }
-
-        return state.getCurrVerdict();
+        FilterVerdict aggregated_verdict(state.getCurrVerdict(), state.getCurrWebUserResponse());
+        if (aggregated_verdict.getVerdict() == ngx_http_cp_verdict_e::TRAFFIC_VERDICT_DROP) {
+            SecurityAppsDropEvent(state.getCurrentDropVerdictCausers()).notify();
+        }
+        return aggregated_verdict;
     }
 
     static void

components/http_manager/http_manager_opaque.cc

@@ -32,6 +32,13 @@ HttpManagerOpaque::setApplicationVerdict(const string &app_name, ngx_http_cp_ver
     applications_verdicts[app_name] = verdict;
 }
 
+void
+HttpManagerOpaque::setApplicationWebResponse(const string &app_name, string web_user_response_id)
+{
+    dbgTrace(D_HTTP_MANAGER) << "Security app: " << app_name << ", has web user response: " << web_user_response_id;
+    applications_web_user_response[app_name] = web_user_response_id;
+}
+
 ngx_http_cp_verdict_e
 HttpManagerOpaque::getApplicationsVerdict(const string &app_name) const
 {
@@ -51,8 +58,12 @@ HttpManagerOpaque::getCurrVerdict() const
     for (const auto &app_verdic_pair : applications_verdicts) {
         switch (app_verdic_pair.second) {
             case ngx_http_cp_verdict_e::TRAFFIC_VERDICT_DROP:
+                dbgTrace(D_HTTP_MANAGER) << "Verdict DROP for app: " << app_verdic_pair.first;
+                current_web_user_response = applications_web_user_response.at(app_verdic_pair.first);
+                dbgTrace(D_HTTP_MANAGER) << "current_web_user_response=" << current_web_user_response;
                 return app_verdic_pair.second;
             case ngx_http_cp_verdict_e::TRAFFIC_VERDICT_INJECT:
+                // Sent in ResponseHeaders and ResponseBody.
                 verdict = ngx_http_cp_verdict_e::TRAFFIC_VERDICT_INJECT;
                 break;
             case ngx_http_cp_verdict_e::TRAFFIC_VERDICT_ACCEPT:
@@ -60,15 +71,21 @@ HttpManagerOpaque::getCurrVerdict() const
                 break;
             case ngx_http_cp_verdict_e::TRAFFIC_VERDICT_INSPECT:
                 break;
+            case ngx_http_cp_verdict_e::LIMIT_RESPONSE_HEADERS:
+                // Sent in End Request.
+                verdict = ngx_http_cp_verdict_e::LIMIT_RESPONSE_HEADERS;
+                break;
             case ngx_http_cp_verdict_e::TRAFFIC_VERDICT_IRRELEVANT:
                 dbgTrace(D_HTTP_MANAGER) << "Verdict 'Irrelevant' is not yet supported. Returning Accept";
                 accepted_apps++;
                 break;
             case ngx_http_cp_verdict_e::TRAFFIC_VERDICT_WAIT:
+                // Sent in Request Headers and Request Body.
                 verdict = ngx_http_cp_verdict_e::TRAFFIC_VERDICT_WAIT;
                 break;
             default:
                 dbgAssert(false)
+                    << AlertInfo(AlertTeam::CORE, "http manager")
                     << "Received unknown verdict "
                     << static_cast<int>(app_verdic_pair.second);
         }
@@ -77,6 +94,25 @@ HttpManagerOpaque::getCurrVerdict() const
     return accepted_apps == applications_verdicts.size() ? ngx_http_cp_verdict_e::TRAFFIC_VERDICT_ACCEPT : verdict;
 }
 
+std::set<std::string>
+HttpManagerOpaque::getCurrentDropVerdictCausers() const
+{
+    std::set<std::string> causers;
+    if (manager_verdict == ngx_http_cp_verdict_e::TRAFFIC_VERDICT_DROP) {
+        causers.insert(HTTP_MANAGER_NAME);
+    }
+    for (const auto &app_verdic_pair : applications_verdicts) {
+        bool was_dropped = app_verdic_pair.second == ngx_http_cp_verdict_e::TRAFFIC_VERDICT_DROP;
+        dbgTrace(D_HTTP_MANAGER)
+            << "The verdict from: " << app_verdic_pair.first
+            << (was_dropped ? " is \"drop\"" : " is not \"drop\" ");
+        if (was_dropped) {
+            causers.insert(app_verdic_pair.first);
+        }
+    }
+    return causers;
+}
+
 void
 HttpManagerOpaque::saveCurrentDataToCache(const Buffer &full_data)
 {

components/http_manager/http_manager_opaque.h

@@ -20,16 +20,21 @@
 #include "table_opaque.h"
 #include "nginx_attachment_common.h"
 
+static const std::string HTTP_MANAGER_NAME = "HTTP Manager";
+
 class HttpManagerOpaque : public TableOpaqueSerialize<HttpManagerOpaque>
 {
 public:
     HttpManagerOpaque();
 
     void setApplicationVerdict(const std::string &app_name, ngx_http_cp_verdict_e verdict);
+    void setApplicationWebResponse(const std::string &app_name, std::string web_user_response_id);
     ngx_http_cp_verdict_e getApplicationsVerdict(const std::string &app_name) const;
     void setManagerVerdict(ngx_http_cp_verdict_e verdict) { manager_verdict = verdict; }
     ngx_http_cp_verdict_e getManagerVerdict() const { return manager_verdict; }
     ngx_http_cp_verdict_e getCurrVerdict() const;
+    const std::string & getCurrWebUserResponse() const { return current_web_user_response; };
+    std::set<std::string> getCurrentDropVerdictCausers() const;
     void saveCurrentDataToCache(const Buffer &full_data);
     void setUserDefinedValue(const std::string &value) { user_defined_value = value; }
     Maybe<std::string> getUserDefinedValue() const { return user_defined_value; }
@@ -49,6 +54,8 @@ public:
 
 private:
     std::unordered_map<std::string, ngx_http_cp_verdict_e> applications_verdicts;
+    std::unordered_map<std::string, std::string> applications_web_user_response;
+    mutable std::string current_web_user_response;
     ngx_http_cp_verdict_e manager_verdict = ngx_http_cp_verdict_e::TRAFFIC_VERDICT_INSPECT;
     Buffer prev_data_cache;
     uint aggregated_payload_size = 0;

components/include/central_nginx_manager.h

@@ -0,0 +1,45 @@
+// Copyright (C) 2022 Check Point Software Technologies Ltd. All rights reserved.
+
+// Licensed under the Apache License, Version 2.0 (the "License");
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#ifndef __CENTRAL_NGINX_MANAGER_H__
+#define __CENTRAL_NGINX_MANAGER_H__
+
+#include "component.h"
+#include "singleton.h"
+#include "i_messaging.h"
+#include "i_rest_api.h"
+#include "i_mainloop.h"
+#include "i_agent_details.h"
+
+class CentralNginxManager
+        :
+    public Component,
+    Singleton::Consume<I_RestApi>,
+    Singleton::Consume<I_Messaging>,
+    Singleton::Consume<I_MainLoop>,
+    Singleton::Consume<I_AgentDetails>
+{
+public:
+    CentralNginxManager();
+    ~CentralNginxManager();
+
+    void preload() override;
+    void init() override;
+    void fini() override;
+
+private:
+    class Impl;
+    std::unique_ptr<Impl> pimpl;
+};
+
+#endif // __CENTRAL_NGINX_MANAGER_H__

components/include/external_sdk_server.h

@@ -24,7 +24,8 @@ class ExternalSdkServer
         :
     public Component,
     Singleton::Provide<I_ExternalSdkServer>,
-    Singleton::Consume<I_RestApi>
+    Singleton::Consume<I_RestApi>,
+    Singleton::Consume<I_Messaging>
 {
 public:
     ExternalSdkServer();

components/include/generic_rulebase/evaluators/http_transaction_data_eval.h

@@ -45,6 +45,19 @@ private:
     std::string host;
 };
 
+class EqualWafTag : public EnvironmentEvaluator<bool>, Singleton::Consume<I_Environment>
+{
+public:
+EqualWafTag(const std::vector<std::string> &params);
+
+    static std::string getName() { return "EqualWafTag"; }
+
+    Maybe<bool, Context::Error> evalVariable() const override;
+
+private:
+    std::string waf_tag;
+};
+
 class EqualListeningIP : public EnvironmentEvaluator<bool>, Singleton::Consume<I_Environment>
 {
 public:

components/include/generic_rulebase/match_query.h

@@ -89,7 +89,9 @@ private:
     bool matchAttributesRegEx(const std::set<std::string> &values,
             std::set<std::string> &matched_override_keywords) const;
     bool matchAttributesString(const std::set<std::string> &values) const;
+    bool matchAttributesIp(const std::set<std::string> &values) const;
     bool isRegEx() const;
+    void sortAndMergeIpRangesValues();
 
     MatchType type;
     Operators operator_type;

components/include/generic_rulebase/triggers_config.h

@@ -317,12 +317,12 @@ public:
     {
         return url_for_cef;
     }
+    Flags<ReportIS::StreamType> getStreams(SecurityType security_type, bool is_action_drop_or_prevent) const;
+    Flags<ReportIS::Enreachments> getEnrechments(SecurityType security_type) const;
 
 private:
     ReportIS::Severity getSeverity(bool is_action_drop_or_prevent) const;
     ReportIS::Priority getPriority(bool is_action_drop_or_prevent) const;
-    Flags<ReportIS::StreamType> getStreams(SecurityType security_type, bool is_action_drop_or_prevent) const;
-    Flags<ReportIS::Enreachments> getEnrechments(SecurityType security_type) const;
 
     std::string name;
     std::string verbosity;
@@ -339,4 +339,32 @@ private:
     bool should_format_output = false;
 };
 
+class ReportTriggerConf
+{
+public:
+    /// \brief Default constructor for ReportTriggerConf.
+    ReportTriggerConf() {}
+
+    /// \brief Preload function to register expected configuration.
+    static void
+    preload()
+    {
+        registerExpectedConfiguration<ReportTriggerConf>("rulebase", "report");
+    }
+
+    /// \brief Load function to deserialize configuration from JSONInputArchive.
+    /// \param archive_in The JSON input archive.
+    void load(cereal::JSONInputArchive &archive_in);
+
+    /// \brief Get the name.
+    /// \return The name.
+    const std::string &
+    getName() const
+    {
+        return name;
+    }
+private:
+    std::string name;
+};
+
 #endif //__TRIGGERS_CONFIG_H__

components/include/health_checker.h

@@ -21,6 +21,7 @@
 #include "i_shell_cmd.h"
 #include "i_orchestration_status.h"
 #include "component.h"
+#include "i_service_controller.h"
 
 class HealthChecker
         :
@@ -29,7 +30,8 @@ class HealthChecker
     Singleton::Consume<I_Socket>,
     Singleton::Consume<I_Health_Check_Manager>,
     Singleton::Consume<I_ShellCmd>,
-    Singleton::Consume<I_OrchestrationStatus>
+    Singleton::Consume<I_OrchestrationStatus>,
+    Singleton::Consume<I_ServiceController>
 {
 public:
     HealthChecker();

components/include/http_event_impl/filter_verdict.h

@@ -27,9 +27,18 @@ public:
         verdict(_verdict)
     {}
 
+    FilterVerdict(
+        ngx_http_cp_verdict_e _verdict,
+        const std::string &_web_reponse_id)
+            :
+        verdict(_verdict),
+        web_user_response_id(_web_reponse_id)
+    {}
+
     FilterVerdict(const EventVerdict &_verdict, ModifiedChunkIndex _event_idx = -1)
             :
-        verdict(_verdict.getVerdict())
+        verdict(_verdict.getVerdict()),
+        web_user_response_id(_verdict.getWebUserResponseByPractice())
     {
         if (verdict == ngx_http_cp_verdict_e::TRAFFIC_VERDICT_INJECT) {
             addModifications(_verdict.getModifications(), _event_idx);
@@ -59,10 +68,12 @@ public:
     uint getModificationsAmount() const { return total_modifications; }
     ngx_http_cp_verdict_e getVerdict() const { return verdict; }
     const std::vector<EventModifications> & getModifications() const { return modifications; }
+    const std::string getWebUserResponseID() const { return web_user_response_id; }
 
 private:
     ngx_http_cp_verdict_e verdict = ngx_http_cp_verdict_e::TRAFFIC_VERDICT_INSPECT;
     std::vector<EventModifications> modifications;
+    std::string web_user_response_id;
     uint total_modifications = 0;
 };
 

components/include/http_event_impl/i_http_event_impl.h

@@ -50,9 +50,11 @@ public:
         position(mod_position)
     {
         dbgAssert(mod_type != ModificationType::APPEND || position == injection_pos_irrelevant)
+            << AlertInfo(AlertTeam::CORE, "http manager")
             << "Injection position is not applicable to a modification of type \"Append\"";
 
         dbgAssert(mod_type != ModificationType::INJECT || position >= 0)
+            << AlertInfo(AlertTeam::CORE, "http manager")
             << "Invalid injection position: must be non-negative. Position: "
             << position;
     }
@@ -166,6 +168,7 @@ private:
             }
             default:
                 dbgAssert(false)
+                    << AlertInfo(AlertTeam::CORE, "http manager")
                     << "Unknown type of ModificationType: "
                     << static_cast<int>(modification_type);
         }
@@ -236,6 +239,7 @@ public:
     const Buffer & getValue() const { return value; }
 
     bool isLastHeader() const { return is_last_header; }
+    void setIsLastHeader() { is_last_header = true; }
     uint8_t getHeaderIndex() const { return header_index; }
 
 private:
@@ -372,16 +376,31 @@ public:
         verdict(event_verdict)
     {}
 
+    EventVerdict(
+        const ModificationList &mods,
+        ngx_http_cp_verdict_e event_verdict,
+        std::string response_id) :
+        modifications(mods),
+        verdict(event_verdict),
+        webUserResponseByPractice(response_id)
+    {}
+
 // LCOV_EXCL_START - sync functions, can only be tested once the sync module exists
     template <typename T> void serialize(T &ar, uint) { ar(verdict); }
 // LCOV_EXCL_STOP
 
     const ModificationList & getModifications() const { return modifications; }
     ngx_http_cp_verdict_e getVerdict() const { return verdict; }
+    const std::string getWebUserResponseByPractice() const { return webUserResponseByPractice; }
+    void setWebUserResponseByPractice(const std::string id) {
+        dbgTrace(D_HTTP_MANAGER) << "current verdict web user response set to: " << id;
+        webUserResponseByPractice = id;
+    }
 
 private:
     ModificationList modifications;
     ngx_http_cp_verdict_e verdict = ngx_http_cp_verdict_e::TRAFFIC_VERDICT_INSPECT;
+    std::string webUserResponseByPractice;
 };
 
 #endif // __I_HTTP_EVENT_IMPL_H__

components/include/http_inspection_events.h

@@ -183,4 +183,16 @@ class WaitTransactionEvent : public Event<WaitTransactionEvent, EventVerdict>
 {
 };
 
+class SecurityAppsDropEvent : public Event<SecurityAppsDropEvent>
+{
+public:
+    SecurityAppsDropEvent(
+        const std::set<std::string> &apps_names)
+            :
+        apps_names(apps_names) {}
+    const std::set<std::string> & getAppsNames() const { return apps_names; }
+
+private:
+    const std::set<std::string> apps_names;
+};
 #endif // __HTTP_INSPECTION_EVENTS_H__

components/include/http_transaction_data.h

@@ -72,7 +72,8 @@ public:
             parsed_uri,
             client_ip,
             client_port,
-            response_content_encoding
+            response_content_encoding,
+            waf_tag
         );
     }
 
@@ -91,7 +92,8 @@ public:
             parsed_uri,
             client_ip,
             client_port,
-            response_content_encoding
+            response_content_encoding,
+            waf_tag
         );
     }
 // LCOV_EXCL_STOP
@@ -122,6 +124,9 @@ public:
         response_content_encoding = _response_content_encoding;
     }
 
+    const std::string & getWafTag() const { return waf_tag; }
+    void setWafTag(const std::string &_waf_tag) { waf_tag = _waf_tag; }
+
     static const std::string http_proto_ctx;
     static const std::string method_ctx;
     static const std::string host_name_ctx;
@@ -137,6 +142,7 @@ public:
     static const std::string source_identifier;
     static const std::string proxy_ip_ctx;
     static const std::string xff_vals_ctx;
+    static const std::string waf_tag_ctx;
 
     static const CompressionType default_response_content_encoding;
 
@@ -153,6 +159,7 @@ private:
     uint16_t client_port;
     bool is_request;
     CompressionType response_content_encoding;
+    std::string waf_tag;
 };
 
 #endif // __HTTP_TRANSACTION_DATA_H__

components/include/i_details_resolver.h

@@ -26,11 +26,12 @@ public:
     virtual Maybe<std::string> getArch() = 0;
     virtual std::string getAgentVersion() = 0;
     virtual bool isKernelVersion3OrHigher() = 0;
+    virtual bool isGw() = 0;
     virtual bool isGwNotVsx() = 0;
     virtual bool isVersionAboveR8110() = 0;
     virtual bool isReverseProxy() = 0;
     virtual bool isCloudStorageEnabled() = 0;
-    virtual Maybe<std::tuple<std::string, std::string, std::string>> parseNginxMetadata() = 0;
+    virtual Maybe<std::tuple<std::string, std::string, std::string, std::string>> parseNginxMetadata() = 0;
     virtual Maybe<std::tuple<std::string, std::string, std::string, std::string, std::string>> readCloudMetadata() = 0;
     virtual std::map<std::string, std::string> getResolvedDetails() = 0;
 #if defined(gaia) || defined(smb)

components/include/i_service_controller.h

@@ -66,6 +66,8 @@ public:
 
     virtual std::map<std::string, std::vector<PortNumber>> getServiceToPortMap() = 0;
 
+    virtual bool getServicesPolicyStatus() const = 0;
+
 protected:
     virtual ~I_ServiceController() {}
 };

components/include/i_waap_telemetry.h

@@ -27,6 +27,7 @@ struct DecisionTelemetryData
     int responseCode;
     uint64_t elapsedTime;
     std::set<std::string> attackTypes;
+    bool temperatureDetected;
 
     DecisionTelemetryData() :
         blockType(NOT_BLOCKING),
@@ -38,7 +39,8 @@ struct DecisionTelemetryData
         method(POST),
         responseCode(0),
         elapsedTime(0),
-        attackTypes()
+        attackTypes(),
+        temperatureDetected(false)
     {
     }
 };

components/include/ip_utilities.h

@@ -28,8 +28,9 @@
 
 // LCOV_EXCL_START Reason: temporary until we add relevant UT until 07/10
 bool operator<(const IpAddress &this_ip_addr, const IpAddress &other_ip_addr);
-
 bool operator==(const IpAddress &this_ip_addr, const IpAddress &other_ip_addr);
+bool operator<=(const IpAddress &this_ip_addr, const IpAddress &other_ip_addr);
+bool operator<(const IPRange &range1, const IPRange &range2);
 // LCOV_EXCL_STOP
 
 Maybe<std::pair<std::string, int>> extractAddressAndMaskSize(const std::string &cidr);

components/include/ips_comp.h

@@ -4,6 +4,7 @@
 #include "singleton.h"
 #include "i_keywords_rule.h"
 #include "i_table.h"
+#include "i_mainloop.h"
 #include "i_http_manager.h"
 #include "i_environment.h"
 #include "http_inspection_events.h"
@@ -16,7 +17,8 @@ class IPSComp
     Singleton::Consume<I_KeywordsRule>,
     Singleton::Consume<I_Table>,
     Singleton::Consume<I_Environment>,
-    Singleton::Consume<I_GenericRulebase>
+    Singleton::Consume<I_GenericRulebase>,
+    Singleton::Consume<I_MainLoop>
 {
 public:
     IPSComp();

components/include/manifest_handler.h

@@ -62,6 +62,7 @@ public:
 
 private:
     Maybe<std::string> downloadPackage(const Package &package, bool is_clean_installation);
+    std::string getCurrentTimestamp();
 
     std::string manifest_file_path;
     std::string temp_ext;

components/include/nginx_message_reader.h

@@ -0,0 +1,28 @@
+#ifndef __NGINX_MESSAGE_READER_H__
+#define __NGINX_MESSAGE_READER_H__
+
+#include "singleton.h"
+#include "i_mainloop.h"
+#include "i_socket_is.h"
+#include "component.h"
+
+class NginxMessageReader
+        :
+    public Component,
+    Singleton::Consume<I_MainLoop>,
+    Singleton::Consume<I_Socket>
+{
+public:
+    NginxMessageReader();
+    ~NginxMessageReader();
+
+    void init() override;
+    void fini() override;
+    void preload() override;
+
+private:
+    class Impl;
+    std::unique_ptr<Impl> pimpl;
+};
+
+#endif //__NGINX_MESSAGE_READER_H__

components/include/nginx_utils.h

@@ -0,0 +1,51 @@
+// Copyright (C) 2022 Check Point Software Technologies Ltd. All rights reserved.
+
+// Licensed under the Apache License, Version 2.0 (the "License");
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#ifndef __NGINX_UTILS_H__
+#define __NGINX_UTILS_H__
+
+#include <string>
+
+#include "maybe_res.h"
+#include "singleton.h"
+#include "i_shell_cmd.h"
+
+class NginxConfCollector
+{
+public:
+    NginxConfCollector(const std::string &nginx_conf_input_path, const std::string &nginx_conf_output_path);
+    Maybe<std::string> generateFullNginxConf() const;
+
+private:
+    std::vector<std::string> expandIncludes(const std::string &includePattern) const;
+    void processConfigFile(
+        const std::string &path,
+        std::ostringstream &conf_output,
+        std::vector<std::string> &errors
+    ) const;
+
+    std::string main_conf_input_path;
+    std::string main_conf_output_path;
+    std::string main_conf_directory_path;
+};
+
+class NginxUtils : Singleton::Consume<I_ShellCmd>
+{
+public:
+    static std::string getModulesPath();
+    static std::string getMainNginxConfPath();
+    static Maybe<void> validateNginxConf(const std::string &nginx_conf_path);
+    static Maybe<void> reloadNginx(const std::string &nginx_conf_path);
+};
+
+#endif // __NGINX_UTILS_H__

components/include/orchestrator/rest_api/get_resource_file.h

@@ -115,7 +115,7 @@ public:
             case ResourceFileType::VIRTUAL_SETTINGS: return "virtualSettings";
             case ResourceFileType::VIRTUAL_POLICY:   return "virtualPolicy";
             default:
-                dbgAssert(false) << "Unknown file type";
+                dbgAssert(false) << AlertInfo(AlertTeam::CORE, "update process") << "Unknown file type";
         }
         return std::string();
     }

components/include/package.h

@@ -56,7 +56,7 @@ private:
             if (mapped_type.second == type) return mapped_type.first;
         }
 
-        dbgAssert(false) << "Unsupported type " << static_cast<int>(type);
+        dbgAssert(false) << AlertInfo(AlertTeam::CORE, "packaging") << "Unsupported type " << static_cast<int>(type);
         // Just satisfying the compiler, this return never reached
         return std::string();
     }

components/include/prometheus_comp.h

@@ -0,0 +1,30 @@
+#ifndef __PROMETHEUS_COMP_H__
+#define __PROMETHEUS_COMP_H__
+
+#include <memory>
+
+#include "component.h"
+#include "singleton.h"
+
+#include "i_rest_api.h"
+#include "i_messaging.h"
+#include "generic_metric.h"
+
+class PrometheusComp
+        :
+    public Component,
+    Singleton::Consume<I_RestApi>,
+    Singleton::Consume<I_Messaging>
+{
+public:
+    PrometheusComp();
+    ~PrometheusComp();
+
+    void init() override;
+
+private:
+    class Impl;
+    std::unique_ptr<Impl> pimpl;
+};
+
+#endif // __PROMETHEUS_COMP_H__

components/include/rate_limit.h

@@ -7,15 +7,21 @@
 #include "singleton.h"
 #include "i_mainloop.h"
 #include "i_environment.h"
+#include "i_geo_location.h"
 #include "i_generic_rulebase.h"
+#include "i_shell_cmd.h"
+#include "i_env_details.h"
 
 class RateLimit
     :
     public Component,
     Singleton::Consume<I_MainLoop>,
     Singleton::Consume<I_TimeGet>,
+    Singleton::Consume<I_GeoLocation>,
     Singleton::Consume<I_Environment>,
-    Singleton::Consume<I_GenericRulebase>
+    Singleton::Consume<I_GenericRulebase>,
+    Singleton::Consume<I_ShellCmd>,
+    Singleton::Consume<I_EnvDetails>
 {
 public:
     RateLimit();

components/include/reverse_proxy_defaults.h

@@ -7,24 +7,28 @@ static const std::string product_name = getenv("DOCKER_RPM_ENABLED") ? "CloudGua
 static const std::string default_cp_cert_file = "/etc/cp/cpCert.pem";
 static const std::string default_cp_key_file = "/etc/cp/cpKey.key";
 static const std::string default_rpm_conf_path = "/etc/cp/conf/rpmanager/";
+
 static const std::string default_certificate_path = "/etc/cp/rpmanager/certs";
+static const std::string default_manual_certs_path = "/etc/cp/rpmanager/manualCerts/";
+static const std::string default_config_path = "/etc/cp/conf/rpmanager/servers";
+static const std::string default_rpm_prepare_path = "/etc/cp/conf/rpmanager/prepare/servers";
+
+static const std::string default_nginx_log_files_path = "/var/log/nginx/";
 static const std::string default_additional_files_path = "/etc/cp/conf/rpmanager/include";
 static const std::string default_server_config = "additional_server_config.conf";
 static const std::string default_location_config = "additional_location_config.conf";
 static const std::string default_trusted_ca_suffix = "_user_ca_bundle.crt";
-static const std::string default_nginx_log_files_path = "/var/log/nginx/";
 static const std::string default_log_files_host_path = "/var/log/nano_agent/rpmanager/nginx_log/";
-static const std::string default_config_path = "/etc/cp/conf/rpmanager/servers";
 static const std::string default_template_path = "/etc/cp/conf/rpmanager/nginx-template-clear";
-static const std::string default_manual_certs_path = "/etc/cp/rpmanager/manualCerts/";
 static const std::string default_server_certificate_path = "/etc/cp/rpmanager/certs/sslCertificate_";
 static const std::string default_server_certificate_key_path = "/etc/cp/rpmanager/certs/sslPrivateKey_";
 static const std::string default_container_name = "cp_nginx_gaia";
 static const std::string default_docker_image = "cp_nginx_gaia";
 static const std::string default_nginx_config_file = "/etc/cp/conf/rpmanager/nginx.conf";
+static const std::string default_prepare_nginx_config_file = "/etc/cp/conf/rpmanager/nginx_prepare.conf";
 static const std::string default_global_conf_template = "/etc/cp/conf/rpmanager/nginx-conf-template";
 static const std::string default_nginx_config_include_file =
-    "/etc/cp/conf/rpmanager/servers/nginx_conf_include";
+    "/etc/cp/conf/rpmanager/servers/00_nginx_conf_include.conf";
 static const std::string default_global_conf_include_template =
     "/etc/cp/conf/rpmanager/nginx-conf-include-template";
 static const std::string default_global_conf_include_template_no_responses =

components/include/service_health_status.h

@@ -0,0 +1,39 @@
+// Copyright (C) 2022 Check Point Software Technologies Ltd. All rights reserved.
+
+// Licensed under the Apache License, Version 2.0 (the "License");
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#ifndef __SERVICE_HEALTH_STATUS_H__
+#define __SERVICE_HEALTH_STATUS_H__
+
+#include "singleton.h"
+#include "i_rest_api.h"
+#include "i_environment.h"
+#include "component.h"
+
+class ServiceHealthStatus
+        :
+    public Component,
+    Singleton::Consume<I_RestApi>,
+    Singleton::Consume<I_Environment>
+{
+public:
+    ServiceHealthStatus();
+    ~ServiceHealthStatus();
+
+    void init() override;
+
+private:
+    class Impl;
+    std::unique_ptr<Impl> pimpl;
+};
+
+#endif // __SERVICE_HEALTH_STATUS_H__

components/include/telemetry.h

@@ -30,6 +30,7 @@
 #include "generic_metric.h"
 
 #define LOGGING_INTERVAL_IN_MINUTES 10
+USE_DEBUG_FLAG(D_WAAP);
 enum class AssetType { API, WEB, ALL, COUNT };
 
 class WaapTelemetryEvent : public Event<WaapTelemetryEvent>
@@ -75,6 +76,20 @@ private:
     std::unordered_set<std::string> sources_seen;
 };
 
+class WaapAdditionalTrafficTelemetrics : public WaapTelemetryBase
+{
+public:
+    void updateMetrics(const std::string &asset_id, const DecisionTelemetryData &data);
+    void initMetrics();
+
+private:
+    MetricCalculations::Counter requests{this, "reservedNgenA"};
+    MetricCalculations::Counter sources{this, "reservedNgenB"};
+    MetricCalculations::Counter blocked{this, "reservedNgenC"};
+    MetricCalculations::Counter temperature_count{this, "reservedNgenD"};
+    std::unordered_set<std::string> sources_seen;
+};
+
 class WaapTrafficTelemetrics : public WaapTelemetryBase
 {
 public:
@@ -123,6 +138,7 @@ private:
     std::map<std::string, std::shared_ptr<WaapTrafficTelemetrics>> traffic_telemetries;
     std::map<std::string, std::shared_ptr<WaapAttackTypesMetrics>> attack_types;
     std::map<std::string, std::shared_ptr<WaapAttackTypesMetrics>> attack_types_telemetries;
+    std::map<std::string, std::shared_ptr<WaapAdditionalTrafficTelemetrics>> additional_traffic_telemetries;
 
     template <typename T>
     void initializeTelemetryData(
@@ -132,6 +148,7 @@ private:
         std::map<std::string, std::shared_ptr<T>>& telemetryMap
     ) {
         if (!telemetryMap.count(asset_id)) {
+            dbgTrace(D_WAAP) << "creating telemetry data for asset: " << data.assetName;
             telemetryMap.emplace(asset_id, std::make_shared<T>());
             telemetryMap[asset_id]->init(
                 telemetryName,
@@ -139,7 +156,9 @@ private:
                 ReportIS::IssuingEngine::AGENT_CORE,
                 std::chrono::minutes(LOGGING_INTERVAL_IN_MINUTES),
                 true,
-                ReportIS::Audience::SECURITY
+                ReportIS::Audience::SECURITY,
+                false,
+                asset_id
             );
 
             telemetryMap[asset_id]->template registerContext<std::string>(
@@ -152,29 +171,30 @@ private:
                 std::string("Web Application"),
                 EnvKeyAttr::LogSection::SOURCE
             );
-            telemetryMap[asset_id]->template registerContext<std::string>(
-                "assetId",
-                asset_id,
-                EnvKeyAttr::LogSection::SOURCE
-            );
-            telemetryMap[asset_id]->template registerContext<std::string>(
-                "assetName",
-                data.assetName,
-                EnvKeyAttr::LogSection::SOURCE
-            );
-            telemetryMap[asset_id]->template registerContext<std::string>(
-                "practiceId",
-                data.practiceId,
-                EnvKeyAttr::LogSection::SOURCE
-            );
-            telemetryMap[asset_id]->template registerContext<std::string>(
-                "practiceName",
-                data.practiceName,
-                EnvKeyAttr::LogSection::SOURCE
-            );
-
             telemetryMap[asset_id]->registerListener();
         }
+        dbgTrace(D_WAAP) << "updating telemetry data for asset: " << data.assetName;
+
+        telemetryMap[asset_id]->template registerContext<std::string>(
+            "assetId",
+            asset_id,
+            EnvKeyAttr::LogSection::SOURCE
+        );
+        telemetryMap[asset_id]->template registerContext<std::string>(
+            "assetName",
+            data.assetName,
+            EnvKeyAttr::LogSection::SOURCE
+        );
+        telemetryMap[asset_id]->template registerContext<std::string>(
+            "practiceId",
+            data.practiceId,
+            EnvKeyAttr::LogSection::SOURCE
+        );
+        telemetryMap[asset_id]->template registerContext<std::string>(
+            "practiceName",
+            data.practiceName,
+            EnvKeyAttr::LogSection::SOURCE
+        );
     }
 };
 

components/include/user_identifiers_config.h

@@ -30,6 +30,7 @@ public:
     void parseRequestHeaders(const HttpHeader &header) const;
     std::vector<std::string> getHeaderValuesFromConfig(const std::string &header_key) const;
     void setXFFValuesToOpaqueCtx(const HttpHeader &header, ExtractType type) const;
+    void setWafTagValuesToOpaqueCtx(const HttpHeader &header) const;
 
 private:
     class UsersIdentifiersConfig
@@ -58,7 +59,7 @@ private:
         const std::string::const_iterator &end,
         const std::string &key) const;
     Buffer extractKeyValueFromCookie(const std::string &cookie_value, const std::string &key) const;
-    Maybe<std::string> parseXForwardedFor(const std::string &str) const;
+    Maybe<std::string> parseXForwardedFor(const std::string &str, ExtractType type) const;
 
     std::vector<UsersIdentifiersConfig> user_identifiers;
 };

components/include/waap.h

@@ -33,6 +33,9 @@ class I_WaapAssetStatesManager;
 class I_Messaging;
 class I_AgentDetails;
 class I_Encryptor;
+class I_WaapModelResultLogger;
+
+const std::string WAAP_APPLICATION_NAME = "waap application";
 
 class WaapComponent
         :
@@ -48,7 +51,8 @@ class WaapComponent
     Singleton::Consume<I_AgentDetails>,
     Singleton::Consume<I_Messaging>,
     Singleton::Consume<I_Encryptor>,
-    Singleton::Consume<I_Environment>
+    Singleton::Consume<I_Environment>,
+    Singleton::Consume<I_WaapModelResultLogger>
 {
 public:
     WaapComponent();

components/nginx_message_reader/CMakeLists.txt

@@ -0,0 +1,3 @@
+link_directories(${BOOST_ROOT}/lib)
+
+add_library(nginx_message_reader nginx_message_reader.cc)

components/nginx_message_reader/nginx_message_reader.cc

@@ -0,0 +1,735 @@
+#include "nginx_message_reader.h"
+
+#include <string>
+#include <boost/regex.hpp>
+#include <boost/algorithm/string.hpp>
+#include <boost/algorithm/string/regex.hpp>
+
+#include "config.h"
+#include "singleton.h"
+#include "i_mainloop.h"
+#include "enum_array.h"
+#include "log_generator.h"
+#include "maybe_res.h"
+#include "http_transaction_data.h"
+#include "generic_rulebase/rulebase_config.h"
+#include "generic_rulebase/evaluators/asset_eval.h"
+#include "generic_rulebase/triggers_config.h"
+#include "agent_core_utilities.h"
+#include "rate_limit_config.h"
+
+USE_DEBUG_FLAG(D_NGINX_MESSAGE_READER);
+
+using namespace std;
+
+static const string syslog_regex_string = (
+    "<[0-9]+>([A-Z][a-z][a-z]\\s{1,2}\\d{1,2}\\s\\d{2}"
+    "[:]\\d{2}[:]\\d{2})\\s([\\w][\\w\\d\\.@-]*)\\s(nginx:)"
+);
+
+static const boost::regex socket_address_regex("(\\d+\\.\\d+\\.\\d+\\.\\d+):(\\d+)");
+static const boost::regex syslog_regex(syslog_regex_string);
+static const boost::regex alert_log_regex(
+    "("
+    + syslog_regex_string + ") "
+    + "(.+?\\[alert\\] )(.+?)"
+    ", (client: .+?)"
+    ", (server: .+?)"
+    ", (request: \".+?\")"
+    ", (upstream: \".+?\")"
+    ", (host: \".+?\")$"
+);
+
+static const boost::regex error_log_regex(
+    "("
+    + syslog_regex_string + ") "
+    + "(.+?\\[error\\] )(.+?)"
+    ", (client: .+?)"
+    ", (server: .+?)"
+    ", (request: \".+?\")"
+    ", (upstream: \".+?\")"
+    ", (host: \".+?\")$"
+);
+
+static const boost::regex server_regex("(\\d+\\.\\d+\\.\\d+\\.\\d+)|(\\w+\\.\\w+)");
+static const boost::regex uri_regex("^/");
+static const boost::regex port_regex("\\d+");
+static const boost::regex response_code_regex("[0-9]{3}");
+static const boost::regex http_method_regex("[A-Za-z]+");
+
+class NginxMessageReader::Impl
+{
+public:
+    void
+    init()
+    {
+        dbgFlow(D_NGINX_MESSAGE_READER);
+        I_MainLoop *mainloop = Singleton::Consume<I_MainLoop>::by<NginxMessageReader>();
+        mainloop->addOneTimeRoutine(
+            I_MainLoop::RoutineType::System,
+            [this] ()
+            {
+                initSyslogServerSocket();
+                handleNginxLogs();
+            },
+            "Initialize nginx syslog",
+            true
+        );
+    }
+
+    void
+    preload()
+    {
+        registerConfigLoadCb([this]() { loadNginxMessageReaderConfig(); });
+    }
+
+    void
+    fini()
+    {
+        I_Socket *i_socket = Singleton::Consume<I_Socket>::by<NginxMessageReader>();
+        i_socket->closeSocket(syslog_server_socket);
+    }
+
+    void
+    loadNginxMessageReaderConfig()
+    {
+        rate_limit_status_code = getProfileAgentSettingWithDefault<string>(
+            "429",
+            "accessControl.rateLimit.returnCode"
+        );
+        dbgTrace(D_NGINX_MESSAGE_READER) << "Selected rate-limit status code: " << rate_limit_status_code;
+    }
+
+private:
+    enum class LogInfo {
+        HTTP_METHOD,
+        URI,
+        RESPONSE_CODE,
+        HOST,
+        SOURCE,
+        DESTINATION_IP,
+        DESTINATION_PORT,
+        EVENT_MESSAGE,
+        ASSET_ID,
+        ASSET_NAME,
+        RULE_NAME,
+        RULE_ID,
+        COUNT
+    };
+
+    void
+    initSyslogServerSocket()
+    {
+        dbgFlow(D_NGINX_MESSAGE_READER);
+        I_MainLoop *mainloop = Singleton::Consume<I_MainLoop>::by<NginxMessageReader>();
+        I_Socket *i_socket = Singleton::Consume<I_Socket>::by<NginxMessageReader>();
+        string nginx_syslog_server_address = getProfileAgentSettingWithDefault<string>(
+            "127.0.0.1:1514",
+            "reverseProxy.nginx.syslogAddress"
+        );
+        dbgInfo(D_NGINX_MESSAGE_READER) << "Attempting to open a socket: " << nginx_syslog_server_address;
+        do {
+            Maybe<I_Socket::socketFd> new_socket = i_socket->genSocket(
+                I_Socket::SocketType::UDP,
+                false,
+                true,
+                nginx_syslog_server_address
+            );
+            if (!new_socket.ok()) {
+                dbgError(D_NGINX_MESSAGE_READER) << "Failed to open a socket. Error: " << new_socket.getErr();
+                mainloop->yield(chrono::milliseconds(500));
+                continue;
+            }
+
+            if (new_socket.unpack() < 0) {
+                dbgError(D_NGINX_MESSAGE_READER)<< "Generated socket is OK yet negative";
+                mainloop->yield(chrono::milliseconds(500));
+                continue;
+            }
+            syslog_server_socket = new_socket.unpack();
+            dbgInfo(D_NGINX_MESSAGE_READER)
+                << "Opened socket for nginx logs over syslog. Socket: "
+                << syslog_server_socket;
+        } while (syslog_server_socket < 0);
+    }
+
+    void
+    handleNginxLogs()
+    {
+        dbgFlow(D_NGINX_MESSAGE_READER);
+        I_MainLoop::Routine read_logs =
+        [this] ()
+        {
+            Maybe<string> logs = getLogsFromSocket(syslog_server_socket);
+
+            if (!logs.ok()) {
+                dbgWarning(D_NGINX_MESSAGE_READER)
+                    << "Failed to get NGINX logs from the socket. Error: "
+                    << logs.getErr();
+                return;
+            }
+            string raw_logs_to_parse = logs.unpackMove();
+            vector<string> logs_to_parse = separateLogs(raw_logs_to_parse);
+
+            for (auto const &log: logs_to_parse) {
+                bool log_sent;
+                if (isAccessLog(log)) {
+                    log_sent = sendAccessLog(log);
+                } else if (isAlertErrorLog(log) || isErrorLog(log)) {
+                    log_sent = sendErrorLog(log);
+                } else {
+                    dbgWarning(D_NGINX_MESSAGE_READER) << "Unexpected nginx log format";
+                    continue;
+                }
+                if (!log_sent) {
+                    dbgWarning(D_NGINX_MESSAGE_READER) << "Failed to send Log to Infinity Portal";
+                } else {
+                    dbgTrace(D_NGINX_MESSAGE_READER) << "Succesfully sent nginx log to Infinity Portal";
+                }
+            }
+        };
+        I_MainLoop *mainloop = Singleton::Consume<I_MainLoop>::by<NginxMessageReader>();
+        mainloop->addFileRoutine(
+            I_MainLoop::RoutineType::RealTime,
+            syslog_server_socket,
+            read_logs,
+            "Process nginx logs",
+            true
+        );
+    }
+
+    bool
+    sendAccessLog(const string &log)
+    {
+        dbgFlow(D_NGINX_MESSAGE_READER) << "Access log" << log;
+        Maybe<EnumArray<LogInfo, string>> log_info = parseAccessLog(log);
+        if (!log_info.ok()) {
+            dbgWarning(D_NGINX_MESSAGE_READER)
+                << "Failed parsing the NGINX logs. Error: "
+                << log_info.getErr();
+            return false;
+        }
+        auto unpacked_log_info = log_info.unpack();
+
+        if (unpacked_log_info[LogInfo::RESPONSE_CODE] == rate_limit_status_code) {
+            return sendRateLimitLog(unpacked_log_info);
+        }
+        return sendLog(unpacked_log_info);
+    }
+
+    bool
+    sendErrorLog(const string &log)
+    {
+        dbgFlow(D_NGINX_MESSAGE_READER) << "Error log" << log;
+        Maybe<EnumArray<LogInfo, string>> log_info = parseErrorLog(log);
+        if (!log_info.ok()) {
+            dbgWarning(D_NGINX_MESSAGE_READER)
+                << "Failed parsing the NGINX logs. Error: "
+                << log_info.getErr();
+            return false;
+        }
+        return sendLog(log_info.unpack());
+    }
+
+    bool
+    isAccessLog(const string &log) const
+    {
+        dbgFlow(D_NGINX_MESSAGE_READER) << "Chekck if string contains \"accessLog\"" << log;
+        return log.find("accessLog") != string::npos;
+    }
+
+    bool
+    isAlertErrorLog(const string &log) const
+    {
+        dbgFlow(D_NGINX_MESSAGE_READER) << "Check if log is of type 'error log'. Log: " << log;
+        return log.find("[alert]") != string::npos;
+    }
+
+    bool
+    isErrorLog(const string &log) const
+    {
+        dbgFlow(D_NGINX_MESSAGE_READER) << "Check if log is of type 'error log'. Log: " << log;
+        return log.find("[error]") != string::npos;
+    }
+
+    bool
+    sendLog(const EnumArray<LogInfo, string> &log_info)
+    {
+        dbgFlow(D_NGINX_MESSAGE_READER);
+        string event_name;
+        switch (log_info[LogInfo::RESPONSE_CODE][0]) {
+            case '4': {
+                event_name = "Invalid request or incorrect reverse proxy configuration - Request dropped."
+                " Please check the reverse proxy configuration of your relevant assets";
+                break;
+            }
+            case '5': {
+                event_name = "AppSec Gateway reverse proxy error - Request dropped. "
+                "Please verify the reverse proxy configuration of your relevant assets. "
+                "If the issue persists please contact Check Point Support";
+                break;
+            }
+            default: {
+                dbgError(D_NGINX_MESSAGE_READER) << "Irrelevant status code";
+                return false;
+            }
+        }
+
+        dbgTrace(D_NGINX_MESSAGE_READER)
+            << "Nginx log's event name and response code: "
+            << event_name
+            << ", "
+            << log_info[LogInfo::RESPONSE_CODE];
+        LogGen log(
+            event_name,
+            ReportIS::Audience::SECURITY,
+            ReportIS::Severity::INFO,
+            ReportIS::Priority::LOW,
+            ReportIS::Tags::REVERSE_PROXY
+        );
+        log << LogField("eventConfidence", "High");
+
+        for (LogInfo field : makeRange<LogInfo>()) {
+            Maybe<string> string_field = convertLogFieldToString(field);
+            if (!string_field.ok()) {
+                dbgDebug(D_NGINX_MESSAGE_READER) << "Enum field was not converted: " <<  string_field.getErr();
+                return false;
+            }
+
+            if (field != LogInfo::DESTINATION_PORT) {
+                log << LogField(string_field.unpack(), log_info[field]);
+                continue;
+            }
+
+            try {
+                log << LogField(string_field.unpack(), stoi(log_info[field]));
+            } catch (const exception &e) {
+                dbgError(D_NGINX_MESSAGE_READER)
+                    << "Unable to convert port to numeric value: "
+                    << e.what();
+                log << LogField(string_field.unpack(), 0);
+            }
+        }
+        return true;
+    }
+
+    bool
+    sendRateLimitLog(const EnumArray<LogInfo, string> &log_info)
+    {
+        dbgFlow(D_NGINX_MESSAGE_READER) << "Getting rate-limit rules of asset ID: " << log_info[LogInfo::ASSET_ID];
+
+        ScopedContext rate_limit_ctx;
+
+        rate_limit_ctx.registerValue<GenericConfigId>(AssetMatcher::ctx_key, log_info[LogInfo::ASSET_ID]);
+        auto rate_limit_config = getConfiguration<RateLimitConfig>("rulebase", "rateLimit");
+        if (!rate_limit_config.ok()) {
+            dbgTrace(D_NGINX_MESSAGE_READER)
+                << "Rate limit context does not match asset ID: " << log_info[LogInfo::ASSET_ID];
+            return false;
+        }
+        RateLimitConfig unpacked_rate_limit_config = rate_limit_config.unpack();
+
+        string nginx_uri = log_info[LogInfo::URI];
+        const LogTriggerConf &rate_limit_trigger = unpacked_rate_limit_config.getRateLimitTrigger(nginx_uri);
+
+        dbgTrace(D_NGINX_MESSAGE_READER)<< "About to generate NGINX rate-limit log";
+
+        string event_name = "Rate limit";
+        string security_action = "Drop";
+        bool is_log_required = false;
+
+        // Prevent events checkbox (in triggers)
+        if (rate_limit_trigger.isPreventLogActive(LogTriggerConf::SecurityType::AccessControl)) {
+            is_log_required = true;
+        }
+
+        if (!is_log_required) {
+            dbgTrace(D_NGINX_MESSAGE_READER) << "Not sending NGINX rate-limit log as it is not required";
+            return false;
+        }
+
+        ostringstream src_ip;
+        ostringstream dst_ip;
+        src_ip << log_info[LogInfo::SOURCE];
+        dst_ip << log_info[LogInfo::DESTINATION_IP];
+
+        ReportIS::Severity log_severity = ReportIS::Severity::MEDIUM;
+        ReportIS::Priority log_priority = ReportIS::Priority::MEDIUM;
+
+        LogGen log = rate_limit_trigger(
+            event_name,
+            LogTriggerConf::SecurityType::AccessControl,
+            log_severity,
+            log_priority,
+            true, // is drop
+            LogField("practiceType", "Rate Limit"),
+            ReportIS::Tags::RATE_LIMIT
+        );
+
+        for (LogInfo field : makeRange<LogInfo>()) {
+            Maybe<string> string_field = convertLogFieldToString(field);
+            if (!string_field.ok()) {
+                dbgDebug(D_NGINX_MESSAGE_READER) << "Enum field was not converted: " <<  string_field.getErr();
+                return false;
+            }
+            
+            if (
+                field == LogInfo::HOST ||
+                field == LogInfo::URI ||
+                field == LogInfo::HTTP_METHOD ||
+                field == LogInfo::SOURCE ||
+                field == LogInfo::DESTINATION_IP ||
+                field == LogInfo::ASSET_ID ||
+                field == LogInfo::ASSET_NAME ||
+                field == LogInfo::RESPONSE_CODE
+            ) {
+                if (!log_info[field].empty()) {
+                    log << LogField(string_field.unpack(), log_info[field]);
+                    continue;
+                }
+            }
+
+            if (field == LogInfo::DESTINATION_PORT) {
+                try {
+                    int numeric_dst_port = stoi(log_info[field]);
+                    log << LogField(string_field.unpack(), numeric_dst_port);
+                } catch (const exception &e) {
+                    dbgWarning(D_NGINX_MESSAGE_READER)
+                        << "Unable to convert dst port: "
+                        << log_info[field]
+                        << " to numberic value. Error: "
+                        << e.what();
+                }
+            }
+        }
+
+        return true;
+    }
+
+    Maybe<string>
+    convertLogFieldToString(LogInfo field)
+    {
+        dbgFlow(D_NGINX_MESSAGE_READER);
+        switch (field) {
+            case LogInfo::HTTP_METHOD:
+                return string("httpMethod");
+            case LogInfo::URI:
+                return string("httpUriPath");
+            case LogInfo::RESPONSE_CODE:
+                return string("httpResponseCode");
+            case LogInfo::HOST:
+                return string("httpHostName");
+            case LogInfo::SOURCE:
+                return string("httpSourceId");
+            case LogInfo::DESTINATION_IP:
+                return string("destinationIp");
+            case LogInfo::DESTINATION_PORT:
+                return string("destinationPort");
+            case LogInfo::ASSET_ID:
+                return string("assetId");
+            case LogInfo::ASSET_NAME:
+                return string("assetName");
+            case LogInfo::EVENT_MESSAGE:
+                return string("httpResponseBody");
+            case LogInfo::RULE_ID:
+                return string("ruleId");
+            case LogInfo::RULE_NAME:
+                return string("ruleName");
+            case LogInfo::COUNT:
+                dbgError(D_NGINX_MESSAGE_READER) << "LogInfo::COUNT is not allowed";
+                return genError("LogInfo::COUNT is not allowed");
+        }
+        dbgError(D_NGINX_MESSAGE_READER) << "No Enum found, int value: " << static_cast<int>(field);
+        return genError("No Enum found");
+    }
+
+    static vector<string>
+    separateLogs(const string &raw_logs_to_parse)
+    {
+        dbgFlow(D_NGINX_MESSAGE_READER) << "separating logs. logs: " << raw_logs_to_parse;
+        dbgTrace(D_NGINX_MESSAGE_READER) << "separateLogs start of function. Logs to parse: " << raw_logs_to_parse;
+        boost::smatch matcher;
+        vector<string> logs;
+
+        if (raw_logs_to_parse.empty()) return logs;
+
+        size_t pos = 0;
+        while (NGEN::Regex::regexSearch(__FILE__, __LINE__, raw_logs_to_parse.substr(pos), matcher, syslog_regex)) {
+            if (pos == 0) {
+                dbgTrace(D_NGINX_MESSAGE_READER) << "separateLogs pos = 0";
+                pos++;
+                continue;
+            }
+            auto log_length = matcher.position();
+            logs.push_back(raw_logs_to_parse.substr(pos - 1, log_length));
+
+            pos += log_length + 1;
+        }
+        logs.push_back(raw_logs_to_parse.substr(pos - 1));
+        dbgTrace(D_NGINX_MESSAGE_READER) << "separateLogs end of function";
+
+        return logs;
+    }
+
+    static pair<string, string>
+    parseErrorLogRequestField(const string &request)
+    {
+        dbgFlow(D_NGINX_MESSAGE_READER) << "parsing request field. request: " << request;
+        string formatted_request = request;
+        vector<string> result;
+        boost::erase_all(formatted_request, "\"");
+        boost::erase_all(formatted_request, "\n");
+        boost::split(result, formatted_request, boost::is_any_of(" "), boost::token_compress_on);
+
+        const int http_method_index = 1;
+        const int uri_index = 2;
+        return pair<string, string>(result[http_method_index], result[uri_index]);
+    }
+
+    static string
+    parseErrorLogField(const string &field)
+    {
+        dbgFlow(D_NGINX_MESSAGE_READER) << "parsing error log field " << field;
+        string formatted_field = field;
+        vector<string> result;
+        boost::erase_all(formatted_field, "\"");
+        boost::erase_all(formatted_field, "\n");
+        boost::split(result, formatted_field, boost::is_any_of(" "), boost::token_compress_on);
+
+        const int field_index = 1;
+        return result[field_index];
+    }
+
+    void
+    addContextFieldsToLogInfo(EnumArray<LogInfo, string> &log_info)
+    {
+        dbgFlow(D_NGINX_MESSAGE_READER);
+        ScopedContext ctx;
+
+        try {
+            ctx.registerValue<uint16_t>(
+                HttpTransactionData::listening_port_ctx,
+                static_cast<uint16_t>(stoi(log_info[LogInfo::DESTINATION_PORT]))
+            );
+        } catch (const exception &e) {
+            dbgError(D_NGINX_MESSAGE_READER) << "Failed register values for context " << e.what();
+        }
+        ctx.registerValue<string>(HttpTransactionData::host_name_ctx, log_info[LogInfo::HOST]);
+        ctx.registerValue<string>(HttpTransactionData::uri_ctx, log_info[LogInfo::URI]);
+        auto rule_by_ctx = getConfiguration<BasicRuleConfig>("rulebase", "rulesConfig");
+        if (!rule_by_ctx.ok()) {
+            dbgWarning(D_NGINX_MESSAGE_READER)
+                << "AssetId was not found by the given context. Reason: "
+                << rule_by_ctx.getErr();
+            return;
+        }
+
+        BasicRuleConfig context = rule_by_ctx.unpack();
+        log_info[LogInfo::ASSET_ID] = context.getAssetId();
+        log_info[LogInfo::ASSET_NAME] = context.getAssetName();
+        log_info[LogInfo::RULE_ID] = context.getRuleId();
+        log_info[LogInfo::RULE_NAME] = context.getRuleName();
+    }
+
+    Maybe<EnumArray<LogInfo, string>>
+    parseErrorLog(const string &log_line)
+    {
+        dbgFlow(D_NGINX_MESSAGE_READER) << "Handling log line:" << log_line;
+        string port;
+        EnumArray<LogInfo, string> log_info(EnumArray<LogInfo, string>::Fill(), string(""));
+
+        boost::smatch matcher;
+        vector<string> result;
+        if (
+            !NGEN::Regex::regexSearch(
+                __FILE__,
+                __LINE__,
+                log_line,
+                matcher,
+                isAlertErrorLog(log_line) ? alert_log_regex : error_log_regex
+            )
+        ) {
+            dbgWarning(D_NGINX_MESSAGE_READER) << "Unexpected nginx log format";
+            return genError("Unexpected nginx log format");
+        }
+
+        const int event_message_index = 6;
+        const int source_index = 7;
+        const int request_index = 9;
+        const int host_index = 11;
+        string host = string(matcher[host_index].first, matcher[host_index].second);
+        string source = string(matcher[source_index].first, matcher[source_index].second);
+        string event_message = string(matcher[event_message_index].first, matcher[event_message_index].second);
+        string request = string(matcher[request_index].first, matcher[request_index].second);
+
+        host = parseErrorLogField(host);
+        source = parseErrorLogField(source);
+        pair<string, string> parsed_request = parseErrorLogRequestField(request);
+        string http_method = parsed_request.first;
+        string uri = parsed_request.second;
+
+        if (NGEN::Regex::regexSearch(__FILE__, __LINE__, host, matcher, socket_address_regex)) {
+            int host_index = 1;
+            int port_index = 2;
+            host = string(matcher[host_index].first, matcher[host_index].second);
+            port = string(matcher[port_index].first, matcher[port_index].second);
+        } else if (NGEN::Regex::regexSearch(__FILE__, __LINE__, host, matcher, boost::regex("https://"))) {
+            port = "443";
+        } else {
+            port = "80";
+        }
+
+        log_info[LogInfo::HOST] = host;
+        log_info[LogInfo::URI] = uri;
+        log_info[LogInfo::RESPONSE_CODE] = "500";
+        log_info[LogInfo::HTTP_METHOD] = http_method;
+        log_info[LogInfo::SOURCE] = source;
+        log_info[LogInfo::DESTINATION_IP] = host;
+        log_info[LogInfo::DESTINATION_PORT] = port;
+        log_info[LogInfo::EVENT_MESSAGE] = event_message;
+
+        addContextFieldsToLogInfo(log_info);
+
+        if (!validateLog(log_info)) {
+            dbgWarning(D_NGINX_MESSAGE_READER) << "Unexpected nginx log format";
+            return genError("Unexpected nginx log format");
+        }
+
+        return log_info;
+    }
+
+    Maybe<EnumArray<LogInfo, string>>
+    parseAccessLog(const string &log_line)
+    {
+        dbgFlow(D_NGINX_MESSAGE_READER) << "Parsing log line: " << log_line;
+        string formatted_log = log_line;
+        EnumArray<LogInfo, string> log_info(EnumArray<LogInfo, string>::Fill(), string(""));
+        vector<string> result;
+        boost::erase_all(formatted_log, "\"");
+        boost::erase_all(formatted_log, "\n");
+        boost::split(result, formatted_log, boost::is_any_of(" "), boost::token_compress_on);
+
+        const int valid_log_size = 20;
+
+        if (result.size() < valid_log_size) {
+            dbgWarning(D_NGINX_MESSAGE_READER) << "Unexpected nginx log format";
+            return genError("Unexpected nginx log format");
+        }
+
+        const int host_index = 6;
+        const int host_port_index = 7;
+        const int http_method_index = 13;
+        const int uri_index = 14;
+        const int response_cod_index = 16;
+        const int source_index = 8;
+
+        log_info[LogInfo::HOST] = result[host_index];
+        log_info[LogInfo::URI] = result[uri_index];
+        log_info[LogInfo::RESPONSE_CODE] = result[response_cod_index];
+        log_info[LogInfo::HTTP_METHOD] = result[http_method_index];
+        log_info[LogInfo::SOURCE] = result[source_index];
+        log_info[LogInfo::DESTINATION_IP] = result[host_index];
+        log_info[LogInfo::DESTINATION_PORT] = result[host_port_index];
+        log_info[LogInfo::EVENT_MESSAGE] = "Invalid request or incorrect reverse proxy configuration - "
+        "Request dropped. Please check the reverse proxy configuration of your relevant assets";
+
+        addContextFieldsToLogInfo(log_info);
+
+        if (!validateLog(log_info)) {
+            dbgWarning(D_NGINX_MESSAGE_READER) << "Unexpected nginx log format";
+            return genError("Unexpected nginx log format");
+        }
+        return log_info;
+    }
+
+    static bool
+    validateLog(const EnumArray<LogInfo, string> &log_info)
+    {
+        dbgFlow(D_NGINX_MESSAGE_READER);
+
+        boost::smatch matcher;
+        if (!NGEN::Regex::regexSearch(__FILE__, __LINE__, log_info[LogInfo::HOST], matcher, server_regex)) {
+            dbgTrace(D_NGINX_MESSAGE_READER) << "Could not validate server (Host): " << log_info[LogInfo::HOST];
+            return false;
+        }
+        if (!NGEN::Regex::regexSearch(__FILE__, __LINE__, log_info[LogInfo::URI], matcher, uri_regex)) {
+            dbgTrace(D_NGINX_MESSAGE_READER) << "Could not validate Uri: " << log_info[LogInfo::URI];
+            return false;
+        }
+
+        if (
+            !NGEN::Regex::regexSearch(
+                __FILE__,
+                __LINE__,
+                log_info[LogInfo::RESPONSE_CODE],
+                matcher, response_code_regex
+            )
+        ) {
+            dbgTrace(D_NGINX_MESSAGE_READER)
+                << "Could not validate response code: "
+                << log_info[LogInfo::RESPONSE_CODE];
+            return false;
+        }
+
+        if (
+            !NGEN::Regex::regexSearch(__FILE__, __LINE__, log_info[LogInfo::HTTP_METHOD], matcher, http_method_regex)
+        ) {
+            dbgTrace(D_NGINX_MESSAGE_READER) << "Could not validate HTTP method: " << log_info[LogInfo::HTTP_METHOD];
+            return false;
+        }
+
+        if (!NGEN::Regex::regexSearch(__FILE__, __LINE__, log_info[LogInfo::DESTINATION_PORT], matcher, port_regex)) {
+            dbgTrace(D_NGINX_MESSAGE_READER)
+                << "Could not validate destination port : "
+                << log_info[LogInfo::DESTINATION_PORT];
+            return false;
+        }
+
+        if (!NGEN::Regex::regexSearch(__FILE__, __LINE__, log_info[LogInfo::SOURCE], matcher, server_regex)) {
+            dbgTrace(D_NGINX_MESSAGE_READER) << "Could not validate source : " << log_info[LogInfo::SOURCE];
+            return false;
+        }
+
+        return true;
+    }
+
+    Maybe<string>
+    getLogsFromSocket(const I_Socket::socketFd &client_socket) const
+    {
+        dbgFlow(D_NGINX_MESSAGE_READER) << "Reading logs from socket. fd: " << client_socket;
+        I_Socket *i_socket = Singleton::Consume<I_Socket>::by<NginxMessageReader>();
+        Maybe<vector<char>> raw_log_data = i_socket->receiveData(client_socket, 0, false);
+        if (!raw_log_data.ok()) {
+            dbgWarning(D_NGINX_MESSAGE_READER) << "Error receiving data from socket";
+            return genError("Error receiving data from socket");
+        }
+
+        string raw_log(raw_log_data.unpack().begin(), raw_log_data.unpack().end());
+        return move(raw_log);
+    }
+
+    I_Socket::socketFd syslog_server_socket = -1;
+    string rate_limit_status_code = "429";
+};
+
+NginxMessageReader::NginxMessageReader() : Component("NginxMessageReader"), pimpl(make_unique<Impl>()) {}
+
+NginxMessageReader::~NginxMessageReader() {}
+
+void
+NginxMessageReader::init()
+{
+    pimpl->init();
+}
+
+void
+NginxMessageReader::preload()
+{
+    pimpl->preload();
+}
+
+void
+NginxMessageReader::fini()
+{
+    pimpl->fini();
+}

components/packet/packet.cc

@@ -563,7 +563,10 @@ Packet::parsePacket(PktType type, IPType proto)
             return parseFromL3v6();
         }
         default: {
-            dbgAssert(false) << "Unknown (neither IPv4, nor IPv6), or uninitialized packet type: " << proto;
+            dbgAssert(false)
+                << AlertInfo(AlertTeam::CORE, "packet")
+                << "Unknown (neither IPv4, nor IPv6), or uninitialized packet type: "
+                << proto;
         }
     }
 

components/pending_key/pending_key.cc

@@ -43,7 +43,9 @@ PendingKey::print(ostream &os) const
 size_t
 PendingKey::hash() const
 {
-    dbgAssert(src.type != IPType::UNINITIALIZED) << "PendingKey::hash was called on an uninitialized object";
+    dbgAssert(src.type != IPType::UNINITIALIZED)
+        << AlertInfo(AlertTeam::CORE, "pending key")
+        << "PendingKey::hash was called on an uninitialized object";
     size_t seed = 0;
     hashCombine(seed, static_cast<u_char>(src.type));
     hashCombine(seed, src.proto);

components/report_messaging/report_messaging_ut/CMakeLists.txt

@@ -1,3 +0,0 @@
-link_directories(${BOOST_ROOT}/lib)
-
-add_unit_test(report_messaging_ut "report_messaging_ut.cc" "report_messaging;report;messaging;singleton;-lboost_regex")

components/security_apps/CMakeLists.txt

@@ -3,5 +3,7 @@ add_subdirectory(ips)
 add_subdirectory(layer_7_access_control)
 add_subdirectory(local_policy_mgmt_gen)
 add_subdirectory(orchestration)
+add_subdirectory(prometheus)
 add_subdirectory(rate_limit)
 add_subdirectory(waap)
+add_subdirectory(central_nginx_manager)

components/security_apps/central_nginx_manager/CMakeLists.txt

@@ -0,0 +1,3 @@
+include_directories(include)
+
+add_library(central_nginx_manager central_nginx_manager.cc lets_encrypt_listener.cc)

components/security_apps/central_nginx_manager/central_nginx_manager.cc

@@ -0,0 +1,418 @@
+// Copyright (C) 2022 Check Point Software Technologies Ltd. All rights reserved.
+
+// Licensed under the Apache License, Version 2.0 (the "License");
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "central_nginx_manager.h"
+#include "lets_encrypt_listener.h"
+
+#include <string>
+#include <vector>
+#include <cereal/external/base64.hpp>
+
+#include "debug.h"
+#include "config.h"
+#include "rest.h"
+#include "log_generator.h"
+#include "nginx_utils.h"
+#include "agent_core_utilities.h"
+
+using namespace std;
+
+USE_DEBUG_FLAG(D_NGINX_MANAGER);
+
+class CentralNginxConfig
+{
+public:
+    void load(cereal::JSONInputArchive &ar)
+    {
+        try {
+            string nginx_conf_base64;
+            ar(cereal::make_nvp("id", file_id));
+            ar(cereal::make_nvp("name", file_name));
+            ar(cereal::make_nvp("data", nginx_conf_base64));
+            nginx_conf_content = cereal::base64::decode(nginx_conf_base64);
+            central_nginx_conf_path = getCentralNginxConfPath();
+            shared_config_path = getSharedConfigPath();
+            if (!nginx_conf_content.empty()) configureCentralNginx();
+        } catch (const cereal::Exception &e) {
+            dbgDebug(D_NGINX_MANAGER) << "Could not load Central Management Config JSON. Error: " << e.what();
+            ar.setNextName(nullptr);
+        }
+    }
+
+    const string & getFileId() const { return file_id; }
+    const string & getFileName() const { return file_name; }
+    const string & getFileContent() const { return nginx_conf_content; }
+
+    static string
+    getCentralNginxConfPath()
+    {
+        string central_nginx_conf_path = getProfileAgentSettingWithDefault<string>(
+            string("/tmp/central_nginx.conf"),
+            "centralNginxManagement.confDownloadPath"
+        );
+        dbgInfo(D_NGINX_MANAGER) << "Central NGINX configuration path: " << central_nginx_conf_path;
+
+        return central_nginx_conf_path;
+    }
+
+    static string
+    getSharedConfigPath()
+    {
+        string central_shared_conf_path = getConfigurationWithDefault<string>(
+            "/etc/cp/conf",
+            "Config Component",
+            "configuration path"
+        );
+        central_shared_conf_path += "/centralNginxManager/shared/central_nginx_shared.conf";
+        dbgInfo(D_NGINX_MANAGER) << "Shared NGINX configuration path: " << central_shared_conf_path;
+
+        return central_shared_conf_path;
+    }
+
+private:
+    void
+    loadAttachmentModule()
+    {
+        string attachment_module_path = NginxUtils::getModulesPath() + "/ngx_cp_attachment_module.so";
+        if (!NGEN::Filesystem::exists(attachment_module_path)) {
+            dbgTrace(D_NGINX_MANAGER) << "Attachment module " << attachment_module_path << " does not exist";
+            return;
+        }
+
+        string attachment_module_conf = "load_module " + attachment_module_path + ";";
+        if (nginx_conf_content.find(attachment_module_conf) != string::npos) {
+            dbgTrace(D_NGINX_MANAGER) << "Attachment module " << attachment_module_path << " already loaded";
+            return;
+        }
+
+        nginx_conf_content = attachment_module_conf + "\n" + nginx_conf_content;
+    }
+
+    Maybe<void>
+    loadSharedDirective(const string &directive)
+    {
+        dbgFlow(D_NGINX_MANAGER) << "Loading shared directive into the servers " << directive;
+
+        if (!NGEN::Filesystem::copyFile(shared_config_path, shared_config_path + ".bak", true)) {
+            return genError("Could not create a backup of the shared NGINX configuration file");
+        }
+
+        ifstream shared_config(shared_config_path);
+        if (!shared_config.is_open()) {
+            return genError("Could not open shared NGINX configuration file");
+        }
+
+        string shared_config_content((istreambuf_iterator<char>(shared_config)), istreambuf_iterator<char>());
+        shared_config.close();
+
+        if (shared_config_content.find(directive) != string::npos) {
+            dbgTrace(D_NGINX_MANAGER) << "Shared directive " << directive << " already loaded";
+            return {};
+        }
+
+        ofstream new_shared_config(shared_config_path, ios::app);
+        if (!new_shared_config.is_open()) {
+            return genError("Could not open shared NGINX configuration file");
+        }
+
+        dbgTrace(D_NGINX_MANAGER) << "Adding shared directive " << directive;
+        new_shared_config << directive << "\n";
+        new_shared_config.close();
+
+        auto validation = NginxUtils::validateNginxConf(central_nginx_conf_path);
+        if (!validation.ok()) {
+            if (!NGEN::Filesystem::copyFile(shared_config_path + ".bak", shared_config_path, true)) {
+                return genError("Could not restore the shared NGINX configuration file");
+            }
+            return genError("Could not validate shared NGINX configuration file. Error: " + validation.getErr());
+        }
+
+        return {};
+    }
+
+    Maybe<void>
+    loadSharedConfig()
+    {
+        dbgFlow(D_NGINX_MANAGER) << "Loading shared configuration into the servers";
+
+        ofstream shared_config(shared_config_path);
+        if (!shared_config.is_open()) {
+            return genError("Could not create shared NGINX configuration file");
+        }
+        shared_config.close();
+
+        string shared_config_directive = "include " + shared_config_path + ";\n";
+        boost::regex server_regex("server\\s*\\{");
+        nginx_conf_content = NGEN::Regex::regexReplace(
+            __FILE__,
+            __LINE__,
+            nginx_conf_content,
+            server_regex,
+            "server {\n" + shared_config_directive
+        );
+
+        ofstream nginx_conf_file(central_nginx_conf_path);
+        if (!nginx_conf_file.is_open()) {
+            return genError("Could not open a temporary central NGINX configuration file");
+        }
+        nginx_conf_file << nginx_conf_content;
+        nginx_conf_file.close();
+
+        auto validation = NginxUtils::validateNginxConf(central_nginx_conf_path);
+        if (!validation.ok()) {
+            return genError("Could not validate central NGINX configuration file. Error: " + validation.getErr());
+        }
+
+        return {};
+    }
+
+    Maybe<void>
+    configureSyslog()
+    {
+        if (!getProfileAgentSettingWithDefault<bool>(false, "centralNginxManagement.syslogEnabled")) {
+            dbgTrace(D_NGINX_MANAGER) << "Syslog is disabled via settings";
+            return {};
+        }
+
+        string syslog_directive = "error_log syslog:server=127.0.0.1:1514 warn;";
+        auto load_shared_directive_result = loadSharedDirective(syslog_directive);
+        if (!load_shared_directive_result.ok()) {
+            return genError("Could not configure syslog directive, error: " + load_shared_directive_result.getErr());
+        }
+
+        return {};
+    }
+
+    Maybe<void>
+    saveBaseCentralNginxConf()
+    {
+        ofstream central_nginx_conf_base_file(central_nginx_conf_path + ".base");
+        if (!central_nginx_conf_base_file.is_open()) {
+            return genError("Could not open a temporary central NGINX configuration file");
+        }
+        central_nginx_conf_base_file << nginx_conf_content;
+        central_nginx_conf_base_file.close();
+
+        return {};
+    }
+
+    void
+    configureCentralNginx()
+    {
+        loadAttachmentModule();
+        auto save_base_nginx_conf = saveBaseCentralNginxConf();
+        if (!save_base_nginx_conf.ok()) {
+            dbgWarning(D_NGINX_MANAGER)
+                << "Could not save base NGINX configuration. Error: "
+                << save_base_nginx_conf.getErr();
+            return;
+        }
+
+        string nginx_conf_content_backup = nginx_conf_content;
+        auto shared_config_result = loadSharedConfig();
+        if (!shared_config_result.ok()) {
+            dbgWarning(D_NGINX_MANAGER)
+                << "Could not load shared configuration. Error: "
+                << shared_config_result.getErr();
+            nginx_conf_content = nginx_conf_content_backup;
+            return;
+        }
+
+        auto syslog_result = configureSyslog();
+        if (!syslog_result.ok()) {
+            dbgWarning(D_NGINX_MANAGER) << "Could not configure syslog. Error: " << syslog_result.getErr();
+        }
+    }
+
+    string file_id;
+    string file_name;
+    string nginx_conf_content;
+    string central_nginx_conf_path;
+    string shared_config_path;
+};
+
+class CentralNginxManager::Impl
+{
+public:
+    void
+    init()
+    {
+        dbgInfo(D_NGINX_MANAGER) << "Starting Central NGINX Manager";
+
+        string main_nginx_conf_path = NginxUtils::getMainNginxConfPath();
+        if (
+            NGEN::Filesystem::exists(main_nginx_conf_path)
+            && !NGEN::Filesystem::exists(main_nginx_conf_path + ".orig")
+        ) {
+            dbgInfo(D_NGINX_MANAGER) << "Creating a backup of the original main NGINX configuration file";
+            NGEN::Filesystem::copyFile(main_nginx_conf_path, main_nginx_conf_path + ".orig", true);
+        }
+
+        i_mainloop = Singleton::Consume<I_MainLoop>::by<CentralNginxManager>();
+        if (!lets_encrypt_listener.init()) {
+            dbgWarning(D_NGINX_MANAGER) << "Could not start Lets Encrypt Listener, scheduling retry";
+            i_mainloop->addOneTimeRoutine(
+                I_MainLoop::RoutineType::System,
+                [this] ()
+                {
+                    while(!lets_encrypt_listener.init()) {
+                        dbgWarning(D_NGINX_MANAGER) << "Could not start Lets Encrypt Listener, will retry";
+                        i_mainloop->yield(chrono::seconds(5));
+                    }
+                },
+                "Lets Encrypt Listener initializer",
+                false
+            );
+        }
+    }
+
+    void
+    loadPolicy()
+    {
+        auto central_nginx_config = getSetting<vector<CentralNginxConfig>>("centralNginxManagement");
+        if (!central_nginx_config.ok() || central_nginx_config.unpack().empty()) {
+            dbgWarning(D_NGINX_MANAGER)
+                << "Could not load Central NGINX Management settings. Error: "
+                << central_nginx_config.getErr();
+            return;
+        }
+
+        auto &config = central_nginx_config.unpack().front();
+        if (config.getFileContent().empty()) {
+            dbgWarning(D_NGINX_MANAGER) << "Empty NGINX configuration file";
+            return;
+        }
+
+        dbgTrace(D_NGINX_MANAGER)
+            << "Handling Central NGINX Management settings: "
+            << config.getFileId()
+            << ", "
+            << config.getFileName()
+            << ", "
+            << config.getFileContent();
+
+        string central_nginx_conf_path = config.getCentralNginxConfPath();
+        ofstream central_nginx_conf_file(central_nginx_conf_path);
+        if (!central_nginx_conf_file.is_open()) {
+            dbgWarning(D_NGINX_MANAGER)
+                << "Could not open central NGINX configuration file: "
+                << central_nginx_conf_path;
+            return;
+        }
+        central_nginx_conf_file << config.getFileContent();
+        central_nginx_conf_file.close();
+
+        auto validation_result = NginxUtils::validateNginxConf(central_nginx_conf_path);
+        if (!validation_result.ok()) {
+            dbgWarning(D_NGINX_MANAGER)
+                << "Could not validate central NGINX configuration file. Error: "
+                << validation_result.getErr();
+            logError(validation_result.getErr());
+            return;
+        }
+
+        dbgTrace(D_NGINX_MANAGER) << "Validated central NGINX configuration file";
+
+        auto reload_result = NginxUtils::reloadNginx(central_nginx_conf_path);
+        if (!reload_result.ok()) {
+            dbgWarning(D_NGINX_MANAGER)
+                << "Could not reload central NGINX configuration. Error: "
+                << reload_result.getErr();
+            logError("Could not reload central NGINX configuration. Error: " + reload_result.getErr());
+            return;
+        }
+
+        logInfo("Central NGINX configuration has been successfully reloaded");
+    }
+
+    void
+    fini()
+    {
+        string central_nginx_base_path = CentralNginxConfig::getCentralNginxConfPath() + ".base";
+        if (!NGEN::Filesystem::exists(central_nginx_base_path)) {
+            dbgWarning(D_NGINX_MANAGER) << "Could not find base NGINX configuration file: " << central_nginx_base_path;
+            return;
+        }
+
+        NginxUtils::reloadNginx(central_nginx_base_path);
+    }
+
+private:
+    void
+    logError(const string &error)
+    {
+        LogGen log(
+            error,
+            ReportIS::Level::ACTION,
+            ReportIS::Audience::SECURITY,
+            ReportIS::Severity::CRITICAL,
+            ReportIS::Priority::URGENT,
+            ReportIS::Tags::POLICY_INSTALLATION
+        );
+
+        log.addToOrigin(LogField("eventTopic", "Central NGINX Management"));
+        log << LogField("notificationId", "4165c3b1-e9bc-44c3-888b-863e204c1bfb");
+        log << LogField(
+            "eventRemediation",
+            "Please verify your NGINX configuration and enforce policy again. "
+            "Contact Check Point support if the issue persists."
+        );
+    }
+
+    void
+    logInfo(const string &info)
+    {
+        LogGen log(
+            info,
+            ReportIS::Level::ACTION,
+            ReportIS::Audience::SECURITY,
+            ReportIS::Severity::INFO,
+            ReportIS::Priority::LOW,
+            ReportIS::Tags::POLICY_INSTALLATION
+        );
+
+        log.addToOrigin(LogField("eventTopic", "Central NGINX Management"));
+        log << LogField("notificationId", "4165c3b1-e9bc-44c3-888b-863e204c1bfb");
+        log << LogField("eventRemediation", "No action required");
+    }
+
+    I_MainLoop *i_mainloop = nullptr;
+    LetsEncryptListener lets_encrypt_listener;
+};
+
+CentralNginxManager::CentralNginxManager()
+    :
+    Component("Central NGINX Manager"),
+    pimpl(make_unique<CentralNginxManager::Impl>()) {}
+
+CentralNginxManager::~CentralNginxManager() {}
+
+void
+CentralNginxManager::init()
+{
+    pimpl->init();
+}
+
+void
+CentralNginxManager::fini()
+{
+    pimpl->fini();
+}
+
+void
+CentralNginxManager::preload()
+{
+    registerExpectedSetting<vector<CentralNginxConfig>>("centralNginxManagement");
+    registerExpectedConfiguration<string>("Config Component", "configuration path");
+    registerConfigLoadCb([this]() { pimpl->loadPolicy(); });
+}

components/security_apps/central_nginx_manager/include/lets_encrypt_listener.h

@@ -0,0 +1,30 @@
+// Copyright (C) 2022 Check Point Software Technologies Ltd. All rights reserved.
+
+// Licensed under the Apache License, Version 2.0 (the "License");
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#ifndef __LETS_ENCRYPT_HANDLER_H__
+#define __LETS_ENCRYPT_HANDLER_H__
+
+#include <string>
+
+#include "maybe_res.h"
+
+class LetsEncryptListener
+{
+public:
+    bool init();
+
+private:
+    Maybe<std::string> getChallengeValue(const std::string &uri) const;
+};
+
+#endif // __LETS_ENCRYPT_HANDLER_H__

components/security_apps/central_nginx_manager/lets_encrypt_listener.cc

@@ -0,0 +1,76 @@
+// Copyright (C) 2022 Check Point Software Technologies Ltd. All rights reserved.
+
+// Licensed under the Apache License, Version 2.0 (the "License");
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "lets_encrypt_listener.h"
+
+#include <string>
+
+#include "central_nginx_manager.h"
+#include "debug.h"
+
+using namespace std;
+
+USE_DEBUG_FLAG(D_NGINX_MANAGER);
+
+bool
+LetsEncryptListener::init()
+{
+    dbgInfo(D_NGINX_MANAGER) << "Starting Lets Encrypt Listener";
+    return Singleton::Consume<I_RestApi>::by<CentralNginxManager>()->addWildcardGetCall(
+        ".well-known/acme-challenge/",
+        [&] (const string &uri) -> string
+        {
+            Maybe<string> maybe_challenge_value = getChallengeValue(uri);
+            if (!maybe_challenge_value.ok()) {
+                dbgWarning(D_NGINX_MANAGER)
+                    << "Could not get challenge value for uri: "
+                    << uri
+                    << ", error: "
+                    << maybe_challenge_value.getErr();
+                return string{""};
+            };
+
+            dbgTrace(D_NGINX_MANAGER) << "Got challenge value: " << maybe_challenge_value.unpack();
+            return maybe_challenge_value.unpack();
+        }
+    );
+}
+
+Maybe<string>
+LetsEncryptListener::getChallengeValue(const string &uri) const
+{
+    string challenge_key = uri.substr(uri.find_last_of('/') + 1);
+    string api_query = "/api/lets-encrypt-challenge?http_challenge_key=" + challenge_key;
+
+    dbgInfo(D_NGINX_MANAGER) << "Getting challenge value via: " << api_query;
+
+    MessageMetadata md;
+    md.insertHeader("X-Tenant-Id", Singleton::Consume<I_AgentDetails>::by<CentralNginxManager>()->getTenantId());
+    Maybe<HTTPResponse, HTTPResponse> maybe_http_challenge_value =
+        Singleton::Consume<I_Messaging>::by<CentralNginxManager>()->sendSyncMessage(
+            HTTPMethod::GET,
+            api_query,
+            string("{}"),
+            MessageCategory::GENERIC,
+            md
+        );
+
+    if (!maybe_http_challenge_value.ok()) return genError(maybe_http_challenge_value.getErr().getBody());
+
+    string challenge_value = maybe_http_challenge_value.unpack().getBody();
+    if (!challenge_value.empty() && challenge_value.front() == '"' && challenge_value.back() == '"') {
+        challenge_value = challenge_value.substr(1, challenge_value.size() - 2);
+    }
+
+    return challenge_value;
+}

components/security_apps/http_geo_filter/http_geo_filter.cc

@@ -67,18 +67,18 @@ public:
         dbgTrace(D_GEO_FILTER) << getListenerName() << " new transaction event";
 
         if (!event.isLastHeader()) return EventVerdict(ngx_http_cp_verdict_e::TRAFFIC_VERDICT_INSPECT);
-        std::set<std::string> xff_set;
+        std::set<std::string> ip_set;
         auto env = Singleton::Consume<I_Environment>::by<HttpGeoFilter>();
         auto maybe_xff = env->get<std::string>(HttpTransactionData::xff_vals_ctx);
         if (!maybe_xff.ok()) {
             dbgTrace(D_GEO_FILTER) << "failed to get xff vals from env";
         } else {
-            xff_set = split(maybe_xff.unpack(), ',');
+            ip_set = split(maybe_xff.unpack(), ',');
         }
         dbgDebug(D_GEO_FILTER) << getListenerName() << " last header, start lookup";
 
-        if (xff_set.size() > 0) {
-            removeTrustedIpsFromXff(xff_set);
+        if (ip_set.size() > 0) {
+            removeTrustedIpsFromXff(ip_set);
         } else {
             dbgDebug(D_GEO_FILTER) << "xff not found in headers";
         }
@@ -88,16 +88,25 @@ public:
             dbgWarning(D_GEO_FILTER) << "failed to get source ip from env";
             return EventVerdict(default_action);
         }
-
         auto source_ip = convertIpAddrToString(maybe_source_ip.unpack());
-        xff_set.insert(source_ip);
 
-        ngx_http_cp_verdict_e exception_verdict = getExceptionVerdict(xff_set);
+        // saas profile setting
+        bool ignore_source_ip =
+            getProfileAgentSettingWithDefault<bool>(false, "agent.geoProtaction.ignoreSourceIP");
+        if (ignore_source_ip){
+            dbgDebug(D_GEO_FILTER) << "Geo protection ignoring source ip: " << source_ip;
+        } else {
+            dbgTrace(D_GEO_FILTER) << "Geo protection source ip: " << source_ip;
+            ip_set.insert(convertIpAddrToString(maybe_source_ip.unpack()));
+        }
+
+
+        ngx_http_cp_verdict_e exception_verdict = getExceptionVerdict(ip_set);
         if (exception_verdict != ngx_http_cp_verdict_e::TRAFFIC_VERDICT_IRRELEVANT) {
             return EventVerdict(exception_verdict);
         }
 
-        ngx_http_cp_verdict_e geo_lookup_verdict = getGeoLookupVerdict(xff_set);
+        ngx_http_cp_verdict_e geo_lookup_verdict = getGeoLookupVerdict(ip_set);
         if (geo_lookup_verdict != ngx_http_cp_verdict_e::TRAFFIC_VERDICT_IRRELEVANT) {
             return EventVerdict(geo_lookup_verdict);
         }
@@ -327,6 +336,14 @@ private:
         ngx_http_cp_verdict_e verdict = ngx_http_cp_verdict_e::TRAFFIC_VERDICT_IRRELEVANT;
         I_GeoLocation *i_geo_location = Singleton::Consume<I_GeoLocation>::by<HttpGeoFilter>();
         EnumArray<I_GeoLocation::GeoLocationField, std::string> geo_location_data;
+        auto env = Singleton::Consume<I_Environment>::by<HttpGeoFilter>();
+        string source_id;
+        auto maybe_source_id = env->get<std::string>(HttpTransactionData::source_identifier);
+        if (!maybe_source_id.ok()) {
+            dbgTrace(D_GEO_FILTER) << "failed to get source identifier from env";
+        } else {
+            source_id = maybe_source_id.unpack();
+        }
 
         for (const std::string& source : sources) {
 
@@ -343,7 +360,7 @@ private:
 
             auto asset_location = i_geo_location->lookupLocation(maybe_source_ip.unpack());
             if (!asset_location.ok()) {
-                dbgWarning(D_GEO_FILTER) << "Lookup location failed for source: " <<
+                dbgDebug(D_GEO_FILTER) << "Lookup location failed for source: " <<
                 source <<
                 ", Error: " <<
                 asset_location.getErr();
@@ -358,11 +375,15 @@ private:
             << country_code
             << ", country name: "
             << country_name
-            << ", source ip address: "
-            << source;
+            << ", ip address: "
+            << source
+            << ", source identifier: "
+            << source_id;
+
 
             unordered_map<string, set<string>> exception_value_country_code = {
-                {"countryCode", {country_code}}
+                {"countryCode", {country_code}},
+                {"sourceIdentifier", {source_id}}
             };
             auto matched_behavior_maybe = getBehaviorsVerdict(exception_value_country_code, geo_location_data);
             if (matched_behavior_maybe.ok()) {
@@ -374,7 +395,8 @@ private:
             }
 
             unordered_map<string, set<string>> exception_value_country_name = {
-                {"countryName", {country_name}}
+                {"countryName", {country_name}},
+                {"sourceIdentifier", {source_id}}
             };
             matched_behavior_maybe = getBehaviorsVerdict(exception_value_country_name, geo_location_data);
             if (matched_behavior_maybe.ok()) {
@@ -469,5 +491,6 @@ void
 HttpGeoFilter::preload()
 {
     registerExpectedConfiguration<GeoConfig>("rulebase", "httpGeoFilter");
+    registerExpectedConfiguration<UsersAllIdentifiersConfig>("rulebase", "usersIdentifiers");
     registerConfigLoadCb([this]() { pimpl->loadDefaultAction(); });
 }

components/security_apps/ips/compound_protection.cc

@@ -43,7 +43,10 @@ CompoundProtection::Impl::getMatch(const set<PMPattern> &matched) const
         case Operation::ORDERED_AND: return getMatchOrderedAnd(matched);
     }
 
-    dbgAssert(false) << "Unknown compound operation: " << static_cast<uint>(operation);
+    dbgAssert(false)
+        << AlertInfo(AlertTeam::CORE, "ips")
+        << "Unknown compound operation: "
+        << static_cast<uint>(operation);
     return MatchType::NO_MATCH;
 }
 

components/security_apps/ips/include/ips_signatures.h

@@ -29,6 +29,8 @@
 #include "pm_hook.h"
 #include "i_generic_rulebase.h"
 
+#define DEFAULT_IPS_YIELD_COUNT 500
+
 /// \namespace IPSSignatureSubTypes
 /// \brief Namespace containing subtypes for IPS signatures.
 namespace IPSSignatureSubTypes
@@ -336,9 +338,23 @@ public:
         return metadata.getYear();
     }
 
+    bool
+    isOk() const
+    {
+        return is_loaded;
+    }
+
+    static void
+    setYieldCounter(int new_yield_cnt)
+    {
+        yield_on_load_cnt = new_yield_cnt;
+    }
+
 private:
     IPSSignatureMetaData metadata;
     std::shared_ptr<BaseSignature> rule;
+    bool is_loaded;
+    static int yield_on_load_cnt;
 };
 
 /// \class SignatureAndAction

components/security_apps/ips/ips_comp.cc

@@ -98,6 +98,7 @@ public:
         registerListener();
         table = Singleton::Consume<I_Table>::by<IPSComp>();
         env = Singleton::Consume<I_Environment>::by<IPSComp>();
+        updateSigsYieldCount();
     }
 
     void
@@ -307,6 +308,20 @@ public:
 
     EventVerdict respond (const EndTransactionEvent &) override { return ACCEPT; }
 
+    void
+    updateSigsYieldCount()
+    {
+        const char *ips_yield_env_str = getenv("CPNANO_IPS_LOAD_YIELD_CNT");
+        int ips_yield_default = DEFAULT_IPS_YIELD_COUNT;
+        if (ips_yield_env_str != nullptr) {
+            dbgDebug(D_IPS) << "CPNANO_IPS_LOAD_YIELD_CNT env variable is set to " << ips_yield_env_str;
+            ips_yield_default = atoi(ips_yield_env_str);
+        }
+        int yield_limit = getProfileAgentSettingWithDefault<int>(ips_yield_default, "ips.sigsYieldCnt");
+        dbgDebug(D_IPS) << "Setting IPS yield count to  " << yield_limit;
+        IPSSignatureSubTypes::CompleteSignature::setYieldCounter(yield_limit);
+    }
+
 private:
     static void setDrop(IPSEntry &state) { state.setDrop(); }
     static bool isDrop(const IPSEntry &state) { return state.isDrop(); }
@@ -373,6 +388,7 @@ IPSComp::preload()
     registerExpectedConfigFile("ips", Config::ConfigFileType::Policy);
     registerExpectedConfigFile("ips", Config::ConfigFileType::Data);
     registerExpectedConfigFile("snort", Config::ConfigFileType::Policy);
+    registerConfigLoadCb([this]() { pimpl->updateSigsYieldCount(); });
 
     ParameterException::preload();
 

components/security_apps/ips/ips_configuration.cc

@@ -8,7 +8,9 @@ IPSConfiguration::Context::Context(ContextType _type, uint history) : type(_type
 uint
 IPSConfiguration::Context::getHistorySize() const
 {
-    dbgAssert(type == ContextType::HISTORY) << "Try to access history size for non-history context";
+    dbgAssert(type == ContextType::HISTORY)
+        << AlertInfo(AlertTeam::CORE, "ips")
+        << "Try to access history size for non-history context";
     return history_size;
 }
 
@@ -69,6 +71,8 @@ uint
 IPSConfiguration::getHistorySize(const string &name) const
 {
     auto context = context_config.find(name);
-    dbgAssert(context != context_config.end()) << "Try to access history size for non-exiting context";
+    dbgAssert(context != context_config.end())
+        << AlertInfo(AlertTeam::CORE, "ips")
+        << "Try to access history size for non-exiting context";
     return context->second.getHistorySize();
 }

components/security_apps/ips/ips_entry.cc

@@ -26,6 +26,8 @@ static const map<string, IPSConfiguration::Context> default_conf_mapping = {
 };
 
 static const IPSConfiguration default_conf(default_conf_mapping);
+static const IPSSignatures default_ips_sigs;
+static const SnortSignatures default_snort_sigs;
 
 IPSEntry::IPSEntry() : TableOpaqueSerialize<IPSEntry>(this) {}
 
@@ -51,9 +53,9 @@ IPSEntry::respond(const ParsedContext &parsed)
     ctx.registerValue(name, buf);
 
     ctx.activate();
-    auto &signatures = getConfigurationWithDefault(IPSSignatures(), "IPS", "IpsProtections");
+    auto &signatures = getConfigurationWithDefault(default_ips_sigs, "IPS", "IpsProtections");
     bool should_drop = signatures.isMatchedPrevent(parsed.getName(), buf);
-    auto &snort_signatures = getConfigurationWithDefault(SnortSignatures(), "IPSSnortSigs", "SnortProtections");
+    auto &snort_signatures = getConfigurationWithDefault(default_snort_sigs, "IPSSnortSigs", "SnortProtections");
     should_drop |= snort_signatures.isMatchedPrevent(parsed.getName(), buf);
     ctx.deactivate();
 

components/security_apps/ips/ips_signatures.cc

@@ -45,6 +45,8 @@ static const map<string, IPSLevel> levels = {
     { "Very Low",    IPSLevel::VERY_LOW }
 };
 
+int CompleteSignature::yield_on_load_cnt = DEFAULT_IPS_YIELD_COUNT;
+
 static IPSLevel
 getLevel(const string &level_string, const string &attr_name)
 {
@@ -84,7 +86,7 @@ IPSSignatureMetaData::getSeverityString() const
             return "Critical";
     }
 
-    dbgAssert(false) << "Illegal severity value: " << static_cast<uint>(severity);
+    dbgAssert(false) << AlertInfo(AlertTeam::CORE, "ips") << "Illegal severity value: " << static_cast<uint>(severity);
     return "Critical";
 }
 
@@ -116,7 +118,10 @@ IPSSignatureMetaData::getPerformanceString() const
             return "Critical";
     }
 
-    dbgAssert(false) << "Illegal performance value: " << static_cast<uint>(performance);
+    dbgAssert(false)
+        << AlertInfo(AlertTeam::CORE, "ips")
+        << "Illegal performance value: "
+        << static_cast<uint>(performance);
     return "Critical";
 }
 
@@ -216,10 +221,37 @@ IPSSignatureMetaData::getYear() const
 void
 CompleteSignature::load(cereal::JSONInputArchive &ar)
 {
-    ar(cereal::make_nvp("protectionMetadata", metadata));
-    RuleDetection rule_detection(metadata.getName());
-    ar(cereal::make_nvp("detectionRules", rule_detection));
-    rule = rule_detection.getRule();
+    static int sigs_load_counter = 0;
+    static I_Environment *env = Singleton::Consume<I_Environment>::by<IPSComp>();
+    static bool post_init = false;
+
+    if (!post_init) {
+        auto routine_id = Singleton::Consume<I_MainLoop>::by<IPSComp>()->getCurrentRoutineId();
+        if (routine_id.ok()) {
+            post_init = true;
+            dbgInfo(D_IPS) << "Loading signatures post init, enabling yield with limit " << yield_on_load_cnt;
+        }
+    }
+
+    try {
+        ar(cereal::make_nvp("protectionMetadata", metadata));
+        RuleDetection rule_detection(metadata.getName());
+        ar(cereal::make_nvp("detectionRules", rule_detection));
+        rule = rule_detection.getRule();
+        is_loaded = true;
+    } catch (cereal::Exception &e) {
+        is_loaded = false;
+        dbgWarning(D_IPS) << "Failed to load signature: " << e.what();
+    }
+
+    if (post_init && (yield_on_load_cnt > 0) && (++sigs_load_counter == yield_on_load_cnt)) {
+        sigs_load_counter = 0;
+        auto maybe_is_async = env->get<bool>("Is Async Config Load");
+        if (maybe_is_async.ok() && *maybe_is_async == true) {
+            dbgTrace(D_IPS) << "Yielding after " << yield_on_load_cnt << " signatures";
+            Singleton::Consume<I_MainLoop>::by<IPSComp>()->yield(false);
+        }
+    }
 }
 
 MatchType
@@ -364,7 +396,16 @@ SignatureAndAction::matchSilent(const Buffer &sample) const
     if (method.ok()) log << LogField("httpMethod", method.unpack());
 
     auto path  = env->get<Buffer>("HTTP_PATH_DECODED");
-    if (path.ok()) log << LogField("httpUriPath", getSubString(path, 1536), LogFieldOption::XORANDB64);
+    if (path.ok()) {
+        log << LogField("httpUriPath", getSubString(path, 1536), LogFieldOption::XORANDB64);
+    } else {
+        auto transaction_path = env->get<string>(HttpTransactionData::uri_path_decoded);
+        if (transaction_path.ok()) {
+            auto uri_path = transaction_path.unpack();
+            auto question_mark = uri_path.find('?');
+            log << LogField("httpUriPath", uri_path.substr(0, question_mark), LogFieldOption::XORANDB64);
+        }
+    }
 
     auto req_header = ips_state.getTransactionData(IPSCommonTypes::requests_header_for_log);
     if (req_header.ok()) log << LogField("httpRequestHeaders", getSubString(req_header), LogFieldOption::XORANDB64);
@@ -482,13 +523,30 @@ SignatureAndAction::isMatchedPrevent(const Buffer &context_buffer, const set<PMP
     auto method = env->get<string>(HttpTransactionData::method_ctx);
     if (method.ok()) log << LogField("httpMethod", method.unpack());
     uint max_size = getConfigurationWithDefault<uint>(1536, "IPS", "Max Field Size");
-    auto path  = env->get<Buffer>("HTTP_PATH_DECODED");
-    if (path.ok() && trigger.isWebLogFieldActive(url_path)) {
-        log << LogField("httpUriPath", getSubString(path, max_size), LogFieldOption::XORANDB64);
+
+    if (trigger.isWebLogFieldActive(url_path)) {
+        auto path  = env->get<Buffer>("HTTP_PATH_DECODED");
+        if (path.ok()) {
+            log << LogField("httpUriPath", getSubString(path, max_size), LogFieldOption::XORANDB64);
+        } else {
+            auto transaction_path = env->get<string>(HttpTransactionData::uri_path_decoded);
+            if (transaction_path.ok()) {
+                auto uri_path = transaction_path.unpack();
+                auto question_mark = uri_path.find('?');
+                log << LogField("httpUriPath", uri_path.substr(0, question_mark), LogFieldOption::XORANDB64);
+            }
+        }
     }
-    auto query = env->get<Buffer>("HTTP_QUERY_DECODED");
-    if (query.ok() && trigger.isWebLogFieldActive(url_query)) {
-        log << LogField("httpUriQuery", getSubString(query, max_size), LogFieldOption::XORANDB64);
+    if (trigger.isWebLogFieldActive(url_query)) {
+        auto query = env->get<Buffer>("HTTP_QUERY_DECODED");
+        if (query.ok()) {
+            log << LogField("httpUriQuery", getSubString(query, max_size), LogFieldOption::XORANDB64);
+        } else {
+            auto transaction_query = env->get<string>(HttpTransactionData::uri_query_decoded);
+            if (transaction_query.ok()) {
+                log << LogField("httpUriQuery", transaction_query.unpack());
+            }
+        }
     }
 
     auto res_code = env->get<Buffer>("HTTP_RESPONSE_CODE");
@@ -530,7 +588,9 @@ IPSSignaturesResource::load(cereal::JSONInputArchive &ar)
 
     all_signatures.reserve(sigs.size());
     for (auto &sig : sigs) {
-        all_signatures.emplace_back(make_shared<CompleteSignature>(move(sig)));
+        if (sig.isOk()) {
+            all_signatures.emplace_back(make_shared<CompleteSignature>(move(sig)));
+        }
     }
 }
 

components/security_apps/ips/ips_ut/component_ut.cc

@@ -29,6 +29,8 @@ public:
     {
         comp.preload();
         comp.init();
+        auto err = genError("not coroutine");
+        EXPECT_CALL(mainloop, getCurrentRoutineId()).WillRepeatedly(Return(Maybe<I_MainLoop::RoutineID>(err)));
     }
 
     ~ComponentTest()

components/security_apps/ips/ips_ut/configuration.cc

@@ -7,7 +7,7 @@ TEST(configuration, basic_context)
 
     IPSConfiguration::Context ctx1(IPSConfiguration::ContextType::HISTORY, 254);
     EXPECT_EQ(ctx1.getType(), IPSConfiguration::ContextType::HISTORY);
-    EXPECT_EQ(ctx1.getHistorySize(), 254);
+    EXPECT_EQ(ctx1.getHistorySize(), 254u);
 
     IPSConfiguration::Context ctx2(IPSConfiguration::ContextType::NORMAL, 0);
     EXPECT_EQ(ctx2.getType(), IPSConfiguration::ContextType::NORMAL);
@@ -42,7 +42,7 @@ TEST(configuration, read_configuration)
 
     auto body = conf.getContext("HTTP_REQUEST_BODY");
     EXPECT_EQ(body.getType(), IPSConfiguration::ContextType::HISTORY);
-    EXPECT_EQ(conf.getHistorySize("HTTP_REQUEST_BODY"), 100);
+    EXPECT_EQ(conf.getHistorySize("HTTP_REQUEST_BODY"), 100u);
 
     auto header = conf.getContext("HTTP_REQUEST_HEADER");
     EXPECT_EQ(header.getType(), IPSConfiguration::ContextType::KEEP);

components/security_apps/ips/ips_ut/entry_ut.cc

@@ -41,6 +41,8 @@ public:
     EntryTest()
     {
         ON_CALL(table, getState(_)).WillByDefault(Return(ptr));
+        auto err = genError("not coroutine");
+        EXPECT_CALL(mock_mainloop, getCurrentRoutineId()).WillRepeatedly(Return(Maybe<I_MainLoop::RoutineID>(err)));
     }
 
     void
@@ -137,8 +139,8 @@ private:
 TEST_F(EntryTest, basic_inherited_functions)
 {
     EXPECT_EQ(IPSEntry::name(), "IPS");
-    EXPECT_EQ(IPSEntry::currVer(), 0);
-    EXPECT_EQ(IPSEntry::minVer(), 0);
+    EXPECT_EQ(IPSEntry::currVer(), 0u);
+    EXPECT_EQ(IPSEntry::minVer(), 0u);
     EXPECT_NE(IPSEntry::prototype(), nullptr);
     EXPECT_EQ(entry.getListenerName(), IPSEntry::name());
 

components/security_apps/ips/ips_ut/resource_ut.cc

@@ -2,6 +2,7 @@
 #include "cptest.h"
 #include "environment.h"
 #include "config_component.h"
+#include "mock/mock_mainloop.h"
 
 using namespace std;
 using namespace testing;
@@ -61,6 +62,9 @@ TEST(resources, basic_resource)
 {
     ConfigComponent conf;
     ::Environment env;
+    NiceMock<MockMainLoop> mock_mainloop;
+    auto err = genError("not coroutine");
+    EXPECT_CALL(mock_mainloop, getCurrentRoutineId()).WillRepeatedly(Return(Maybe<I_MainLoop::RoutineID>(err)));
 
     conf.preload();
 
@@ -71,7 +75,7 @@ TEST(resources, basic_resource)
     Singleton::Consume<Config::I_Config>::from(conf)->loadConfiguration(resource);
 
     auto loaded_resources = getSettingWithDefault(IPSSignaturesResource(), "IPS", "protections");
-    EXPECT_EQ(loaded_resources.getSignatures().size(), 2);
+    EXPECT_EQ(loaded_resources.getSignatures().size(), 2u);
     auto version = getSettingWithDefault<string>("", "IPS", "VersionId");
     EXPECT_EQ(version, "1234567");
 }

components/security_apps/ips/ips_ut/signatures_ut.cc

@@ -60,7 +60,12 @@ public:
     {
         IPSHelper::has_deobfuscation = true;
         generic_rulebase.preload();
+        env.preload();
+        env.init();
+
         EXPECT_CALL(logs, getCurrentLogId()).Times(AnyNumber());
+        auto err = genError("not coroutine");
+        EXPECT_CALL(mock_mainloop, getCurrentRoutineId()).WillRepeatedly(Return(Maybe<I_MainLoop::RoutineID>(err)));
         ON_CALL(table, getState(_)).WillByDefault(Return(&ips_state));
         {
             stringstream ss;
@@ -104,6 +109,12 @@ public:
             cereal::JSONInputArchive ar(ss);
             high_medium_confidance_signatures.load(ar);
         }
+        {
+            stringstream ss;
+            ss << "[" << signature_performance_high << ", " << signature_broken << "]";
+            cereal::JSONInputArchive ar(ss);
+            single_broken_signature.load(ar);
+        }
     }
 
     ~SignatureTest()
@@ -117,9 +128,6 @@ public:
     void
     loadExceptions()
     {
-        env.preload();
-        env.init();
-
         BasicRuleConfig::preload();
         registerExpectedConfiguration<ParameterException>("rulebase", "exception");
 
@@ -189,6 +197,7 @@ public:
     void
     load(const IPSSignaturesResource &policy, const string &severity, const string &confidence)
     {
+        Singleton::Consume<I_Environment>::from(env)->registerValue<bool>("Is Async Config Load", false);
         setResource(policy, "IPS", "protections");
         stringstream ss;
         ss << "{";
@@ -250,6 +259,7 @@ public:
     IPSSignaturesResource performance_signatures1;
     IPSSignaturesResource performance_signatures2;
     IPSSignaturesResource performance_signatures3;
+    IPSSignaturesResource single_broken_signature;
     NiceMock<MockTable> table;
     MockAgg mock_agg;
 
@@ -483,6 +493,26 @@ private:
                 "\"context\": [\"HTTP_REQUEST_BODY\", \"HTTP_RESPONSE_BODY\"]"
             "}"
         "}";
+
+    string signature_broken =
+    "{"
+        "\"protectionMetadata\": {"
+            "\"protectionName\": \"BrokenTest\","
+            "\"maintrainId\": \"101\","
+            "\"severity\": \"Medium High\","
+            "\"confidenceLevel\": \"Low\","
+            "\"performanceImpact\": \"High\","
+            "\"lastUpdate\": \"20210420\","
+            "\"tags\": [],"
+            "\"cveList\": []"
+        "},"
+        "\"detectionRules\": {"
+            "\"type\": \"simple\","
+            "\"SSM\": \"\","
+            "\"keywosrds\": \"data: \\\"www\\\";\","
+            "\"context\": [\"HTTP_REQUEST_BODY\", \"HTTP_RESPONSE_BODY\"]"
+        "}"
+    "}";
 };
 
 TEST_F(SignatureTest, basic_load_of_signatures)
@@ -665,3 +695,14 @@ TEST_F(SignatureTest, high_confidance_signatures_matching)
     expectLog("\"protectionId\": \"Test4\"", "\"matchedSignatureConfidence\": \"Medium\"");
     EXPECT_FALSE(checkData("mmm"));
 }
+
+TEST_F(SignatureTest, broken_signature)
+{
+    load(single_broken_signature, "Low or above", "Low");
+    EXPECT_FALSE(checkData("ggg"));
+
+    expectLog("\"matchedSignaturePerformance\": \"High\"");
+    EXPECT_TRUE(checkData("fff"));
+
+    EXPECT_FALSE(checkData("www"));
+}

components/security_apps/layer_7_access_control/layer_7_access_control.cc

@@ -131,8 +131,12 @@ public:
     EventVerdict
     respond(const WaitTransactionEvent &) override
     {
-        dbgFlow(D_L7_ACCESS_CONTROL) << "Handling wait verdict";
+        if (!isAppEnabled()) {
+            dbgTrace(D_L7_ACCESS_CONTROL) << "Returning Accept verdict as the Layer-7 Access Control app is disabled";
+            return ngx_http_cp_verdict_e::TRAFFIC_VERDICT_ACCEPT;
+        }
 
+        dbgTrace(D_L7_ACCESS_CONTROL) << "Handling wait verdict";
         return handleEvent();
     }
 
@@ -385,8 +389,29 @@ Layer7AccessControl::Impl::init()
     i_intelligence = Singleton::Consume<I_Intelligence_IS_V2>::by<Layer7AccessControl>();
     i_mainloop = Singleton::Consume<I_MainLoop>::by<Layer7AccessControl>();
 
-    chrono::minutes expiration(
-        getProfileAgentSettingWithDefault<uint>(60u, "layer7AccessControl.crowdsec.cacheExpiration")
+    int cache_expiration_in_seconds = 30;
+    string cache_expiration_env = getenv("CROWDSEC_CACHE_EXPIRATION") ? getenv("CROWDSEC_CACHE_EXPIRATION") : "";
+    if (!cache_expiration_env.empty()) {
+        if (
+            all_of(cache_expiration_env.begin(), cache_expiration_env.end(), ::isdigit)
+            && stoi(cache_expiration_env) > 0
+        ) {
+            cache_expiration_in_seconds = stoi(cache_expiration_env);
+            dbgInfo(D_L7_ACCESS_CONTROL)
+                << "Successfully read cache expiration value from env: "
+                << cache_expiration_env;
+        } else {
+            dbgWarning(D_L7_ACCESS_CONTROL)
+                << "An invalid cache expiration value was provided in env: "
+                << cache_expiration_env;
+        }
+    }
+
+    chrono::seconds expiration(
+        getProfileAgentSettingWithDefault<uint>(
+            cache_expiration_in_seconds,
+            "layer7AccessControl.crowdsec.cacheExpiration"
+        )
     );
 
     ip_reputation_cache.startExpiration(

components/security_apps/layer_7_access_control/layer_7_access_control_ut/layer_7_access_control_ut.cc

@@ -247,7 +247,9 @@ Layer7AccessControlTest::verifyReport(
     string log = reportToStr(report);
     dbgTrace(D_L7_ACCESS_CONTROL) << "Report: " << log;
 
-    if (!source_identifier.empty()) EXPECT_THAT(log, HasSubstr("\"httpSourceId\": \"" + source_identifier + "\""));
+    if (!source_identifier.empty()) {
+        EXPECT_THAT(log, HasSubstr("\"httpSourceId\": \"" + source_identifier + "\""));
+    }
     EXPECT_THAT(log, HasSubstr("\"securityAction\": \"" + security_action + "\""));
     EXPECT_THAT(log, HasSubstr("\"eventName\": \"Access Control External Vendor Reputation\""));
     EXPECT_THAT(log, HasSubstr("\"httpHostName\": \"juice-shop.checkpoint.com\""));

components/security_apps/local_policy_mgmt_gen/CMakeLists.txt

@@ -22,4 +22,5 @@ add_library(local_policy_mgmt_gen
     access_control_practice.cc
     configmaps.cc
     reverse_proxy_section.cc
+    policy_activation_data.cc
 )

components/security_apps/local_policy_mgmt_gen/access_control_practice.cc

@@ -228,7 +228,11 @@ AccessControlPracticeSpec::load(cereal::JSONInputArchive &archive_in)
     dbgTrace(D_LOCAL_POLICY) << "Loading AppSec practice spec";
 
     parseAppsecJSONKey<string>("name", practice_name, archive_in);
-    parseAppsecJSONKey<string>("practiceMode", mode, archive_in);
+    parseAppsecJSONKey<string>("practiceMode", mode, archive_in, "inherited");
+    if (valid_modes.count(mode) == 0) {
+        dbgWarning(D_LOCAL_POLICY) << "AppSec Access control practice mode invalid: " << mode;
+        throw PolicyGenException("AppSec Access control practice mode invalid: " + mode);
+    }
     parseAppsecJSONKey<string>("appsecClassName", appsec_class_name, archive_in);
     parseMandatoryAppsecJSONKey<AccessControlRateLimit>("rateLimit", rate_limit, archive_in);
 }

components/security_apps/local_policy_mgmt_gen/appsec_practice_section.cc

@@ -19,7 +19,14 @@ using namespace std;
 USE_DEBUG_FLAG(D_LOCAL_POLICY);
 // LCOV_EXCL_START Reason: no test exist
 
-static const set<string> valid_modes = {"prevent-learn", "detect-learn", "prevent", "detect", "inactive"};
+static const set<string> valid_modes = {
+    "prevent-learn",
+    "detect-learn",
+    "prevent",
+    "detect",
+    "inactive",
+    "as-top-level"
+};
 static const set<string> valid_confidences = {"medium", "high", "critical"};
 
 void
@@ -138,15 +145,11 @@ AppSecPracticeWebAttacks::load(cereal::JSONInputArchive &archive_in)
         dbgWarning(D_LOCAL_POLICY) << "AppSec practice override mode invalid: " << mode;
     }
 
-    if (getMode() == "Prevent") {
-        parseAppsecJSONKey<string>("minimum-confidence", minimum_confidence, archive_in, "critical");
-        if (valid_confidences.count(minimum_confidence) == 0) {
-            dbgWarning(D_LOCAL_POLICY)
-                << "AppSec practice override minimum confidence invalid: "
-                << minimum_confidence;
-        }
-    } else {
-        minimum_confidence = "Transparent";
+    parseAppsecJSONKey<string>("minimum-confidence", minimum_confidence, archive_in, "critical");
+    if (valid_confidences.count(minimum_confidence) == 0) {
+        dbgWarning(D_LOCAL_POLICY)
+            << "AppSec practice override minimum confidence invalid: "
+            << minimum_confidence;
     }
     parseAppsecJSONKey<int>("max-body-size-kb", max_body_size_kb, archive_in, 1000000);
     parseAppsecJSONKey<int>("max-header-size-bytes", max_header_size_bytes, archive_in, 102400);
@@ -189,7 +192,10 @@ AppSecPracticeWebAttacks::getMode(const string &default_mode) const
 {
     if (isModeInherited(mode) || (key_to_practices_val2.find(mode) == key_to_practices_val2.end())) {
         dbgError(D_LOCAL_POLICY) << "Couldn't find a value for key: " << mode << ". Returning " << default_mode;
-        return default_mode;
+        if(key_to_practices_val2.find(default_mode) == key_to_practices_val2.end()) {
+            return default_mode;
+        }
+        return key_to_practices_val2.at(default_mode);
     }
     return key_to_practices_val2.at(mode);
 }
@@ -404,6 +410,7 @@ AppsecPracticeAntiBotSection::save(cereal::JSONOutputArchive &out_ar) const
 }
 
 // LCOV_EXCL_START Reason: no test exist
+// Used for V1Beta1
 WebAppSection::WebAppSection(
     const string &_application_urls,
     const string &_asset_id,
@@ -417,7 +424,7 @@ WebAppSection::WebAppSection(
     const LogTriggerSection &parsed_log_trigger,
     const string &default_mode,
     const AppSecTrustedSources &parsed_trusted_sources,
-    const vector<InnerException> &parsed_exceptions)
+    const std::map<std::string, std::vector<InnerException>> &exceptions)
     :
     application_urls(_application_urls),
     asset_id(_asset_id),
@@ -427,21 +434,34 @@ WebAppSection::WebAppSection(
     practice_id(_practice_id),
     practice_name(_practice_name),
     context(_context),
-    web_attack_mitigation_severity(parsed_appsec_spec.getWebAttacks().getMinimumConfidence()),
     web_attack_mitigation_mode(parsed_appsec_spec.getWebAttacks().getMode(default_mode)),
     csrf_protection_mode("Disabled"),
     open_redirect_mode("Disabled"),
     error_disclosure_mode("Disabled"),
+    schema_validation_mode("Disabled"),
+    schema_validation_enforce_level("fullSchema"),
     practice_advanced_config(parsed_appsec_spec),
     anti_bots(parsed_appsec_spec.getAntiBot()),
     trusted_sources({ parsed_trusted_sources })
 {
+    auto mitigation_sevirity = parsed_appsec_spec.getWebAttacks().getMinimumConfidence();
+    if (key_to_mitigation_severity.find(mitigation_sevirity) == key_to_mitigation_severity.end()) {
+        dbgWarning(D_LOCAL_POLICY)
+        << "web attack mitigation severity invalid: "
+        << mitigation_sevirity;
+        throw PolicyGenException("web attack mitigation severity invalid: " + mitigation_sevirity);
+    } else {
+        web_attack_mitigation_severity = key_to_mitigation_severity.at(mitigation_sevirity);
+    }
     web_attack_mitigation = web_attack_mitigation_mode != "Disabled";
+    web_attack_mitigation_severity =
+        web_attack_mitigation_mode != "Prevent" ? "Transparent" :
+        web_attack_mitigation_severity;
     web_attack_mitigation_action =
         web_attack_mitigation_mode != "Prevent" ? "Transparent" :
-        web_attack_mitigation_severity == "critical" ? "low" :
-        web_attack_mitigation_severity == "high" ? "balanced" :
-        web_attack_mitigation_severity == "medium" ? "high" :
+        web_attack_mitigation_severity == "Critical" ? "Low" :
+        web_attack_mitigation_severity == "High" ? "Balanced" :
+        web_attack_mitigation_severity == "Medium" ? "High" :
         "Error";
 
     triggers.push_back(TriggersInWaapSection(parsed_log_trigger));
@@ -449,8 +469,11 @@ WebAppSection::WebAppSection(
         overrides.push_back(AppSecOverride(source_ident));
     }
 
-    for (const InnerException &exception : parsed_exceptions) {
-        overrides.push_back(AppSecOverride(exception));
+    for (const auto &exception : exceptions) {
+
+        for (const auto &inner_exception : exception.second) {
+            overrides.push_back(AppSecOverride(inner_exception));
+        }
     }
 }
 
@@ -466,11 +489,16 @@ WebAppSection::WebAppSection(
     const string &_context,
     const string &_web_attack_mitigation_severity,
     const string &_web_attack_mitigation_mode,
+    const string &_bot_protection,
+    const string &_schema_validation_mode,
+    const string &_schema_validation_enforce_level,
+    const vector<string> &_schema_validation_oas,
     const PracticeAdvancedConfig &_practice_advanced_config,
     const AppsecPracticeAntiBotSection &_anti_bots,
     const LogTriggerSection &parsed_log_trigger,
     const AppSecTrustedSources &parsed_trusted_sources,
-    const NewAppSecWebAttackProtections &protections)
+    const NewAppSecWebAttackProtections &protections,
+    const vector<InnerException> &exceptions)
     :
     application_urls(_application_urls),
     asset_id(_asset_id),
@@ -480,18 +508,29 @@ WebAppSection::WebAppSection(
     practice_id(_practice_id),
     practice_name(_practice_name),
     context(_context),
-    web_attack_mitigation_severity(_web_attack_mitigation_severity),
     web_attack_mitigation_mode(_web_attack_mitigation_mode),
+    bot_protection(_bot_protection),
+    schema_validation_mode(_schema_validation_mode),
+    schema_validation_enforce_level(_schema_validation_enforce_level),
+    schema_validation_oas(_schema_validation_oas),
     practice_advanced_config(_practice_advanced_config),
     anti_bots(_anti_bots),
     trusted_sources({ parsed_trusted_sources })
 {
+    if (key_to_mitigation_severity.find(_web_attack_mitigation_severity) == key_to_mitigation_severity.end()) {
+        dbgWarning(D_LOCAL_POLICY)
+        << "web attack mitigation severity invalid: "
+        << _web_attack_mitigation_severity;
+        throw PolicyGenException("web attack mitigation severity invalid: " + _web_attack_mitigation_severity);
+    } else {
+        web_attack_mitigation_severity = key_to_mitigation_severity.at(_web_attack_mitigation_severity);
+    }
     web_attack_mitigation = web_attack_mitigation_mode != "Disabled";
     web_attack_mitigation_action =
         web_attack_mitigation_mode != "Prevent" ? "Transparent" :
-        web_attack_mitigation_severity == "critical" ? "low" :
-        web_attack_mitigation_severity == "high" ? "balanced" :
-        web_attack_mitigation_severity == "medium" ? "high" :
+        web_attack_mitigation_severity == "Critical" ? "Low" :
+        web_attack_mitigation_severity == "High" ? "Balanced" :
+        web_attack_mitigation_severity == "Medium" ? "High" :
         "Error";
 
     csrf_protection_mode = protections.getCsrfProtectionMode(_web_attack_mitigation_mode);
@@ -502,6 +541,11 @@ WebAppSection::WebAppSection(
     for (const SourcesIdentifiers &source_ident : parsed_trusted_sources.getSourcesIdentifiers()) {
         overrides.push_back(AppSecOverride(source_ident));
     }
+
+    for (const auto &exception : exceptions) {
+        overrides.push_back(AppSecOverride(exception));
+    }
+
 }
 
 // LCOV_EXCL_STOP
@@ -509,36 +553,35 @@ WebAppSection::WebAppSection(
 void
 WebAppSection::save(cereal::JSONOutputArchive &out_ar) const
 {
-    string disabled_str = "Disabled";
-    string detect_str = "Detect";
     vector<string> empty_list;
     out_ar(
-        cereal::make_nvp("context",                     context),
-        cereal::make_nvp("webAttackMitigation",         web_attack_mitigation),
-        cereal::make_nvp("webAttackMitigationSeverity", web_attack_mitigation_severity),
-        cereal::make_nvp("webAttackMitigationAction",   web_attack_mitigation_action),
-        cereal::make_nvp("webAttackMitigationMode",     web_attack_mitigation_mode),
-        cereal::make_nvp("practiceAdvancedConfig",      practice_advanced_config),
-        cereal::make_nvp("csrfProtection",              csrf_protection_mode),
-        cereal::make_nvp("openRedirect",                open_redirect_mode),
-        cereal::make_nvp("errorDisclosure",             error_disclosure_mode),
-        cereal::make_nvp("practiceId",                  practice_id),
-        cereal::make_nvp("practiceName",                practice_name),
-        cereal::make_nvp("assetId",                     asset_id),
-        cereal::make_nvp("assetName",                   asset_name),
-        cereal::make_nvp("ruleId",                      rule_id),
-        cereal::make_nvp("ruleName",                    rule_name),
-        cereal::make_nvp("schemaValidation",               false),
-        cereal::make_nvp("schemaValidation_v2",            disabled_str),
-        cereal::make_nvp("oas",                            empty_list),
-        cereal::make_nvp("triggers",                    triggers),
-        cereal::make_nvp("applicationUrls",             application_urls),
-        cereal::make_nvp("overrides",                   overrides),
-        cereal::make_nvp("trustedSources",              trusted_sources),
-        cereal::make_nvp("waapParameters",              empty_list),
-        cereal::make_nvp("botProtection",               false),
-        cereal::make_nvp("antiBot",                     anti_bots),
-        cereal::make_nvp("botProtection_v2",            detect_str)
+        cereal::make_nvp("context",                         context),
+        cereal::make_nvp("webAttackMitigation",             web_attack_mitigation),
+        cereal::make_nvp("webAttackMitigationSeverity",     web_attack_mitigation_severity),
+        cereal::make_nvp("webAttackMitigationAction",       web_attack_mitigation_action),
+        cereal::make_nvp("webAttackMitigationMode",         web_attack_mitigation_mode),
+        cereal::make_nvp("practiceAdvancedConfig",          practice_advanced_config),
+        cereal::make_nvp("csrfProtection",                  csrf_protection_mode),
+        cereal::make_nvp("openRedirect",                    open_redirect_mode),
+        cereal::make_nvp("errorDisclosure",                 error_disclosure_mode),
+        cereal::make_nvp("practiceId",                      practice_id),
+        cereal::make_nvp("practiceName",                    practice_name),
+        cereal::make_nvp("assetId",                         asset_id),
+        cereal::make_nvp("assetName",                       asset_name),
+        cereal::make_nvp("ruleId",                          rule_id),
+        cereal::make_nvp("ruleName",                        rule_name),
+        cereal::make_nvp("schemaValidation",                schema_validation_mode == "Prevent"),
+        cereal::make_nvp("schemaValidation_v2",             schema_validation_mode),
+        cereal::make_nvp("oas",                             schema_validation_oas),
+        cereal::make_nvp("schemaValidationEnforceLevel",    schema_validation_enforce_level),
+        cereal::make_nvp("triggers",                        triggers),
+        cereal::make_nvp("applicationUrls",                 application_urls),
+        cereal::make_nvp("overrides",                       overrides),
+        cereal::make_nvp("trustedSources",                  trusted_sources),
+        cereal::make_nvp("waapParameters",                  empty_list),
+        cereal::make_nvp("botProtection",                   false),
+        cereal::make_nvp("antiBot",                         anti_bots),
+        cereal::make_nvp("botProtection_v2",                bot_protection != "" ? bot_protection : string("Detect"))
     );
 }
 

components/security_apps/local_policy_mgmt_gen/exceptions_section.cc

@@ -146,7 +146,9 @@ AppsecException::load(cereal::JSONInputArchive &archive_in)
 {
     dbgTrace(D_LOCAL_POLICY) << "Loading AppSec exception";
     parseAppsecJSONKey<string>("name", name, archive_in);
-    archive_in(CEREAL_NVP(exception_spec));
+    AppsecExceptionSpec single_exception_spec;
+    single_exception_spec.load(archive_in);
+    exception_spec.push_back(single_exception_spec);
 }
 
 void
@@ -174,7 +176,7 @@ ExceptionMatch::ExceptionMatch(const AppsecExceptionSpec &parsed_exception)
 {
     bool single_condition = parsed_exception.isOneCondition();
     for (auto &attrib : attributes) {
-        auto &attrib_name = attrib.first;
+        auto attrib_name = (attrib.first == "sourceIp" ? "sourceIP" : attrib.first);
         auto &attrib_getter = attrib.second;
         auto exceptions_value = attrib_getter(parsed_exception);
         if (exceptions_value.empty()) continue;

components/security_apps/local_policy_mgmt_gen/include/appsec_practice_section.h

@@ -275,7 +275,7 @@ public:
         const LogTriggerSection &parsed_log_trigger,
         const std::string &default_mode,
         const AppSecTrustedSources &parsed_trusted_sources,
-        const std::vector<InnerException> &parsed_exceptions
+        const std::map<std::string, std::vector<InnerException>> &exceptions
     );
 
     // used for V1beta2
@@ -290,37 +290,46 @@ public:
         const std::string &_context,
         const std::string &_web_attack_mitigation_severity,
         const std::string &_web_attack_mitigation_mode,
+        const std::string &_bot_protection,
+        const std::string &schema_validation_mode,
+        const std::string &schema_validation_enforce_level,
+        const std::vector<std::string> &schema_validation_oas,
         const PracticeAdvancedConfig &_practice_advanced_config,
         const AppsecPracticeAntiBotSection &_anti_bots,
         const LogTriggerSection &parsed_log_trigger,
         const AppSecTrustedSources &parsed_trusted_sources,
-        const NewAppSecWebAttackProtections &protections);
+        const NewAppSecWebAttackProtections &protections,
+        const std::vector<InnerException> &exceptions);
 
     void save(cereal::JSONOutputArchive &out_ar) const;
 
     bool operator< (const WebAppSection &other) const;
 
 private:
-    std::string application_urls;
-    std::string asset_id;
-    std::string asset_name;
-    std::string rule_id;
-    std::string rule_name;
-    std::string practice_id;
-    std::string practice_name;
-    std::string context;
-    std::string web_attack_mitigation_action;
-    std::string web_attack_mitigation_severity;
-    std::string web_attack_mitigation_mode;
-    std::string csrf_protection_mode;
-    std::string open_redirect_mode;
-    std::string error_disclosure_mode;
-    bool web_attack_mitigation;
-    std::vector<TriggersInWaapSection> triggers;
-    PracticeAdvancedConfig practice_advanced_config;
-    AppsecPracticeAntiBotSection anti_bots;
-    std::vector<AppSecTrustedSources> trusted_sources;
-    std::vector<AppSecOverride> overrides;
+    bool                                    web_attack_mitigation;
+    std::string                             application_urls;
+    std::string                             asset_id;
+    std::string                             asset_name;
+    std::string                             rule_id;
+    std::string                             rule_name;
+    std::string                             practice_id;
+    std::string                             practice_name;
+    std::string                             context;
+    std::string                             web_attack_mitigation_action;
+    std::string                             web_attack_mitigation_severity;
+    std::string                             web_attack_mitigation_mode;
+    std::string                             csrf_protection_mode;
+    std::string                             open_redirect_mode;
+    std::string                             error_disclosure_mode;
+    std::string                             bot_protection;
+    std::string                             schema_validation_mode;
+    std::string                             schema_validation_enforce_level;
+    std::vector<std::string>                schema_validation_oas;
+    PracticeAdvancedConfig                  practice_advanced_config;
+    AppsecPracticeAntiBotSection            anti_bots;
+    std::vector<AppSecOverride>             overrides;
+    std::vector<AppSecTrustedSources>       trusted_sources;
+    std::vector<TriggersInWaapSection>      triggers;
 };
 
 class WebAPISection
@@ -408,7 +417,7 @@ class ParsedRule
 {
 public:
     ParsedRule() {}
-    ParsedRule(const std::string &_host) : host(_host) {}
+    ParsedRule(const std::string &_host, const std::string &_mode) : host(_host), mode(_mode) {}
 
     void load(cereal::JSONInputArchive &archive_in);
     const std::vector<std::string> & getExceptions() const;

components/security_apps/local_policy_mgmt_gen/include/exceptions_section.h

@@ -44,7 +44,7 @@ public:
     bool isOneCondition() const;
 
 private:
-    int conditions_number;
+    int conditions_number = 0;
     std::string action;
     std::vector<std::string> country_code;
     std::vector<std::string> country_name;

components/security_apps/local_policy_mgmt_gen/include/ingress_data.h

@@ -79,6 +79,7 @@ class DefaultBackend
 {
 public:
     void load(cereal::JSONInputArchive &);
+    bool doesExist() const;
 
 private:
     bool is_exists = false;
@@ -90,6 +91,7 @@ public:
     void load(cereal::JSONInputArchive &archive_in);
 
     const std::vector<IngressDefinedRule> & getRules() const;
+    bool doesDefaultBackendExist() const;
 
 private:
     std::string ingress_class_name;

components/security_apps/local_policy_mgmt_gen/include/k8s_policy_utils.h

@@ -24,6 +24,7 @@
 #include "maybe_res.h"
 #include "i_orchestration_tools.h"
 #include "i_shell_cmd.h"
+#include "i_encryptor.h"
 #include "i_messaging.h"
 #include "i_env_details.h"
 #include "i_agent_details.h"
@@ -40,13 +41,14 @@ class K8sPolicyUtils
     Singleton::Consume<I_Messaging>,
     Singleton::Consume<I_ShellCmd>,
     Singleton::Consume<I_EnvDetails>,
+    Singleton::Consume<I_Encryptor>,
     Singleton::Consume<I_AgentDetails>
 {
 public:
     void init();
 
     std::tuple<std::map<std::string, AppsecLinuxPolicy>, std::map<std::string, V1beta2AppsecLinuxPolicy>>
-    createAppsecPoliciesFromIngresses();
+    createAppsecPolicies();
     void getClusterId() const;
 
 private:
@@ -80,6 +82,8 @@ private:
 
     void createSnortFile(std::vector<NewAppSecPracticeSpec> &practices) const;
 
+    void createSchemaValidationOas(std::vector<NewAppSecPracticeSpec> &practices) const;
+
     template<class T>
     std::vector<T> extractV1Beta2ElementsFromCluster(
         const std::string &crd_plural,
@@ -97,12 +101,18 @@ private:
     ) const;
 
     template<class T, class K>
-    void createPolicy(
+    void createPolicyFromIngress(
         T &appsec_policy,
         std::map<std::string, T> &policies,
         std::map<AnnotationKeys, std::string> &annotations_values,
         const SingleIngressData &item) const;
 
+    template<class T, class K>
+    void createPolicyFromActivation(
+        T &appsec_policy,
+        std::map<std::string, T> &policies,
+        const EnabledPolicy &policy) const;
+
     std::tuple<Maybe<AppsecLinuxPolicy>, Maybe<V1beta2AppsecLinuxPolicy>> createAppsecPolicyK8s(
         const std::string &policy_name,
         const std::string &ingress_mode
@@ -112,6 +122,7 @@ private:
     I_Messaging* messaging = nullptr;
     EnvType env_type;
     std::string token;
+    std::string agent_ns;
 };
 
 #endif // __K8S_POLICY_UTILS_H__

components/security_apps/local_policy_mgmt_gen/include/local_policy_common.h

@@ -49,6 +49,13 @@ static const std::unordered_map<std::string, TriggerType> string_to_trigger_type
     { "WebUserResponse", TriggerType::WebUserResponse }
 };
 
+static const std::unordered_map<std::string, std::string> key_to_mitigation_severity = {
+    { "high", "High"},
+    { "medium", "Medium"},
+    { "critical", "Critical"},
+    { "Transparent", "Transparent"}
+};
+
 static const std::unordered_map<std::string, std::string> key_to_practices_val = {
     { "prevent-learn", "Prevent"},
     { "detect-learn", "Learn"},
@@ -57,6 +64,14 @@ static const std::unordered_map<std::string, std::string> key_to_practices_val =
     { "inactive", "Inactive"}
 };
 
+static const std::unordered_map<std::string, std::string> key_to_practices_mode_val = {
+    { "prevent-learn", "Prevent"},
+    { "detect-learn", "Detect"},
+    { "prevent", "Prevent"},
+    { "detect", "Detect"},
+    { "inactive", "Disabled"}
+};
+
 static const std::unordered_map<std::string, std::string> key_to_practices_val2 = {
     { "prevent-learn", "Prevent"},
     { "detect-learn", "Learn"},
@@ -66,6 +81,8 @@ static const std::unordered_map<std::string, std::string> key_to_practices_val2
 };
 
 static const std::string default_appsec_url = "http://*:*";
+static const std::string default_appsec_name = "Any";
+
 
 class PolicyGenException : public std::exception
 {
@@ -153,6 +170,7 @@ public:
         ss.str(modified_json);
         try {
             cereal::JSONInputArchive in_ar(ss);
+            in_ar(cereal::make_nvp("apiVersion", api_version));
             in_ar(cereal::make_nvp("spec", spec));
             in_ar(cereal::make_nvp("metadata", meta_data));
         } catch (cereal::Exception &e) {
@@ -174,11 +192,18 @@ public:
         return meta_data;
     }
 
+    const std::string &
+    getApiVersion() const
+    {
+        return api_version;
+    }
+
     const T & getSpec() const { return spec; }
 
 private:
     T spec;
     AppsecSpecParserMetaData meta_data;
+    std::string api_version;
 };
 
 #endif // __LOCAL_POLICY_COMMON_H__

components/security_apps/local_policy_mgmt_gen/include/new_appsec_policy_crd_parser.h

@@ -31,7 +31,7 @@ class NewParsedRule
 {
 public:
     NewParsedRule() {}
-    NewParsedRule(const std::string &_host) : host(_host) {}
+    NewParsedRule(const std::string &_host, const std::string &_mode) : host(_host), mode(_mode) {}
 
     void load(cereal::JSONInputArchive &archive_in);
 

components/security_apps/local_policy_mgmt_gen/include/new_log_trigger.h

@@ -129,7 +129,7 @@ public:
     bool shouldBeautifyLogs() const;
 
     bool getCloud() const;
-    bool isK8SNeeded() const;
+    bool isContainerNeeded() const;
     bool isCefNeeded() const;
     bool isSyslogNeeded() const;
     const std::string & getSyslogServerIpv4Address() const;
@@ -140,7 +140,7 @@ private:
     const NewLoggingService & getCefServiceData() const;
 
     bool cloud = false;
-    bool k8s_service = false;
+    bool container_service = false;
     bool agent_local = true;
     bool beautify_logs = true;
     NewLoggingService syslog_service;

components/security_apps/local_policy_mgmt_gen/include/new_practice.h

@@ -23,6 +23,8 @@
 #include "config.h"
 #include "debug.h"
 #include "local_policy_common.h"
+#include "i_orchestration_tools.h"
+#include "i_encryptor.h"
 
 bool isModeInherited(const std::string &mode);
 
@@ -88,6 +90,8 @@ public:
 
     void save(cereal::JSONOutputArchive &out_ar) const;
 
+    bool operator<(const IpsProtectionsSection &other) const;
+
 private:
     std::string                                 context;
     std::string                                 name;
@@ -105,7 +109,7 @@ public:
     // LCOV_EXCL_START Reason: no test exist
     IPSSection() {};
 
-    IPSSection(const std::vector<IpsProtectionsSection> &_ips) : ips(_ips) {};
+    IPSSection(const std::vector<IpsProtectionsSection> &_ips);
     // LCOV_EXCL_STOP
 
     void save(cereal::JSONOutputArchive &out_ar) const;
@@ -138,6 +142,12 @@ public:
     const std::string & getMode(const std::string &default_mode = "inactive") const;
 
 private:
+
+    const std::string & getRulesMode(
+        const std::string &mode,
+        const std::string &default_mode = "inactive"
+    ) const;
+
     std::string override_mode;
     std::string max_performance_impact;
     std::string min_severity_level;
@@ -487,15 +497,16 @@ private:
     SnortSection snort;
 };
 
-class NewSnortSignaturesAndOpenSchemaAPI
+class NewSnortSignatures
 {
 public:
-    NewSnortSignaturesAndOpenSchemaAPI() : is_temporary(false) {};
+    NewSnortSignatures() : is_temporary(false) {};
 
     void load(cereal::JSONInputArchive &archive_in);
 
     void addFile(const std::string &file_name);
     const std::string & getOverrideMode(const std::string &default_mode = "inactive") const;
+    const std::string & getEnforceLevel() const;
     const std::vector<std::string> & getConfigMap() const;
     const std::vector<std::string> & getFiles() const;
     bool isTemporary() const;
@@ -503,35 +514,48 @@ public:
 
 private:
     std::string override_mode;
+    std::string enforcement_level;
     std::vector<std::string> config_map;
     std::vector<std::string> files;
     bool is_temporary;
 };
 
-class NewAppSecWebBotsURI
+class NewOpenApiSchema : Singleton::Consume<I_OrchestrationTools>, Singleton::Consume<I_Encryptor>
 {
 public:
+    NewOpenApiSchema() {};
+
     void load(cereal::JSONInputArchive &archive_in);
 
-    const std::string & getURI() const;
+    void addOas(const std::string &file);
+    const std::string & getOverrideMode(const std::string &default_mode = "inactive") const;
+    const std::string & getEnforceLevel() const;
+    const std::vector<std::string> & getConfigMap() const;
+    const std::vector<std::string> & getFiles() const;
+    const std::vector<std::string> & getOas() const;
 
 private:
-    std::string uri;
+    std::string override_mode;
+    std::string enforcement_level;
+    std::vector<std::string> config_map;
+    std::vector<std::string> files;
+    std::vector<std::string> oas;
 };
 
 class NewAppSecPracticeAntiBot
 {
 public:
-    std::vector<std::string> getIjectedUris() const;
-    std::vector<std::string> getValidatedUris() const;
+    const std::vector<std::string> & getIjectedUris() const;
+    const std::vector<std::string> & getValidatedUris() const;
+    const std::string & getMode(const std::string &default_mode = "inactive") const;
 
     void load(cereal::JSONInputArchive &archive_in);
     void save(cereal::JSONOutputArchive &out_ar) const;
 
 private:
     std::string override_mode;
-    std::vector<NewAppSecWebBotsURI> injected_uris;
-    std::vector<NewAppSecWebBotsURI> validated_uris;
+    std::vector<std::string> injected_uris;
+    std::vector<std::string> validated_uris;
 };
 
 class NewAppSecWebAttackProtections
@@ -579,8 +603,8 @@ class NewAppSecPracticeSpec
 public:
     void load(cereal::JSONInputArchive &archive_in);
 
-    NewSnortSignaturesAndOpenSchemaAPI & getSnortSignatures();
-    const NewSnortSignaturesAndOpenSchemaAPI & getOpenSchemaValidation() const;
+    NewSnortSignatures & getSnortSignatures();
+    NewOpenApiSchema & getOpenSchemaValidation();
     const NewAppSecPracticeWebAttacks & getWebAttacks() const;
     const NewAppSecPracticeAntiBot & getAntiBot() const;
     const NewIntrusionPrevention & getIntrusionPrevention() const;
@@ -593,8 +617,8 @@ public:
 private:
     NewFileSecurity                             file_security;
     NewIntrusionPrevention                      intrusion_prevention;
-    NewSnortSignaturesAndOpenSchemaAPI          openapi_schema_validation;
-    NewSnortSignaturesAndOpenSchemaAPI          snort_signatures;
+    NewOpenApiSchema                            openapi_schema_validation;
+    NewSnortSignatures                          snort_signatures;
     NewAppSecPracticeWebAttacks                 web_attacks;
     NewAppSecPracticeAntiBot                    anti_bot;
     std::string                                 appsec_class_name;

components/security_apps/local_policy_mgmt_gen/include/policy_activation_data.h

@@ -0,0 +1,89 @@
+// Copyright (C) 2022 Check Point Software Technologies Ltd. All rights reserved.
+
+// Licensed under the Apache License, Version 2.0 (the "License");
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#ifndef __POLICY_ACTIVATION_DATA_H__
+#define __POLICY_ACTIVATION_DATA_H__
+
+#include <vector>
+#include <map>
+
+#include "config.h"
+#include "debug.h"
+#include "rest.h"
+#include "cereal/archives/json.hpp"
+#include <cereal/types/map.hpp>
+#include "customized_cereal_map.h"
+
+#include "local_policy_common.h"
+
+class PolicyActivationMetadata
+{
+public:
+    void load(cereal::JSONInputArchive &archive_in);
+
+private:
+    std::string name;
+};
+
+class EnabledPolicy
+{
+public:
+    void load(cereal::JSONInputArchive &archive_in);
+
+    const std::string & getName() const;
+    const std::vector<std::string> & getHosts() const;
+
+private:
+    std::string name;
+    std::vector<std::string> hosts;
+};
+
+class PolicyActivationSpec
+{
+public:
+    void load(cereal::JSONInputArchive &archive_in);
+
+    const std::vector<EnabledPolicy> & getPolicies() const;
+
+private:
+    std::string appsec_class_name;
+    std::vector<EnabledPolicy> policies;
+};
+
+class SinglePolicyActivationData
+{
+public:
+    void load(cereal::JSONInputArchive &archive_in);
+
+    const PolicyActivationSpec & getSpec() const;
+
+private:
+    std::string api_version;
+    std::string kind;
+    PolicyActivationMetadata metadata;
+    PolicyActivationSpec spec;
+};
+
+class PolicyActivationData : public ClientRest
+{
+public:
+    bool loadJson(const std::string &json);
+
+    const std::vector<SinglePolicyActivationData> & getItems() const;
+
+private:
+    std::string api_version;
+    std::vector<SinglePolicyActivationData> items;
+};
+
+#endif // __POLICY_ACTIVATION_DATA_H__

components/security_apps/local_policy_mgmt_gen/include/policy_maker_utils.h

@@ -32,6 +32,7 @@
 #include "i_messaging.h"
 #include "appsec_practice_section.h"
 #include "ingress_data.h"
+#include "policy_activation_data.h"
 #include "settings_section.h"
 #include "triggers_section.h"
 #include "local_policy_common.h"
@@ -111,7 +112,7 @@ private:
     SecurityAppsWrapper security_apps;
 };
 
-class PolicyMakerUtils
+class PolicyMakerUtils : Singleton::Consume<I_EnvDetails>
 {
 public:
     std::string proccesSingleAppsecPolicy(
@@ -205,7 +206,8 @@ private:
         const RulesConfigRulebase& rule_config,
         const std::string &practice_id, const std::string &full_url,
         const std::string &default_mode,
-        std::map<AnnotationTypes, std::string> &rule_annotations
+        std::map<AnnotationTypes, std::string> &rule_annotations,
+        std::vector<InnerException>
     );
 
     void

components/security_apps/local_policy_mgmt_gen/include/rules_config_section.h

@@ -123,6 +123,7 @@ public:
     );
 
     const std::string & getIdentifier() const;
+    const std::string & getIdentifierValue() const;
 
     void save(cereal::JSONOutputArchive &out_ar) const;
 
@@ -145,6 +146,7 @@ public:
     );
 
     const std::string & getIdentifier() const;
+    const std::string & getIdentifierValue() const;
 
     void save(cereal::JSONOutputArchive &out_ar) const;
 

components/security_apps/local_policy_mgmt_gen/include/triggers_section.h

@@ -39,7 +39,7 @@ public:
         bool _logToAgent,
         bool _logToCef,
         bool _logToCloud,
-        bool _logToK8sService,
+        bool _logTolocalTuning,
         bool _logToSyslog,
         bool _responseBody,
         bool _tpDetect,
@@ -73,7 +73,7 @@ private:
     bool logToAgent;
     bool logToCef;
     bool logToCloud;
-    bool logToK8sService;
+    bool logTolocalTuning;
     bool logToSyslog;
     bool responseBody;
     bool tpDetect;
@@ -258,7 +258,7 @@ public:
     bool shouldBeautifyLogs() const;
 
     bool getCloud() const;
-    bool isK8SNeeded() const;
+    bool isContainerNeeded() const;
     bool isCefNeeded() const;
     bool isSyslogNeeded() const;
     const std::string & getSyslogServerIpv4Address() const;
@@ -269,7 +269,7 @@ private:
     const LoggingService & getCefServiceData() const;
 
     bool cloud = false;
-    bool k8s_service = false;
+    bool container_service = false;
     bool agent_local = true;
     bool beautify_logs = true;
     LoggingService syslog_service;

components/security_apps/local_policy_mgmt_gen/ingress_data.cc

@@ -86,6 +86,12 @@ DefaultBackend::load(cereal::JSONInputArchive &)
     is_exists = true;
 }
 
+bool
+DefaultBackend::doesExist() const
+{
+    return is_exists;
+}
+
 void
 IngressSpec::load(cereal::JSONInputArchive &archive_in)
 {
@@ -101,6 +107,12 @@ IngressSpec::getRules() const
     return rules;
 }
 
+bool
+IngressSpec::doesDefaultBackendExist() const
+{
+    return default_backend.doesExist();
+}
+
 void
 SingleIngressData::load(cereal::JSONInputArchive &archive_in)
 {

components/security_apps/local_policy_mgmt_gen/k8s_policy_utils.cc

@@ -35,6 +35,14 @@ convertAnnotationKeysTostring(const AnnotationKeys &key)
     }
 }
 
+string
+getAppSecScopeType()
+{
+    auto env_res = getenv("CRDS_SCOPE");
+    if (env_res != nullptr) return env_res;
+    return "cluster";
+}
+
 void
 K8sPolicyUtils::init()
 {
@@ -42,6 +50,7 @@ K8sPolicyUtils::init()
     env_type = env_details->getEnvType();
     if (env_type == EnvType::K8S) {
         token = env_details->getToken();
+        agent_ns = getAppSecScopeType() == "namespaced" ? env_details->getNameSpace() + "/" : "";
         messaging = Singleton::Consume<I_Messaging>::by<K8sPolicyUtils>();
     }
 }
@@ -140,10 +149,12 @@ extractElementsFromNewRule(
     const NewParsedRule &rule,
     map<AnnotationTypes, unordered_set<string>> &policy_elements_names)
 {
-    policy_elements_names[AnnotationTypes::EXCEPTION].insert(
-        rule.getExceptions().begin(),
-        rule.getExceptions().end()
-    );
+    if (rule.getExceptions().size() > 0) {
+        policy_elements_names[AnnotationTypes::EXCEPTION].insert(
+            rule.getExceptions().begin(),
+            rule.getExceptions().end()
+        );
+    }
     policy_elements_names[AnnotationTypes::THREAT_PREVENTION_PRACTICE].insert(
         rule.getPractices().begin(),
         rule.getPractices().end()
@@ -152,14 +163,24 @@ extractElementsFromNewRule(
         rule.getAccessControlPractices().begin(),
         rule.getAccessControlPractices().end()
     );
-    policy_elements_names[AnnotationTypes::TRIGGER].insert(
-        rule.getLogTriggers().begin(),
-        rule.getLogTriggers().end()
-    );
-    policy_elements_names[AnnotationTypes::WEB_USER_RES].insert(rule.getCustomResponse());
-    policy_elements_names[AnnotationTypes::SOURCE_IDENTIFIERS].insert(rule.getSourceIdentifiers());
-    policy_elements_names[AnnotationTypes::TRUSTED_SOURCES].insert(rule.getTrustedSources());
-    policy_elements_names[AnnotationTypes::UPGRADE_SETTINGS].insert(rule.getUpgradeSettings());
+    if (rule.getLogTriggers().size() > 0) {
+        policy_elements_names[AnnotationTypes::TRIGGER].insert(
+            rule.getLogTriggers().begin(),
+            rule.getLogTriggers().end()
+        );
+    }
+    if (rule.getCustomResponse() != "" ) {
+        policy_elements_names[AnnotationTypes::WEB_USER_RES].insert(rule.getCustomResponse());
+    }
+    if (rule.getSourceIdentifiers() != "" ) {
+        policy_elements_names[AnnotationTypes::SOURCE_IDENTIFIERS].insert(rule.getSourceIdentifiers());
+    }
+    if (rule.getTrustedSources() != "" ) {
+        policy_elements_names[AnnotationTypes::TRUSTED_SOURCES].insert(rule.getTrustedSources());
+    }
+    if (rule.getUpgradeSettings() != "" ) {
+        policy_elements_names[AnnotationTypes::UPGRADE_SETTINGS].insert(rule.getUpgradeSettings());
+    }
 }
 
 map<AnnotationTypes, unordered_set<string>>
@@ -259,9 +280,11 @@ K8sPolicyUtils::extractV1Beta2ElementsFromCluster(
     dbgTrace(D_LOCAL_POLICY) << "Retrieve AppSec elements. type: " << crd_plural;
     vector<T> elements;
     for (const string &element_name : elements_names) {
+        string ns_suffix = getAppSecScopeType() == "namespaced" ? "ns" : "";
+        string ns = getAppSecScopeType() == "namespaced" ? "namespaces/" : "";
         dbgTrace(D_LOCAL_POLICY) << "AppSec element name: " << element_name;
         auto maybe_appsec_element = getObjectFromCluster<AppsecSpecParser<T>>(
-            "/apis/openappsec.io/v1beta2/" + crd_plural + "/" + element_name
+            "/apis/openappsec.io/v1beta2/" + ns + agent_ns + crd_plural + ns_suffix + "/" + element_name
         );
 
         if (!maybe_appsec_element.ok()) {
@@ -362,8 +385,9 @@ K8sPolicyUtils::createSnortFile(vector<NewAppSecPracticeSpec> &practices) const
         practice.getSnortSignatures().setTemporary(true);
         for (const string &config_map : practice.getSnortSignatures().getConfigMap())
         {
+            string ns = agent_ns == "" ? "default/" : agent_ns;
             auto maybe_configmap = getObjectFromCluster<ConfigMaps>(
-                "/api/v1/namespaces/default/configmaps/" + config_map
+                "/api/v1/namespaces/" + ns + "configmaps/" + config_map
             );
             if (!maybe_configmap.ok())  {
                 dbgWarning(D_LOCAL_POLICY) << "Failed to get configMaps from the cluster.";
@@ -381,6 +405,28 @@ K8sPolicyUtils::createSnortFile(vector<NewAppSecPracticeSpec> &practices) const
     }
 }
 
+void
+K8sPolicyUtils::createSchemaValidationOas(vector<NewAppSecPracticeSpec> &practices) const
+{
+    for (NewAppSecPracticeSpec &practice : practices) {
+        vector<string> res;
+        for (const string &config_map : practice.getOpenSchemaValidation().getConfigMap())
+        {
+            string ns = agent_ns == "" ? "default/" : agent_ns;
+            auto maybe_configmap = getObjectFromCluster<ConfigMaps>(
+                "/api/v1/namespaces/" + ns + "configmaps/" + config_map
+            );
+            if (!maybe_configmap.ok())  {
+                dbgWarning(D_LOCAL_POLICY) << "Failed to get configMaps from the cluster.";
+                continue;
+            }
+            string file_content = maybe_configmap.unpack().getFileContent();
+            string res = Singleton::Consume<I_Encryptor>::by<K8sPolicyUtils>()->base64Encode(file_content);
+            practice.getOpenSchemaValidation().addOas(res);
+        }
+    }
+}
+
 Maybe<V1beta2AppsecLinuxPolicy>
 K8sPolicyUtils::createAppsecPolicyK8sFromV1beta2Crds(
     const AppsecSpecParser<NewAppsecPolicySpec> &appsec_policy_spec,
@@ -396,6 +442,7 @@ K8sPolicyUtils::createAppsecPolicyK8sFromV1beta2Crds(
     }
 
     if (default_rule.getMode().empty() && !ingress_mode.empty()) {
+        dbgTrace(D_LOCAL_POLICY) << "setting the policy default rule mode to the ingress mode: " << ingress_mode;
         default_rule.setMode(ingress_mode);
     }
 
@@ -411,6 +458,7 @@ K8sPolicyUtils::createAppsecPolicyK8sFromV1beta2Crds(
         );
 
     createSnortFile(threat_prevention_practices);
+    createSchemaValidationOas(threat_prevention_practices);
 
     vector<AccessControlPracticeSpec> access_control_practices =
         extractV1Beta2ElementsFromCluster<AccessControlPracticeSpec>(
@@ -467,17 +515,6 @@ K8sPolicyUtils::createAppsecPolicyK8sFromV1beta2Crds(
 }
 // LCOV_EXCL_STOP
 
-bool
-doesVersionExist(const map<string, string> &annotations, const string &version)
-{
-    for (auto annotation : annotations) {
-        if(annotation.second.find(version) != std::string::npos) {
-            return true;
-        }
-    }
-    return false;
-}
-
 std::tuple<Maybe<AppsecLinuxPolicy>, Maybe<V1beta2AppsecLinuxPolicy>>
 K8sPolicyUtils::createAppsecPolicyK8s(const string &policy_name, const string &ingress_mode) const
 {
@@ -486,16 +523,19 @@ K8sPolicyUtils::createAppsecPolicyK8s(const string &policy_name, const string &i
     );
 
     if (!maybe_appsec_policy_spec.ok() ||
-        !doesVersionExist(maybe_appsec_policy_spec.unpack().getMetaData().getAnnotations(), "v1beta1")
+        maybe_appsec_policy_spec.unpack().getApiVersion().find("v1beta1") == std::string::npos
     ) {
         try {
             std::string v1beta1_error =
                 maybe_appsec_policy_spec.ok() ? "There is no v1beta1 policy" : maybe_appsec_policy_spec.getErr();
             dbgWarning(D_LOCAL_POLICY
             ) << "Failed to retrieve Appsec policy with crds version: v1beta1, Trying version: v1beta2";
+            string ns_suffix = getAppSecScopeType() == "namespaced" ? "ns" : "";
+            string ns = getAppSecScopeType() == "namespaced" ? "namespaces/" : "";
             auto maybe_v1beta2_appsec_policy_spec = getObjectFromCluster<AppsecSpecParser<NewAppsecPolicySpec>>(
-                "/apis/openappsec.io/v1beta2/policies/" + policy_name
+                "/apis/openappsec.io/v1beta2/" + ns + agent_ns + "policies" + ns_suffix + "/" + policy_name
             );
+
             if (!maybe_v1beta2_appsec_policy_spec.ok()) {
                 dbgWarning(D_LOCAL_POLICY)
                     << "Failed to retrieve AppSec policy. Error: " << maybe_v1beta2_appsec_policy_spec.getErr();
@@ -526,36 +566,73 @@ K8sPolicyUtils::createAppsecPolicyK8s(const string &policy_name, const string &i
 
 template<class T, class K>
 void
-K8sPolicyUtils::createPolicy(
+K8sPolicyUtils::createPolicyFromIngress(
     T &appsec_policy,
     map<std::string, T> &policies,
     map<AnnotationKeys, string> &annotations_values,
     const SingleIngressData &item) const
 {
+    if (policies.find(annotations_values[AnnotationKeys::PolicyKey]) == policies.end()) {
+        policies[annotations_values[AnnotationKeys::PolicyKey]] = appsec_policy;
+    }
+    auto default_mode = appsec_policy.getAppsecPolicySpec().getDefaultRule().getMode();
+    if (item.getSpec().doesDefaultBackendExist()) {
+        dbgTrace(D_LOCAL_POLICY)
+            << "Inserting Any host rule to the specific asset set";
+        K ingress_rule = K("*", default_mode);
+        policies[annotations_values[AnnotationKeys::PolicyKey]].addSpecificRule(ingress_rule);
+    }
+
     for (const IngressDefinedRule &rule : item.getSpec().getRules()) {
-        string url = rule.getHost();
+        string host = rule.getHost();
         for (const IngressRulePath &uri : rule.getPathsWrapper().getRulePaths()) {
-            if (!appsec_policy.getAppsecPolicySpec().isAssetHostExist(url + uri.getPath())) {
+            if (uri.getPath() != "/") {
+                host = host + uri.getPath();
+            }
+            if (!appsec_policy.getAppsecPolicySpec().isAssetHostExist(host)) {
                 dbgTrace(D_LOCAL_POLICY)
                     << "Inserting Host data to the specific asset set:"
                     << "URL: '"
-                    << url
+                    << rule.getHost()
                     << "' uri: '"
                     << uri.getPath()
                     << "'";
-                K ingress_rule = K(url + uri.getPath());
-                appsec_policy.addSpecificRule(ingress_rule);
+                K ingress_rule = K(host, default_mode);
+                policies[annotations_values[AnnotationKeys::PolicyKey]].addSpecificRule(ingress_rule);
             }
         }
     }
-    policies[annotations_values[AnnotationKeys::PolicyKey]] = appsec_policy;
 }
 
+template<class T, class K>
+void
+K8sPolicyUtils::createPolicyFromActivation(
+    T &appsec_policy,
+    map<std::string, T> &policies,
+    const EnabledPolicy &policy) const
+{
+    if (policies.find(policy.getName()) == policies.end()) {
+        policies[policy.getName()] = appsec_policy;
+    }
+    auto default_mode = appsec_policy.getAppsecPolicySpec().getDefaultRule().getMode();
+
+    for (const string &host : policy.getHosts()) {
+        if (!appsec_policy.getAppsecPolicySpec().isAssetHostExist(host)) {
+            dbgTrace(D_LOCAL_POLICY)
+                << "Inserting Host data to the specific asset set:"
+                << "URL: '"
+                << host
+                << "'";
+            K ingress_rule = K(host, default_mode);
+            policies[policy.getName()].addSpecificRule(ingress_rule);
+        }
+    }
+}
 
 std::tuple<map<string, AppsecLinuxPolicy>, map<string, V1beta2AppsecLinuxPolicy>>
-K8sPolicyUtils::createAppsecPoliciesFromIngresses()
+K8sPolicyUtils::createAppsecPolicies()
 {
-    dbgFlow(D_LOCAL_POLICY) << "Getting all policy object from Ingresses";
+    dbgFlow(D_LOCAL_POLICY) << "Getting all policy object from Ingresses and PolicyActivation";
     map<string, AppsecLinuxPolicy> v1bet1_policies;
     map<string, V1beta2AppsecLinuxPolicy> v1bet2_policies;
     auto maybe_ingress = getObjectFromCluster<IngressData>("/apis/networking.k8s.io/v1/ingresses");
@@ -565,7 +642,7 @@ K8sPolicyUtils::createAppsecPoliciesFromIngresses()
         dbgWarning(D_LOCAL_POLICY)
             << "Failed to retrieve K8S Ingress configurations. Error: "
             << maybe_ingress.getErr();
-        return make_tuple(v1bet1_policies, v1bet2_policies);
+        maybe_ingress = IngressData{};
     }
 
 
@@ -595,19 +672,54 @@ K8sPolicyUtils::createAppsecPoliciesFromIngresses()
 
         if (!std::get<0>(maybe_appsec_policy).ok()) {
             auto appsec_policy=std::get<1>(maybe_appsec_policy).unpack();
-            createPolicy<V1beta2AppsecLinuxPolicy, NewParsedRule>(
+            createPolicyFromIngress<V1beta2AppsecLinuxPolicy, NewParsedRule>(
                 appsec_policy,
                 v1bet2_policies,
                 annotations_values,
                 item);
         } else {
             auto appsec_policy=std::get<0>(maybe_appsec_policy).unpack();
-            createPolicy<AppsecLinuxPolicy, ParsedRule>(
+            createPolicyFromIngress<AppsecLinuxPolicy, ParsedRule>(
                 appsec_policy,
                 v1bet1_policies,
                 annotations_values,
                 item);
         }
     }
+
+
+    string ns_suffix = getAppSecScopeType() == "namespaced" ? "ns" : "";
+    string ns = getAppSecScopeType() == "namespaced" ? "namespaces/" : "";
+    auto maybe_policy_activation = getObjectFromCluster<PolicyActivationData>(
+        "/apis/openappsec.io/v1beta2/" + ns + agent_ns + "policyactivations" + ns_suffix
+    );
+
+    if (!maybe_policy_activation.ok()) {
+        dbgWarning(D_LOCAL_POLICY)
+            << "Failed to retrieve K8S PolicyActivation configurations. Error: "
+            << maybe_policy_activation.getErr();
+        return make_tuple(v1bet1_policies, v1bet2_policies);
+    }
+
+    PolicyActivationData policy_activation = maybe_policy_activation.unpack();
+    for (const SinglePolicyActivationData &item : policy_activation.getItems()) {
+        for (const auto &policy : item.getSpec().getPolicies()) {
+            auto maybe_appsec_policy = createAppsecPolicyK8s(policy.getName(), "");
+
+            if (!std::get<1>(maybe_appsec_policy).ok()) {
+                dbgWarning(D_LOCAL_POLICY)
+                    << "Failed to create appsec policy. v1beta2 Error: "
+                    << std::get<1>(maybe_appsec_policy).getErr();
+                continue;
+            } else {
+                auto appsec_policy=std::get<1>(maybe_appsec_policy).unpack();
+                createPolicyFromActivation<V1beta2AppsecLinuxPolicy, NewParsedRule>(
+                    appsec_policy,
+                    v1bet2_policies,
+                    policy);
+            }
+        }
+    }
+
     return make_tuple(v1bet1_policies, v1bet2_policies);
 }

components/security_apps/local_policy_mgmt_gen/local_policy_mgmt_gen.cc

@@ -36,6 +36,7 @@
 #include "customized_cereal_map.h"
 #include "include/appsec_practice_section.h"
 #include "include/ingress_data.h"
+#include "include/policy_activation_data.h"
 #include "include/settings_section.h"
 #include "include/triggers_section.h"
 #include "include/local_policy_common.h"
@@ -85,7 +86,7 @@ public:
         K8sPolicyUtils k8s_policy_utils;
         k8s_policy_utils.init();
 
-        auto appsec_policies = k8s_policy_utils.createAppsecPoliciesFromIngresses();
+        auto appsec_policies = k8s_policy_utils.createAppsecPolicies();
         if (!std::get<0>(appsec_policies).empty()) {
             return policy_maker_utils.proccesMultipleAppsecPolicies<AppsecLinuxPolicy, ParsedRule>(
                 std::get<0>(appsec_policies),

components/security_apps/local_policy_mgmt_gen/new_appsec_policy_crd_parser.cc

@@ -126,6 +126,7 @@ NewAppsecPolicySpec::load(cereal::JSONInputArchive &archive_in)
     dbgTrace(D_LOCAL_POLICY) << "Loading AppSec policy spec";
     parseAppsecJSONKey<string>("appsecClassName", appsec_class_name, archive_in);
     parseAppsecJSONKey<NewParsedRule>("default", default_rule, archive_in);
+    default_rule.setHost("*");
     parseAppsecJSONKey<vector<NewParsedRule>>("specificRules", specific_rules, archive_in);
 }
 

components/security_apps/local_policy_mgmt_gen/new_log_trigger.cc

@@ -180,10 +180,16 @@ NewAppsecTriggerLogDestination::load(cereal::JSONInputArchive &archive_in)
     } else {
         cloud = false;
     }
-    auto mode = Singleton::Consume<I_AgentDetails>::by<NewAppsecTriggerLogDestination>()->getOrchestrationMode();
-    auto env_type = Singleton::Consume<I_EnvDetails>::by<NewAppsecTriggerLogDestination>()->getEnvType();
-    bool k8s_service_default = (mode == OrchestrationMode::HYBRID && env_type == EnvType::K8S);
-    parseAppsecJSONKey<bool>("k8s-service", k8s_service, archive_in, k8s_service_default);
+    bool local_tuning_default = false;
+    // check ENV VAR LOCAL_TUNING_ENABLED
+    char * tuning_enabled = getenv("LOCAL_TUNING_ENABLED");
+    if (tuning_enabled != NULL) {
+        for (unsigned int i = 0; i < strlen(tuning_enabled); i++) {
+            tuning_enabled[i] = tolower(tuning_enabled[i]);
+        }
+        local_tuning_default = string(tuning_enabled) == "true";
+    }
+    parseAppsecJSONKey<bool>("local-tuning", container_service, archive_in, local_tuning_default);
 
     NewStdoutLogging stdout_log;
     parseAppsecJSONKey<NewStdoutLogging>("stdout", stdout_log, archive_in);
@@ -224,9 +230,9 @@ NewAppsecTriggerLogDestination::getCloud() const
 }
 
 bool
-NewAppsecTriggerLogDestination::isK8SNeeded() const
+NewAppsecTriggerLogDestination::isContainerNeeded() const
 {
-    return k8s_service;
+    return container_service;
 }
 
 bool

components/security_apps/local_policy_mgmt_gen/new_practice.cc

@@ -22,6 +22,7 @@ static const set<string> performance_impacts    = {"low", "medium", "high"};
 static const set<string> severity_levels        = {"low", "medium", "high", "critical"};
 static const set<string> size_unit              = {"bytes", "KB", "MB", "GB"};
 static const set<string> confidences_actions    = {"prevent", "detect", "inactive", "as-top-level", "inherited"};
+static const set<string> valied_enforcement_level = {"fullSchema", "endpointOnly"};
 static const set<string> valid_modes = {
     "prevent",
     "detect",
@@ -32,31 +33,38 @@ static const set<string> valid_modes = {
     "inherited"
 };
 static const set<string> valid_confidences      = {"medium", "high", "critical"};
-static const std::unordered_map<std::string, std::string> key_to_performance_impact_val = {
+static const unordered_map<string, string> key_to_performance_impact_val = {
     { "low", "Low or lower"},
     { "medium", "Medium or lower"},
     { "high", "High or lower"}
 };
-static const std::unordered_map<std::string, std::string> key_to_severity_level_val = {
+static const unordered_map<string, string> key_to_severity_level_val = {
     { "low", "Low or above"},
     { "medium", "Medium or above"},
     { "high", "High or above"},
     { "critical", "Critical"}
 };
-static const std::unordered_map<std::string, std::string> key_to_mode_val = {
+static const unordered_map<string, string> key_to_mode_val = {
     { "prevent-learn", "Prevent"},
     { "detect-learn", "Detect"},
     { "prevent", "Prevent"},
     { "detect", "Detect"},
     { "inactive", "Inactive"}
 };
-static const std::unordered_map<std::string, uint64_t> unit_to_int = {
+static const unordered_map<string, string> anti_bot_key_to_mode_val = {
+    { "prevent-learn", "Prevent"},
+    { "detect-learn", "Detect"},
+    { "prevent", "Prevent"},
+    { "detect", "Detect"},
+    { "inactive", "Disabled"}
+};
+static const unordered_map<string, uint64_t> unit_to_int = {
     { "bytes", 1},
     { "KB", 1024},
     { "MB", 1048576},
     { "GB", 1073741824}
 };
-static const std::string TRANSPARENT_MODE = "Transparent";
+static const string TRANSPARENT_MODE = "Transparent";
 
 bool
 isModeInherited(const string &mode)
@@ -64,11 +72,11 @@ isModeInherited(const string &mode)
     return mode == "as-top-level" || mode == "inherited";
 }
 
-const std::string &
+const string &
 getModeWithDefault(
-    const std::string &mode,
-    const std::string &default_mode,
-    const std::unordered_map<std::string, std::string> &key_to_val)
+    const string &mode,
+    const string &default_mode,
+    const unordered_map<string, string> &key_to_val)
 {
     if (isModeInherited(mode) && (key_to_val.find(default_mode) != key_to_val.end())) {
         dbgError(D_LOCAL_POLICY) << "Setting to top-level mode: " << default_mode;
@@ -81,57 +89,43 @@ getModeWithDefault(
     return key_to_val.at(mode);
 }
 
-void
-NewAppSecWebBotsURI::load(cereal::JSONInputArchive &archive_in)
+const vector<string> &
+NewAppSecPracticeAntiBot::getIjectedUris() const
 {
-    dbgTrace(D_LOCAL_POLICY) << "Loading AppSec Web Bots URI";
-    parseAppsecJSONKey<string>("uri", uri, archive_in);
+    return injected_uris;
+}
+
+const vector<string> &
+NewAppSecPracticeAntiBot::getValidatedUris() const
+{
+    return validated_uris;
 }
 
 const string &
-NewAppSecWebBotsURI::getURI() const
+NewAppSecPracticeAntiBot::getMode(const string &default_mode) const
 {
-    return uri;
-}
-
-std::vector<std::string>
-NewAppSecPracticeAntiBot::getIjectedUris() const
-{
-    vector<string> injected;
-    for (const NewAppSecWebBotsURI &uri : injected_uris) injected.push_back(uri.getURI());
-    return injected;
-}
-
-std::vector<std::string>
-NewAppSecPracticeAntiBot::getValidatedUris() const
-{
-    vector<string> validated;
-    for (const NewAppSecWebBotsURI &uri : validated_uris) validated.push_back(uri.getURI());
-    return validated;
+    return getModeWithDefault(override_mode, default_mode, anti_bot_key_to_mode_val);
 }
 
 void
 NewAppSecPracticeAntiBot::load(cereal::JSONInputArchive &archive_in)
 {
     dbgTrace(D_LOCAL_POLICY) << "Loading AppSec Web Bots";
-    parseAppsecJSONKey<vector<NewAppSecWebBotsURI>>("injectedUris", injected_uris, archive_in);
-    parseAppsecJSONKey<vector<NewAppSecWebBotsURI>>("validatedUris", validated_uris, archive_in);
+    parseAppsecJSONKey<vector<string>>("injectedUris", injected_uris, archive_in);
+    parseAppsecJSONKey<vector<string>>("validatedUris", validated_uris, archive_in);
     parseMandatoryAppsecJSONKey<string>("overrideMode", override_mode, archive_in, "inactive");
     if (valid_modes.count(override_mode) == 0) {
         dbgWarning(D_LOCAL_POLICY) << "AppSec Web Bots override mode invalid: " << override_mode;
+        throw PolicyGenException("AppSec Web Bots override mode invalid: " + override_mode);
     }
 }
 
 void
 NewAppSecPracticeAntiBot::save(cereal::JSONOutputArchive &out_ar) const
 {
-    vector<string> injected;
-    vector<string> validated;
-    for (const NewAppSecWebBotsURI &uri : injected_uris) injected.push_back(uri.getURI());
-    for (const NewAppSecWebBotsURI &uri : validated_uris) validated.push_back(uri.getURI());
     out_ar(
-        cereal::make_nvp("injected", injected),
-        cereal::make_nvp("validated", validated)
+        cereal::make_nvp("injected", injected_uris),
+        cereal::make_nvp("validated", validated_uris)
     );
 }
 
@@ -248,14 +242,14 @@ NewAppSecPracticeWebAttacks::getProtections() const
 }
 
 SnortProtectionsSection::SnortProtectionsSection(
-    const std::string               &_context,
-    const std::string               &_asset_name,
-    const std::string               &_asset_id,
-    const std::string               &_practice_name,
-    const std::string               &_practice_id,
-    const std::string               &_source_identifier,
-    const std::string               &_mode,
-    const std::vector<std::string>  &_files)
+    const string               &_context,
+    const string               &_asset_name,
+    const string               &_asset_id,
+    const string               &_practice_name,
+    const string               &_practice_id,
+    const string               &_source_identifier,
+    const string               &_mode,
+    const vector<string>  &_files)
         :
     context(_context),
     asset_name(_asset_name),
@@ -284,10 +278,10 @@ SnortProtectionsSection::save(cereal::JSONOutputArchive &out_ar) const
 }
 
 DetectionRules::DetectionRules(
-    const std::string                   &_type,
-    const std::string                   &_SSM,
-    const std::string                   &_keywords,
-    const std::vector<std::string>      &_context)
+    const string                   &_type,
+    const string                   &_SSM,
+    const string                   &_keywords,
+    const vector<string>      &_context)
         :
     type(_type),
     SSM(_SSM),
@@ -320,14 +314,14 @@ DetectionRules::save(cereal::JSONOutputArchive &out_ar) const
 
 ProtectionMetadata::ProtectionMetadata(
     bool                                _silent,
-    const std::string                   &_protection_name,
-    const std::string                   &_severity,
-    const std::string                   &_confidence_level,
-    const std::string                   &_performance_impact,
-    const std::string                   &_last_update,
-    const std::string                   &_maintrain_id,
-    const std::vector<std::string>      &_tags,
-    const std::vector<std::string>      &_cve_list)
+    const string                   &_protection_name,
+    const string                   &_severity,
+    const string                   &_confidence_level,
+    const string                   &_performance_impact,
+    const string                   &_last_update,
+    const string                   &_maintrain_id,
+    const vector<string>      &_tags,
+    const vector<string>      &_cve_list)
         :
     silent(_silent),
     protection_name(_protection_name),
@@ -400,9 +394,9 @@ ProtectionsProtectionsSection::save(cereal::JSONOutputArchive &out_ar) const
 }
 
 ProtectionsSection::ProtectionsSection(
-    const std::vector<ProtectionsProtectionsSection>    &_protections,
-    const std::string                                   &_name,
-    const std::string                                   &_modification_time)
+    const vector<ProtectionsProtectionsSection>    &_protections,
+    const string                                   &_name,
+    const string                                   &_modification_time)
         :
     protections(_protections),
     name(_name),
@@ -466,12 +460,16 @@ SnortSectionWrapper::save(cereal::JSONOutputArchive &out_ar) const
 }
 
 void
-NewSnortSignaturesAndOpenSchemaAPI::load(cereal::JSONInputArchive &archive_in)
+NewSnortSignatures::load(cereal::JSONInputArchive &archive_in)
 {
     dbgTrace(D_LOCAL_POLICY) << "Loading AppSec Snort Signatures practice";
     parseMandatoryAppsecJSONKey<string>("overrideMode", override_mode, archive_in, "inactive");
     parseAppsecJSONKey<vector<string>>("configmap", config_map, archive_in);
     parseAppsecJSONKey<vector<string>>("files", files, archive_in);
+    if (valid_modes.count(override_mode) == 0) {
+        dbgWarning(D_LOCAL_POLICY) << "AppSec Snort Signatures override mode invalid: " << override_mode;
+        throw PolicyGenException("AppSec Snort Signatures override mode invalid: " + override_mode);
+    }
     is_temporary = false;
     if (valid_modes.count(override_mode) == 0) {
         dbgWarning(D_LOCAL_POLICY) << "AppSec Snort Signatures override mode invalid: " << override_mode;
@@ -480,42 +478,107 @@ NewSnortSignaturesAndOpenSchemaAPI::load(cereal::JSONInputArchive &archive_in)
 }
 
 void
-NewSnortSignaturesAndOpenSchemaAPI::addFile(const string &file_name)
+NewSnortSignatures::addFile(const string &file_name)
 {
     files.push_back(file_name);
 }
 
 const string &
-NewSnortSignaturesAndOpenSchemaAPI::getOverrideMode(const string &default_mode) const
+NewSnortSignatures::getOverrideMode(const string &default_mode) const
 {
-    const string &res = getModeWithDefault(override_mode, default_mode, key_to_practices_val);
+    const string &res = getModeWithDefault(override_mode, default_mode, key_to_practices_mode_val);
     return res;
 }
 
 const vector<string> &
-NewSnortSignaturesAndOpenSchemaAPI::getFiles() const
+NewSnortSignatures::getFiles() const
 {
     return files;
 }
 
 const vector<string> &
-NewSnortSignaturesAndOpenSchemaAPI::getConfigMap() const
+NewSnortSignatures::getConfigMap() const
 {
     return config_map;
 }
 
 bool
-NewSnortSignaturesAndOpenSchemaAPI::isTemporary() const
+NewSnortSignatures::isTemporary() const
 {
     return is_temporary;
 }
 
 void
-NewSnortSignaturesAndOpenSchemaAPI::setTemporary(bool val)
+NewSnortSignatures::setTemporary(bool val)
 {
     is_temporary = val;
 }
 
+void
+NewOpenApiSchema::load(cereal::JSONInputArchive &archive_in)
+{
+    dbgTrace(D_LOCAL_POLICY) << "Loading AppSec Schema Validation practice";
+    parseMandatoryAppsecJSONKey<string>("overrideMode", override_mode, archive_in, "inactive");
+    parseAppsecJSONKey<vector<string>>("configmap", config_map, archive_in);
+    parseAppsecJSONKey<vector<string>>("files", files, archive_in);
+    parseAppsecJSONKey<string>("enforcementLevel", enforcement_level, archive_in, "fullSchema");
+    if (valied_enforcement_level.count(enforcement_level) == 0) {
+        dbgWarning(D_LOCAL_POLICY) << "AppSec Schema Validation enforcement level invalid: " << enforcement_level;
+        throw PolicyGenException("AppSec Schema Validation enforcement level invalid: " + enforcement_level);
+    }
+    if (valid_modes.count(override_mode) == 0) {
+        dbgWarning(D_LOCAL_POLICY) << "AppSec Schema Validation override mode invalid: " << override_mode;
+        throw PolicyGenException("AppSec Schema Validation override mode invalid: " + override_mode);
+    }
+    for (const string &file : files)
+    {
+        auto i_orchestration_tools = Singleton::Consume<I_OrchestrationTools>::by<NewOpenApiSchema>();
+        auto file_content = i_orchestration_tools->readFile(file);
+        if (!file_content.ok()) {
+            dbgWarning(D_LOCAL_POLICY) << "Couldn't open the schema validation file";
+            continue;
+        }
+        oas.push_back(Singleton::Consume<I_Encryptor>::by<NewOpenApiSchema>()->base64Encode(file_content.unpack()));
+    }
+}
+
+void
+NewOpenApiSchema::addOas(const string &file)
+{
+    oas.push_back(file);
+}
+
+const string &
+NewOpenApiSchema::getOverrideMode(const string &default_mode) const
+{
+    const string &res = getModeWithDefault(override_mode, default_mode, key_to_practices_val2);
+    return res;
+}
+
+const string &
+NewOpenApiSchema::getEnforceLevel() const
+{
+    return enforcement_level;
+}
+
+const vector<string> &
+NewOpenApiSchema::getFiles() const
+{
+    return files;
+}
+
+const vector<string> &
+NewOpenApiSchema::getConfigMap() const
+{
+    return config_map;
+}
+
+const vector<string> &
+NewOpenApiSchema::getOas() const
+{
+    return oas;
+}
+
 void
 IpsProtectionsRulesSection::save(cereal::JSONOutputArchive &out_ar) const
 {
@@ -554,7 +617,7 @@ IpsProtectionsSection::IpsProtectionsSection(
 {
 }
 
-std::string &
+string &
 IpsProtectionsSection::getMode()
 {
     return mode;
@@ -576,6 +639,20 @@ IpsProtectionsSection::save(cereal::JSONOutputArchive &out_ar) const
     );
 }
 
+bool
+IpsProtectionsSection::operator<(const IpsProtectionsSection &other) const
+{
+    // for sorting from the most specific to the least specific rule
+    if (name == default_appsec_name) return false;
+    if (other.name == default_appsec_name) return true;
+    return name.size() > other.name.size();
+}
+
+IPSSection::IPSSection(const vector<IpsProtectionsSection> &_ips) : ips(_ips)
+{
+    sort(ips.begin(), ips.end());
+}
+
 void
 IPSSection::save(cereal::JSONOutputArchive &out_ar) const
 {
@@ -654,7 +731,7 @@ NewIntrusionPrevention::createIpsRules(const string &default_mode) const
     vector<IpsProtectionsRulesSection> ips_rules;
     IpsProtectionsRulesSection high_rule(
         min_cve_Year,
-        getModeWithDefault(high_confidence_event_action, default_mode, key_to_practices_val),
+        getRulesMode(high_confidence_event_action, default_mode),
         string("High"),
         max_performance_impact,
         string(""),
@@ -664,7 +741,7 @@ NewIntrusionPrevention::createIpsRules(const string &default_mode) const
 
     IpsProtectionsRulesSection med_rule(
         min_cve_Year,
-        getModeWithDefault(medium_confidence_event_action, default_mode, key_to_practices_val),
+        getRulesMode(medium_confidence_event_action, default_mode),
         string("Medium"),
         max_performance_impact,
         string(""),
@@ -674,7 +751,7 @@ NewIntrusionPrevention::createIpsRules(const string &default_mode) const
 
     IpsProtectionsRulesSection low_rule(
         min_cve_Year,
-        getModeWithDefault(low_confidence_event_action, default_mode, key_to_practices_val),
+        getRulesMode(low_confidence_event_action, default_mode),
         string("Low"),
         max_performance_impact,
         string(""),
@@ -685,33 +762,45 @@ NewIntrusionPrevention::createIpsRules(const string &default_mode) const
     return ips_rules;
 }
 
-const std::string &
-NewIntrusionPrevention::getMode(const std::string &default_mode) const
+const string &
+NewIntrusionPrevention::getMode(const string &default_mode) const
 {
-    const string &res = getModeWithDefault(override_mode, default_mode, key_to_practices_val);
+    const string &res = getModeWithDefault(override_mode, default_mode, key_to_practices_mode_val);
     return res;
 }
 
+const string &
+NewIntrusionPrevention::getRulesMode(const string &mode, const string &default_mode) const
+{
+    if (isModeInherited(mode)) return default_mode;
+
+    if (key_to_practices_mode_val.find(mode) == key_to_practices_mode_val.end()) {
+        dbgError(D_LOCAL_POLICY) << "Given mode: " << mode << " or top-level: " << default_mode << " is invalid.";
+        return key_to_practices_mode_val.at("inactive");
+    }
+    return key_to_practices_mode_val.at(mode);
+}
+
 FileSecurityProtectionsSection::FileSecurityProtectionsSection(
     uint64_t                _file_size_limit,
     uint64_t                _archive_file_size_limit,
     bool                    _allow_files_without_name,
     bool                    _required_file_size_limit,
     bool                    _required_archive_extraction,
-    const std::string       &_context,
-    const std::string       &_name,
-    const std::string       &_asset_id,
-    const std::string       &_practice_name,
-    const std::string       &_practice_id,
-    const std::string       &_action,
-    const std::string       &_files_without_name_action,
-    const std::string       &_high_confidence_action,
-    const std::string       &_medium_confidence_action,
-    const std::string       &_low_confidence_action,
-    const std::string       &_severity_level,
-    const std::string       &_file_size_limit_action,
-    const std::string       &_multi_level_archive_action,
-    const std::string       &_unopened_archive_action)
+    const string       &_context,
+    const string       &_name,
+    const string       &_asset_id,
+    const string       &_practice_name,
+    const string       &_practice_id,
+    const string       &_action,
+    const string       &_files_without_name_action,
+    const string       &_high_confidence_action,
+    const string       &_medium_confidence_action,
+    const string       &_low_confidence_action,
+    const string       &_severity_level,
+    const string       &_file_size_limit_action,
+    const string       &_multi_level_archive_action,
+    const string       &_unopened_archive_action)
         :
     file_size_limit(_file_size_limit),
     archive_file_size_limit(_archive_file_size_limit),
@@ -837,13 +926,13 @@ NewFileSecurityArchiveInspection::getrequiredArchiveExtraction() const
     return extract_archive_files;
 }
 
-const std::string &
+const string &
 NewFileSecurityArchiveInspection::getMultiLevelArchiveAction() const
 {
     return archived_files_within_archived_files;
 }
 
-const std::string &
+const string &
 NewFileSecurityArchiveInspection::getUnopenedArchiveAction() const
 {
     return archived_files_where_content_extraction_failed;
@@ -892,7 +981,7 @@ NewFileSecurityLargeFileInspection::getFileSizeLimit() const
     return (file_size_limit * unit_to_int.at(file_size_limit_unit));
 }
 
-const std::string &
+const string &
 NewFileSecurityLargeFileInspection::getFileSizeLimitAction() const
 {
     return files_exceeding_size_limit_action;
@@ -1013,7 +1102,7 @@ void
 NewAppSecPracticeSpec::load(cereal::JSONInputArchive &archive_in)
 {
     dbgTrace(D_LOCAL_POLICY) << "Loading AppSec practice spec";
-    parseAppsecJSONKey<NewSnortSignaturesAndOpenSchemaAPI>(
+    parseAppsecJSONKey<NewOpenApiSchema>(
         "schemaValidation",
         openapi_schema_validation,
         archive_in
@@ -1021,11 +1110,15 @@ NewAppSecPracticeSpec::load(cereal::JSONInputArchive &archive_in)
     parseAppsecJSONKey<string>("appsecClassName", appsec_class_name, archive_in);
     parseMandatoryAppsecJSONKey<NewFileSecurity>("fileSecurity", file_security, archive_in);
     parseMandatoryAppsecJSONKey<NewIntrusionPrevention>("intrusionPrevention", intrusion_prevention, archive_in);
-    parseMandatoryAppsecJSONKey<NewSnortSignaturesAndOpenSchemaAPI>("snortSignatures", snort_signatures, archive_in);
+    parseMandatoryAppsecJSONKey<NewSnortSignatures>("snortSignatures", snort_signatures, archive_in);
     parseMandatoryAppsecJSONKey<NewAppSecPracticeWebAttacks>("webAttacks", web_attacks, archive_in);
     parseAppsecJSONKey<NewAppSecPracticeAntiBot>("antiBot", anti_bot, archive_in);
     parseAppsecJSONKey<string>("name", practice_name, archive_in);
     parseAppsecJSONKey<string>("practiceMode", mode, archive_in, "inherited");
+    if (valid_modes.count(mode) == 0) {
+        dbgWarning(D_LOCAL_POLICY) << "AppSec Threat prevention practice mode invalid: " << mode;
+        throw PolicyGenException("AppSec Threat prevention practice mode invalid: " + mode);
+    }
 }
 
 void
@@ -1034,13 +1127,13 @@ NewAppSecPracticeSpec::setName(const string &_name)
     practice_name = _name;
 }
 
-const NewSnortSignaturesAndOpenSchemaAPI &
-NewAppSecPracticeSpec::getOpenSchemaValidation() const
+NewOpenApiSchema &
+NewAppSecPracticeSpec::getOpenSchemaValidation()
 {
     return openapi_schema_validation;
 }
 
-NewSnortSignaturesAndOpenSchemaAPI &
+NewSnortSignatures &
 NewAppSecPracticeSpec::getSnortSignatures()
 {
     return snort_signatures;

components/security_apps/local_policy_mgmt_gen/new_trusted_sources.cc

@@ -69,7 +69,7 @@ Identifier::load(cereal::JSONInputArchive &archive_in)
         dbgWarning(D_LOCAL_POLICY) << "AppSec identifier invalid: " << identifier;
         identifier = "sourceip";
     }
-    parseMandatoryAppsecJSONKey<vector<string>>("value", value, archive_in);
+    parseAppsecJSONKey<vector<string>>("value", value, archive_in);
 }
 
 const string &

components/security_apps/local_policy_mgmt_gen/policy_activation_data.cc

@@ -0,0 +1,103 @@
+// Copyright (C) 2022 Check Point Software Technologies Ltd. All rights reserved.
+
+// Licensed under the Apache License, Version 2.0 (the "License");
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "policy_activation_data.h"
+#include "customized_cereal_map.h"
+
+using namespace std;
+
+USE_DEBUG_FLAG(D_LOCAL_POLICY);
+
+void
+PolicyActivationMetadata::load(cereal::JSONInputArchive &archive_in)
+{
+    dbgTrace(D_LOCAL_POLICY) << "PolicyActivationMetadata load";
+    parseAppsecJSONKey<string>("name", name, archive_in);
+}
+
+void
+EnabledPolicy::load(cereal::JSONInputArchive &archive_in)
+{
+    dbgTrace(D_LOCAL_POLICY) << "Loading policyActivation enabled policy";
+    parseMandatoryAppsecJSONKey<vector<string>>("hosts", hosts, archive_in);
+    parseAppsecJSONKey<string>("name", name, archive_in);
+}
+
+const string &
+EnabledPolicy::getName() const
+{
+    return name;
+}
+
+const vector<string> &
+EnabledPolicy::getHosts() const
+{
+    return hosts;
+}
+
+void
+PolicyActivationSpec::load(cereal::JSONInputArchive &archive_in)
+{
+    dbgTrace(D_LOCAL_POLICY) << "PolicyActivationSpec load";
+    parseAppsecJSONKey<string>("appsecClassName", appsec_class_name, archive_in);
+    parseMandatoryAppsecJSONKey<vector<EnabledPolicy>>("enabledPolicies", policies, archive_in);
+}
+
+const vector<EnabledPolicy> &
+PolicyActivationSpec::getPolicies() const
+{
+    return policies;
+}
+
+void
+SinglePolicyActivationData::load(cereal::JSONInputArchive &archive_in)
+{
+    dbgTrace(D_LOCAL_POLICY) << "Loading single policy activation data";
+    parseAppsecJSONKey<string>("apiVersion", api_version, archive_in);
+    parseAppsecJSONKey<string>("kind", kind, archive_in);
+    parseAppsecJSONKey<PolicyActivationMetadata>("metadata", metadata, archive_in);
+    parseAppsecJSONKey<PolicyActivationSpec>("spec", spec, archive_in);
+}
+
+const PolicyActivationSpec &
+SinglePolicyActivationData::getSpec() const
+{
+    return spec;
+}
+
+bool
+PolicyActivationData::loadJson(const string &json)
+{
+    string modified_json = json;
+    modified_json.pop_back();
+    stringstream in;
+    in.str(modified_json);
+    dbgTrace(D_LOCAL_POLICY) << "Loading policy activations data";
+    try {
+        cereal::JSONInputArchive in_ar(in);
+        in_ar(
+            cereal::make_nvp("apiVersion", api_version),
+            cereal::make_nvp("items", items)
+        );
+    } catch (cereal::Exception &e) {
+        dbgError(D_LOCAL_POLICY) << "Failed to load policy activations data JSON. Error: " << e.what();
+        return false;
+    }
+    return true;
+}
+
+const vector<SinglePolicyActivationData> &
+PolicyActivationData::getItems() const
+{
+    return items;
+}

components/security_apps/local_policy_mgmt_gen/policy_maker_utils.cc

@@ -21,6 +21,15 @@
 using namespace std;
 
 USE_DEBUG_FLAG(D_NGINX_POLICY);
+USE_DEBUG_FLAG(D_LOCAL_POLICY);
+
+static const std::unordered_map<std::string, std::string> key_to_source_identefier_val = {
+    { "sourceip", "Source IP"},
+    { "cookie", "Cookie:"},
+    { "headerkey", "Header:"},
+    { "JWTKey", ""},
+    { "x-forwarded-for", "X-Forwarded-For"}
+};
 
 void
 SecurityAppsWrapper::save(cereal::JSONOutputArchive &out_ar) const
@@ -185,6 +194,33 @@ PolicyMakerUtils::dumpPolicyToFile(
     return policy_str;
 }
 
+template<class R>
+vector<string>
+extractExceptionAnnotationNames(
+    const R &parsed_rule,
+    const R &default_rule,
+    const string &policy_name)
+{
+    vector<string> annotation_names;
+
+    const R &rule = (!parsed_rule.getExceptions().empty() ? parsed_rule : default_rule);
+    for (const string &exception_name : rule.getExceptions()) {
+        if (exception_name.empty()) {
+            continue;
+        }
+
+        const auto policy_exception = policy_name + "/" + exception_name;
+
+        dbgTrace(D_NGINX_POLICY) << "Adding " << policy_exception << " to exception vector";
+
+        annotation_names.push_back(policy_exception);
+    }
+
+    dbgTrace(D_NGINX_POLICY) << "Number of exceptions related to rule: " << annotation_names.size();
+
+    return annotation_names;
+}
+
 template<class R>
 map<AnnotationTypes, string>
 extractAnnotationsNames(
@@ -217,18 +253,6 @@ extractAnnotationsNames(
         rule_annotation[AnnotationTypes::TRIGGER] = policy_name + "/" + trigger_annotation_name;
     }
 
-    string exception_annotation_name;
-    // TBD: support multiple exceptions
-    if (!parsed_rule.getExceptions().empty() && !parsed_rule.getExceptions()[0].empty()) {
-        exception_annotation_name = parsed_rule.getExceptions()[0];
-    } else if (!default_rule.getExceptions().empty() && !default_rule.getExceptions()[0].empty()) {
-        exception_annotation_name = default_rule.getExceptions()[0];
-    }
-
-    if (!exception_annotation_name.empty()) {
-        rule_annotation[AnnotationTypes::EXCEPTION] = policy_name + "/" + exception_annotation_name;
-    }
-
     string web_user_res_annotation_name =
         parsed_rule.getCustomResponse().empty() ?
         default_rule.getCustomResponse() :
@@ -444,6 +468,7 @@ template<class T, class R>
 R
 getAppsecExceptionSpec(const string &exception_annotation_name, const T &policy)
 {
+    dbgFlow(D_NGINX_POLICY) << "anotation name: " << exception_annotation_name;
     auto exceptions_vec = policy.getAppsecExceptions();
     auto exception_it = extractElement(exceptions_vec.begin(), exceptions_vec.end(), exception_annotation_name);
 
@@ -538,7 +563,7 @@ extractLogTriggerData(const string &trigger_annotation_name, const T &trigger_sp
     bool webHeaders = trigger_spec.getAppsecTriggerExtendedLogging().isHttpHeaders();
     bool webBody = trigger_spec.getAppsecTriggerExtendedLogging().isRequestBody();
     bool logToCloud = trigger_spec.getAppsecTriggerLogDestination().getCloud();
-    bool logToK8sService = trigger_spec.getAppsecTriggerLogDestination().isK8SNeeded();
+    bool logTolocalTuning = trigger_spec.getAppsecTriggerLogDestination().isContainerNeeded();
     bool logToAgent = trigger_spec.getAppsecTriggerLogDestination().isAgentLocal();
     bool beautify_logs = trigger_spec.getAppsecTriggerLogDestination().shouldBeautifyLogs();
     bool logToCef = trigger_spec.getAppsecTriggerLogDestination().isCefNeeded();
@@ -565,7 +590,7 @@ extractLogTriggerData(const string &trigger_annotation_name, const T &trigger_sp
         logToAgent,
         logToCef,
         logToCloud,
-        logToK8sService,
+        logTolocalTuning,
         logToSyslog,
         responseBody,
         tpDetect,
@@ -776,6 +801,7 @@ createExceptionSection(
     const string &exception_annotation_name,
     const T &policy)
 {
+    dbgFlow(D_NGINX_POLICY) << "exception annotation name" << exception_annotation_name;
     AppsecException exception_spec =
         getAppsecExceptionSpec<T, AppsecException>(exception_annotation_name, policy);
     vector<InnerException> res;
@@ -784,6 +810,7 @@ createExceptionSection(
         ExceptionBehavior exception_behavior(exception.getAction());
         res.push_back(InnerException(exception_behavior, exception_match));
     }
+
     return res;
 }
 
@@ -896,13 +923,15 @@ createMultiRulesSections(
     const string &web_user_res_vec_id,
     const string &web_user_res_vec_type,
     const string &asset_name,
-    const string &exception_name,
-    const vector<InnerException> &exceptions)
+    const std::map<std::string, std::vector<InnerException>> &exceptions)
 {
     PracticeSection practice = PracticeSection(practice_id, practice_type, practice_name);
     vector<ParametersSection> exceptions_result;
     for (auto exception : exceptions) {
-        exceptions_result.push_back(ParametersSection(exception.getBehaviorId(), exception_name));
+        const auto &exception_name = exception.first;
+        for (const auto &inner_exception : exception.second) {
+            exceptions_result.push_back(ParametersSection(inner_exception.getBehaviorId(), exception_name));
+        }
     }
 
     vector<RulesTriggerSection> triggers;
@@ -1016,7 +1045,7 @@ PolicyMakerUtils::createIpsSections(
         practice_name,
         practice_id,
         source_identifier,
-        override_mode,
+        "Inactive",
         apssec_practice.getIntrusionPrevention().createIpsRules(override_mode)
     );
 
@@ -1026,8 +1055,7 @@ PolicyMakerUtils::createIpsSections(
 void
 PolicyMakerUtils::createSnortProtecionsSection(const string &file_name, bool is_temporary)
 {
-    auto path = getFilesystemPathConfig() + "/conf/snort/" + file_name;
-    string in_file = is_temporary ? path + ".rule" : path;
+    auto path = is_temporary ? getFilesystemPathConfig() + "/conf/snort/" + file_name + ".rule" : file_name;
 
     if (snort_protections.find(path) != snort_protections.end()) {
         dbgTrace(D_LOCAL_POLICY) << "Snort protections section for file " << file_name << " already exists";
@@ -1038,7 +1066,9 @@ PolicyMakerUtils::createSnortProtecionsSection(const string &file_name, bool is_
         << (is_temporary ? " temporary" : "") << " file " << path;
 
     auto snort_script_path = getFilesystemPathConfig() + "/scripts/snort_to_ips_local.py";
-    auto cmd = "python3 " + snort_script_path + " " + in_file + " " + path + ".out " + path + ".err";
+    auto tmp_out = "/tmp/" + file_name + ".out";
+    auto tmp_err = "/tmp/" + file_name + ".err";
+    auto cmd = "python3 " + snort_script_path + " " + path + " " + tmp_out + " " + tmp_err;
 
     auto res = Singleton::Consume<I_ShellCmd>::by<LocalPolicyMgmtGenerator>()->getExecOutput(cmd);
 
@@ -1047,16 +1077,16 @@ PolicyMakerUtils::createSnortProtecionsSection(const string &file_name, bool is_
         return;
     }
 
-    Maybe<ProtectionsSectionWrapper> maybe_protections = openFileAsJson<ProtectionsSectionWrapper>(path + ".out");
+    Maybe<ProtectionsSectionWrapper> maybe_protections = openFileAsJson<ProtectionsSectionWrapper>(tmp_out);
     if (!maybe_protections.ok()){
         dbgWarning(D_LOCAL_POLICY) << maybe_protections.getErr();
         return;
     }
 
     auto i_orchestration_tools = Singleton::Consume<I_OrchestrationTools>::by<LocalPolicyMgmtGenerator>();
-    if (is_temporary) i_orchestration_tools->removeFile(in_file);
-    i_orchestration_tools->removeFile(path + ".out");
-    i_orchestration_tools->removeFile(path + ".err");
+    if (is_temporary) i_orchestration_tools->removeFile(path);
+    i_orchestration_tools->removeFile(tmp_out);
+    i_orchestration_tools->removeFile(tmp_err);
 
     snort_protections[path] = ProtectionsSection(
         maybe_protections.unpack().getProtections(),
@@ -1186,9 +1216,11 @@ void
 PolicyMakerUtils::createWebAppSection(
     const V1beta2AppsecLinuxPolicy &policy,
     const RulesConfigRulebase& rule_config,
-    const string &practice_id, const string &full_url,
+    const string &practice_id,
+    const string &full_url,
     const string &default_mode,
-    map<AnnotationTypes, string> &rule_annotations)
+    map<AnnotationTypes, string> &rule_annotations,
+    vector<InnerException> rule_inner_exceptions)
 {
     auto apssec_practice =
         getAppsecPracticeSpec<V1beta2AppsecLinuxPolicy, NewAppSecPracticeSpec>(
@@ -1203,6 +1235,7 @@ PolicyMakerUtils::createWebAppSection(
         apssec_practice.getWebAttacks().getMaxObjectDepth(),
         apssec_practice.getWebAttacks().getMaxUrlSizeBytes()
     );
+
     WebAppSection web_app = WebAppSection(
         full_url == "Any" ? default_appsec_url : full_url,
         rule_config.getAssetId(),
@@ -1214,11 +1247,16 @@ PolicyMakerUtils::createWebAppSection(
         rule_config.getContext(),
         apssec_practice.getWebAttacks().getMinimumConfidence(practice_mode),
         apssec_practice.getWebAttacks().getMode(practice_mode),
+        apssec_practice.getAntiBot().getMode(practice_mode),
+        apssec_practice.getOpenSchemaValidation().getOverrideMode(practice_mode),
+        apssec_practice.getOpenSchemaValidation().getEnforceLevel(),
+        apssec_practice.getOpenSchemaValidation().getOas(),
         practice_advance_config,
         apssec_practice.getAntiBot(),
         log_triggers[rule_annotations[AnnotationTypes::TRIGGER]],
         trusted_sources[rule_annotations[AnnotationTypes::TRUSTED_SOURCES]],
-        apssec_practice.getWebAttacks().getProtections()
+        apssec_practice.getWebAttacks().getProtections(),
+        rule_inner_exceptions
     );
     web_apps[rule_config.getAssetName()] = web_app;
 }
@@ -1267,7 +1305,7 @@ PolicyMakerUtils::createThreatPreventionPracticeSections(
     );
     rules_config[rule_config.getAssetName()] = rule_config;
 
-    string current_identifier;
+    string current_identifier, current_identifier_value;
     if (!rule_annotations[AnnotationTypes::SOURCE_IDENTIFIERS].empty()) {
         UsersIdentifiersRulebase user_identifiers = createUserIdentifiers<V1beta2AppsecLinuxPolicy>(
             rule_annotations[AnnotationTypes::SOURCE_IDENTIFIERS],
@@ -1276,6 +1314,15 @@ PolicyMakerUtils::createThreatPreventionPracticeSections(
         );
         users_identifiers[rule_annotations[AnnotationTypes::SOURCE_IDENTIFIERS]] = user_identifiers;
         current_identifier = user_identifiers.getIdentifier();
+        current_identifier_value = user_identifiers.getIdentifierValue();
+    }
+
+    string ips_identifier, ips_identifier_value;
+    if(key_to_source_identefier_val.find(current_identifier) != key_to_source_identefier_val.end()) {
+        ips_identifier = key_to_source_identefier_val.at(current_identifier);
+    }
+    if (current_identifier == "cookie" || current_identifier == "headerkey") {
+        ips_identifier_value = current_identifier_value;
     }
 
     createIpsSections(
@@ -1283,7 +1330,7 @@ PolicyMakerUtils::createThreatPreventionPracticeSections(
         rule_config.getAssetName(),
         practice_id,
         rule_annotations[AnnotationTypes::PRACTICE],
-        current_identifier,
+        ips_identifier + ips_identifier_value,
         rule_config.getContext(),
         policy,
         rule_annotations,
@@ -1320,7 +1367,8 @@ PolicyMakerUtils::createThreatPreventionPracticeSections(
             practice_id,
             asset_name,
             default_mode,
-            rule_annotations);
+            rule_annotations,
+            inner_exceptions[rule_annotations[AnnotationTypes::EXCEPTION]]);
     }
 
 }
@@ -1344,6 +1392,7 @@ PolicyMakerUtils::combineElementsToPolicy(const string &policy_version)
             convertMapToVector(log_triggers), convertMapToVector(web_user_res_triggers)
         )
     );
+
     ExceptionsWrapper exceptions_section({
         ExceptionsRulebase(convertExceptionsMapToVector(inner_exceptions))
     });
@@ -1381,6 +1430,7 @@ PolicyMakerUtils::createPolicyElementsByRule(
     const string &policy_name)
 {
     map<AnnotationTypes, string> rule_annotations = extractAnnotationsNames(rule, default_rule, policy_name);
+
     if (
         !rule_annotations[AnnotationTypes::TRIGGER].empty() &&
         !log_triggers.count(rule_annotations[AnnotationTypes::TRIGGER])
@@ -1403,15 +1453,27 @@ PolicyMakerUtils::createPolicyElementsByRule(
             );
     }
 
-    if (
-        !rule_annotations[AnnotationTypes::EXCEPTION].empty() &&
-        !inner_exceptions.count(rule_annotations[AnnotationTypes::EXCEPTION])
-    ) {
-        inner_exceptions[rule_annotations[AnnotationTypes::EXCEPTION]] =
-            createExceptionSection<T>(
-                rule_annotations[AnnotationTypes::EXCEPTION],
-                policy
-            );
+    const auto exceptions_annotations = extractExceptionAnnotationNames(rule, default_rule, policy_name);
+    std::map<std::string, std::vector<InnerException>> rule_inner_exceptions;
+    if (!exceptions_annotations.empty()) {
+        for (const auto &exception_name :exceptions_annotations) {
+            dbgWarning(D_LOCAL_POLICY) << "exceptions name: " << exception_name;
+
+            if (rule_inner_exceptions.count(exception_name)) {
+                dbgWarning(D_LOCAL_POLICY) << "exception name already exists for that rule: " << exception_name;
+                continue;
+            }
+
+            if (inner_exceptions.count(exception_name)) {
+                dbgWarning(D_LOCAL_POLICY) << "exception name already exists in inner exceptions: " << exception_name;
+                rule_inner_exceptions[exception_name] = inner_exceptions[exception_name];
+                continue;
+            }
+
+            auto exception_section = createExceptionSection<T>(exception_name, policy);
+            rule_inner_exceptions[exception_name] = exception_section;
+            inner_exceptions[exception_name] = exception_section;
+        }
     }
 
     if (
@@ -1470,8 +1532,7 @@ PolicyMakerUtils::createPolicyElementsByRule(
             web_user_res_triggers[rule_annotations[AnnotationTypes::WEB_USER_RES]].getTriggerId(),
             "WebUserResponse",
             full_url,
-            rule_annotations[AnnotationTypes::EXCEPTION],
-            inner_exceptions[rule_annotations[AnnotationTypes::EXCEPTION]]
+            rule_inner_exceptions
         );
         rules_config[rule_config.getAssetName()] = rule_config;
 
@@ -1498,7 +1559,7 @@ PolicyMakerUtils::createPolicyElementsByRule(
                 log_triggers[rule_annotations[AnnotationTypes::TRIGGER]],
                 rule.getMode(),
                 trusted_sources[rule_annotations[AnnotationTypes::TRUSTED_SOURCES]],
-                inner_exceptions[rule_annotations[AnnotationTypes::EXCEPTION]]
+                rule_inner_exceptions
             );
             web_apps[rule_config.getAssetName()] = web_app;
         }
@@ -1636,7 +1697,9 @@ PolicyMakerUtils::createAgentPolicyFromAppsecPolicy(const string &policy_name, c
     createPolicyElements<T, R>(specific_rules, default_rule, appsec_policy, policy_name);
 
     // add default rule to policy
-    createPolicyElementsByRule<T, R>(default_rule, default_rule, appsec_policy, policy_name);
+    if (Singleton::Consume<I_EnvDetails>::by<PolicyMakerUtils>()->getEnvType() != EnvType::K8S) {
+        createPolicyElementsByRule<T, R>(default_rule, default_rule, appsec_policy, policy_name);
+    }
 }
 
 // LCOV_EXCL_START Reason: no test exist
@@ -1659,11 +1722,13 @@ PolicyMakerUtils::createAgentPolicyFromAppsecPolicy<V1beta2AppsecLinuxPolicy, Ne
     );
 
     // add default rule to policy
-    createPolicyElementsByRule<V1beta2AppsecLinuxPolicy, NewParsedRule>(
-        default_rule,
-        default_rule,
-        appsec_policy,
-        policy_name);
+    if (Singleton::Consume<I_EnvDetails>::by<PolicyMakerUtils>()->getEnvType() != EnvType::K8S) {
+        createPolicyElementsByRule<V1beta2AppsecLinuxPolicy, NewParsedRule>(
+            default_rule,
+            default_rule,
+            appsec_policy,
+            policy_name);
+    }
 }
 // LCOV_EXCL_STOP