prometheus support locally managed

* fix image entrypoint sigterm

* fix image entrypoint sigterm

---------

Co-authored-by: avigailo <avigailo@checkpoint.com>

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,36 @@
+---
+name: "Bug Report"
+about: "Report a bug with open-appsec"
+labels: [bug]
+---
+
+**Checklist**
+- Have you checked the open-appsec troubleshooting guides - https://docs.openappsec.io/troubleshooting/troubleshooting
+  - Yes / No
+- Have you checked the existing issues and discussions in github for the same issue 
+  - Yes / No
+- Have you checked the knwon limitations same issue - https://docs.openappsec.io/release-notes#limitations
+  - Yes / No
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Run '...'
+3. See error '...'
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots or Logs**
+If applicable, add screenshots or logs to help explain the issue.
+
+**Environment (please complete the following information):**
+- open-appsec version: 
+- Deployment type (Docker, Kubernetes, etc.): 
+- OS:
+
+**Additional context**
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,8 @@
+blank_issues_enabled: false
+contact_links:
+  - name: "Documentation & Troubleshooting"
+    url: "https://docs.openappsec.io/"
+    about: "Check the documentation before submitting an issue."
+  - name: "Feature Requests & Discussions"
+    url: "https://github.com/openappsec/openappsec/discussions"
+    about: "Please open a discussion for feature requests."

.github/ISSUE_TEMPLATE/nginx_version_support.md

@@ -0,0 +1,17 @@
+---
+name: "Nginx Version Support Request"
+about: "Request for a specific Nginx version to be supported"
+---
+
+**Nginx & OS Version:**
+Which Nginx and OS version are you using? 
+
+**Output of nginx -V**
+Share the output of nginx -v
+
+**Expected Behavior:**
+What do you expect to happen with this version?
+
+**Checklist**
+- Have you considered a docker based deployment - find more information here https://docs.openappsec.io/getting-started/start-with-docker?
+  - Yes / No

README.md

@@ -6,7 +6,7 @@
 [![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/6629/badge)](https://bestpractices.coreinfrastructure.org/projects/6629)
 
 # About
-[open-appsec](https://www.openappsec.io) (openappsec.io) builds on machine learning to provide preemptive web app & API threat protection against OWASP-Top-10 and zero-day attacks. It can be deployed as an add-on to Kubernetes Ingress, NGINX, Envoy (soon), and API Gateways.
+[open-appsec](https://www.openappsec.io) (openappsec.io) builds on machine learning to provide preemptive web app & API threat protection against OWASP-Top-10 and zero-day attacks. It can be deployed as an add-on to Linux, Docker or K8s deployments, on NGINX, Kong, APISIX, or Envoy.
 
 The open-appsec engine learns how users normally interact with your web application. It then uses this information to automatically detect requests that fall outside of normal operations, and conducts further analysis to decide whether the request is malicious or not.
 
@@ -39,13 +39,13 @@ open-appsec can be managed using multiple methods:
 * [Using SaaS Web Management](https://docs.openappsec.io/getting-started/using-the-web-ui-saas)
 
 open-appsec Web UI:
-![image](https://github.com/openappsec/openappsec/assets/114033741/22d99379-df52-45c8-984f-1b820635f3b9)
+<img width="1854" height="775" alt="image" src="https://github.com/user-attachments/assets/4c6f7b0a-14f3-4f02-9ab0-ddadc9979b8d" />
+
 
 
 ## Deployment Playgrounds (Virtual labs)
 You can experiment with open-appsec using [Playgrounds](https://www.openappsec.io/playground)
-
-![image](https://github.com/openappsec/openappsec/assets/114033741/14d35d69-4577-48fc-ae87-ea344888e94d)
+<img width="781" height="878" alt="image" src="https://github.com/user-attachments/assets/0ddee216-5cdf-4288-8c41-cc28cfbf3297" />
 
 # Resources
 * [Project Website](https://openappsec.io)
@@ -54,21 +54,15 @@ You can experiment with open-appsec using [Playgrounds](https://www.openappsec.i
 
 # Installation
 
-For Kubernetes (NGINX Ingress) using the installer:
+For Kubernetes (NGINX /Kong / APISIX / Istio) using Helm: follow [documentation](https://docs.openappsec.io/getting-started/start-with-kubernetes)
 
-```bash
-$ wget https://downloads.openappsec.io/open-appsec-k8s-install && chmod +x open-appsec-k8s-install
-$ ./open-appsec-k8s-install
-```
-
-For Kubernetes (NGINX or Kong) using Helm: follow [documentation](https://docs.openappsec.io/getting-started/start-with-kubernetes/install-using-helm-ingress-nginx-and-kong) – use this method if you’ve built your own containers. 
-
-For Linux (NGINX or Kong) using the installer (list of supported/pre-compiled NGINX attachments is available [here](https://downloads.openappsec.io/packages/supported-nginx.txt)):
+For Linux (NGINX / Kong / APISIX) using the installer (list of supported/pre-compiled NGINX attachments is available [here](https://downloads.openappsec.io/packages/supported-nginx.txt)):
 
 ```bash
 $ wget https://downloads.openappsec.io/open-appsec-install && chmod +x open-appsec-install
 $ ./open-appsec-install --auto
 ```
+For kong Lua Based plug in follow [documentation](https://docs.openappsec.io/getting-started/start-with-linux)
 
 For Linux, if you’ve built your own package use the following commands:
 
@@ -177,7 +171,7 @@ open-appsec code was audited by an independent third party in September-October
 See the [full report](https://github.com/openappsec/openappsec/blob/main/LEXFO-CHP20221014-Report-Code_audit-OPEN-APPSEC-v1.2.pdf).
 
 ### Reporting security vulnerabilities
-If you've found a vulnerability or a potential vulnerability in open-appsec please let us know at securityalert@openappsec.io. We'll send a confirmation email to acknowledge your report within 24 hours, and we'll send an additional email when we've identified the issue positively or negatively.
+If you've found a vulnerability or a potential vulnerability in open-appsec please let us know at security-alert@openappsec.io. We'll send a confirmation email to acknowledge your report within 24 hours, and we'll send an additional email when we've identified the issue positively or negatively.
 
 
 # License

build_system/docker/CMakeLists.txt

@@ -1,4 +1,4 @@
-install(FILES Dockerfile entry.sh install-cp-agent-intelligence-service.sh install-cp-crowdsec-aux.sh DESTINATION .)
+install(FILES Dockerfile entry.sh install-cp-agent-intelligence-service.sh install-cp-crowdsec-aux.sh self_managed_openappsec_manifest.json DESTINATION .)
 
 add_custom_command(
     OUTPUT ${CMAKE_INSTALL_PREFIX}/agent-docker.img

build_system/docker/Dockerfile

@@ -1,5 +1,7 @@
 FROM alpine
 
+ENV OPENAPPSEC_NANO_AGENT=TRUE
+
 RUN apk add --no-cache -u busybox
 RUN apk add --no-cache -u zlib
 RUN apk add --no-cache bash
@@ -11,8 +13,12 @@ RUN apk add --no-cache libunwind
 RUN apk add --no-cache gdb
 RUN apk add --no-cache libxml2
 RUN apk add --no-cache pcre2
+RUN apk add --no-cache ca-certificates
 RUN apk add --update coreutils
 
+
+COPY self_managed_openappsec_manifest.json /tmp/self_managed_openappsec_manifest.json
+
 COPY install*.sh /nano-service-installers/
 COPY entry.sh /entry.sh
 

build_system/docker/entry.sh

@@ -6,6 +6,8 @@ HTTP_TRANSACTION_HANDLER_SERVICE="install-cp-nano-service-http-transaction-handl
 ATTACHMENT_REGISTRATION_SERVICE="install-cp-nano-attachment-registration-manager.sh"
 ORCHESTRATION_INSTALLATION_SCRIPT="install-cp-nano-agent.sh"
 CACHE_INSTALLATION_SCRIPT="install-cp-nano-agent-cache.sh"
+PROMETHEUS_INSTALLATION_SCRIPT="install-cp-nano-service-prometheus.sh"
+NGINX_CENTRAL_MANAGER_INSTALLATION_SCRIPT="install-cp-nano-central-nginx-manager.sh"
 
 var_fog_address=
 var_proxy=
@@ -13,6 +15,21 @@ var_mode=
 var_token=
 var_ignore=
 init=
+active_watchdog_pid=
+
+cleanup() {
+    local signal="$1"
+    echo "[$(date '+%Y-%m-%d %H:%M:%S')] Signal ${signal} was received, exiting gracefully..." >&2
+    if [ -n "${active_watchdog_pid}" ] && ps -p ${active_watchdog_pid} > /dev/null 2>&1; then
+        kill -TERM ${active_watchdog_pid} 2>/dev/null || true
+        wait ${active_watchdog_pid} 2>/dev/null || true
+    fi
+    echo "Cleanup completed. Exiting now." >&2
+    exit 0
+}
+
+trap 'cleanup SIGTERM' SIGTERM
+trap 'cleanup SIGINT' SIGINT
 
 if [ ! -f /nano-service-installers/$ORCHESTRATION_INSTALLATION_SCRIPT ]; then
     echo "Error: agent installation package doesn't exist."
@@ -81,6 +98,14 @@ fi
 /nano-service-installers/$CACHE_INSTALLATION_SCRIPT --install
 /nano-service-installers/$HTTP_TRANSACTION_HANDLER_SERVICE --install
 
+if [ "$PROMETHEUS" == "true" ]; then
+    /nano-service-installers/$PROMETHEUS_INSTALLATION_SCRIPT --install
+fi
+
+if [ "$CENTRAL_NGINX_MANAGER" == "true" ]; then
+    /nano-service-installers/$NGINX_CENTRAL_MANAGER_INSTALLATION_SCRIPT --install
+fi
+
 if [ "$CROWDSEC_ENABLED" == "true" ]; then
     /nano-service-installers/$INTELLIGENCE_INSTALLATION_SCRIPT --install
     /nano-service-installers/$CROWDSEC_INSTALLATION_SCRIPT --install
@@ -93,25 +118,16 @@ if [ -f "$FILE" ]; then
 fi
 
 touch /etc/cp/watchdog/wd.startup
+/etc/cp/watchdog/cp-nano-watchdog >/dev/null 2>&1 &
+active_watchdog_pid=$!
 while true; do
-    if [ -z "$init" ]; then
-        init=true
-        /etc/cp/watchdog/cp-nano-watchdog >/dev/null 2>&1 &
-        sleep 5
-        active_watchdog_pid=$(pgrep -f -x -o "/bin/(bash|sh) /etc/cp/watchdog/cp-nano-watchdog")
-    fi
-
-    current_watchdog_pid=$(pgrep -f -x -o "/bin/(bash|sh) /etc/cp/watchdog/cp-nano-watchdog")
-    if [ ! -f /tmp/restart_watchdog ] && [ "$current_watchdog_pid" != "$active_watchdog_pid" ]; then
-        echo "Error: Watchdog exited abnormally"
-        exit 1
-    elif [ -f /tmp/restart_watchdog ]; then
+    if [ -f /tmp/restart_watchdog ]; then
         rm -f /tmp/restart_watchdog
-        kill -9 "$(pgrep -f -x -o "/bin/(bash|sh) /etc/cp/watchdog/cp-nano-watchdog")"
-        /etc/cp/watchdog/cp-nano-watchdog >/dev/null 2>&1 &
-        sleep 5
-        active_watchdog_pid=$(pgrep -f -x -o "/bin/(bash|sh) /etc/cp/watchdog/cp-nano-watchdog")
+        kill -9 ${active_watchdog_pid}
+    fi
+    if [ ! "$(ps -f | grep cp-nano-watchdog | grep ${active_watchdog_pid})" ]; then
+        /etc/cp/watchdog/cp-nano-watchdog >/dev/null 2>&1 &
+        active_watchdog_pid=$!
     fi
-
     sleep 5
 done

components/attachment-intakers/nginx_attachment/nginx_attachment.cc

@@ -36,6 +36,7 @@
 
 #include "nginx_attachment_config.h"
 #include "nginx_attachment_opaque.h"
+#include "generic_rulebase/evaluators/trigger_eval.h"
 #include "nginx_parser.h"
 #include "i_instance_awareness.h"
 #include "common.h"
@@ -130,6 +131,7 @@ class NginxAttachment::Impl
     Singleton::Provide<I_StaticResourcesHandler>::From<NginxAttachment>
 {
     static constexpr auto INSPECT = ngx_http_cp_verdict_e::TRAFFIC_VERDICT_INSPECT;
+    static constexpr auto LIMIT_RESPONSE_HEADERS = ngx_http_cp_verdict_e::LIMIT_RESPONSE_HEADERS;
     static constexpr auto ACCEPT = ngx_http_cp_verdict_e::TRAFFIC_VERDICT_ACCEPT;
     static constexpr auto DROP = ngx_http_cp_verdict_e::TRAFFIC_VERDICT_DROP;
     static constexpr auto INJECT = ngx_http_cp_verdict_e::TRAFFIC_VERDICT_INJECT;
@@ -1146,10 +1148,18 @@ private:
     handleCustomWebResponse(
         SharedMemoryIPC *ipc,
         vector<const char *> &verdict_data,
-        vector<uint16_t> &verdict_data_sizes)
+        vector<uint16_t> &verdict_data_sizes,
+        string web_user_response_id)
     {
         ngx_http_cp_web_response_data_t web_response_data;
-
+        ScopedContext ctx;
+        if (web_user_response_id != "") {
+            dbgTrace(D_NGINX_ATTACHMENT)
+            << "web user response ID registered in contex: "
+            << web_user_response_id;
+            set<string> triggers_set{web_user_response_id};
+            ctx.registerValue<set<GenericConfigId>>(TriggerMatcher::ctx_key, triggers_set);
+        }
         WebTriggerConf web_trigger_conf = getConfigurationWithDefault<WebTriggerConf>(
             WebTriggerConf::default_trigger_conf,
             "rulebase",
@@ -1271,7 +1281,7 @@ private:
         if (verdict.getVerdict() == DROP) {
             nginx_attachment_event.addTrafficVerdictCounter(nginxAttachmentEvent::trafficVerdict::DROP);
             verdict_to_send.modification_count = 1;
-            return handleCustomWebResponse(ipc, verdict_fragments, fragments_sizes);
+            return handleCustomWebResponse(ipc, verdict_fragments, fragments_sizes, verdict.getWebUserResponseID());
         }
 
         if (verdict.getVerdict() == ACCEPT) {
@@ -1497,11 +1507,17 @@ private:
         opaque.activateContext();
 
         FilterVerdict verdict = handleChunkedData(*chunked_data_type, inspection_data, opaque);
-
         bool is_header =
             *chunked_data_type == ChunkType::REQUEST_HEADER  ||
             *chunked_data_type == ChunkType::RESPONSE_HEADER ||
             *chunked_data_type == ChunkType::CONTENT_LENGTH;
+
+        if (verdict.getVerdict() == LIMIT_RESPONSE_HEADERS) {
+            handleVerdictResponse(verdict, attachment_ipc, transaction_data->session_id, is_header);
+            popData(attachment_ipc);
+            verdict = FilterVerdict(INSPECT);
+        }
+
         handleVerdictResponse(verdict, attachment_ipc, transaction_data->session_id, is_header);
 
         bool is_final_verdict = verdict.getVerdict() == ACCEPT ||
@@ -1614,6 +1630,8 @@ private:
                 return "INJECT";
             case INSPECT:
                 return "INSPECT";
+            case LIMIT_RESPONSE_HEADERS:
+                return "LIMIT_RESPONSE_HEADERS";
             case IRRELEVANT:
                 return "IRRELEVANT";
             case RECONF:

components/attachment-intakers/nginx_attachment/nginx_attachment_opaque.cc

@@ -70,6 +70,12 @@ NginxAttachmentOpaque::NginxAttachmentOpaque(HttpTransactionData _transaction_da
         ctx.registerValue(HttpTransactionData::uri_query_decoded, decoded_url.substr(question_mark_location + 1));
     }
     ctx.registerValue(HttpTransactionData::uri_path_decoded, decoded_url);
+
+    // Register waf_tag from transaction data if available
+    const std::string& waf_tag = transaction_data.getWafTag();
+    if (!waf_tag.empty()) {
+        ctx.registerValue(HttpTransactionData::waf_tag_ctx, waf_tag);
+    }
 }
 
 NginxAttachmentOpaque::~NginxAttachmentOpaque()

components/attachment-intakers/nginx_attachment/nginx_parser.cc

@@ -231,16 +231,17 @@ NginxParser::parseRequestHeaders(const Buffer &data, const unordered_set<string>
             static_cast<string>(header.getKey()) + ": " + static_cast<string>(header.getValue()) + "\r\n"
         );
 
-        if (NginxParser::tenant_header_key == header.getKey()) {
+        const auto &header_key = header.getKey();
+        if (NginxParser::tenant_header_key == header_key) {
             dbgDebug(D_NGINX_ATTACHMENT_PARSER)
                 << "Identified active tenant header. Key: "
-                << dumpHex(header.getKey())
+                << dumpHex(header_key)
                 << ", Value: "
                 << dumpHex(header.getValue());
 
             auto active_tenant_and_profile = getActivetenantAndProfile(header.getValue());
             opaque.setSessionTenantAndProfile(active_tenant_and_profile[0], active_tenant_and_profile[1]);
-        } else if (proxy_ip_header_key == header.getKey()) {
+        } else if (proxy_ip_header_key == header_key) {
             source_identifiers.setXFFValuesToOpaqueCtx(header, UsersAllIdentifiersConfig::ExtractType::PROXYIP);
         }
     }
@@ -378,12 +379,15 @@ NginxParser::parseResponseBody(const Buffer &raw_response_body, CompressionStrea
 Maybe<CompressionType>
 NginxParser::parseContentEncoding(const vector<HttpHeader> &headers)
 {
-    static const Buffer content_encoding_header_key("Content-Encoding");
+    dbgFlow(D_NGINX_ATTACHMENT_PARSER) << "Parsing \"Content-Encoding\" header";
+    static const Buffer content_encoding_header_key("content-encoding");
 
     auto it = find_if(
         headers.begin(),
         headers.end(),
-        [&] (const HttpHeader &http_header) { return http_header.getKey() == content_encoding_header_key; }
+        [&] (const HttpHeader &http_header) {
+            return http_header.getKey().isEqualLowerCase(content_encoding_header_key);
+        }
     );
     if (it == headers.end()) {
         dbgTrace(D_NGINX_ATTACHMENT_PARSER)

components/attachment-intakers/nginx_attachment/user_identifiers_config.cc

@@ -366,6 +366,24 @@ UsersAllIdentifiersConfig::setCustomHeaderToOpaqueCtx(const HttpHeader &header)
     return;
 }
 
+void
+UsersAllIdentifiersConfig::setWafTagValuesToOpaqueCtx(const HttpHeader &header) const
+{
+    auto i_transaction_table = Singleton::Consume<I_TableSpecific<SessionID>>::by<NginxAttachment>();
+    if (!i_transaction_table || !i_transaction_table->hasState<NginxAttachmentOpaque>()) {
+        dbgDebug(D_NGINX_ATTACHMENT_PARSER) << "Can't get the transaction table";
+        return;
+    }
+
+    NginxAttachmentOpaque &opaque = i_transaction_table->getState<NginxAttachmentOpaque>();
+    opaque.setSavedData(HttpTransactionData::waf_tag_ctx, static_cast<string>(header.getValue()));
+
+    dbgDebug(D_NGINX_ATTACHMENT_PARSER)
+        << "Added waf tag to context: "
+        <<  static_cast<string>(header.getValue());
+    return;
+}
+
 Maybe<string>
 UsersAllIdentifiersConfig::parseCookieElement(
     const string::const_iterator &start,

components/gradual_deployment/gradual_deployment.cc

@@ -142,7 +142,7 @@ private:
         if (temp_params_list.size() == 1) {
             Maybe<IPAddr> maybe_ip = IPAddr::createIPAddr(temp_params_list[0]);
             if (!maybe_ip.ok()) return genError("Could not create IP address, " + maybe_ip.getErr());
-            IpAddress addr = move(ConvertToIpAddress(maybe_ip.unpackMove()));
+            IpAddress addr = ConvertToIpAddress(maybe_ip.unpackMove());
 
             return move(IPRange{.start = addr, .end = addr});
         }
@@ -157,11 +157,11 @@ private:
             IPAddr max_addr = maybe_ip_max.unpackMove();
             if (min_addr > max_addr) return genError("Could not create ip range - start greater then end");
 
-            IpAddress addr_min = move(ConvertToIpAddress(move(min_addr)));
-            IpAddress addr_max = move(ConvertToIpAddress(move(max_addr)));
+            IpAddress addr_min = ConvertToIpAddress(move(min_addr));
+            IpAddress addr_max = ConvertToIpAddress(move(max_addr));
             if (addr_max.ip_type != addr_min.ip_type) return genError("Range IP's type does not match");
 
-            return move(IPRange{.start = move(addr_min), .end = move(addr_max)});
+            return IPRange{.start = move(addr_min), .end = move(addr_max)};
         }
 
         return genError("Illegal range received: " + range);

components/http_manager/http_manager.cc

@@ -37,6 +37,7 @@ operator<<(ostream &os, const EventVerdict &event)
 {
     switch (event.getVerdict()) {
         case ngx_http_cp_verdict_e::TRAFFIC_VERDICT_INSPECT: return os << "Inspect";
+        case ngx_http_cp_verdict_e::LIMIT_RESPONSE_HEADERS: return os << "Limit Response Headers";
         case ngx_http_cp_verdict_e::TRAFFIC_VERDICT_ACCEPT: return os << "Accept";
         case ngx_http_cp_verdict_e::TRAFFIC_VERDICT_DROP: return os << "Drop";
         case ngx_http_cp_verdict_e::TRAFFIC_VERDICT_INJECT: return os << "Inject";
@@ -93,13 +94,14 @@ public:
         ctx.registerValue(app_sec_marker_key, i_transaction_table->keyToString(), EnvKeyAttr::LogSection::MARKER);
 
         HttpManagerOpaque &state = i_transaction_table->getState<HttpManagerOpaque>();
-        string event_key = static_cast<string>(event.getKey());
-
-        if (event_key == getProfileAgentSettingWithDefault<string>("", "agent.customHeaderValueLogging")) {
+        
+        const auto &custom_header = getProfileAgentSettingWithDefault<string>("", "agent.customHeaderValueLogging");
+        
+        if (event.getKey().isEqualLowerCase(custom_header)) {
             string event_value = static_cast<string>(event.getValue());
             dbgTrace(D_HTTP_MANAGER)
                 << "Found header key and value - ("
-                << event_key
+                << custom_header
                 << ": "
                 << event_value
                 << ") that matched agent settings";
@@ -195,7 +197,6 @@ public:
         if (state.getUserDefinedValue().ok()) {
             ctx.registerValue("UserDefined", state.getUserDefinedValue().unpack(), EnvKeyAttr::LogSection::DATA);
         }
-
         return handleEvent(EndRequestEvent().performNamedQuery());
     }
 
@@ -323,8 +324,9 @@ private:
                 << respond.second.getVerdict();
 
             state.setApplicationVerdict(respond.first, respond.second.getVerdict());
+            state.setApplicationWebResponse(respond.first, respond.second.getWebUserResponseByPractice());
         }
-        FilterVerdict aggregated_verdict = state.getCurrVerdict();
+        FilterVerdict aggregated_verdict(state.getCurrVerdict(), state.getCurrWebUserResponse());
         if (aggregated_verdict.getVerdict() == ngx_http_cp_verdict_e::TRAFFIC_VERDICT_DROP) {
             SecurityAppsDropEvent(state.getCurrentDropVerdictCausers()).notify();
         }

components/http_manager/http_manager_opaque.cc

@@ -32,6 +32,13 @@ HttpManagerOpaque::setApplicationVerdict(const string &app_name, ngx_http_cp_ver
     applications_verdicts[app_name] = verdict;
 }
 
+void
+HttpManagerOpaque::setApplicationWebResponse(const string &app_name, string web_user_response_id)
+{
+    dbgTrace(D_HTTP_MANAGER) << "Security app: " << app_name << ", has web user response: " << web_user_response_id;
+    applications_web_user_response[app_name] = web_user_response_id;
+}
+
 ngx_http_cp_verdict_e
 HttpManagerOpaque::getApplicationsVerdict(const string &app_name) const
 {
@@ -51,8 +58,12 @@ HttpManagerOpaque::getCurrVerdict() const
     for (const auto &app_verdic_pair : applications_verdicts) {
         switch (app_verdic_pair.second) {
             case ngx_http_cp_verdict_e::TRAFFIC_VERDICT_DROP:
+                dbgTrace(D_HTTP_MANAGER) << "Verdict DROP for app: " << app_verdic_pair.first;
+                current_web_user_response = applications_web_user_response.at(app_verdic_pair.first);
+                dbgTrace(D_HTTP_MANAGER) << "current_web_user_response=" << current_web_user_response;
                 return app_verdic_pair.second;
             case ngx_http_cp_verdict_e::TRAFFIC_VERDICT_INJECT:
+                // Sent in ResponseHeaders and ResponseBody.
                 verdict = ngx_http_cp_verdict_e::TRAFFIC_VERDICT_INJECT;
                 break;
             case ngx_http_cp_verdict_e::TRAFFIC_VERDICT_ACCEPT:
@@ -60,11 +71,16 @@ HttpManagerOpaque::getCurrVerdict() const
                 break;
             case ngx_http_cp_verdict_e::TRAFFIC_VERDICT_INSPECT:
                 break;
+            case ngx_http_cp_verdict_e::LIMIT_RESPONSE_HEADERS:
+                // Sent in End Request.
+                verdict = ngx_http_cp_verdict_e::LIMIT_RESPONSE_HEADERS;
+                break;
             case ngx_http_cp_verdict_e::TRAFFIC_VERDICT_IRRELEVANT:
                 dbgTrace(D_HTTP_MANAGER) << "Verdict 'Irrelevant' is not yet supported. Returning Accept";
                 accepted_apps++;
                 break;
             case ngx_http_cp_verdict_e::TRAFFIC_VERDICT_WAIT:
+                // Sent in Request Headers and Request Body.
                 verdict = ngx_http_cp_verdict_e::TRAFFIC_VERDICT_WAIT;
                 break;
             default:

components/http_manager/http_manager_opaque.h

@@ -28,10 +28,12 @@ public:
     HttpManagerOpaque();
 
     void setApplicationVerdict(const std::string &app_name, ngx_http_cp_verdict_e verdict);
+    void setApplicationWebResponse(const std::string &app_name, std::string web_user_response_id);
     ngx_http_cp_verdict_e getApplicationsVerdict(const std::string &app_name) const;
     void setManagerVerdict(ngx_http_cp_verdict_e verdict) { manager_verdict = verdict; }
     ngx_http_cp_verdict_e getManagerVerdict() const { return manager_verdict; }
     ngx_http_cp_verdict_e getCurrVerdict() const;
+    const std::string & getCurrWebUserResponse() const { return current_web_user_response; };
     std::set<std::string> getCurrentDropVerdictCausers() const;
     void saveCurrentDataToCache(const Buffer &full_data);
     void setUserDefinedValue(const std::string &value) { user_defined_value = value; }
@@ -52,6 +54,8 @@ public:
 
 private:
     std::unordered_map<std::string, ngx_http_cp_verdict_e> applications_verdicts;
+    std::unordered_map<std::string, std::string> applications_web_user_response;
+    mutable std::string current_web_user_response;
     ngx_http_cp_verdict_e manager_verdict = ngx_http_cp_verdict_e::TRAFFIC_VERDICT_INSPECT;
     Buffer prev_data_cache;
     uint aggregated_payload_size = 0;

components/include/generic_rulebase/evaluators/http_transaction_data_eval.h

@@ -45,6 +45,19 @@ private:
     std::string host;
 };
 
+class EqualWafTag : public EnvironmentEvaluator<bool>, Singleton::Consume<I_Environment>
+{
+public:
+EqualWafTag(const std::vector<std::string> &params);
+
+    static std::string getName() { return "EqualWafTag"; }
+
+    Maybe<bool, Context::Error> evalVariable() const override;
+
+private:
+    std::string waf_tag;
+};
+
 class EqualListeningIP : public EnvironmentEvaluator<bool>, Singleton::Consume<I_Environment>
 {
 public:

components/include/generic_rulebase/triggers_config.h

@@ -317,12 +317,12 @@ public:
     {
         return url_for_cef;
     }
+    Flags<ReportIS::StreamType> getStreams(SecurityType security_type, bool is_action_drop_or_prevent) const;
+    Flags<ReportIS::Enreachments> getEnrechments(SecurityType security_type) const;
 
 private:
     ReportIS::Severity getSeverity(bool is_action_drop_or_prevent) const;
     ReportIS::Priority getPriority(bool is_action_drop_or_prevent) const;
-    Flags<ReportIS::StreamType> getStreams(SecurityType security_type, bool is_action_drop_or_prevent) const;
-    Flags<ReportIS::Enreachments> getEnrechments(SecurityType security_type) const;
 
     std::string name;
     std::string verbosity;
@@ -339,4 +339,32 @@ private:
     bool should_format_output = false;
 };
 
+class ReportTriggerConf
+{
+public:
+    /// \brief Default constructor for ReportTriggerConf.
+    ReportTriggerConf() {}
+
+    /// \brief Preload function to register expected configuration.
+    static void
+    preload()
+    {
+        registerExpectedConfiguration<ReportTriggerConf>("rulebase", "report");
+    }
+
+    /// \brief Load function to deserialize configuration from JSONInputArchive.
+    /// \param archive_in The JSON input archive.
+    void load(cereal::JSONInputArchive &archive_in);
+
+    /// \brief Get the name.
+    /// \return The name.
+    const std::string &
+    getName() const
+    {
+        return name;
+    }
+private:
+    std::string name;
+};
+
 #endif //__TRIGGERS_CONFIG_H__

components/include/http_event_impl/filter_verdict.h

@@ -27,9 +27,18 @@ public:
         verdict(_verdict)
     {}
 
+    FilterVerdict(
+        ngx_http_cp_verdict_e _verdict,
+        const std::string &_web_reponse_id)
+            :
+        verdict(_verdict),
+        web_user_response_id(_web_reponse_id)
+    {}
+
     FilterVerdict(const EventVerdict &_verdict, ModifiedChunkIndex _event_idx = -1)
             :
-        verdict(_verdict.getVerdict())
+        verdict(_verdict.getVerdict()),
+        web_user_response_id(_verdict.getWebUserResponseByPractice())
     {
         if (verdict == ngx_http_cp_verdict_e::TRAFFIC_VERDICT_INJECT) {
             addModifications(_verdict.getModifications(), _event_idx);
@@ -59,10 +68,12 @@ public:
     uint getModificationsAmount() const { return total_modifications; }
     ngx_http_cp_verdict_e getVerdict() const { return verdict; }
     const std::vector<EventModifications> & getModifications() const { return modifications; }
+    const std::string getWebUserResponseID() const { return web_user_response_id; }
 
 private:
     ngx_http_cp_verdict_e verdict = ngx_http_cp_verdict_e::TRAFFIC_VERDICT_INSPECT;
     std::vector<EventModifications> modifications;
+    std::string web_user_response_id;
     uint total_modifications = 0;
 };
 

components/include/http_event_impl/i_http_event_impl.h

@@ -376,16 +376,31 @@ public:
         verdict(event_verdict)
     {}
 
+    EventVerdict(
+        const ModificationList &mods,
+        ngx_http_cp_verdict_e event_verdict,
+        std::string response_id) :
+        modifications(mods),
+        verdict(event_verdict),
+        webUserResponseByPractice(response_id)
+    {}
+
 // LCOV_EXCL_START - sync functions, can only be tested once the sync module exists
     template <typename T> void serialize(T &ar, uint) { ar(verdict); }
 // LCOV_EXCL_STOP
 
     const ModificationList & getModifications() const { return modifications; }
     ngx_http_cp_verdict_e getVerdict() const { return verdict; }
+    const std::string getWebUserResponseByPractice() const { return webUserResponseByPractice; }
+    void setWebUserResponseByPractice(const std::string id) {
+        dbgTrace(D_HTTP_MANAGER) << "current verdict web user response set to: " << id;
+        webUserResponseByPractice = id;
+    }
 
 private:
     ModificationList modifications;
     ngx_http_cp_verdict_e verdict = ngx_http_cp_verdict_e::TRAFFIC_VERDICT_INSPECT;
+    std::string webUserResponseByPractice;
 };
 
 #endif // __I_HTTP_EVENT_IMPL_H__

components/include/http_transaction_data.h

@@ -72,7 +72,8 @@ public:
             parsed_uri,
             client_ip,
             client_port,
-            response_content_encoding
+            response_content_encoding,
+            waf_tag
         );
     }
 
@@ -91,7 +92,8 @@ public:
             parsed_uri,
             client_ip,
             client_port,
-            response_content_encoding
+            response_content_encoding,
+            waf_tag
         );
     }
 // LCOV_EXCL_STOP
@@ -122,6 +124,9 @@ public:
         response_content_encoding = _response_content_encoding;
     }
 
+    const std::string & getWafTag() const { return waf_tag; }
+    void setWafTag(const std::string &_waf_tag) { waf_tag = _waf_tag; }
+
     static const std::string http_proto_ctx;
     static const std::string method_ctx;
     static const std::string host_name_ctx;
@@ -137,6 +142,7 @@ public:
     static const std::string source_identifier;
     static const std::string proxy_ip_ctx;
     static const std::string xff_vals_ctx;
+    static const std::string waf_tag_ctx;
 
     static const CompressionType default_response_content_encoding;
 
@@ -153,6 +159,7 @@ private:
     uint16_t client_port;
     bool is_request;
     CompressionType response_content_encoding;
+    std::string waf_tag;
 };
 
 #endif // __HTTP_TRANSACTION_DATA_H__

components/include/i_details_resolver.h

@@ -26,6 +26,7 @@ public:
     virtual Maybe<std::string> getArch() = 0;
     virtual std::string getAgentVersion() = 0;
     virtual bool isKernelVersion3OrHigher() = 0;
+    virtual bool isGw() = 0;
     virtual bool isGwNotVsx() = 0;
     virtual bool isVersionAboveR8110() = 0;
     virtual bool isReverseProxy() = 0;

components/include/i_waap_telemetry.h

@@ -27,6 +27,7 @@ struct DecisionTelemetryData
     int responseCode;
     uint64_t elapsedTime;
     std::set<std::string> attackTypes;
+    bool temperatureDetected;
 
     DecisionTelemetryData() :
         blockType(NOT_BLOCKING),
@@ -38,7 +39,8 @@ struct DecisionTelemetryData
         method(POST),
         responseCode(0),
         elapsedTime(0),
-        attackTypes()
+        attackTypes(),
+        temperatureDetected(false)
     {
     }
 };

components/include/ips_comp.h

@@ -4,6 +4,7 @@
 #include "singleton.h"
 #include "i_keywords_rule.h"
 #include "i_table.h"
+#include "i_mainloop.h"
 #include "i_http_manager.h"
 #include "i_environment.h"
 #include "http_inspection_events.h"
@@ -16,7 +17,8 @@ class IPSComp
     Singleton::Consume<I_KeywordsRule>,
     Singleton::Consume<I_Table>,
     Singleton::Consume<I_Environment>,
-    Singleton::Consume<I_GenericRulebase>
+    Singleton::Consume<I_GenericRulebase>,
+    Singleton::Consume<I_MainLoop>
 {
 public:
     IPSComp();

components/include/prometheus_comp.h

@@ -0,0 +1,30 @@
+#ifndef __PROMETHEUS_COMP_H__
+#define __PROMETHEUS_COMP_H__
+
+#include <memory>
+
+#include "component.h"
+#include "singleton.h"
+
+#include "i_rest_api.h"
+#include "i_messaging.h"
+#include "generic_metric.h"
+
+class PrometheusComp
+        :
+    public Component,
+    Singleton::Consume<I_RestApi>,
+    Singleton::Consume<I_Messaging>
+{
+public:
+    PrometheusComp();
+    ~PrometheusComp();
+
+    void init() override;
+
+private:
+    class Impl;
+    std::unique_ptr<Impl> pimpl;
+};
+
+#endif // __PROMETHEUS_COMP_H__

components/include/telemetry.h

@@ -76,6 +76,20 @@ private:
     std::unordered_set<std::string> sources_seen;
 };
 
+class WaapAdditionalTrafficTelemetrics : public WaapTelemetryBase
+{
+public:
+    void updateMetrics(const std::string &asset_id, const DecisionTelemetryData &data);
+    void initMetrics();
+
+private:
+    MetricCalculations::Counter requests{this, "reservedNgenA"};
+    MetricCalculations::Counter sources{this, "reservedNgenB"};
+    MetricCalculations::Counter blocked{this, "reservedNgenC"};
+    MetricCalculations::Counter temperature_count{this, "reservedNgenD"};
+    std::unordered_set<std::string> sources_seen;
+};
+
 class WaapTrafficTelemetrics : public WaapTelemetryBase
 {
 public:
@@ -124,6 +138,7 @@ private:
     std::map<std::string, std::shared_ptr<WaapTrafficTelemetrics>> traffic_telemetries;
     std::map<std::string, std::shared_ptr<WaapAttackTypesMetrics>> attack_types;
     std::map<std::string, std::shared_ptr<WaapAttackTypesMetrics>> attack_types_telemetries;
+    std::map<std::string, std::shared_ptr<WaapAdditionalTrafficTelemetrics>> additional_traffic_telemetries;
 
     template <typename T>
     void initializeTelemetryData(

components/include/user_identifiers_config.h

@@ -30,6 +30,7 @@ public:
     void parseRequestHeaders(const HttpHeader &header) const;
     std::vector<std::string> getHeaderValuesFromConfig(const std::string &header_key) const;
     void setXFFValuesToOpaqueCtx(const HttpHeader &header, ExtractType type) const;
+    void setWafTagValuesToOpaqueCtx(const HttpHeader &header) const;
 
 private:
     class UsersIdentifiersConfig

components/security_apps/CMakeLists.txt

@@ -3,6 +3,7 @@ add_subdirectory(ips)
 add_subdirectory(layer_7_access_control)
 add_subdirectory(local_policy_mgmt_gen)
 add_subdirectory(orchestration)
+add_subdirectory(prometheus)
 add_subdirectory(rate_limit)
 add_subdirectory(waap)
 add_subdirectory(central_nginx_manager)

components/security_apps/http_geo_filter/http_geo_filter.cc

@@ -96,6 +96,7 @@ public:
         if (ignore_source_ip){
             dbgDebug(D_GEO_FILTER) << "Geo protection ignoring source ip: " << source_ip;
         } else {
+            dbgTrace(D_GEO_FILTER) << "Geo protection source ip: " << source_ip;
             ip_set.insert(convertIpAddrToString(maybe_source_ip.unpack()));
         }
 
@@ -335,6 +336,14 @@ private:
         ngx_http_cp_verdict_e verdict = ngx_http_cp_verdict_e::TRAFFIC_VERDICT_IRRELEVANT;
         I_GeoLocation *i_geo_location = Singleton::Consume<I_GeoLocation>::by<HttpGeoFilter>();
         EnumArray<I_GeoLocation::GeoLocationField, std::string> geo_location_data;
+        auto env = Singleton::Consume<I_Environment>::by<HttpGeoFilter>();
+        string source_id;
+        auto maybe_source_id = env->get<std::string>(HttpTransactionData::source_identifier);
+        if (!maybe_source_id.ok()) {
+            dbgTrace(D_GEO_FILTER) << "failed to get source identifier from env";
+        } else {
+            source_id = maybe_source_id.unpack();
+        }
 
         for (const std::string& source : sources) {
 
@@ -366,11 +375,15 @@ private:
             << country_code
             << ", country name: "
             << country_name
-            << ", source ip address: "
-            << source;
+            << ", ip address: "
+            << source
+            << ", source identifier: "
+            << source_id;
+
 
             unordered_map<string, set<string>> exception_value_country_code = {
-                {"countryCode", {country_code}}
+                {"countryCode", {country_code}},
+                {"sourceIdentifier", {source_id}}
             };
             auto matched_behavior_maybe = getBehaviorsVerdict(exception_value_country_code, geo_location_data);
             if (matched_behavior_maybe.ok()) {
@@ -382,7 +395,8 @@ private:
             }
 
             unordered_map<string, set<string>> exception_value_country_name = {
-                {"countryName", {country_name}}
+                {"countryName", {country_name}},
+                {"sourceIdentifier", {source_id}}
             };
             matched_behavior_maybe = getBehaviorsVerdict(exception_value_country_name, geo_location_data);
             if (matched_behavior_maybe.ok()) {

components/security_apps/ips/include/ips_signatures.h

@@ -29,6 +29,8 @@
 #include "pm_hook.h"
 #include "i_generic_rulebase.h"
 
+#define DEFAULT_IPS_YIELD_COUNT 500
+
 /// \namespace IPSSignatureSubTypes
 /// \brief Namespace containing subtypes for IPS signatures.
 namespace IPSSignatureSubTypes
@@ -342,10 +344,17 @@ public:
         return is_loaded;
     }
 
+    static void
+    setYieldCounter(int new_yield_cnt)
+    {
+        yield_on_load_cnt = new_yield_cnt;
+    }
+
 private:
     IPSSignatureMetaData metadata;
     std::shared_ptr<BaseSignature> rule;
     bool is_loaded;
+    static int yield_on_load_cnt;
 };
 
 /// \class SignatureAndAction

components/security_apps/ips/ips_comp.cc

@@ -98,6 +98,7 @@ public:
         registerListener();
         table = Singleton::Consume<I_Table>::by<IPSComp>();
         env = Singleton::Consume<I_Environment>::by<IPSComp>();
+        updateSigsYieldCount();
     }
 
     void
@@ -307,6 +308,20 @@ public:
 
     EventVerdict respond (const EndTransactionEvent &) override { return ACCEPT; }
 
+    void
+    updateSigsYieldCount()
+    {
+        const char *ips_yield_env_str = getenv("CPNANO_IPS_LOAD_YIELD_CNT");
+        int ips_yield_default = DEFAULT_IPS_YIELD_COUNT;
+        if (ips_yield_env_str != nullptr) {
+            dbgDebug(D_IPS) << "CPNANO_IPS_LOAD_YIELD_CNT env variable is set to " << ips_yield_env_str;
+            ips_yield_default = atoi(ips_yield_env_str);
+        }
+        int yield_limit = getProfileAgentSettingWithDefault<int>(ips_yield_default, "ips.sigsYieldCnt");
+        dbgDebug(D_IPS) << "Setting IPS yield count to  " << yield_limit;
+        IPSSignatureSubTypes::CompleteSignature::setYieldCounter(yield_limit);
+    }
+
 private:
     static void setDrop(IPSEntry &state) { state.setDrop(); }
     static bool isDrop(const IPSEntry &state) { return state.isDrop(); }
@@ -373,6 +388,7 @@ IPSComp::preload()
     registerExpectedConfigFile("ips", Config::ConfigFileType::Policy);
     registerExpectedConfigFile("ips", Config::ConfigFileType::Data);
     registerExpectedConfigFile("snort", Config::ConfigFileType::Policy);
+    registerConfigLoadCb([this]() { pimpl->updateSigsYieldCount(); });
 
     ParameterException::preload();
 

components/security_apps/ips/ips_signatures.cc

@@ -45,6 +45,8 @@ static const map<string, IPSLevel> levels = {
     { "Very Low",    IPSLevel::VERY_LOW }
 };
 
+int CompleteSignature::yield_on_load_cnt = DEFAULT_IPS_YIELD_COUNT;
+
 static IPSLevel
 getLevel(const string &level_string, const string &attr_name)
 {
@@ -219,6 +221,18 @@ IPSSignatureMetaData::getYear() const
 void
 CompleteSignature::load(cereal::JSONInputArchive &ar)
 {
+    static int sigs_load_counter = 0;
+    static I_Environment *env = Singleton::Consume<I_Environment>::by<IPSComp>();
+    static bool post_init = false;
+
+    if (!post_init) {
+        auto routine_id = Singleton::Consume<I_MainLoop>::by<IPSComp>()->getCurrentRoutineId();
+        if (routine_id.ok()) {
+            post_init = true;
+            dbgInfo(D_IPS) << "Loading signatures post init, enabling yield with limit " << yield_on_load_cnt;
+        }
+    }
+
     try {
         ar(cereal::make_nvp("protectionMetadata", metadata));
         RuleDetection rule_detection(metadata.getName());
@@ -229,6 +243,15 @@ CompleteSignature::load(cereal::JSONInputArchive &ar)
         is_loaded = false;
         dbgWarning(D_IPS) << "Failed to load signature: " << e.what();
     }
+
+    if (post_init && (yield_on_load_cnt > 0) && (++sigs_load_counter == yield_on_load_cnt)) {
+        sigs_load_counter = 0;
+        auto maybe_is_async = env->get<bool>("Is Async Config Load");
+        if (maybe_is_async.ok() && *maybe_is_async == true) {
+            dbgTrace(D_IPS) << "Yielding after " << yield_on_load_cnt << " signatures";
+            Singleton::Consume<I_MainLoop>::by<IPSComp>()->yield(false);
+        }
+    }
 }
 
 MatchType

components/security_apps/ips/ips_ut/component_ut.cc

@@ -29,6 +29,8 @@ public:
     {
         comp.preload();
         comp.init();
+        auto err = genError("not coroutine");
+        EXPECT_CALL(mainloop, getCurrentRoutineId()).WillRepeatedly(Return(Maybe<I_MainLoop::RoutineID>(err)));
     }
 
     ~ComponentTest()

components/security_apps/ips/ips_ut/entry_ut.cc

@@ -41,6 +41,8 @@ public:
     EntryTest()
     {
         ON_CALL(table, getState(_)).WillByDefault(Return(ptr));
+        auto err = genError("not coroutine");
+        EXPECT_CALL(mock_mainloop, getCurrentRoutineId()).WillRepeatedly(Return(Maybe<I_MainLoop::RoutineID>(err)));
     }
 
     void

components/security_apps/ips/ips_ut/resource_ut.cc

@@ -2,6 +2,7 @@
 #include "cptest.h"
 #include "environment.h"
 #include "config_component.h"
+#include "mock/mock_mainloop.h"
 
 using namespace std;
 using namespace testing;
@@ -61,6 +62,9 @@ TEST(resources, basic_resource)
 {
     ConfigComponent conf;
     ::Environment env;
+    NiceMock<MockMainLoop> mock_mainloop;
+    auto err = genError("not coroutine");
+    EXPECT_CALL(mock_mainloop, getCurrentRoutineId()).WillRepeatedly(Return(Maybe<I_MainLoop::RoutineID>(err)));
 
     conf.preload();
 

components/security_apps/ips/ips_ut/signatures_ut.cc

@@ -60,7 +60,12 @@ public:
     {
         IPSHelper::has_deobfuscation = true;
         generic_rulebase.preload();
+        env.preload();
+        env.init();
+
         EXPECT_CALL(logs, getCurrentLogId()).Times(AnyNumber());
+        auto err = genError("not coroutine");
+        EXPECT_CALL(mock_mainloop, getCurrentRoutineId()).WillRepeatedly(Return(Maybe<I_MainLoop::RoutineID>(err)));
         ON_CALL(table, getState(_)).WillByDefault(Return(&ips_state));
         {
             stringstream ss;
@@ -123,9 +128,6 @@ public:
     void
     loadExceptions()
     {
-        env.preload();
-        env.init();
-
         BasicRuleConfig::preload();
         registerExpectedConfiguration<ParameterException>("rulebase", "exception");
 
@@ -195,6 +197,7 @@ public:
     void
     load(const IPSSignaturesResource &policy, const string &severity, const string &confidence)
     {
+        Singleton::Consume<I_Environment>::from(env)->registerValue<bool>("Is Async Config Load", false);
         setResource(policy, "IPS", "protections");
         stringstream ss;
         ss << "{";

components/security_apps/layer_7_access_control/layer_7_access_control.cc

@@ -131,8 +131,12 @@ public:
     EventVerdict
     respond(const WaitTransactionEvent &) override
     {
-        dbgFlow(D_L7_ACCESS_CONTROL) << "Handling wait verdict";
+        if (!isAppEnabled()) {
+            dbgTrace(D_L7_ACCESS_CONTROL) << "Returning Accept verdict as the Layer-7 Access Control app is disabled";
+            return ngx_http_cp_verdict_e::TRAFFIC_VERDICT_ACCEPT;
+        }
 
+        dbgTrace(D_L7_ACCESS_CONTROL) << "Handling wait verdict";
         return handleEvent();
     }
 

components/security_apps/local_policy_mgmt_gen/include/local_policy_common.h

@@ -170,6 +170,7 @@ public:
         ss.str(modified_json);
         try {
             cereal::JSONInputArchive in_ar(ss);
+            in_ar(cereal::make_nvp("apiVersion", api_version));
             in_ar(cereal::make_nvp("spec", spec));
             in_ar(cereal::make_nvp("metadata", meta_data));
         } catch (cereal::Exception &e) {
@@ -191,11 +192,18 @@ public:
         return meta_data;
     }
 
+    const std::string &
+    getApiVersion() const
+    {
+        return api_version;
+    }
+
     const T & getSpec() const { return spec; }
 
 private:
     T spec;
     AppsecSpecParserMetaData meta_data;
+    std::string api_version;
 };
 
 #endif // __LOCAL_POLICY_COMMON_H__

components/security_apps/local_policy_mgmt_gen/k8s_policy_utils.cc

@@ -515,17 +515,6 @@ K8sPolicyUtils::createAppsecPolicyK8sFromV1beta2Crds(
 }
 // LCOV_EXCL_STOP
 
-bool
-doesVersionExist(const map<string, string> &annotations, const string &version)
-{
-    for (auto annotation : annotations) {
-        if(annotation.second.find(version) != std::string::npos) {
-            return true;
-        }
-    }
-    return false;
-}
-
 std::tuple<Maybe<AppsecLinuxPolicy>, Maybe<V1beta2AppsecLinuxPolicy>>
 K8sPolicyUtils::createAppsecPolicyK8s(const string &policy_name, const string &ingress_mode) const
 {
@@ -534,7 +523,7 @@ K8sPolicyUtils::createAppsecPolicyK8s(const string &policy_name, const string &i
     );
 
     if (!maybe_appsec_policy_spec.ok() ||
-        !doesVersionExist(maybe_appsec_policy_spec.unpack().getMetaData().getAnnotations(), "v1beta1")
+        maybe_appsec_policy_spec.unpack().getApiVersion().find("v1beta1") == std::string::npos
     ) {
         try {
             std::string v1beta1_error =

components/security_apps/orchestration/details_resolver/details_resolver.cc

@@ -41,6 +41,7 @@ public:
 
     string getAgentVersion() override;
     bool isKernelVersion3OrHigher() override;
+    bool isGw() override;
     bool isGwNotVsx() override;
     bool isVersionAboveR8110() override;
     bool isReverseProxy() override;
@@ -167,6 +168,19 @@ DetailsResolver::Impl::isKernelVersion3OrHigher()
     return false;
 }
 
+bool
+DetailsResolver::Impl::isGw()
+{
+#if defined(gaia) || defined(smb)
+    static const string is_gw_cmd = "cpprod_util FwIsFirewallModule";
+    auto is_gw = DetailsResolvingHanlder::getCommandOutput(is_gw_cmd);
+    if (is_gw.ok() && !is_gw.unpack().empty()) {
+        return is_gw.unpack().front() == '1';
+    }
+#endif
+    return false;
+}
+
 bool
 DetailsResolver::Impl::isGwNotVsx()
 {
@@ -238,15 +252,21 @@ DetailsResolver::Impl::parseNginxMetadata()
         "orchestration",
         "Nginx metadata temp file"
     );
+
+    const string &filesystem_path_config = getFilesystemPathConfig();
+
     const string srcipt_exe_cmd =
-        getFilesystemPathConfig() +
+        filesystem_path_config +
         "/scripts/cp-nano-makefile-generator.sh -f -o " +
         output_path;
 
     const string script_fresh_exe_cmd =
-        getFilesystemPathConfig() +
+        filesystem_path_config +
         "/scripts/cp-nano-makefile-generator-fresh.sh save --save-location " +
-        output_path;
+        output_path +
+        " --strings_bin_path " +
+        filesystem_path_config +
+        "/bin/strings";
 
     dbgTrace(D_ORCHESTRATOR) << "Details resolver, srcipt exe cmd: " << srcipt_exe_cmd;
     if (isNoResponse("which nginx") && isNoResponse("which kong")) {

components/security_apps/orchestration/details_resolver/details_resolver_handlers/checkpoint_product_handlers.h

@@ -26,9 +26,7 @@
 Maybe<string>
 checkSAMLSupportedBlade(const string &command_output)
 {
-    // uncomment when vpn will support SAML authentication
-    // string supportedBlades[3] = {"identityServer", "vpn", "cvpn"};
-    string supportedBlades[1] = {"identityServer"};
+    string supportedBlades[3] = {"identityServer", "vpn", "cvpn"};
     for(const string &blade : supportedBlades) {
         if (command_output.find(blade) != string::npos) {
             return string("true");
@@ -49,6 +47,17 @@ checkIDABlade(const string &command_output)
     return string("false");
 }
 
+Maybe<string>
+checkVPNBlade(const string &command_output)
+{
+    string vpnBlade = "vpn";
+    if (command_output.find(vpnBlade) != string::npos) {
+        return string("true");
+    }
+
+    return string("false");
+}
+
 Maybe<string>
 checkSAMLPortal(const string &command_output)
 {
@@ -60,9 +69,9 @@ checkSAMLPortal(const string &command_output)
 }
 
 Maybe<string>
-checkPepIdaIdnStatus(const string &command_output)
+checkInfinityIdentityEnabled(const string &command_output)
 {
-    if (command_output.find("nac_pep_identity_next_enabled = 1") != string::npos) {
+    if (command_output.find("get_identities_from_infinity_identity (true)") != string::npos) {
         return string("true");
     }
     return string("false");
@@ -90,9 +99,6 @@ checkIDP(shared_ptr<istream> file_stream)
 {
     string line;
     while (getline(*file_stream, line)) {
-        if (line.find("<identity_portal/>") != string::npos) {
-            return string("false");
-        }
         if (line.find("<central_idp ") != string::npos) {
             return string("true");
         }
@@ -101,6 +107,26 @@ checkIDP(shared_ptr<istream> file_stream)
     return string("false");
 }
 
+Maybe<string>
+checkVPNCIDP(shared_ptr<istream> file_stream)
+{
+    string line;
+    while (getline(*file_stream, line)) {
+        if (line.find("<vpn") != string::npos) {
+            while (getline(*file_stream, line)) {
+                if (line.find("<central_idp ") != string::npos) {
+                    return string("true");
+                }
+                if (line.find("</vpn>") != string::npos) {
+                    break;
+                }
+            }
+        }
+    }
+
+    return string("false");
+}
+
 #endif // gaia
 
 #if defined(gaia) || defined(smb)
@@ -140,6 +166,17 @@ getIsAiopsRunning(const string &command_output)
     return command_output;
 }
 
+Maybe<string>
+getInterfaceMgmtIp(const string &command_output)
+{
+    if (!command_output.empty()) {
+        return command_output;
+    }
+
+    return genError("Eth Management IP was not found");
+}
+
+
 Maybe<string>
 checkHasSDWan(const string &command_output)
 {
@@ -451,6 +488,14 @@ extractManagements(const string &command_output)
     json_output += "]";
     return json_output;
 }
+
+Maybe<string>
+checkQosLegacyBlade(const string &command_output)
+{
+    if (command_output == "true" || command_output == "false") return command_output;
+
+    return string("false");
+}
 #endif // gaia || smb
 
 #if defined(gaia)

components/security_apps/orchestration/details_resolver/details_resolver_handlers/details_resolver_impl.h

@@ -79,6 +79,14 @@ SHELL_CMD_HANDLER("MGMT_QUID", "[ -d /opt/CPquid ] "
 SHELL_CMD_HANDLER("AIOPS_AGENT_ROLE", "[ -d /opt/CPOtlpAgent/custom_scripts ] "
     "&& ENV_NO_FORMAT=1 /opt/CPOtlpAgent/custom_scripts/agent_role.sh",
     getOtlpAgentGaiaOsRole)
+SHELL_CMD_HANDLER("ETH_MGMT_IP",
+    "FS_PATH=<FILESYSTEM-PREFIX>;"
+    "VS_ID=$(echo \"${FS_PATH}\" | grep -o -E \"vs[0-9]+\" | grep -o -E \"[0-9]+\");"
+    "[ -z \"${VS_ID}\" ] && "
+    "(eth=\"$(grep 'management:interface' /config/active | awk '{print $2}')\" &&"
+    " ip addr show \"${eth}\" | grep inet | awk '{print $2}' | cut -d '/' -f1) || "
+    "(ip a | grep UP | grep -v lo | head -n 1 | cut -d ':' -f2 | tr -d ' ')",
+    getInterfaceMgmtIp)
 #endif
 #if defined(smb) || defined(smb_thx_v3) || defined(smb_sve_v2) || defined(smb_mrv_v1)
 SHELL_CMD_HANDLER("GLOBAL_QUID",
@@ -89,6 +97,8 @@ SHELL_CMD_HANDLER("QUID",
     "cat $FWDIR/database/myown.C "
     "| awk -F'[()]' '/:name/ { found=1; next } found && /:uuid/ { uid=tolower($2); print uid; exit }'",
     getQUID)
+
+
 SHELL_CMD_HANDLER("SMO_QUID", "echo ''", getQUID)
 SHELL_CMD_HANDLER("MGMT_QUID", "echo ''", getQUID)
 SHELL_CMD_HANDLER("AIOPS_AGENT_ROLE", "echo 'SMB'", getOtlpAgentGaiaOsRole)
@@ -114,12 +124,6 @@ SHELL_CMD_HANDLER(
     "jq -r .lsm_profile_uuid /tmp/cpsdwan_getdata_orch.json",
     checkLsmProfileUuid
 )
-SHELL_CMD_HANDLER(
-    "IP Address",
-    "[ $(cpprod_util FWisDAG) -eq 1 ] && echo \"Dynamic Address\" "
-    "|| (jq -r .main_ip /tmp/cpsdwan_getdata_orch.json)",
-    getGWIPAddress
-)
 SHELL_CMD_HANDLER(
     "Version",
     "cat /etc/cp-release | grep -oE 'R[0-9]+(\\.[0-9]+)?'",
@@ -138,13 +142,22 @@ SHELL_CMD_HANDLER(
     "fw ctl get int support_fec |& grep -sq \"support_fec =\";echo $?",
     getFecApplicable
 )
+SHELL_CMD_HANDLER("is_legacy_qos_blade_enabled",
+    "cpprod_util CPPROD_GetValue FG1 ProdActive 1 | grep -q '^1$' "
+    "&& (cpprod_util CPPROD_GetValue FG1 FgSDWAN 1 | grep -q '^1$' && echo false || echo true) || "
+    "echo false",
+    checkQosLegacyBlade)
 #endif //gaia || smb
 
 #if defined(gaia)
 SHELL_CMD_HANDLER("hasSAMLSupportedBlade", "enabled_blades", checkSAMLSupportedBlade)
 SHELL_CMD_HANDLER("hasIDABlade", "enabled_blades", checkIDABlade)
+SHELL_CMD_HANDLER("hasVPNBlade", "enabled_blades", checkVPNBlade)
 SHELL_CMD_HANDLER("hasSAMLPortal", "mpclient status nac", checkSAMLPortal)
-SHELL_CMD_HANDLER("hasIdaIdnEnabled", "fw ctl get int nac_pep_identity_next_enabled", checkPepIdaIdnStatus)
+SHELL_CMD_HANDLER("hasInfinityIdentityEnabled",
+    "cat $FWDIR/database/myself_objects.C | grep get_identities_from_infinity_identity",
+    checkInfinityIdentityEnabled
+)
 SHELL_CMD_HANDLER("requiredNanoServices", "echo ida", getRequiredNanoServices)
 SHELL_CMD_HANDLER(
     "cpProductIntegrationMgmtObjectName",
@@ -209,6 +222,14 @@ SHELL_CMD_HANDLER(
     "echo 1",
     extractManagements
 )
+SHELL_CMD_HANDLER(
+    "IP Address",
+    "( [ $(cpprod_util FwIsHighAvail) -eq 1 ] && [ $(cpprod_util FwIsVSX) -eq 1 ]"
+    "&& (jq -r .cluster_main_ip /tmp/cpsdwan_getdata_orch.json) )"
+    "|| ( [ $(cpprod_util FWisDAG) -eq 1 ] && echo \"Dynamic Address\" )"
+    "|| (jq -r .main_ip /tmp/cpsdwan_getdata_orch.json)",
+    getGWIPAddress
+)
 #endif //gaia
 
 #if defined(smb) || defined(smb_thx_v3) || defined(smb_sve_v2) || defined(smb_mrv_v1)
@@ -270,6 +291,17 @@ SHELL_CMD_HANDLER(
     "echo 1",
     extractManagements
 )
+SHELL_CMD_HANDLER(
+    "IP Address",
+    "[ $(cpprod_util FWisDAG) -eq 1 ] && echo \"Dynamic Address\" "
+    "|| (jq -r .main_ip /tmp/cpsdwan_getdata_orch.json)",
+    getGWIPAddress
+)
+SHELL_CMD_HANDLER(
+    "Hardware",
+    R"(ver | sed -E 's/^This is Check Point'\''s +([^ ]+).*$/\1/')",
+    getHardware
+)
 #endif//smb
 
 SHELL_CMD_OUTPUT("kernel_version", "uname -r")
@@ -287,6 +319,11 @@ FILE_CONTENT_HANDLER(
     (getenv("SAMLPORTAL_HOME") ? string(getenv("SAMLPORTAL_HOME")) : "") + "/phpincs/spPortal/idpPolicy.xml",
     checkIDP
 )
+FILE_CONTENT_HANDLER(
+    "hasVPNCidpConfigured",
+    (getenv("SAMLPORTAL_HOME") ? string(getenv("SAMLPORTAL_HOME")) : "") + "/phpincs/spPortal/idpPolicy.xml",
+    checkVPNCIDP
+)
 #endif //gaia
 
 #if defined(alpine)

components/security_apps/orchestration/downloader/https_client.cc

@@ -41,8 +41,13 @@ HTTPSClient::getFile(const URLParser &url, const string &out_file, bool auth_req
 
     if (!url.isOverSSL()) return genError("URL is not over SSL.");
 
-    if (getFileSSLDirect(url, out_file, token).ok()) return Maybe<void>();
-    dbgWarning(D_ORCHESTRATOR) << "Failed to get file over SSL directly. Trying indirectly.";
+    bool skip_direct_download = (url.getQuery().find("/resources/") != string::npos);
+    if (skip_direct_download) {
+        dbgWarning(D_ORCHESTRATOR) << "Resources path: " << url.getQuery() << ". Skipping direct download.";
+    } else {
+        if (getFileSSLDirect(url, out_file, token).ok()) return Maybe<void>();
+        dbgWarning(D_ORCHESTRATOR) << "Failed to get file over SSL directly. Trying indirectly.";
+    }
 
     if (getFileSSL(url, out_file, token).ok()) return Maybe<void>();
     dbgWarning(D_ORCHESTRATOR) << "Failed to get file over SSL. Trying via CURL (SSL).";

components/security_apps/orchestration/include/mock/mock_details_resolver.h

@@ -42,13 +42,14 @@ public:
     MOCK_METHOD0(getPlatform,                Maybe<std::string>());
     MOCK_METHOD0(getArch,                    Maybe<std::string>());
     MOCK_METHOD0(getAgentVersion,            std::string());
-    MOCK_METHOD0(isCloudStorageEnabled,        bool());
+    MOCK_METHOD0(isCloudStorageEnabled,      bool());
     MOCK_METHOD0(isReverseProxy,             bool());
     MOCK_METHOD0(isKernelVersion3OrHigher,   bool());
+    MOCK_METHOD0(isGw,                       bool());
     MOCK_METHOD0(isGwNotVsx,                 bool());
     MOCK_METHOD0(getResolvedDetails,         std::map<std::string, std::string>());
-    MOCK_METHOD0(isVersionAboveR8110, bool());
-    MOCK_METHOD0(parseNginxMetadata, Maybe<std::tuple<std::string, std::string, std::string, std::string>>());
+    MOCK_METHOD0(isVersionAboveR8110,        bool());
+    MOCK_METHOD0(parseNginxMetadata,         Maybe<std::tuple<std::string, std::string, std::string, std::string>>());
     MOCK_METHOD0(
         readCloudMetadata, Maybe<std::tuple<std::string, std::string, std::string, std::string, std::string>>());
 };

components/security_apps/orchestration/manifest_controller/manifest_diff_calculator.cc

@@ -115,9 +115,9 @@ ManifestDiffCalculator::buildRecInstallationQueue(
     const map<string, Package> &current_packages,
     const map<string, Package> &new_packages)
 {
-    const vector<string> &requires = package.getRequire();
+    const vector<string> &requires_packages = package.getRequire();
 
-    for (const auto &require : requires) {
+    for (const auto &require : requires_packages) {
         auto installed_package = current_packages.find(require);
         auto new_package = new_packages.find(require);
 

components/security_apps/orchestration/orchestration_comp.cc

@@ -1471,7 +1471,8 @@ private:
             string cc_opt;
             tie(config_opt, cc_opt, nginx_version, nginx_signature) = nginx_data.unpack();
             agent_data_report
-                << make_pair("attachmentVersion", "Legacy")
+                << make_pair("configureOptStatus", "Enabled")
+                << make_pair("moduleSignatureStatus", "Enabled")
                 << make_pair("nginxSignature",    nginx_signature)
                 << make_pair("nginxVersion",      nginx_version)
                 << make_pair("configureOpt",      config_opt)
@@ -1496,6 +1497,10 @@ private:
             agent_data_report << AgentReportFieldWithLabel("isKernelVersion3OrHigher", "true");
         }
 
+        if (i_details_resolver->isGw()) {
+            agent_data_report << AgentReportFieldWithLabel("isGw", "true");
+        }
+
         if (i_details_resolver->isGwNotVsx()) {
             agent_data_report << AgentReportFieldWithLabel("isGwNotVsx", "true");
         }
@@ -1517,6 +1522,12 @@ private:
 
         agent_data_report << make_pair("registeredServer", i_agent_details->getRegisteredServer());
 
+        const char *prometheus_env = getenv("PROMETHEUS");
+        if (prometheus_env != nullptr) {
+            auto enable_prometheus = string(prometheus_env) == "true";
+            agent_data_report << AgentReportFieldWithLabel("enablePrometheus", enable_prometheus ? "true" : "false");
+        }
+
 #if defined(gaia) || defined(smb)
         if (i_details_resolver->compareCheckpointVersion(8100, greater_equal<int>())) {
             agent_data_report << AgentReportFieldWithLabel("isCheckpointVersionGER81", "true");
@@ -2267,4 +2278,4 @@ OrchestrationComp::preload()
     registerExpectedSetting<uint>("successUpgradeInterval");
     registerExpectedConfigFile("orchestration", Config::ConfigFileType::Policy);
     registerExpectedConfigFile("registration-data", Config::ConfigFileType::Policy);
-}
+}

components/security_apps/orchestration/orchestration_tools/orchestration_tools.cc

@@ -150,7 +150,8 @@ getNamespaceDataFromCluster()
     string auth_header = "Authorization: Bearer " + token;
     string connection_header = "Connection: close";
     string host = "https://kubernetes.default.svc:443/api/v1/namespaces/";
-    string culr_cmd = "curl -s -k -H \"" + auth_header + "\" -H \"" + connection_header + "\" " + host +
+    string culr_cmd =
+        "LD_LIBRARY_PATH=\"\" curl -s -k -H \"" + auth_header + "\" -H \"" + connection_header + "\" " + host +
         " | /etc/cp/bin/cpnano_json";
 
     auto output_res = Singleton::Consume<I_ShellCmd>::by<OrchestrationTools>()->getExecOutput(culr_cmd);

components/security_apps/orchestration/orchestration_tools/orchestration_tools_ut/orchestration_tools_ut.cc

@@ -86,7 +86,7 @@ TEST_F(OrchestrationToolsTest, setClusterId)
     EXPECT_CALL(
         mock_shell_cmd,
         getExecOutput(
-            "curl -s -k -H \"Authorization: Bearer 123\" -H \"Connection: close\" "
+            "LD_LIBRARY_PATH=\"\" curl -s -k -H \"Authorization: Bearer 123\" -H \"Connection: close\" "
             "https://kubernetes.default.svc:443/api/v1/namespaces/ | /etc/cp/bin/cpnano_json",
             200,
             false

components/security_apps/orchestration/orchestration_ut/orchestration_multitenant_ut.cc

@@ -145,6 +145,7 @@ public:
         EXPECT_CALL(mock_details_resolver, getArch()).WillRepeatedly(Return(string("x86_64")));
         EXPECT_CALL(mock_details_resolver, isReverseProxy()).WillRepeatedly(Return(false));
         EXPECT_CALL(mock_details_resolver, isKernelVersion3OrHigher()).WillRepeatedly(Return(false));
+        EXPECT_CALL(mock_details_resolver, isGw()).WillRepeatedly(Return(false));
         EXPECT_CALL(mock_details_resolver, isGwNotVsx()).WillRepeatedly(Return(false));
         EXPECT_CALL(mock_details_resolver, isVersionAboveR8110()).WillRepeatedly(Return(false));
         EXPECT_CALL(mock_details_resolver, parseNginxMetadata()).WillRepeatedly(Return(no_nginx));

components/security_apps/orchestration/orchestration_ut/orchestration_ut.cc

@@ -174,6 +174,7 @@ public:
         EXPECT_CALL(mock_details_resolver, isReverseProxy()).WillRepeatedly(Return(false));
         EXPECT_CALL(mock_details_resolver, isCloudStorageEnabled()).WillRepeatedly(Return(false));
         EXPECT_CALL(mock_details_resolver, isKernelVersion3OrHigher()).WillRepeatedly(Return(false));
+        EXPECT_CALL(mock_details_resolver, isGw()).WillRepeatedly(Return(false));
         EXPECT_CALL(mock_details_resolver, isGwNotVsx()).WillRepeatedly(Return(false));
         EXPECT_CALL(mock_details_resolver, isVersionAboveR8110()).WillRepeatedly(Return(false));
         EXPECT_CALL(mock_details_resolver, parseNginxMetadata()).WillRepeatedly(Return(no_nginx));

components/security_apps/orchestration/service_controller/service_controller.cc

@@ -209,6 +209,7 @@ ServiceDetails::sendNewConfigurations(int configuration_id, const string &policy
     new_config_req_md.setConnectioFlag(MessageConnectionConfig::ONE_TIME_CONN);
     new_config_req_md.setConnectioFlag(MessageConnectionConfig::UNSECURE_CONN);
     new_config_req_md.setSuspension(false);
+    new_config_req_md.setShouldSendAccessToken(false);
     auto res = messaging->sendSyncMessage(
         HTTPMethod::POST,
         "/set-new-configuration",

components/security_apps/orchestration/update_communication/fog_authenticator.cc

@@ -139,6 +139,25 @@ FogAuthenticator::RegistrationData::serialize(JSONOutputArchive &out_ar) const
     );
 }
 
+static string
+getDeplymentType()
+{
+    auto deplyment_type = Singleton::Consume<I_EnvDetails>::by<FogAuthenticator>()->getEnvType();
+    switch (deplyment_type) {
+        case EnvType::LINUX: return "Embedded";
+        case EnvType::DOCKER: return "Docker";
+        case EnvType::NON_CRD_K8S:
+        case EnvType::K8S: return "K8S";
+        case EnvType::COUNT: break;
+    }
+
+    dbgAssertOpt(false)
+        << AlertInfo(AlertTeam::CORE, "fog communication")
+        << "Failed to get a legitimate deployment type: "
+        << static_cast<uint>(deplyment_type);
+    return "Embedded";
+}
+
 Maybe<FogAuthenticator::UserCredentials>
 FogAuthenticator::registerAgent(
     const FogAuthenticator::RegistrationData &reg_data,
@@ -208,6 +227,18 @@ FogAuthenticator::registerAgent(
 
     request << make_pair("userEdition", getUserEdition());
 
+    const char *prometheus_env = getenv("PROMETHEUS");
+    if (prometheus_env != nullptr) {
+        request << make_pair("enablePrometheus", string(prometheus_env) == "true" ? "true" : "false");
+    }
+
+    if (getDeplymentType() == "Docker" || getDeplymentType() == "K8S") {
+        const char *image_version_otp = getenv("IMAGE_VERSION");
+        if (image_version_otp) {
+            request << make_pair("imageVersion", image_version_otp);
+        }
+    }
+
     if (details_resolver->isReverseProxy()) {
         request << make_pair("reverse_proxy", "true");
     }
@@ -220,6 +251,10 @@ FogAuthenticator::registerAgent(
         request << make_pair("isKernelVersion3OrHigher", "true");
     }
 
+    if (details_resolver->isGw()) {
+        request << make_pair("isGw", "true");
+    }
+
     if (details_resolver->isGwNotVsx()) {
         request << make_pair("isGwNotVsx", "true");
     }
@@ -283,11 +318,14 @@ FogAuthenticator::getAccessToken(const UserCredentials &user_credentials) const
     static const string grant_type_string = "/oauth/token?grant_type=client_credentials";
     TokenRequest request = TokenRequest();
 
-    MessageMetadata request_token_md;
+    MessageMetadata request_token_md(true);
     request_token_md.insertHeader(
         "Authorization",
         buildBasicAuthHeader(user_credentials.getClientId(), user_credentials.getSharedSecret())
     );
+    dbgInfo(D_ORCHESTRATOR)
+        << "Sending request for access token. Trace: "
+        << (request_token_md.getTraceId().ok() ? request_token_md.getTraceId().unpack() : "No trace id");
     auto request_token_status = Singleton::Consume<I_Messaging>::by<FogAuthenticator>()->sendSyncMessage(
         HTTPMethod::POST,
         grant_type_string,
@@ -461,25 +499,6 @@ FogAuthenticator::getCredentialsFromFile() const
     return orchestration_tools->jsonStringToObject<UserCredentials>(encrypted_cred.unpack());
 }
 
-static string
-getDeplymentType()
-{
-    auto deplyment_type = Singleton::Consume<I_EnvDetails>::by<FogAuthenticator>()->getEnvType();
-    switch (deplyment_type) {
-        case EnvType::LINUX: return "Embedded";
-        case EnvType::DOCKER: return "Docker";
-        case EnvType::NON_CRD_K8S:
-        case EnvType::K8S: return "K8S";
-        case EnvType::COUNT: break;
-    }
-
-    dbgAssertOpt(false)
-        << AlertInfo(AlertTeam::CORE, "fog communication")
-        << "Failed to get a legitimate deployment type: "
-        << static_cast<uint>(deplyment_type);
-    return "Embedded";
-}
-
 Maybe<FogAuthenticator::UserCredentials>
 FogAuthenticator::getCredentials()
 {

components/security_apps/prometheus/CMakeLists.txt

@@ -0,0 +1,2 @@
+add_library(prometheus_comp prometheus_comp.cc)
+add_subdirectory(prometheus_ut)

components/security_apps/prometheus/prometheus_comp.cc

@@ -0,0 +1,200 @@
+#include "prometheus_comp.h"
+
+#include <string>
+#include <map>
+#include <vector>
+#include <cereal/archives/json.hpp>
+#include <cereal/types/map.hpp>
+#include <cereal/types/vector.hpp>
+#include <cereal/types/string.hpp>
+#include <iostream>
+#include <fstream>
+
+#include "common.h"
+#include "report/base_field.h"
+#include "report/report_enums.h"
+#include "log_generator.h"
+#include "debug.h"
+#include "rest.h"
+#include "customized_cereal_map.h"
+#include "i_messaging.h"
+#include "prometheus_metric_names.h"
+
+USE_DEBUG_FLAG(D_PROMETHEUS);
+
+using namespace std;
+using namespace ReportIS;
+
+struct ServiceData
+{
+    template <typename Archive>
+    void
+    serialize(Archive &ar)
+    {
+        ar(cereal::make_nvp("Service port", service_port));
+    }
+
+    int service_port;
+};
+
+class PrometheusMetricData
+{
+public:
+    PrometheusMetricData(const string &n, const string &t, const string &d) : name(n), type(t), description(d) {}
+
+    void
+    addElement(const string &labels, const string &value)
+    {
+        metric_labels_to_values[labels] = value;
+    }
+
+    ostream &
+    print(ostream &os)
+    {
+        if (metric_labels_to_values.empty()) return os;
+
+        string representative_name = "";
+        if (!name.empty()) {
+            auto metric_name = convertMetricName(name);
+            !metric_name.empty() ? representative_name = metric_name : representative_name = name;
+        }
+
+        if (!description.empty()) os << "# HELP " << representative_name << ' ' << description << '\n';
+        if (!name.empty()) os << "# TYPE " << representative_name << ' ' << type << '\n';
+        for (auto &entry : metric_labels_to_values) {
+            os << representative_name << entry.first << ' ' << entry.second << '\n';
+        }
+        os << '\n';
+        metric_labels_to_values.clear();
+
+        return os;
+    }
+
+private:
+
+    string name;
+    string type;
+    string description;
+    map<string, string> metric_labels_to_values;
+};
+
+static ostream & operator<<(ostream &os, PrometheusMetricData &metric) { return metric.print(os); }
+
+class PrometheusComp::Impl
+{
+public:
+    void
+    init()
+    {
+        Singleton::Consume<I_RestApi>::by<PrometheusComp>()->addGetCall(
+            "metrics",
+            [&] () { return getFormatedPrometheusMetrics(); }
+        );
+    }
+
+    void
+    addMetrics(const vector<PrometheusData> &metrics)
+    {
+        for(auto &metric : metrics) {
+            auto &metric_object = getDataObject(
+                metric.name,
+                metric.type,
+                metric.description
+            );
+            metric_object.addElement(metric.label,  metric.value);
+        }
+    }
+
+private:
+    PrometheusMetricData &
+    getDataObject(const string &name, const string &type, const string &description)
+    {
+        auto elem = prometheus_metrics.find(name);
+        if (elem == prometheus_metrics.end()) {
+            elem = prometheus_metrics.emplace(name, PrometheusMetricData(name, type, description)).first;
+        }
+
+        return elem->second;
+    }
+
+    map<string, ServiceData>
+    getServiceDetails()
+    {
+        map<string, ServiceData> registeredServices;
+        auto registered_services_file = getConfigurationWithDefault<string>(
+            getFilesystemPathConfig() + "/conf/orchestrations_registered_services.json",
+            "orchestration",
+            "Orchestration registered services"
+        );
+        ifstream file(registered_services_file);
+        if (!file.is_open()) {
+            dbgWarning(D_PROMETHEUS) << "Failed to open file: " << registered_services_file;
+            return registeredServices;
+        }
+        stringstream buffer;
+        buffer << file.rdbuf();
+        try {
+            cereal::JSONInputArchive archive(buffer);
+            archive(cereal::make_nvp("Registered Services", registeredServices));
+        } catch (const exception& e) {
+            dbgWarning(D_PROMETHEUS) << "Error parsing Registered Services JSON file: " << e.what();
+        }
+
+        return registeredServices;
+    }
+
+    void
+    getServicesMetrics()
+    {
+        dbgTrace(D_PROMETHEUS) << "Get all registered services metrics";
+        map<string, ServiceData> service_names_to_ports = getServiceDetails();
+        for (const auto &service : service_names_to_ports) {
+            I_Messaging *messaging = Singleton::Consume<I_Messaging>::by<PrometheusComp>();
+            MessageMetadata servie_metric_req_md("127.0.0.1", service.second.service_port);
+            servie_metric_req_md.setConnectioFlag(MessageConnectionConfig::ONE_TIME_CONN);
+            servie_metric_req_md.setConnectioFlag(MessageConnectionConfig::UNSECURE_CONN);
+            auto res = messaging->sendSyncMessage(
+                HTTPMethod::GET,
+                "/service-metrics",
+                string(""),
+                MessageCategory::GENERIC,
+                servie_metric_req_md
+            );
+            if (!res.ok()) {
+                dbgWarning(D_PROMETHEUS) << "Failed to get service metrics. Service: " << service.first;
+                continue;
+            }
+            stringstream buffer;
+            buffer << res.unpack().getBody();
+            cereal::JSONInputArchive archive(buffer);
+            vector<PrometheusData> metrics;
+            archive(cereal::make_nvp("metrics", metrics));
+            addMetrics(metrics);
+        }
+    }
+
+    string
+    getFormatedPrometheusMetrics()
+    {
+        MetricScrapeEvent().notify();
+        getServicesMetrics();
+        stringstream result;
+        for (auto &metric : prometheus_metrics) {
+            result << metric.second;
+        }
+        dbgTrace(D_PROMETHEUS) << "Prometheus metrics: " << result.str();
+        return result.str();
+    }
+
+    map<string, PrometheusMetricData> prometheus_metrics;
+};
+
+PrometheusComp::PrometheusComp() : Component("Prometheus"), pimpl(make_unique<Impl>()) {}
+
+PrometheusComp::~PrometheusComp() {}
+
+void
+PrometheusComp::init()
+{
+    pimpl->init();
+}

components/security_apps/prometheus/prometheus_metric_names.h

@@ -0,0 +1,143 @@
+#ifndef __PROMETHEUS_METRIC_NAMES_H__
+#define __PROMETHEUS_METRIC_NAMES_H__
+
+#include <string>
+#include <unordered_map>
+
+#include "debug.h"
+
+USE_DEBUG_FLAG(D_PROMETHEUS);
+
+std::string
+convertMetricName(const std::string &original_metric_name)
+{
+    static const std::unordered_map<std::string, std::string> original_to_representative_names = {
+        // HybridModeMetric
+        {"watchdogProcessStartupEventsSum", "nano_service_restarts_counter"},
+        // nginxAttachmentMetric
+        {"inspectVerdictSum", "traffic_inspection_verdict_inspect_counter"},
+        {"acceptVeridctSum", "traffic_inspection_verdict_accept_counter"},
+        {"dropVerdictSum", "traffic_inspection_verdict_drop_counter"},
+        {"injectVerdictSum", "traffic_inspection_verdict_inject_counter"},
+        {"irrelevantVerdictSum", "traffic_inspection_verdict_irrelevant_counter"},
+        {"irrelevantVerdictSum", "traffic_inspection_verdict_irrelevant_counter"},
+        {"reconfVerdictSum", "traffic_inspection_verdict_reconf_counter"},
+        {"responseInspection", "response_body_inspection_counter"},
+        // nginxIntakerMetric
+        {"successfullInspectionTransactionsSum", "successful_Inspection_counter"},
+        {"failopenTransactionsSum", "fail_open_Inspection_counter"},
+        {"failcloseTransactionsSum", "fail_close_Inspection_counter"},
+        {"transparentModeTransactionsSum", "transparent_mode_counter"},
+        {"totalTimeInTransparentModeSum", "total_time_in_transparent_mode_counter"},
+        {"reachInspectVerdictSum", "inspect_verdict_counter"},
+        {"reachAcceptVerdictSum", "accept_verdict_counter"},
+        {"reachDropVerdictSum", "drop_verdict_counter"},
+        {"reachInjectVerdictSum", "inject_verdict_counter"},
+        {"reachIrrelevantVerdictSum", "irrelevant_verdict_counter"},
+        {"reachReconfVerdictSum", "reconf_verdict_counter"},
+        {"requestCompressionFailureSum", "failed_requests_compression_counter"},
+        {"responseCompressionFailureSum", "failed_response_compression_counter"},
+        {"requestDecompressionFailureSum", "failed_requests_decompression_counter"},
+        {"responseDecompressionFailureSum", "failed_response_decompression_counter"},
+        {"requestCompressionSuccessSum", "successful_request_compression_counter"},
+        {"responseCompressionSuccessSum", "successful_response_compression_counter"},
+        {"requestDecompressionSuccessSum", "successful_request_decompression_counter"},
+        {"responseDecompressionSuccessSum", "successful_response_decompression_counter"},
+        {"skippedSessionsUponCorruptedZipSum", "corrupted_zip_skipped_session_counter"},
+        {"attachmentThreadReachedTimeoutSum", "thread_exceeded_processing_time_counter"},
+        {"registrationThreadReachedTimeoutSum", "failed_registration_thread_counter"},
+        {"requestHeaderThreadReachedTimeoutSum", "request_headers_processing_thread_timeouts_counter"},
+        {"requestBodyThreadReachedTimeoutSum", "request_body_processing_thread_timeouts_counter"},
+        {"respondHeaderThreadReachedTimeoutSum", "response_headers_processing_thread_timeouts_counter"},
+        {"respondBodyThreadReachedTimeoutSum", "response_body_processing_thread_timeouts_counter"},
+        {"attachmentThreadFailureSum", "thread_failures_counter"},
+        {"httpRequestProcessingReachedTimeoutSum", "request_processing_timeouts_counter"},
+        {"httpRequestsSizeSum", "requests_total_size_counter"},
+        {"httpResponsesSizeSum", "response_total_size_counter"},
+        {"httpRequestFailedToReachWebServerUpstreamSum", "requests_failed_reach_upstram_counter"},
+        {"overallSessionProcessTimeToVerdictAvgSample", "overall_processing_time_until_verdict_average"},
+        {"overallSessionProcessTimeToVerdictMaxSample", "overall_processing_time_until_verdict_max"},
+        {"overallSessionProcessTimeToVerdictMinSample", "overall_processing_time_until_verdict_min"},
+        {"requestProcessTimeToVerdictAvgSample", "requests_processing_time_until_verdict_average"},
+        {"requestProcessTimeToVerdictMaxSample", "requests_processing_time_until_verdict_max"},
+        {"requestProcessTimeToVerdictMinSample", "requests_processing_time_until_verdict_min"},
+        {"responseProcessTimeToVerdictAvgSample", "response_processing_time_until_verdict_average"},
+        {"responseProcessTimeToVerdictMaxSample", "response_processing_time_until_verdict_max"},
+        {"responseProcessTimeToVerdictMinSample", "response_processing_time_until_verdict_min"},
+        {"requestBodySizeUponTimeoutAvgSample", "request_body_size_average"},
+        {"requestBodySizeUponTimeoutMaxSample", "request_body_size_max"},
+        {"requestBodySizeUponTimeoutMinSample", "request_body_size_min"},
+        {"responseBodySizeUponTimeoutAvgSample", "response_body_size_average"},
+        {"responseBodySizeUponTimeoutMaxSample", "response_body_size_max"},
+        {"responseBodySizeUponTimeoutMinSample", "response_body_size_min"},
+        // WaapTelemetrics
+        {"reservedNgenA", "total_requests_counter"},
+        {"reservedNgenB", "unique_sources_counter"},
+        {"reservedNgenC", "requests_blocked_by_force_and_exception_counter"},
+        {"reservedNgenD", "requests_blocked_by_waf_counter"},
+        {"reservedNgenE", "requests_blocked_by_open_api_counter"},
+        {"reservedNgenF", "requests_blocked_by_bot_protection_counter"},
+        {"reservedNgenG", "requests_threat_level_info_and_no_threat_counter"},
+        {"reservedNgenH", "requests_threat_level_low_counter"},
+        {"reservedNgenI", "requests_threat_level_medium_counter"},
+        {"reservedNgenJ", "requests_threat_level_high_counter"},
+        // WaapTrafficTelemetrics
+        {"reservedNgenA", "post_requests_counter"},
+        {"reservedNgenB", "get_requests_counter"},
+        {"reservedNgenC", "put_requests_counter"},
+        {"reservedNgenD", "patch_requests_counter"},
+        {"reservedNgenE", "delete_requests_counter"},
+        {"reservedNgenF", "other_requests_counter"},
+        {"reservedNgenG", "2xx_status_code_responses_counter"},
+        {"reservedNgenH", "4xx_status_code_responses_counter"},
+        {"reservedNgenI", "5xx_status_code_responses_counter"},
+        {"reservedNgenJ", "requests_time_latency_average"},
+        // WaapAttackTypesMetrics
+        {"reservedNgenA", "sql_injection_attacks_type_counter"},
+        {"reservedNgenB", "vulnerability_scanning_attacks_type_counter"},
+        {"reservedNgenC", "path_traversal_attacks_type_counter"},
+        {"reservedNgenD", "ldap_injection_attacks_type_counter"},
+        {"reservedNgenE", "evasion_techniques_attacks_type_counter"},
+        {"reservedNgenF", "remote_code_execution_attacks_type_counter"},
+        {"reservedNgenG", "xml_extern_entity_attacks_type_counter"},
+        {"reservedNgenH", "cross_site_scripting_attacks_type_counter"},
+        {"reservedNgenI", "general_attacks_type_counter"},
+        // AssetsMetric
+        {"numberOfProtectedApiAssetsSample", "api_assets_counter"},
+        {"numberOfProtectedWebAppAssetsSample", "web_api_assets_counter"},
+        {"numberOfProtectedAssetsSample", "all_assets_counter"},
+        // IPSMetric
+        {"preventEngineMatchesSample", "prevent_action_matches_counter"},
+        {"detectEngineMatchesSample", "detect_action_matches_counter"},
+        {"ignoreEngineMatchesSample", "ignore_action_matches_counter"},
+        // CPUMetric
+        {"cpuMaxSample", "cpu_usage_percentage_max"},
+        {"cpuAvgSample", "cpu_usage_percentage_average"},
+        {"cpuSample", "cpu_usage_percentage_last_value"},
+        // LogMetric
+        {"logQueueMaxSizeSample", "logs_queue_size_max"},
+        {"logQueueAvgSizeSample", "logs_queue_size_average"},
+        {"logQueueCurrentSizeSample", "logs_queue_size_last_value"},
+        {"sentLogsSum", "logs_sent_counter"},
+        {"sentLogsBulksSum", "bulk_logs_sent_counter"},
+        // MemoryMetric
+        {"serviceVirtualMemorySizeMaxSample", "service_virtual_memory_size_kb_max"},
+        {"serviceVirtualMemorySizeMinSample", "service_virtual_memory_size_kb_min"},
+        {"serviceVirtualMemorySizeAvgSample", "service_virtual_memory_size_kb_average"},
+        {"serviceRssMemorySizeMaxSample", "service_physical_memory_size_kb_max"},
+        {"serviceRssMemorySizeMinSample", "service_physical_memory_size_kb_min"},
+        {"serviceRssMemorySizeAvgSample", "service_physical_memory_size_kb_average"},
+        {"generalTotalMemorySizeMaxSample", "general_total_used_memory_max"},
+        {"generalTotalMemorySizeMinSample", "general_total_used_memory_min"},
+        {"generalTotalMemorySizeAvgSample", "general_total_used_memory_average"},
+    };
+
+    auto metric_names = original_to_representative_names.find(original_metric_name);
+    if (metric_names != original_to_representative_names.end()) return metric_names->second;
+    dbgDebug(D_PROMETHEUS)
+        << "Metric don't have a representative name, originl name: "
+        << original_metric_name;
+    return "";
+}
+
+#endif // __PROMETHEUS_METRIC_NAMES_H__

components/security_apps/prometheus/prometheus_ut/CMakeLists.txt

@@ -0,0 +1,8 @@
+link_directories(${BOOST_ROOT}/lib)
+link_directories(${BOOST_ROOT}/lib ${CMAKE_BINARY_DIR}/core/shmem_ipc)
+
+add_unit_test(
+    prometheus_ut
+    "prometheus_ut.cc"
+    "prometheus_comp;logging;agent_details;waap_clib;table;singleton;time_proxy;metric;event_is;connkey;http_transaction_data;generic_rulebase;generic_rulebase_evaluators;ip_utilities;intelligence_is_v2;-lboost_regex;messaging;"
+)

components/security_apps/prometheus/prometheus_ut/prometheus_ut.cc

@@ -0,0 +1,79 @@
+#include "prometheus_comp.h"
+
+#include <sstream>
+#include <fstream>
+#include <vector>
+
+#include "cmock.h"
+#include "cptest.h"
+#include "maybe_res.h"
+#include "debug.h"
+#include "config.h"
+#include "environment.h"
+#include "config_component.h"
+#include "agent_details.h"
+#include "time_proxy.h"
+#include "mock/mock_mainloop.h"
+#include "mock/mock_rest_api.h"
+#include "mock/mock_messaging.h"
+
+using namespace std;
+using namespace testing;
+
+USE_DEBUG_FLAG(D_PROMETHEUS);
+
+class PrometheusCompTest : public Test
+{
+public:
+    PrometheusCompTest()
+    {
+        EXPECT_CALL(mock_rest, mockRestCall(_, "declare-boolean-variable", _)).WillOnce(Return(false));
+        env.preload();
+        config.preload();
+        env.init();
+
+        EXPECT_CALL(
+            mock_rest,
+            addGetCall("metrics", _)
+        ).WillOnce(DoAll(SaveArg<1>(&get_metrics_func), Return(true)));
+
+        prometheus_comp.init();
+    }
+
+    ::Environment            env;
+    ConfigComponent          config;
+    PrometheusComp           prometheus_comp;
+    StrictMock<MockRestApi>  mock_rest;
+    StrictMock<MockMainLoop> mock_ml;
+    NiceMock<MockMessaging>  mock_messaging;
+    unique_ptr<ServerRest>   agent_uninstall;
+    function<string()>       get_metrics_func;
+    CPTestTempfile           status_file;
+    string                   registered_services_file_path;
+
+};
+
+TEST_F(PrometheusCompTest, checkAddingMetric)
+{
+    registered_services_file_path = cptestFnameInSrcDir(string("registered_services.json"));
+    setConfiguration(registered_services_file_path, "orchestration", "Orchestration registered services");
+    string metric_body = "{\n"
+        "   \"metrics\": [\n"
+        "       {\n"
+        "           \"metric_name\": \"watchdogProcessStartupEventsSum\",\n"
+        "           \"metric_type\": \"counter\",\n"
+        "           \"metric_description\": \"\",\n"
+        "           \"labels\": \"{method=\\\"post\\\",code=\\\"200\\\"}\",\n"
+        "           \"value\": \"1534\"\n"
+        "       }\n"
+        "   ]\n"
+        "}";
+
+    string message_body;
+    EXPECT_CALL(mock_messaging, sendSyncMessage(_, "/service-metrics", _, _, _))
+        .Times(2).WillRepeatedly(Return(HTTPResponse(HTTPStatusCode::HTTP_OK, metric_body)));
+
+    string metric_str = "# TYPE nano_service_restarts_counter counter\n"
+        "nano_service_restarts_counter{method=\"post\",code=\"200\"} 1534\n\n";
+    EXPECT_EQ(metric_str,  get_metrics_func());
+}

components/security_apps/prometheus/prometheus_ut/registered_services.json

@@ -0,0 +1,32 @@
+{
+    "Registered Services": {
+        "cp-nano-orchestration": {
+            "Service name": "cp-nano-orchestration",
+            "Service ID": "cp-nano-orchestration",
+            "Service port": 7777,
+            "Relevant configs": [
+                "zones",
+                "triggers",
+                "rules",
+                "registration-data",
+                "parameters",
+                "orchestration",
+                "exceptions",
+                "agent-intelligence"
+            ]
+        },
+        "cp-nano-prometheus": {
+            "Service name": "cp-nano-prometheus",
+            "Service ID": "cp-nano-prometheus",
+            "Service port": 7465,
+            "Relevant configs": [
+                "zones",
+                "triggers",
+                "rules",
+                "parameters",
+                "exceptions",
+                "agent-intelligence"
+            ]
+        }
+    }
+}

components/security_apps/rate_limit/rate_limit.cc

@@ -250,13 +250,14 @@ public:
     fetchReplicaCount()
     {
         string curl_cmd =
-            "curl -H \"Authorization: Bearer " + kubernetes_token + "\" "
+            base_curl_cmd + " -H \"Authorization: Bearer " + kubernetes_token + "\" "
             "https://kubernetes.default.svc.cluster.local/apis/apps/v1/namespaces/" + kubernetes_namespace +
-            "/deployments/${AGENT_DEPLOYMENT_NAME} -k  -s | jq .status.replicas";
+            "/deployments/${AGENT_DEPLOYMENT_NAME} -k -s | jq .status.replicas";
         auto maybe_replicas = i_shell_cmd->getExecOutput(curl_cmd);
         if (maybe_replicas.ok()) {
             try {
                 replicas = std::stoi(maybe_replicas.unpack());
+                dbgTrace(D_RATE_LIMIT) << "replicas is set to " << replicas;
             } catch (const std::exception &e) {
                 dbgWarning(D_RATE_LIMIT) << "error while converting replicas: " << e.what();
             }
@@ -706,7 +707,9 @@ public:
         i_shell_cmd = Singleton::Consume<I_ShellCmd>::by<RateLimit>();
         i_env_details = Singleton::Consume<I_EnvDetails>::by<RateLimit>();
         env_type = i_env_details->getEnvType();
-        if (env_type == EnvType::K8S) {
+        const char *nexus_env = getenv("KUBERNETES_METADATA");
+        if (nexus_env == nullptr) return;
+        if (env_type == EnvType::K8S && string(nexus_env) == "true") {
             kubernetes_token = i_env_details->getToken();
             kubernetes_namespace = i_env_details->getNameSpace();
             fetchReplicaCount();
@@ -742,6 +745,13 @@ private:
     EnvType env_type;
     string kubernetes_namespace = "";
     string kubernetes_token = "";
+#if defined(gaia)
+    const string base_curl_cmd = "curl_cli";
+#elif defined(alpine)
+    const string base_curl_cmd = "LD_LIBRARY_PATH=/usr/lib/:/usr/lib/cpnano curl";
+#else
+    const string base_curl_cmd = "curl";
+#endif
 };
 
 RateLimit::RateLimit() : Component("RateLimit"), pimpl(make_unique<Impl>()) {}

components/security_apps/waap/include/i_serialize.h

@@ -23,7 +23,7 @@
 static const uint max_send_obj_retries = 3;
 static const std::chrono::microseconds wait_next_attempt(5000000);
 
-USE_DEBUG_FLAG(D_WAAP);
+USE_DEBUG_FLAG(D_WAAP_SERIALIZE);
 
 class RestGetFile : public ClientRest
 {
@@ -151,13 +151,14 @@ protected:
         I_Messaging *messaging = Singleton::Consume<I_Messaging>::by<WaapComponent>();
         I_AgentDetails *agentDetails = Singleton::Consume<I_AgentDetails>::by<WaapComponent>();
         if (agentDetails->getOrchestrationMode() == OrchestrationMode::OFFLINE) {
-            dbgDebug(D_WAAP) << "offline mode not sending object";
+            dbgDebug(D_WAAP_SERIALIZE) << "offline mode not sending object";
             return false;
         }
         if (agentDetails->getOrchestrationMode() == OrchestrationMode::HYBRID) {
             MessageMetadata req_md(getSharedStorageHost(), 80);
             req_md.insertHeader("X-Tenant-Id", agentDetails->getTenantId());
             req_md.setConnectioFlag(MessageConnectionConfig::UNSECURE_CONN);
+            req_md.setConnectioFlag(MessageConnectionConfig::ONE_TIME_CONN);
             auto req_status = messaging->sendSyncMessage(
                 method,
                 uri,
@@ -166,19 +167,22 @@ protected:
                 req_md
             );
             if (!req_status.ok()) {
-                dbgWarning(D_WAAP) << "failed to send request to uri: " << uri
+                dbgWarning(D_WAAP_SERIALIZE) << "failed to send request to uri: " << uri
                     << ", error: " << req_status.getErr().toString();
             }
             return req_status.ok();
         }
+        MessageMetadata req_md;
+        req_md.setConnectioFlag(MessageConnectionConfig::ONE_TIME_FOG_CONN);
         auto req_status = messaging->sendSyncMessage(
             method,
             uri,
             obj,
-            MessageCategory::GENERIC
+            MessageCategory::GENERIC,
+            req_md
         );
         if (!req_status.ok()) {
-            dbgWarning(D_WAAP) << "failed to send request to uri: " << uri
+            dbgWarning(D_WAAP_SERIALIZE) << "failed to send request to uri: " << uri
                 << ", error: " << req_status.getErr().toString();
         }
         return req_status.ok();
@@ -192,14 +196,14 @@ protected:
         {
             if (sendObject(obj, method, uri))
             {
-                dbgTrace(D_WAAP) <<
+                dbgTrace(D_WAAP_SERIALIZE) <<
                     "object sent successfully after " << i << " retry attempts";
                 return true;
             }
-            dbgInfo(D_WAAP) << "Failed to send object. Attempt: " << i;
+            dbgInfo(D_WAAP_SERIALIZE) << "Failed to send object. Attempt: " << i;
             mainloop->yield(wait_next_attempt);
         }
-        dbgWarning(D_WAAP) << "Failed to send object to " << uri << ", reached maximum attempts: " <<
+        dbgWarning(D_WAAP_SERIALIZE) << "Failed to send object to " << uri << ", reached maximum attempts: " <<
             max_send_obj_retries;
         return false;
     }
@@ -210,13 +214,14 @@ protected:
         I_Messaging *messaging = Singleton::Consume<I_Messaging>::by<WaapComponent>();
         I_AgentDetails *agentDetails = Singleton::Consume<I_AgentDetails>::by<WaapComponent>();
         if (agentDetails->getOrchestrationMode() == OrchestrationMode::OFFLINE) {
-            dbgDebug(D_WAAP) << "offline mode not sending object";
+            dbgDebug(D_WAAP_SERIALIZE) << "offline mode not sending object";
             return false;
         }
         if (agentDetails->getOrchestrationMode() == OrchestrationMode::HYBRID) {
             MessageMetadata req_md(getSharedStorageHost(), 80);
             req_md.insertHeader("X-Tenant-Id", agentDetails->getTenantId());
             req_md.setConnectioFlag(MessageConnectionConfig::UNSECURE_CONN);
+            req_md.setConnectioFlag(MessageConnectionConfig::ONE_TIME_CONN);
             return messaging->sendSyncMessageWithoutResponse(
                 method,
                 uri,
@@ -225,11 +230,14 @@ protected:
                 req_md
             );
         }
+        MessageMetadata req_md;
+        req_md.setConnectioFlag(MessageConnectionConfig::ONE_TIME_FOG_CONN);
         return messaging->sendSyncMessageWithoutResponse(
             method,
             uri,
             obj,
-            MessageCategory::GENERIC
+            MessageCategory::GENERIC,
+            req_md
         );
     }
 
@@ -241,14 +249,14 @@ protected:
         {
             if (sendNoReplyObject(obj, method, uri))
             {
-                dbgTrace(D_WAAP) <<
+                dbgTrace(D_WAAP_SERIALIZE) <<
                     "object sent successfully after " << i << " retry attempts";
                 return true;
             }
-            dbgInfo(D_WAAP) << "Failed to send object. Attempt: " << i;
+            dbgInfo(D_WAAP_SERIALIZE) << "Failed to send object. Attempt: " << i;
             mainloop->yield(wait_next_attempt);
         }
-        dbgWarning(D_WAAP) << "Failed to send object to " << uri << ", reached maximum attempts: " <<
+        dbgWarning(D_WAAP_SERIALIZE) << "Failed to send object to " << uri << ", reached maximum attempts: " <<
             max_send_obj_retries;
         return false;
     }
@@ -257,6 +265,7 @@ protected:
     std::chrono::seconds m_interval;
     std::string m_owner;
     const std::string m_assetId;
+    bool m_remoteSyncEnabled;
 
 private:
     bool localSyncAndProcess();
@@ -272,7 +281,6 @@ private:
     size_t m_daysCount;
     size_t m_windowsCount;
     size_t m_intervalsCounter;
-    bool m_remoteSyncEnabled;
     const bool m_isAssetIdUuid;
     std::string m_type;
     std::string m_lastProcessedModified;

components/security_apps/waap/include/i_transaction.h

@@ -84,6 +84,7 @@ public:
     virtual const std::string getUri() const = 0;
     virtual const std::string getUriStr() const = 0;
     virtual const std::string& getSourceIdentifier() const = 0;
+    virtual const std::string getCurrentWebUserResponse() = 0;
     virtual double getScore() const = 0;
     virtual double getOtherModelScore() const = 0;
     virtual const std::vector<double> getScoreArray() const = 0;
@@ -130,6 +131,7 @@ public:
     virtual void add_request_body_chunk(const char* data, int data_len) = 0;
     virtual void end_request_body() = 0;
     virtual void end_request() = 0;
+    virtual bool shouldLimitResponseHeadersInspection() = 0;
     // Response
     virtual void start_response(int response_status, int http_version) = 0;
     virtual void start_response_hdrs() = 0;
@@ -145,4 +147,7 @@ public:
     virtual ReportIS::Severity computeEventSeverityFromDecision() const = 0;
     virtual void finish() = 0;
     virtual Waf2TransactionFlags &getTransactionFlags() = 0;
+
+    virtual void setTemperatureDetected(bool detected) = 0;
+    virtual bool wasTemperatureDetected() const = 0;
 };

components/security_apps/waap/include/i_waapConfig.h

@@ -26,7 +26,6 @@
 #include "../waap_clib/SecurityHeadersPolicy.h"
 #include <memory>
 
-
 enum class BlockingLevel {
     NO_BLOCKING = 0,
     LOW_BLOCKING_LEVEL,

components/security_apps/waap/waap_clib/AutonomousSecurityDecision.cc

@@ -19,7 +19,6 @@ AutonomousSecurityDecision::AutonomousSecurityDecision(DecisionType type) :
     m_fpMitigationScore(0.0f),
     m_finalScore(0.0f),
     m_threatLevel(NO_THREAT),
-    m_overridesLog(false),
     m_relativeReputationMean(0.0),
     m_variance(0.0)
 {}
@@ -52,10 +51,6 @@ void AutonomousSecurityDecision::setThreatLevel(ThreatLevel threatLevel)
     m_threatLevel = threatLevel;
 }
 
-void AutonomousSecurityDecision::setOverridesLog(bool overridesLog)
-{
-    m_overridesLog = overridesLog;
-}
 void AutonomousSecurityDecision::setRelativeReputationMean(double relativeReputationMean)
 {
     m_relativeReputationMean = relativeReputationMean;
@@ -80,10 +75,6 @@ ThreatLevel AutonomousSecurityDecision::getThreatLevel() const
 {
     return m_threatLevel;
 }
-bool AutonomousSecurityDecision::getOverridesLog() const
-{
-    return m_overridesLog;
-}
 double AutonomousSecurityDecision::getRelativeReputationMean() const
 {
     return m_relativeReputationMean;

components/security_apps/waap/waap_clib/AutonomousSecurityDecision.h

@@ -30,14 +30,12 @@ public:
     void setFpMitigationScore(double fpMitigationScore);
     void setFinalScore(double finalScore);
     void setThreatLevel(ThreatLevel threatLevel);
-    void setOverridesLog(bool overridesLog);
     void setRelativeReputationMean(double relativeReputationMean);
     void setVariance(double variance);
     double getRelativeReputation() const;
     double getFpMitigationScore() const;
     double getFinalScore() const;
     ThreatLevel getThreatLevel() const;
-    bool getOverridesLog() const;
     double getRelativeReputationMean() const;
     double getVariance() const;
 
@@ -46,7 +44,6 @@ private:
     double m_fpMitigationScore;
     double m_finalScore;
     ThreatLevel m_threatLevel;
-    bool m_overridesLog;
     double m_relativeReputationMean;
     double m_variance;
 };

components/security_apps/waap/waap_clib/CMakeLists.txt

@@ -12,6 +12,7 @@ add_library(waap_clib
     ParserJson.cc
     ParserMultipartForm.cc
     ParserRaw.cc
+    ParserGzip.cc
     ParserUrlEncode.cc
     ParserXML.cc
     ParserDelimiter.cc

components/security_apps/waap/waap_clib/CidrMatch.cc

@@ -41,6 +41,7 @@ static in6_addr applyMaskV6(const in6_addr& addr, uint8_t prefixLength) {
     in6_addr maskedAddr = addr;
     int fullBytes = prefixLength / 8;
     int remainingBits = prefixLength % 8;
+    uint8_t partialByte = maskedAddr.s6_addr[fullBytes];
 
     // Mask full bytes
     for (int i = fullBytes; i < 16; ++i) {
@@ -50,7 +51,7 @@ static in6_addr applyMaskV6(const in6_addr& addr, uint8_t prefixLength) {
     // Mask remaining bits
     if (remainingBits > 0) {
         uint8_t mask = ~((1 << (8 - remainingBits)) - 1);
-        maskedAddr.s6_addr[fullBytes] &= mask;
+        maskedAddr.s6_addr[fullBytes] = partialByte & mask;
     }
 
     return maskedAddr;

components/security_apps/waap/waap_clib/ConfidenceCalculator.h

@@ -28,6 +28,8 @@
 #include "i_ignoreSources.h"
 #include "TuningDecisions.h"
 
+static constexpr size_t defaultConfidenceMemUsage = 40 * 1024 * 1024; // 40MB
+
 USE_DEBUG_FLAG(D_WAAP_CONFIDENCE_CALCULATOR);
 
 class WaapComponent;
@@ -39,9 +41,10 @@ struct ConfidenceCalculatorParams
     std::chrono::minutes intervalDuration;
     double ratioThreshold;
     bool learnPermanently;
+    size_t maxMemoryUsage;
 
     template <class Archive>
-    void serialize(Archive& ar)
+    void serialize(Archive &ar)
     {
         size_t duration = intervalDuration.count();
         ar(cereal::make_nvp("minSources", minSources),
@@ -50,10 +53,17 @@ struct ConfidenceCalculatorParams
             cereal::make_nvp("ratioThreshold", ratioThreshold),
             cereal::make_nvp("learnPermanently", learnPermanently));
         intervalDuration = std::chrono::minutes(duration);
+        try {
+            ar(cereal::make_nvp("maxMemoryUsage", maxMemoryUsage));
+        } catch (cereal::Exception &e) {
+            maxMemoryUsage = defaultConfidenceMemUsage;
+            dbgTrace(D_WAAP_CONFIDENCE_CALCULATOR) << "maxMemoryUsage not found in serialized data";
+            ar.setNextName(nullptr);
+        }
     }
 
-    bool operator==(const ConfidenceCalculatorParams& other);
-    friend std::ostream& operator<<(std::ostream& os, const ConfidenceCalculatorParams& ccp);
+    bool operator==(const ConfidenceCalculatorParams &other);
+    friend std::ostream & operator<<(std::ostream &os, const ConfidenceCalculatorParams &ccp);
 };
 
 class ConfidenceCalculator : public SerializeToLocalAndRemoteSyncBase
@@ -74,7 +84,6 @@ public:
     typedef std::list<ValuesSet> ValuesList;
     typedef UMap<Key, ValuesList> WindowsConfidentValuesList;
     typedef UMap<Key, UMap<Val, double>> ConfidenceLevels;
-    typedef UMap<Key, int> WindowsCounter;
     typedef UMap<Key, ValueSetWithTime> ConfidenceSet;
 
     ConfidenceCalculator(size_t minSources,
@@ -82,19 +91,19 @@ public:
         std::chrono::minutes intervalDuration,
         double ratioThreshold,
         const Val &nullObj,
-        const std::string& backupPath,
-        const std::string& remotePath,
-        const std::string& assetId,
+        const std::string &backupPath,
+        const std::string &remotePath,
+        const std::string &assetId,
         TuningDecision* tuning = nullptr,
         I_IgnoreSources* ignoreSrc = nullptr);
 
     ~ConfidenceCalculator();
 
-    void setOwner(const std::string& owner);
+    void setOwner(const std::string &owner);
 
     void hardReset();
     void reset();
-    bool reset(ConfidenceCalculatorParams& params);
+    bool reset(ConfidenceCalculatorParams &params);
 
     virtual bool postData();
     virtual void pullData(const std::vector<std::string>& files);
@@ -103,10 +112,12 @@ public:
     virtual void pullProcessedData(const std::vector<std::string>& files);
     virtual void updateState(const std::vector<std::string>& files);
 
-    virtual void serialize(std::ostream& stream);
-    virtual void deserialize(std::istream& stream);
+    virtual void serialize(std::ostream &stream);
+    virtual void deserialize(std::istream &stream);
 
-    void mergeFromRemote(const ConfidenceSet& remote_confidence_set, bool is_first_pull);
+    Maybe<void> writeToFile(const std::string& path, const std::vector<unsigned char>& data);
+
+    void mergeFromRemote(const ConfidenceSet &remote_confidence_set, bool is_first_pull);
 
     bool is_confident(const Key &key, const Val &value) const;
 
@@ -121,35 +132,50 @@ public:
 
     void calculateInterval();
 
-    static void mergeConfidenceSets(ConfidenceSet& confidence_set,
-                                    const ConfidenceSet& confidence_set_to_merge,
-                                    size_t& last_indicators_update);
+    static void mergeConfidenceSets(ConfidenceSet &confidence_set,
+                                    const ConfidenceSet &confidence_set_to_merge,
+                                    size_t &last_indicators_update);
 private:
-    void loadVer0(cereal::JSONInputArchive& archive);
-    void loadVer1(cereal::JSONInputArchive& archive);
-    void loadVer2(cereal::JSONInputArchive& archive);
-    void loadVer3(cereal::JSONInputArchive& archive);
+    void loadVer0(cereal::JSONInputArchive &archive);
+    void loadVer1(cereal::JSONInputArchive &archive);
+    void loadVer2(cereal::JSONInputArchive &archive);
+    void loadVer3(cereal::JSONInputArchive &archive);
     bool tryParseVersionBasedOnNames(
-        cereal::JSONInputArchive& archive,
+        cereal::JSONInputArchive &archive,
         const std::string &params_field_name,
         const std::string &indicators_update_field_name,
         const std::string &windows_summary_field_name,
         const std::string &confident_sets_field_name);
-    void convertWindowSummaryToConfidenceLevel(const WindowsConfidentValuesList& windows);
+    void convertWindowSummaryToConfidenceLevel(const WindowsConfidentValuesList &windows);
 
-    std::string getParamName(const Key& key);
-    size_t sumSourcesWeight(const SourcesSet& sources);
-    void mergeSourcesCounter(const Key& key, const SourcesCounters& counters);
-    void removeBadSources(SourcesSet& sources, const std::vector<std::string>* badSources);
+    void loadConfidenceLevels();
+    void saveConfidenceLevels(Maybe<ConfidenceCalculator::ConfidenceLevels> confidenceLevels);
+    void saveConfidenceLevels();
+
+    void saveTimeWindowLogger();
+    std::shared_ptr<KeyValSourcesLogger> loadTimeWindowLogger();
+
+    std::string getParamName(const Key &key);
+    size_t sumSourcesWeight(const SourcesSet &sources);
+    void removeBadSources(SourcesSet &sources, const std::vector<std::string>* badSources);
+
+    // Delete existing carry-on data files asynchronously with yields
+    void garbageCollector();
 
     ConfidenceCalculatorParams m_params;
     Val m_null_obj;
-    KeyValSourcesLogger m_time_window_logger;
-    KeyValSourcesLogger m_time_window_logger_backup;
+    std::shared_ptr<KeyValSourcesLogger> m_time_window_logger;
+    std::shared_ptr<KeyValSourcesLogger> m_time_window_logger_backup;
+    std::string m_path_to_backup;
     ConfidenceSet m_confident_sets;
     ConfidenceLevels m_confidence_level;
-    WindowsCounter m_windows_counter;
     size_t m_last_indicators_update;
+    size_t m_latest_index;
     I_IgnoreSources* m_ignoreSources;
     TuningDecision* m_tuning;
+    size_t m_estimated_memory_usage; // Variable to track estimated memory usage
+    size_t m_post_index;
+    I_MainLoop *m_mainLoop;
+    I_MainLoop::RoutineID m_routineId;
+    std::vector<std::string> m_filesToRemove;
 };

components/security_apps/waap/waap_clib/Csrf.cc

@@ -45,6 +45,21 @@ State::decide
     }
 
     auto csrfDecision = decision.getDecision(CSRF_DECISION);
+    auto autonomousDecision = decision.getDecision(AUTONOMOUS_SECURITY_DECISION);
+    if (autonomousDecision->shouldForceBlock())
+    {
+        dbgTrace(D_WAAP) << "Waap::CSRF::State::decide(): Autonomous decision force should block.";
+        csrfDecision->setBlock(true);
+        csrfDecision->setForceBlock(true);
+        return true;
+    }
+    if (autonomousDecision->shouldForceAllow())
+    {
+        dbgTrace(D_WAAP) << "Waap::CSRF::State::decide(): Autonomous decision force should allow.";
+        csrfDecision->setBlock(false);
+        csrfDecision->setForceAllow(true);
+        return false;
+    }
     if (csrf_token.empty())
     {
         dbgTrace(D_WAAP) << "Waap::CSRF::State::decide(): missing token.";

components/security_apps/waap/waap_clib/DecisionType.h

@@ -14,6 +14,8 @@
 #ifndef __DECISION_TYPE_H__
 #define __DECISION_TYPE_H__
 
+#include <ostream>
+
 enum DecisionType
 {
     // This order determines the priority of the decisions sent to management
@@ -28,4 +30,35 @@ enum DecisionType
     // Must be kept last
     NO_WAAP_DECISION
 };
+
+inline const char *
+decisionTypeToString(DecisionType type)
+{
+    switch (type) {
+        case DecisionType::AUTONOMOUS_SECURITY_DECISION:
+            return "AUTONOMOUS_SECURITY_DECISION";
+        case DecisionType::CSRF_DECISION:
+            return "CSRF_DECISION";
+        case DecisionType::OPEN_REDIRECT_DECISION:
+            return "OPEN_REDIRECT_DECISION";
+        case DecisionType::ERROR_DISCLOSURE_DECISION:
+            return "ERROR_DISCLOSURE_DECISION";
+        case DecisionType::ERROR_LIMITING_DECISION:
+            return "ERROR_LIMITING_DECISION";
+        case DecisionType::USER_LIMITS_DECISION:
+            return "USER_LIMITS_DECISION";
+        case DecisionType::RATE_LIMITING_DECISION:
+            return "RATE_LIMITING_DECISION";
+        case DecisionType::NO_WAAP_DECISION:
+            return "NO_WAAP_DECISION";
+        default:
+            return "INVALID_DECISION_TYPE";
+    }
+}
+
+inline std::ostream & operator<<(std::ostream& os, const DecisionType& type)
+{
+    return os << decisionTypeToString(type);
+}
+
 #endif

components/security_apps/waap/waap_clib/DeepParser.cc

@@ -22,6 +22,7 @@
 #include "ParserXML.h"
 #include "ParserHTML.h"
 #include "ParserBinary.h"
+#include "ParserGzip.h"
 #include "ParserMultipartForm.h"
 #include "ParserPercentEncode.h"
 #include "ParserPairs.h"
@@ -36,6 +37,7 @@
 #include "debug.h"
 #include "i_transaction.h"
 #include "agent_core_utilities.h"
+#include <boost/algorithm/string.hpp>
 
 USE_DEBUG_FLAG(D_WAAP_DEEP_PARSER);
 USE_DEBUG_FLAG(D_WAAP_ULIMITS);
@@ -93,6 +95,12 @@ DeepParser::depth() const
     return m_depth;
 }
 
+static bool err = false;
+static const SingleRegex temperature_value_re(
+    "^\\s*([0-9](?:\\.\\d+)?)\\s*$",
+    err,
+    "temperature_value");
+
 // Called when another key/value pair is ready
 int
 DeepParser::onKv(const char *k, size_t k_len, const char *v, size_t v_len, int flags, size_t parser_depth)
@@ -195,6 +203,14 @@ DeepParser::onKv(const char *k, size_t k_len, const char *v, size_t v_len, int f
     bool isBodyPayload = (m_key.first().size() == 4 && m_key.first() == "body");
 
 
+    if (isBodyPayload && v_len < 32 && k_len == 11 &&
+        boost::to_lower_copy(std::string(k, k_len)) == "temperature" &&
+        temperature_value_re.hasMatch(std::string(v, v_len))) {
+        m_pTransaction->setTemperatureDetected(true);
+        dbgTrace(D_WAAP_DEEP_PARSER) << "temperature detected, value: " << std::string(v, v_len);
+    }
+
+
     // If csrf/antibot cookie - send to Waf2Transaction for collection of cookie value.
     if (m_depth == 1 && isCookiePayload && (m_key.str() == "x-chkp-csrf-token" || m_key.str() == "__fn1522082288")) {
         std::string cur_val = std::string(v, v_len);
@@ -288,6 +304,11 @@ DeepParser::onKv(const char *k, size_t k_len, const char *v, size_t v_len, int f
             dbgTrace(D_WAAP_DEEP_PARSER) << "removing leading '/' from URL param value";
             base64_offset = 1;
         }
+        if (m_depth == 1 && (isUrlParamPayload || isRefererParamPayload) &&
+            k_len != 0  && (v_len == 0 || (v[0] == '=' && v_len == 1))) {
+            // if the value is empty or starts with '=' - replace it with key
+            cur_val = std::string(k, k_len);
+        }
         std::string decoded_val, decoded_key;
         base64_variants base64_status = Waap::Util::b64Test(
             cur_val,
@@ -477,6 +498,19 @@ DeepParser::onKv(const char *k, size_t k_len, const char *v, size_t v_len, int f
         }
     }
 
+    // If this is url_paran and key is match to nosql_key_evasion_detector_re and this is 1st and last buffer
+    // than add to beginning of cur_val "<key>=" where key is the key
+    if (flags == BUFFERED_RECEIVER_F_BOTH) {
+        std::string key = std::string(k, k_len);
+        if (Waap::Util::testNoSQLKeySuspect(key)) {
+            cur_val = key + "=" + cur_val;
+            dbgTrace(D_WAAP_DEEP_PARSER)
+                << "DeepParser::onKv(): found: key = "
+                << key
+                << " is a candidate for NoSQL key evasion - sending to updated string for scanning.";
+        }
+    }
+
     // If there's a parser in parsers stack, push the value to the top parser
     if (!m_parsersDeque.empty()
         && offset >= 0
@@ -1228,6 +1262,10 @@ DeepParser::createInternalParser(
         dbgTrace(D_WAAP_DEEP_PARSER) << "Starting to parse an HTML file";
         m_parsersDeque.push_back(std::make_shared<BufferedParser<ParserHTML>>(*this, parser_depth + 1));
         offset = 0;
+    } else if (isBodyPayload && Waap::Util::isGzipped(cur_val)){
+        dbgTrace(D_WAAP_DEEP_PARSER) << "Starting to parse a gzip file";
+        m_parsersDeque.push_back(std::make_shared<BufferedParser<ParserGzip>>(*this, parser_depth + 1));
+        offset = 0;
     } else if (cur_val.size() > 0 && signatures->php_serialize_identifier.hasMatch(cur_val)) {
         // PHP value detected
         dbgTrace(D_WAAP_DEEP_PARSER) << "Starting to parse phpSerializedData";
@@ -1326,7 +1364,7 @@ DeepParser::createInternalParser(
         } else if (b64FileType != Waap::Util::BinaryFileType::FILE_TYPE_NONE) {
             dbgTrace(D_WAAP_DEEP_PARSER) << "Starting to parse a known binary file, base64 encoded";
             m_parsersDeque.push_back(
-                std::make_shared<BufferedParser<ParserBinaryFile>>(*this, parser_depth + 1, true, b64FileType)
+                std::make_shared<BufferedParser<ParserBinaryFile>>(*this, parser_depth + 1, false, b64FileType)
             );
             offset = 0;
         }

components/security_apps/waap/waap_clib/IndicatorsFiltersManager.cc

@@ -89,6 +89,7 @@ bool IndicatorsFiltersManager::shouldFilterKeyword(const std::string &key, const
             shouldFilter |= m_keywordsFreqFilter->shouldFilterKeyword(type, keyword);
         }
     }
+
     if (m_matchedOverrideKeywords.size() > 0 &&
             m_matchedOverrideKeywords.find(keyword) != m_matchedOverrideKeywords.end())
     {

components/security_apps/waap/waap_clib/KeywordIndicatorFilter.cc

@@ -86,10 +86,11 @@ bool KeywordIndicatorFilter::loadParams(std::shared_ptr<Waap::Parameters::WaapPa
         std::to_string(CONFIDENCE_THRESHOLD)));
     std::string learnPermanentlyStr = pParams->getParamVal("learnIndicators.learnPermanently", "true");
     params.learnPermanently = !boost::iequals(learnPermanentlyStr.c_str(), "false");
+    params.maxMemoryUsage = std::stoul(pParams->getParamVal("learnIndicators.maxMemoryUsage",
+        std::to_string(CONFIDENCE_MAX_MEMORY_USAGE)));
 
     std::string remoteSyncStr = pParams->getParamVal("remoteSync", "true");
     bool syncEnabled = !boost::iequals(remoteSyncStr, "false");
-
     dbgTrace(D_WAAP) << params << " remote sync: " << remoteSyncStr;
 
     m_confidence_calc.setRemoteSyncEnabled(syncEnabled);

components/security_apps/waap/waap_clib/KeywordIndicatorFilter.h

@@ -21,6 +21,7 @@
 #define CONFIDENCE_MIN_INTERVALS 5
 #define CONFIDENCE_THRESHOLD 0.8
 #define CONFIDENCE_WINDOW_INTERVAL std::chrono::minutes(120)
+#define CONFIDENCE_MAX_MEMORY_USAGE (40 * 1024 * 1024) // 40MB
 
 
 class KeywordIndicatorFilter : public IndicatorFilterBase

components/security_apps/waap/waap_clib/LogGenWrapper.cc

@@ -41,15 +41,16 @@ LogGenWrapper::LogGenWrapper(
     }
     else {
         m_log_gen = std::make_unique<LogGen>(
-            maybe_trigger.unpack(),
             title,
-            security_type,
+            ReportIS::Level::LOG,
+            ReportIS::Audience::SECURITY,
             severity,
             priority,
-            is_action_drop_or_prevent,
             ReportIS::Tags::WAF,
-            ReportIS::Tags::THREAT_PREVENTION
-            );
+            ReportIS::Tags::THREAT_PREVENTION,
+            maybe_trigger.unpack().getStreams(security_type, is_action_drop_or_prevent),
+            maybe_trigger.unpack().getEnrechments(security_type)
+        );
     }
 }
 

components/security_apps/waap/waap_clib/ParserBinaryFile.cc

@@ -78,7 +78,6 @@ ParserBinaryFile::detectBinaryFileHeader(const string &buf)
     return BinaryFileType::FILE_TYPE_NONE;
 }
 
-
 size_t
 ParserBinaryFile::push(const char *buf, size_t len)
 {
@@ -151,7 +150,10 @@ ParserBinaryFile::push(const char *buf, size_t len)
                 } else {
                     dbgTrace(D_WAAP_PARSER_BINARY_FILE) << "parsing binary. Searching for tail: " << tail;
                     size_t tail_lookup_offset = (len > MAX_TAIL_LOOKUP) ? len - MAX_TAIL_LOOKUP : 0;
-                    c = strstr(buf + tail_lookup_offset, tail.c_str());
+                    c = static_cast<const char *>(memmem(buf + tail_lookup_offset,
+                        len - tail_lookup_offset,
+                        tail.c_str(),
+                        tail.size()));
                     dbgTrace(D_WAAP_PARSER_BINARY_FILE) << "search result: c=" << c;
                     if (c) {
                         m_state = s_end;

components/security_apps/waap/waap_clib/ParserGzip.cc

@@ -0,0 +1,115 @@
+// Copyright (C) 2022 Check Point Software Technologies Ltd. All rights reserved.
+
+// Licensed under the Apache License, Version 2.0 (the "License");
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "ParserGzip.h"
+#include "debug.h"
+
+USE_DEBUG_FLAG(D_WAAP_PARSER_GZIP);
+
+const std::string ParserGzip::m_parserName = "ParserGzip";
+
+ParserGzip::ParserGzip(IParserStreamReceiver &receiver, size_t parser_depth)
+:m_receiver(receiver), m_key("gzip"), m_state(s_start), m_stream(nullptr) {
+}
+
+ParserGzip::~ParserGzip() {
+    if (m_stream != nullptr) {
+        finiCompressionStream(m_stream);
+        m_stream = nullptr;
+    }
+}
+
+size_t ParserGzip::push(const char *buf, size_t len) {
+    dbgTrace(D_WAAP_PARSER_GZIP) << "len=" << (unsigned long int)len << ")";
+
+    if (len == 0) {
+        dbgTrace(D_WAAP_PARSER_GZIP) << "end of data signal! m_state=" << m_state;
+
+        // flush
+        if (m_state != s_start) { // only emit if at least something was pushed
+            if (m_receiver.onKvDone() != 0) {
+                m_state = s_error;
+            }
+        }
+
+        return 0;
+    }
+    DecompressionResult res;
+    switch (m_state) {
+    case s_start:
+        dbgTrace(D_WAAP_PARSER_GZIP) << "s_start";
+        if (m_receiver.onKey(m_key.data(), m_key.size()) != 0) {
+            m_state = s_error;
+            return 0;
+        }
+        m_stream = initCompressionStream();
+        m_state = s_forward;
+        // fallthrough //
+        CP_FALL_THROUGH;
+    case s_forward:
+        dbgTrace(D_WAAP_PARSER_GZIP) << "s_forward";
+        res = decompressData(
+            m_stream,
+            len,
+            reinterpret_cast<const unsigned char *>(buf));
+        dbgTrace(D_WAAP_PARSER_GZIP) << "res: " << res.ok
+            << ", size: " << res.num_output_bytes
+            << ", is last: " << res.is_last_chunk;
+
+        if (!res.ok) {
+            m_state = s_error;
+            break;
+        }
+
+        if (res.num_output_bytes != 0 &&
+                m_receiver.onValue(reinterpret_cast<const char *>(res.output), res.num_output_bytes) != 0) {
+            m_state = s_error;
+            break;
+        }
+
+        if (res.is_last_chunk) {
+            m_state = s_done;
+            break;
+        }
+        break;
+    case s_done:
+        if (len > 0) {
+            dbgTrace(D_WAAP_PARSER_GZIP) << " unexpected data after completion, len=" << len;
+            m_state = s_error;
+            return 0; // Return 0 to indicate error
+        }
+        break;
+    case s_error:
+        dbgTrace(D_WAAP_PARSER_GZIP) << "s_error";
+        return 0;
+    }
+
+    return len;
+}
+
+void ParserGzip::finish() {
+    push(NULL, 0);
+    if (m_state != s_done) {
+        m_state = s_error;
+        return;
+    }
+}
+
+const std::string &
+ParserGzip::name() const {
+    return m_parserName;
+}
+
+bool ParserGzip::error() const {
+    return m_state == s_error;
+}

components/security_apps/waap/waap_clib/ParserGzip.h

@@ -0,0 +1,46 @@
+// Copyright (C) 2022 Check Point Software Technologies Ltd. All rights reserved.
+
+// Licensed under the Apache License, Version 2.0 (the "License");
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#ifndef __PARSER_GZIP_H_
+#define __PARSER_GZIP_H_
+
+#include "ParserBase.h"
+#include <string.h>
+#include "compression_utils.h"
+
+class ParserGzip : public ParserBase {
+public:
+    ParserGzip(IParserStreamReceiver &receiver, size_t parser_depth);
+    virtual ~ParserGzip();
+    size_t push(const char *data, size_t data_len);
+    void finish();
+    virtual const std::string &name() const;
+    bool error() const;
+    virtual size_t depth() { return 1; }
+private:
+    enum state {
+        s_start,
+        s_forward,
+        s_done,
+        s_error
+    };
+
+    IParserStreamReceiver &m_receiver;
+    std::string m_key;
+    state m_state;
+    CompressionStream * m_stream;
+
+    static const std::string m_parserName;
+};
+
+#endif // __PARSER_GZIP_H_

components/security_apps/waap/waap_clib/ParserUrlEncode.cc

@@ -21,7 +21,11 @@ USE_DEBUG_FLAG(D_WAAP);
 const std::string ParserUrlEncode::m_parserName = "ParserUrlEncode";
 
 ParserUrlEncode::ParserUrlEncode(
-    IParserStreamReceiver &receiver, size_t parser_depth, char separatorChar, bool should_decode_per
+    IParserStreamReceiver &receiver,
+    size_t parser_depth,
+    char separatorChar,
+    bool should_decode_per,
+    bool should_decode_plus
 ) :
     m_receiver(receiver),
     m_state(s_start),
@@ -29,13 +33,16 @@ ParserUrlEncode::ParserUrlEncode(
     m_separatorChar(separatorChar),
     m_escapedCharCandidate(0),
     should_decode_percent(should_decode_per),
+    m_should_decode_plus(should_decode_plus),
     m_parser_depth(parser_depth)
 {
     dbgTrace(D_WAAP)
         << "should_decode_percent="
         << should_decode_per
         << "parser_depth="
-        << parser_depth;
+        << parser_depth
+        << "m_should_decode_plus="
+        << m_should_decode_plus;
 
     // TODO:: is there a need for this?
     memset(m_escaped, 0, sizeof(m_escaped));
@@ -124,7 +131,7 @@ ParserUrlEncode::push(const char *buf, size_t len)
                 }
                 m_state = s_key_escaped1;
                 break;
-                } else if (c == '+') {
+                } else if (c == '+' && m_should_decode_plus) {
                 // convert plus character to space
                 if (i - mark > 0) {
                     if (m_receiver.onKey(buf + mark, i - mark) != 0) {
@@ -281,7 +288,7 @@ ParserUrlEncode::push(const char *buf, size_t len)
                 }
                 m_state = s_value_escaped1;
                 break;
-                } else if (c == '+') {
+                } else if (c == '+' && m_should_decode_plus) {
                 // convert plus character to space
                 if (i - mark > 0) {
                     if (m_receiver.onValue(buf + mark, i - mark) != 0) {

components/security_apps/waap/waap_clib/ParserUrlEncode.h

@@ -25,7 +25,8 @@ public:
         IParserStreamReceiver &receiver,
         size_t parser_depth,
         char separatorChar = '&',
-        bool should_decode_per = true);
+        bool should_decode_per = true,
+        bool should_decode_plus = true);
     virtual ~ParserUrlEncode();
     size_t push(const char *data, size_t data_len);
     void finish();
@@ -55,6 +56,7 @@ private:
     char m_separatorChar;
     char m_escapedCharCandidate;
     bool should_decode_percent;
+    bool m_should_decode_plus;
     static const std::string m_parserName;
     size_t m_parser_depth;
 };

components/security_apps/waap/waap_clib/ParserXML.cc

@@ -170,19 +170,22 @@ ParserXML::onEntityDeclaration(
 {
     dbgTrace(D_WAAP_PARSER_XML) << "ENTITY FOUND WITH VALUE: '" << (content ? (const char*)content : "null") << "'";
 
-    ParserXML* p = (ParserXML*)ctx;
-    std::string kw = "08a80340-06d3-11ea-9f87-0242ac11000f";
+    if (systmeid != nullptr) {
+        dbgTrace(D_WAAP_PARSER_XML) << "ENTITY FOUND WITH SYSTEM ID: '" << (const char*)systmeid << "'";
+        ParserXML* p = (ParserXML*)ctx;
+        std::string kw = "08a80340-06d3-11ea-9f87-0242ac11000f";
 
-    if (p->m_receiver.onKey(p->m_key.c_str(), p->m_key.size()) != 0) {
-        p->m_state = s_error;
-    }
+        if (p->m_receiver.onKey(p->m_key.c_str(), p->m_key.size()) != 0) {
+            p->m_state = s_error;
+        }
 
-    if (p->m_receiver.onValue(kw.data(), kw.size()) != 0) {
-        p->m_state = s_error;
-    }
+        if (p->m_receiver.onValue(kw.data(), kw.size()) != 0) {
+            p->m_state = s_error;
+        }
 
-    if (p->m_receiver.onKvDone() != 0) {
-        p->m_state = s_error; // error
+        if (p->m_receiver.onKvDone() != 0) {
+            p->m_state = s_error; // error
+        }
     }
 }
 

components/security_apps/waap/waap_clib/RequestsMonitor.cc

@@ -18,9 +18,11 @@ SourcesRequestMonitor::SourcesRequestMonitor(
         filePath,
         remotePath != "" ? remotePath + "/Monitor" : remotePath,
         assetId,
-        owner
-    ), m_sourcesRequests()
+        owner),
+    m_sourcesRequests(),
+    m_enabled(false)
 {
+    m_enabled = getProfileAgentSettingWithDefault<bool>(false, "appsec.sourceRequestsMonitor.enabled");
 }
 
 SourcesRequestMonitor::~SourcesRequestMonitor()
@@ -35,17 +37,18 @@ void SourcesRequestMonitor::syncWorker()
     OrchestrationMode mode = Singleton::exists<I_AgentDetails>() ?
         Singleton::Consume<I_AgentDetails>::by<WaapComponent>()->getOrchestrationMode() : OrchestrationMode::ONLINE;
 
-    bool enabled = getProfileAgentSettingWithDefault<bool>(false, "appsec.sourceRequestsMonitor.enabled");
+    m_enabled = getProfileAgentSettingWithDefault<bool>(false, "appsec.sourceRequestsMonitor.enabled");
 
-    if (mode == OrchestrationMode::OFFLINE || !enabled || isBase() || !postData()) {
+    if (mode == OrchestrationMode::OFFLINE || !m_enabled || isBase() || !postData()) {
         dbgInfo(D_WAAP_CONFIDENCE_CALCULATOR)
             << "Did not report data. for asset: "
             << m_assetId
             << " Remote URL: "
             << m_remotePath
             << " is enabled: "
-            << to_string(enabled)
+            << to_string(m_enabled)
             << ", mode: " << int(mode);
+        m_sourcesRequests.clear();
         return;
     }
 
@@ -72,6 +75,9 @@ void SourcesRequestMonitor::syncWorker()
 
 void SourcesRequestMonitor::logSourceHit(const string& source)
 {
+    if (!m_enabled) {
+        return;
+    }
     m_sourcesRequests[chrono::duration_cast<chrono::minutes>(
         Singleton::Consume<I_TimeGet>::by<WaapComponent>()->getWalltime()
     ).count()][source]++;

components/security_apps/waap/waap_clib/RequestsMonitor.h

@@ -28,6 +28,7 @@ protected:
 private:
     // map of sources and their requests per minute (UNIX)
     MonitorData m_sourcesRequests;
+    bool m_enabled;
 };
 
 #endif // __REQUESTS_MONITOR_H__

components/security_apps/waap/waap_clib/Serializator.cc

@@ -29,7 +29,7 @@
 #include "compression_utils.h"
 #include "config.h"
 
-USE_DEBUG_FLAG(D_WAAP_CONFIDENCE_CALCULATOR);
+USE_DEBUG_FLAG(D_WAAP_SERIALIZE);
 
 namespace ch = std::chrono;
 using namespace std;
@@ -44,23 +44,34 @@ static const string defaultSharedStorageHost = "appsec-shared-storage-svc";
 #define SHARED_STORAGE_HOST_ENV_NAME "SHARED_STORAGE_HOST"
 #define LEARNING_HOST_ENV_NAME "LEARNING_HOST"
 
-static bool
-isGZipped(const string &stream)
+void yieldIfPossible(const string& func, int line)
 {
-    if (stream.size() < 2) return false;
-    auto unsinged_stream = reinterpret_cast<const u_char *>(stream.data());
-    return unsinged_stream[0] == 0x1f && unsinged_stream[1] == 0x8b;
+    // Check if we are in the main loop
+    if (Singleton::exists<I_MainLoop>() &&
+        Singleton::Consume<I_MainLoop>::by<WaapComponent>()->getCurrentRoutineId().ok())
+    {
+        // If we are not in the main loop, yield to allow other routines to run
+        // This is important for the main loop to be able to process other events
+        // and avoid blocking the entire system.
+        dbgTrace(D_WAAP_SERIALIZE) << "Yielding to main loop from: " << func << ":" << line;
+        Singleton::Consume<I_MainLoop>::by<WaapComponent>()->yield(false);
+    }
 }
 
+#define YIELD_IF_POSSIBLE() yieldIfPossible(__FUNCTION__, __LINE__)
+
 bool RestGetFile::loadJson(const string& json)
 {
     string json_str;
 
     json_str = json;
-    if (!isGZipped(json_str))
+    if (!Waap::Util::isGzipped(json_str))
     {
         return ClientRest::loadJson(json_str);
     }
+    YIELD_IF_POSSIBLE();
+    dbgTrace(D_WAAP_SERIALIZE) << "before decompression in loadJson, data size: "
+        << json_str.size() << " bytes";
     auto compression_stream = initCompressionStream();
     DecompressionResult res = decompressData(
         compression_stream,
@@ -75,33 +86,81 @@ bool RestGetFile::loadJson(const string& json)
     }
 
     finiCompressionStream(compression_stream);
+    YIELD_IF_POSSIBLE();
+    dbgTrace(D_WAAP_SERIALIZE) << "Yielded after decompression in loadJson, decompressed size: "
+        << json_str.size() << " bytes";
+
     return ClientRest::loadJson(json_str);
 }
 
 Maybe<string> RestGetFile::genJson() const
 {
     Maybe<string> json = ClientRest::genJson();
+    YIELD_IF_POSSIBLE();
+
     if (json.ok())
     {
         string data = json.unpack();
+
+        // Get chunk size from profile settings for compression chunks
+        const size_t COMPRESSED_CHUNK_SIZE = static_cast<size_t>(
+            getProfileAgentSettingWithDefault<uint>(64 * 1024, "appsecLearningSettings.compressionChunkSize"));
+
         auto compression_stream = initCompressionStream();
-        CompressionResult res = compressData(
-            compression_stream,
-            CompressionType::GZIP,
-            data.size(),
-            reinterpret_cast<const unsigned char *>(data.c_str()),
-            true);
+        size_t offset = 0;
+        std::vector<unsigned char> compressed_data;
+        bool ok = true;
+        size_t chunk_count = 0;
+
+        // Process data in chunks for compression
+        while (offset < data.size()) {
+            size_t chunk_size = std::min(COMPRESSED_CHUNK_SIZE, data.size() - offset);
+            bool is_last = (offset + chunk_size >= data.size());
+            CompressionResult chunk_res = compressData(
+                compression_stream,
+                CompressionType::GZIP,
+                static_cast<uint32_t>(chunk_size),
+                reinterpret_cast<const unsigned char *>(data.c_str() + offset),
+                is_last ? 1 : 0
+            );
+
+            if (!chunk_res.ok) {
+                ok = false;
+                break;
+            }
+
+            if (chunk_res.output && chunk_res.num_output_bytes > 0) {
+                compressed_data.insert(
+                    compressed_data.end(),
+                    chunk_res.output,
+                    chunk_res.output + chunk_res.num_output_bytes
+                );
+                free(chunk_res.output);
+                chunk_res.output = nullptr;
+            }
+
+            offset += chunk_size;
+            chunk_count++;
+            YIELD_IF_POSSIBLE();
+            dbgTrace(D_WAAP_SERIALIZE) << "Processed compression chunk " << chunk_count
+                << ", progress: " << offset << "/" << data.size() << " bytes ("
+                << (offset * 100 / data.size()) << "%) - yielded";
+        }
+
         finiCompressionStream(compression_stream);
-        if (!res.ok) {
-            dbgWarning(D_WAAP_CONFIDENCE_CALCULATOR) << "Failed to gzip data";
+        dbgDebug(D_WAAP_SERIALIZE) << "Yielded after finalizing compression stream. "
+            << "Total chunks: " << chunk_count << ", Compression ratio: "
+            << (data.size() > 0 ? (float)compressed_data.size() / data.size() : 0) << "x";
+
+        if (!ok) {
+            dbgWarning(D_WAAP_SERIALIZE) << "Failed to gzip data";
             return genError("Failed to compress data");
         }
-        data = string((const char *)res.output, res.num_output_bytes);
-        json = data;
 
-        if (res.output) free(res.output);
-        res.output = nullptr;
-        res.num_output_bytes = 0;
+        // Create string from compressed data
+        string compressed_str(reinterpret_cast<const char*>(compressed_data.data()), compressed_data.size());
+
+        json = compressed_str;
     }
     return json;
 }
@@ -128,16 +187,16 @@ void SerializeToFilePeriodically::backupWorker()
     I_TimeGet* timer = Singleton::Consume<I_TimeGet>::by<WaapComponent>();
     auto currentTime = timer->getMonotonicTime();
 
-    dbgTrace(D_WAAP_CONFIDENCE_CALCULATOR) << "backup worker: current time: " << currentTime.count();
+    dbgTrace(D_WAAP_SERIALIZE) << "backup worker: current time: " << currentTime.count();
 
     if (currentTime - m_lastSerialization >= m_interval)
     {
-        dbgTrace(D_WAAP_CONFIDENCE_CALCULATOR) << "backup worker: backing up data";
+        dbgTrace(D_WAAP_SERIALIZE) << "backup worker: backing up data";
         m_lastSerialization = currentTime;
         // save data
         saveData();
 
-        dbgTrace(D_WAAP_CONFIDENCE_CALCULATOR) << "backup worker: data is backed up";
+        dbgTrace(D_WAAP_SERIALIZE) << "backup worker: data is backed up";
     }
 }
 
@@ -153,7 +212,7 @@ void SerializeToFilePeriodically::setInterval(ch::seconds newInterval)
 
 SerializeToFileBase::SerializeToFileBase(string fileName) : m_filePath(fileName)
 {
-    dbgTrace(D_WAAP_CONFIDENCE_CALCULATOR) << "SerializeToFileBase::SerializeToFileBase() fname='" << m_filePath
+    dbgTrace(D_WAAP_SERIALIZE) << "SerializeToFileBase::SerializeToFileBase() fname='" << m_filePath
         << "'";
 }
 
@@ -165,52 +224,119 @@ SerializeToFileBase::~SerializeToFileBase()
 void SerializeToFileBase::saveData()
 {
     fstream filestream;
-
-    dbgTrace(D_WAAP_CONFIDENCE_CALCULATOR) << "saving to file: " << m_filePath;
+    auto maybe_routine = Singleton::Consume<I_MainLoop>::by<WaapComponent>()->getCurrentRoutineId();
+    dbgTrace(D_WAAP_SERIALIZE) << "saving to file: " << m_filePath;
     filestream.open(m_filePath, fstream::out);
 
     stringstream ss;
 
     if (filestream.is_open() == false) {
-        dbgWarning(D_WAAP_CONFIDENCE_CALCULATOR) << "failed to open file: " << m_filePath << " Error: "
+        dbgWarning(D_WAAP_SERIALIZE) << "failed to open file: " << m_filePath << " Error: "
             << strerror(errno);
         return;
     }
-
+    if (maybe_routine.ok()) {
+        Singleton::Consume<I_MainLoop>::by<WaapComponent>()->yield(false);
+    }
     serialize(ss);
 
+    if (maybe_routine.ok()) {
+        Singleton::Consume<I_MainLoop>::by<WaapComponent>()->yield(false);
+    }
     string data = ss.str();
+    dbgDebug(D_WAAP_SERIALIZE) << "Serialized data size: " << data.size() << " bytes";
+
+    // Get chunk size from profile settings, with default of 16 MiB for compression chunks
+    const size_t CHUNK_SIZE = static_cast<size_t>(
+        getProfileAgentSettingWithDefault<uint>(16 * 1024 * 1024, "appsecLearningSettings.writeChunkSize"));
+    // Get chunk size for writing compressed data, with default of 16 MiB
+    const size_t COMPRESSED_CHUNK_SIZE = static_cast<size_t>(
+        getProfileAgentSettingWithDefault<uint>(16 * 1024 * 1024, "appsecLearningSettings.compressionChunkSize"));
 
     auto compression_stream = initCompressionStream();
-    CompressionResult res = compressData(
-        compression_stream,
-        CompressionType::GZIP,
-        data.size(),
-        reinterpret_cast<const unsigned char *>(data.c_str()),
-        true
-    );
-    finiCompressionStream(compression_stream);
-    if (!res.ok) {
-        dbgWarning(D_WAAP_CONFIDENCE_CALCULATOR) << "Failed to gzip data";
-    } else {
-        ss.str(string((const char *)res.output, res.num_output_bytes));
-        // free the memory allocated by compressData
-        if (res.output) free(res.output);
-        res.output = nullptr;
-        res.num_output_bytes = 0;
+    size_t offset = 0;
+    std::vector<unsigned char> compressed_data;
+    bool ok = true;
+    size_t chunk_count = 0;
+
+    // Process data in chunks for compression
+    while (offset < data.size()) {
+        size_t chunk_size = std::min(COMPRESSED_CHUNK_SIZE, data.size() - offset);
+        bool is_last = (offset + chunk_size >= data.size());
+        CompressionResult chunk_res = compressData(
+            compression_stream,
+            CompressionType::GZIP,
+            static_cast<uint32_t>(chunk_size),
+            reinterpret_cast<const unsigned char *>(data.c_str() + offset),
+            is_last ? 1 : 0
+        );
+
+        if (!chunk_res.ok) {
+            ok = false;
+            break;
+        }
+
+        if (chunk_res.output && chunk_res.num_output_bytes > 0) {
+            compressed_data.insert(
+                compressed_data.end(),
+                chunk_res.output,
+                chunk_res.output + chunk_res.num_output_bytes
+            );
+            free(chunk_res.output);
+            chunk_res.output = nullptr;
+        }
+
+        offset += chunk_size;
+        chunk_count++;
+        if (maybe_routine.ok()) {
+            Singleton::Consume<I_MainLoop>::by<WaapComponent>()->yield(false);
+            dbgTrace(D_WAAP_SERIALIZE) << "Compression chunk " << chunk_count
+                << " processed (" << offset << "/" << data.size() << " bytes, "
+                << (offset * 100 / data.size()) << "%) - yielded";
+        }
     }
-    if (res.output) free(res.output);
-    res.output = nullptr;
-    res.num_output_bytes = 0;
+    finiCompressionStream(compression_stream);
+    dbgDebug(D_WAAP_SERIALIZE) << "Finished compression stream. "
+        << "Total chunks: " << chunk_count << ", Compression ratio: "
+        << (data.size() > 0 ? (float)compressed_data.size() / data.size() : 0) << "x";
+
+    if (!ok) {
+        dbgWarning(D_WAAP_SERIALIZE) << "Failed to compress data";
+        filestream.close();
+        return;
+    }
+
+    dbgDebug(D_WAAP_SERIALIZE) << "Compression complete: " << data.size() << " bytes -> "
+        << compressed_data.size() << " bytes (ratio: "
+        << (data.size() > 0 ? (float)compressed_data.size() / data.size() : 0) << "x)";
 
 
-    filestream << ss.str();
+    // Use compressed data directly
+    string data_to_write(reinterpret_cast<const char*>(compressed_data.data()), compressed_data.size());
+
+    // Write data to file in chunks with yield points
+    offset = 0;
+    size_t write_chunks = 0;
+
+    while (offset < data_to_write.size()) {
+        size_t current_chunk_size = std::min(CHUNK_SIZE, data_to_write.size() - offset);
+        filestream.write(data_to_write.c_str() + offset, current_chunk_size);
+        offset += current_chunk_size;
+        write_chunks++;
+        Singleton::Consume<I_MainLoop>::by<WaapComponent>()->yield(false);
+        dbgTrace(D_WAAP_SERIALIZE) << "Write chunk " << write_chunks
+            << " complete: " << offset << "/" << data_to_write.size() << " bytes ("
+            << (offset * 100 / data_to_write.size()) << "%) - yielded";
+    }
+
     filestream.close();
+    dbgDebug(D_WAAP_SERIALIZE) << "Finished writing backup file: " << m_filePath
+        << " (" << data_to_write.size() << " bytes in " << write_chunks << " chunks)";
 }
 
 string decompress(string fileContent) {
-    if (!isGZipped(fileContent)) {
-        dbgTrace(D_WAAP) << "file note zipped";
+    if (!Waap::Util::isGzipped(fileContent)) {
+        dbgTrace(D_WAAP_SERIALIZE) << "file note zipped";
         return fileContent;
     }
     auto compression_stream = initCompressionStream();
@@ -236,13 +362,13 @@ string decompress(string fileContent) {
 
 void SerializeToFileBase::loadFromFile(string filePath)
 {
-    dbgTrace(D_WAAP_CONFIDENCE_CALCULATOR) << "loadFromFile() file: " << filePath;
+    dbgTrace(D_WAAP_SERIALIZE) << "loadFromFile() file: " << filePath;
     fstream filestream;
 
     filestream.open(filePath, fstream::in);
 
     if (filestream.is_open() == false) {
-        dbgTrace(D_WAAP_CONFIDENCE_CALCULATOR) << "failed to open file: " << filePath << " Error: " <<
+        dbgTrace(D_WAAP_SERIALIZE) << "failed to open file: " << filePath << " Error: " <<
             strerror(errno);
         if (!Singleton::exists<I_InstanceAwareness>() || errno != ENOENT)
         {
@@ -262,18 +388,18 @@ void SerializeToFileBase::loadFromFile(string filePath)
         if (idPosition != string::npos)
         {
             filePath.erase(idPosition, idStr.length() - 1);
-            dbgDebug(D_WAAP_CONFIDENCE_CALCULATOR) << "retry to load file from : " << filePath;
+            dbgDebug(D_WAAP_SERIALIZE) << "retry to load file from : " << filePath;
             loadFromFile(filePath);
         }
         return;
     }
 
-    dbgTrace(D_WAAP_CONFIDENCE_CALCULATOR) << "loading from file: " << filePath;
+    dbgTrace(D_WAAP_SERIALIZE) << "loading from file: " << filePath;
 
     int length;
     filestream.seekg(0, ios::end);    // go to the end
     length = filestream.tellg();           // report location (this is the length)
-    dbgTrace(D_WAAP_CONFIDENCE_CALCULATOR) << "file length: " << length;
+    dbgTrace(D_WAAP_SERIALIZE) << "file length: " << length;
     assert(length >= 0); // length -1 really happens if filePath is a directory (!)
     char* buffer = new char[length];       // allocate memory for a buffer of appropriate dimension
     filestream.seekg(0, ios::beg);    // go back to the beginning
@@ -281,7 +407,7 @@ void SerializeToFileBase::loadFromFile(string filePath)
     {
         filestream.close();
         delete[] buffer;
-        dbgWarning(D_WAAP_CONFIDENCE_CALCULATOR) << "Failed to read file, file: " << filePath;
+        dbgWarning(D_WAAP_SERIALIZE) << "Failed to read file, file: " << filePath;
         return;
     }
     filestream.close();
@@ -298,7 +424,7 @@ void SerializeToFileBase::loadFromFile(string filePath)
         deserialize(ss);
     }
     catch (runtime_error & e) {
-        dbgWarning(D_WAAP_CONFIDENCE_CALCULATOR) << "failed to deserialize file: " << m_filePath << ", error: " <<
+        dbgWarning(D_WAAP_SERIALIZE) << "failed to deserialize file: " << m_filePath << ", error: " <<
             e.what();
     }
 }
@@ -318,11 +444,11 @@ RemoteFilesList::RemoteFilesList() : files(), filesPathsList()
 bool RemoteFilesList::loadJson(const string& xml)
 {
     xmlDocPtr doc; // the resulting document tree
-    dbgTrace(D_WAAP_CONFIDENCE_CALCULATOR) << "XML input: " << xml;
+    dbgTrace(D_WAAP_SERIALIZE) << "XML input: " << xml;
     doc = xmlParseMemory(xml.c_str(), xml.length());
 
     if (doc == NULL) {
-        dbgWarning(D_WAAP_CONFIDENCE_CALCULATOR) << "Failed to parse " << xml;
+        dbgWarning(D_WAAP_SERIALIZE) << "Failed to parse " << xml;
         return false;
     }
 
@@ -343,7 +469,7 @@ bool RemoteFilesList::loadJson(const string& xml)
     {
         if (xmlStrEqual(contents_name, node->name) == 1)
         {
-            dbgTrace(D_WAAP_CONFIDENCE_CALCULATOR) << "Found the Contents element";
+            dbgTrace(D_WAAP_SERIALIZE) << "Found the Contents element";
             xmlNodePtr contents_node = node->children;
             string file;
             string lastModified;
@@ -351,21 +477,21 @@ bool RemoteFilesList::loadJson(const string& xml)
             {
                 if (xmlStrEqual(key_name, contents_node->name) == 1)
                 {
-                    dbgTrace(D_WAAP_CONFIDENCE_CALCULATOR) << "Found the Key element";
+                    dbgTrace(D_WAAP_SERIALIZE) << "Found the Key element";
                     xmlChar* xml_file = xmlNodeGetContent(contents_node);
                     file = string(reinterpret_cast<const char*>(xml_file));
                     xmlFree(xml_file);
                 }
                 if (xmlStrEqual(last_modified_name, contents_node->name) == 1)
                 {
-                    dbgTrace(D_WAAP_CONFIDENCE_CALCULATOR) << "Found the LastModified element";
+                    dbgTrace(D_WAAP_SERIALIZE) << "Found the LastModified element";
                     xmlChar* xml_file = xmlNodeGetContent(contents_node);
                     lastModified = string(reinterpret_cast<const char*>(xml_file));
                     xmlFree(xml_file);
                 }
                 if (!file.empty() && !lastModified.empty())
                 {
-                    dbgTrace(D_WAAP_CONFIDENCE_CALCULATOR) << "Adding the file: " << file <<
+                    dbgTrace(D_WAAP_SERIALIZE) << "Adding the file: " << file <<
                         " last modified: " << lastModified;
                     break;
                 }
@@ -408,18 +534,18 @@ SerializeToLocalAndRemoteSyncBase::SerializeToLocalAndRemoteSyncBase(
     m_interval(0),
     m_owner(owner),
     m_assetId(replaceAllCopy(assetId, "/", "")),
+    m_remoteSyncEnabled(true),
     m_pMainLoop(nullptr),
     m_waitForSync(waitForSync),
     m_workerRoutineId(0),
     m_daysCount(0),
     m_windowsCount(0),
     m_intervalsCounter(0),
-    m_remoteSyncEnabled(true),
     m_isAssetIdUuid(Waap::Util::isUuid(assetId)),
     m_shared_storage_host(genError("not set")),
     m_learning_host(genError("not set"))
 {
-    dbgInfo(D_WAAP_CONFIDENCE_CALCULATOR) << "Create SerializeToLocalAndRemoteSyncBase. assetId='" << assetId <<
+    dbgInfo(D_WAAP_SERIALIZE) << "Create SerializeToLocalAndRemoteSyncBase. assetId='" << assetId <<
         "', owner='" << m_owner << "'";
 
     if (Singleton::exists<I_AgentDetails>() &&
@@ -429,7 +555,7 @@ SerializeToLocalAndRemoteSyncBase::SerializeToLocalAndRemoteSyncBase(
         if (sharedStorageHost != NULL) {
             m_shared_storage_host = string(sharedStorageHost);
         } else {
-            dbgWarning(D_WAAP_CONFIDENCE_CALCULATOR) <<
+            dbgWarning(D_WAAP_SERIALIZE) <<
                 "shared storage host name(" <<
                 SHARED_STORAGE_HOST_ENV_NAME <<
                 ") is not set";
@@ -438,7 +564,7 @@ SerializeToLocalAndRemoteSyncBase::SerializeToLocalAndRemoteSyncBase(
         if (learningHost != NULL) {
             m_learning_host = string(learningHost);
         } else {
-            dbgWarning(D_WAAP_CONFIDENCE_CALCULATOR) <<
+            dbgWarning(D_WAAP_SERIALIZE) <<
                 "learning host name(" <<
                 SHARED_STORAGE_HOST_ENV_NAME <<
                 ") is not set";
@@ -515,7 +641,7 @@ string SerializeToLocalAndRemoteSyncBase::getPostDataUrl()
     {
         I_InstanceAwareness* instance = Singleton::Consume<I_InstanceAwareness>::by<WaapComponent>();
         Maybe<string> uniqueId = instance->getUniqueID();
-        if (uniqueId.ok())
+        if (uniqueId.ok() && !uniqueId.unpack().empty())
         {
             agentId += "/" + uniqueId.unpack();
         }
@@ -530,7 +656,7 @@ void SerializeToLocalAndRemoteSyncBase::setRemoteSyncEnabled(bool enabled)
 
 void SerializeToLocalAndRemoteSyncBase::setInterval(ch::seconds newInterval)
 {
-    dbgDebug(D_WAAP_CONFIDENCE_CALCULATOR) << "setInterval: from " << m_interval.count() << " to " <<
+    dbgDebug(D_WAAP_SERIALIZE) << "setInterval: from " << m_interval.count() << " to " <<
         newInterval.count() << " seconds. assetId='" << m_assetId << "', owner='" << m_owner << "'";
 
     if (newInterval == m_interval)
@@ -571,7 +697,7 @@ void SerializeToLocalAndRemoteSyncBase::setInterval(ch::seconds newInterval)
             if (remainingTime > m_interval) {
                 // on load between trigger and offset remaining time is larger than the interval itself
                 remainingTime -= m_interval;
-                dbgDebug(D_WAAP_CONFIDENCE_CALCULATOR) << "adjusting remaining time: " << remainingTime.count();
+                dbgDebug(D_WAAP_SERIALIZE) << "adjusting remaining time: " << remainingTime.count();
                 if (timeBeforeSyncWorker.count() != 0)
                 {
                     auto updateTime = timeBeforeSyncWorker - m_interval;
@@ -585,13 +711,13 @@ void SerializeToLocalAndRemoteSyncBase::setInterval(ch::seconds newInterval)
             if (remainingTime < ch::seconds(0)) {
                 // syncWorker execution time was so large the remaining time became negative
                 remainingTime = ch::seconds(0);
-                dbgError(D_WAAP_CONFIDENCE_CALCULATOR) << "syncWorker execution time (owner='" << m_owner <<
+                dbgError(D_WAAP_SERIALIZE) << "syncWorker execution time (owner='" << m_owner <<
                     "', assetId='" << m_assetId << "') is " <<
                     ch::duration_cast<ch::seconds>(timeAfterSyncWorker - timeBeforeSyncWorker).count() <<
                     " seconds, too long to cause negative remainingTime. Waiting 0 seconds...";
             }
 
-            dbgDebug(D_WAAP_CONFIDENCE_CALCULATOR) << "current time: " << timeBeforeSyncWorker.count() << " \u00b5s" <<
+            dbgDebug(D_WAAP_SERIALIZE) << "current time: " << timeBeforeSyncWorker.count() << " \u00b5s" <<
                 ": assetId='" << m_assetId << "'" <<
                 ", owner='" << m_owner << "'" <<
                 ", daysCount=" << m_daysCount <<
@@ -604,7 +730,7 @@ void SerializeToLocalAndRemoteSyncBase::setInterval(ch::seconds newInterval)
             m_pMainLoop->yield(remainingTime);
 
             timeBeforeSyncWorker = timer->getWalltime();
-            syncWorker();
+            m_pMainLoop->addOneTimeRoutine(I_MainLoop::RoutineType::System, [this]() {syncWorker();}, "Sync worker");
             timeAfterSyncWorker = timer->getWalltime();
         }
     };
@@ -618,11 +744,11 @@ void SerializeToLocalAndRemoteSyncBase::setInterval(ch::seconds newInterval)
 bool SerializeToLocalAndRemoteSyncBase::localSyncAndProcess()
 {
     bool isBackupSyncEnabled = getProfileAgentSettingWithDefault<bool>(
-        true,
+        false,
         "appsecLearningSettings.backupLocalSync");
 
     if (!isBackupSyncEnabled) {
-        dbgInfo(D_WAAP_CONFIDENCE_CALCULATOR) << "Local sync is disabled";
+        dbgInfo(D_WAAP_SERIALIZE) << "Local sync is disabled";
         processData();
         saveData();
         return true;
@@ -630,7 +756,7 @@ bool SerializeToLocalAndRemoteSyncBase::localSyncAndProcess()
 
     RemoteFilesList rawDataFiles;
 
-    dbgTrace(D_WAAP_CONFIDENCE_CALCULATOR) << "Getting files of all agents";
+    dbgTrace(D_WAAP_SERIALIZE) << "Getting files of all agents";
 
     bool isSuccessful = sendObjectWithRetry(rawDataFiles,
         HTTPMethod::GET,
@@ -638,7 +764,7 @@ bool SerializeToLocalAndRemoteSyncBase::localSyncAndProcess()
 
     if (!isSuccessful)
     {
-        dbgError(D_WAAP_CONFIDENCE_CALCULATOR) << "Failed to get the list of files";
+        dbgError(D_WAAP_SERIALIZE) << "Failed to get the list of files";
         return false;
     }
 
@@ -662,7 +788,7 @@ void SerializeToLocalAndRemoteSyncBase::updateStateFromRemoteService()
         RemoteFilesList remoteFiles = getRemoteProcessedFilesList();
         if (remoteFiles.getFilesMetadataList().empty())
         {
-            dbgWarning(D_WAAP_CONFIDENCE_CALCULATOR) << "no files generated by the remote service were found";
+            dbgWarning(D_WAAP_SERIALIZE) << "no files generated by the remote service were found";
             continue;
         }
         string lastModified = remoteFiles.getFilesMetadataList().begin()->modified;
@@ -670,26 +796,26 @@ void SerializeToLocalAndRemoteSyncBase::updateStateFromRemoteService()
         {
             m_lastProcessedModified = lastModified;
             updateState(remoteFiles.getFilesList());
-            dbgInfo(D_WAAP_CONFIDENCE_CALCULATOR) << "Owner: " << m_owner <<
+            dbgInfo(D_WAAP_SERIALIZE) << "Owner: " << m_owner <<
                 ". updated state generated by remote at " << m_lastProcessedModified;
             return;
         }
     }
-    dbgWarning(D_WAAP_CONFIDENCE_CALCULATOR) << "polling for update state timeout. for assetId='"
+    dbgWarning(D_WAAP_SERIALIZE) << "polling for update state timeout. for assetId='"
         << m_assetId << "', owner='" << m_owner;
     localSyncAndProcess();
 }
 
 void SerializeToLocalAndRemoteSyncBase::syncWorker()
 {
-    dbgInfo(D_WAAP_CONFIDENCE_CALCULATOR) << "Running the sync worker for assetId='" << m_assetId << "', owner='" <<
+    dbgInfo(D_WAAP_SERIALIZE) << "Running the sync worker for assetId='" << m_assetId << "', owner='" <<
         m_owner << "'" << " last modified state: " << m_lastProcessedModified;
     incrementIntervalsCount();
     OrchestrationMode mode = Singleton::exists<I_AgentDetails>() ?
         Singleton::Consume<I_AgentDetails>::by<WaapComponent>()->getOrchestrationMode() : OrchestrationMode::ONLINE;
 
     if (mode == OrchestrationMode::OFFLINE  || !m_remoteSyncEnabled || isBase() || !postData()) {
-        dbgDebug(D_WAAP_CONFIDENCE_CALCULATOR)
+        dbgDebug(D_WAAP_SERIALIZE)
             << "Did not synchronize the data. for asset: "
             << m_assetId
             << " Remote URL: "
@@ -702,17 +828,17 @@ void SerializeToLocalAndRemoteSyncBase::syncWorker()
         return;
     }
 
-    dbgTrace(D_WAAP_CONFIDENCE_CALCULATOR) << "Waiting for all agents to post their data";
+    dbgTrace(D_WAAP_SERIALIZE) << "Waiting for all agents to post their data";
     waitSync();
     // check if learning service is operational
     if (m_lastProcessedModified == "")
     {
-        dbgTrace(D_WAAP_CONFIDENCE_CALCULATOR) << "check if remote service is operational";
+        dbgTrace(D_WAAP_SERIALIZE) << "check if remote service is operational";
         RemoteFilesList remoteFiles = getRemoteProcessedFilesList();
         if (!remoteFiles.getFilesMetadataList().empty())
         {
             m_lastProcessedModified = remoteFiles.getFilesMetadataList()[0].modified;
-            dbgInfo(D_WAAP_CONFIDENCE_CALCULATOR) << "First sync by remote service: " << m_lastProcessedModified;
+            dbgInfo(D_WAAP_SERIALIZE) << "First sync by remote service: " << m_lastProcessedModified;
         }
     }
 
@@ -721,15 +847,15 @@ void SerializeToLocalAndRemoteSyncBase::syncWorker()
         true,
         "appsecLearningSettings.remoteServiceEnabled");
 
-    dbgDebug(D_WAAP_CONFIDENCE_CALCULATOR) << "using remote service: " << isRemoteServiceEnabled;
+    dbgDebug(D_WAAP_SERIALIZE) << "using remote service: " << isRemoteServiceEnabled;
     if ((m_lastProcessedModified == "" || !isRemoteServiceEnabled) && !localSyncAndProcess())
     {
-        dbgWarning(D_WAAP_CONFIDENCE_CALCULATOR) << "local sync and process failed";
+        dbgWarning(D_WAAP_SERIALIZE) << "local sync and process failed";
         return;
     }
 
     if (mode == OrchestrationMode::HYBRID) {
-        dbgDebug(D_WAAP_CONFIDENCE_CALCULATOR) << "detected running in standalone mode";
+        dbgDebug(D_WAAP_SERIALIZE) << "detected running in standalone mode";
         I_AgentDetails *agentDetails = Singleton::Consume<I_AgentDetails>::by<WaapComponent>();
         I_Messaging *messaging = Singleton::Consume<I_Messaging>::by<WaapComponent>();
 
@@ -738,6 +864,7 @@ void SerializeToLocalAndRemoteSyncBase::syncWorker()
         MessageMetadata req_md(getLearningHost(), 80);
         req_md.insertHeader("X-Tenant-Id", agentDetails->getTenantId());
         req_md.setConnectioFlag(MessageConnectionConfig::UNSECURE_CONN);
+        req_md.setConnectioFlag(MessageConnectionConfig::ONE_TIME_CONN);
         bool ok = messaging->sendSyncMessageWithoutResponse(
             HTTPMethod::POST,
             "/api/sync",
@@ -745,14 +872,14 @@ void SerializeToLocalAndRemoteSyncBase::syncWorker()
             MessageCategory::GENERIC,
             req_md
         );
-        dbgDebug(D_WAAP_CONFIDENCE_CALCULATOR) << "sent learning sync notification ok: " << ok;
+        dbgDebug(D_WAAP_SERIALIZE) << "sent learning sync notification ok: " << ok;
         if (!ok) {
-            dbgWarning(D_WAAP_CONFIDENCE_CALCULATOR) << "failed to send learning notification";
+            dbgWarning(D_WAAP_SERIALIZE) << "failed to send learning notification";
         }
     } else {
         SyncLearningNotificationObject syncNotification(m_assetId, m_type, getWindowId());
 
-        dbgDebug(D_WAAP_CONFIDENCE_CALCULATOR) << "sending sync notification: " << syncNotification;
+        dbgDebug(D_WAAP_SERIALIZE) << "sending sync notification: " << syncNotification;
 
         ReportMessaging(
             "sync notification for '" + m_assetId + "'",
@@ -766,6 +893,8 @@ void SerializeToLocalAndRemoteSyncBase::syncWorker()
 
     if (m_lastProcessedModified != "" && isRemoteServiceEnabled)
     {
+        // wait for remote service to process the data
+        waitSync();
         updateStateFromRemoteService();
     }
 }
@@ -775,7 +904,7 @@ void SerializeToLocalAndRemoteSyncBase::restore()
     SerializeToFileBase::restore();
     if (!isBase())
     {
-        dbgTrace(D_WAAP_CONFIDENCE_CALCULATOR) << "merge state from remote service";
+        dbgTrace(D_WAAP_SERIALIZE) << "merge state from remote service";
         mergeProcessedFromRemote();
     }
 }
@@ -789,7 +918,7 @@ RemoteFilesList SerializeToLocalAndRemoteSyncBase::getRemoteProcessedFilesList()
 
     if (!isRemoteServiceEnabled)
     {
-        dbgDebug(D_WAAP_CONFIDENCE_CALCULATOR) << "remote service is disabled";
+        dbgDebug(D_WAAP_SERIALIZE) << "remote service is disabled";
         return remoteFiles;
     }
 
@@ -800,7 +929,7 @@ RemoteFilesList SerializeToLocalAndRemoteSyncBase::getRemoteProcessedFilesList()
 
     if (!isSuccessful)
     {
-        dbgWarning(D_WAAP_CONFIDENCE_CALCULATOR) << "Failed to get the list of files";
+        dbgWarning(D_WAAP_SERIALIZE) << "Failed to get the list of files";
     }
     return remoteFiles;
 }
@@ -814,12 +943,12 @@ RemoteFilesList SerializeToLocalAndRemoteSyncBase::getProcessedFilesList()
     {
         const vector<FileMetaData>& filesMD = processedFilesList.getFilesMetadataList();
         if (filesMD.size() > 1) {
-            dbgWarning(D_WAAP_CONFIDENCE_CALCULATOR) << "got more than 1 expected processed file";
+            dbgWarning(D_WAAP_SERIALIZE) << "got more than 1 expected processed file";
         }
         if (!filesMD.empty()) {
             m_lastProcessedModified = filesMD[0].modified;
         }
-        dbgTrace(D_WAAP_CONFIDENCE_CALCULATOR) << "found " << filesMD.size() << " remote service state files. "
+        dbgTrace(D_WAAP_SERIALIZE) << "found " << filesMD.size() << " remote service state files. "
             "last modified: " << m_lastProcessedModified;
 
         return processedFilesList;
@@ -833,11 +962,11 @@ RemoteFilesList SerializeToLocalAndRemoteSyncBase::getProcessedFilesList()
 
     if (!isSuccessful)
     {
-        dbgDebug(D_WAAP_CONFIDENCE_CALCULATOR) << "Failed to get the list of files";
+        dbgDebug(D_WAAP_SERIALIZE) << "Failed to get the list of files";
     }
     else if (!processedFilesList.getFilesList().empty())
     {
-        dbgTrace(D_WAAP_CONFIDENCE_CALCULATOR) << "found state files";
+        dbgTrace(D_WAAP_SERIALIZE) << "found state files";
         return processedFilesList;
     }
     // backward compatibility - try to get backup file with the buggy prefix tenantID/assetID/instanceID/
@@ -846,7 +975,7 @@ RemoteFilesList SerializeToLocalAndRemoteSyncBase::getProcessedFilesList()
     pos = bcRemotePath.find('/', pos + 1);
     if (!Singleton::exists<I_InstanceAwareness>())
     {
-        dbgDebug(D_WAAP_CONFIDENCE_CALCULATOR) << "missing instance of instance awareness,"
+        dbgDebug(D_WAAP_SERIALIZE) << "missing instance of instance awareness,"
             " can't check backward compatibility";
         return processedFilesList;
     }
@@ -854,13 +983,13 @@ RemoteFilesList SerializeToLocalAndRemoteSyncBase::getProcessedFilesList()
     Maybe<string> id = instanceAwareness->getUniqueID();
     if (!id.ok())
     {
-        dbgDebug(D_WAAP_CONFIDENCE_CALCULATOR) << "failed to get instance id err: " << id.getErr() <<
+        dbgDebug(D_WAAP_SERIALIZE) << "failed to get instance id err: " << id.getErr() <<
             ". can't check backward compatibility";
         return processedFilesList;
     }
     string idStr = id.unpack();
     bcRemotePath.insert(pos + 1, idStr + "/");
-    dbgDebug(D_WAAP_CONFIDENCE_CALCULATOR) << "List of files is empty - trying to get the file from " <<
+    dbgDebug(D_WAAP_SERIALIZE) << "List of files is empty - trying to get the file from " <<
         bcRemotePath;
 
     isSuccessful = sendObject(
@@ -870,16 +999,16 @@ RemoteFilesList SerializeToLocalAndRemoteSyncBase::getProcessedFilesList()
 
     if (!isSuccessful)
     {
-        dbgWarning(D_WAAP_CONFIDENCE_CALCULATOR) << "Failed to get the list of files";
+        dbgWarning(D_WAAP_SERIALIZE) << "Failed to get the list of files";
     }
-    dbgDebug(D_WAAP_CONFIDENCE_CALCULATOR) << "backwards computability: got "
+    dbgDebug(D_WAAP_SERIALIZE) << "backwards computability: got "
         << processedFilesList.getFilesList().size() << " state files";
     return processedFilesList;
 }
 
 void SerializeToLocalAndRemoteSyncBase::mergeProcessedFromRemote()
 {
-    dbgDebug(D_WAAP_CONFIDENCE_CALCULATOR) << "Merging processed data from remote. assetId='" << m_assetId <<
+    dbgDebug(D_WAAP_SERIALIZE) << "Merging processed data from remote. assetId='" << m_assetId <<
         "', owner='" << m_owner << "'";
     m_pMainLoop->addOneTimeRoutine(
         I_MainLoop::RoutineType::Offline,
@@ -903,7 +1032,7 @@ SerializeToLocalAndRemoteSyncBase::getLearningHost()
             m_learning_host = string(learningHost);
             return learningHost;
         }
-        dbgWarning(D_WAAP_CONFIDENCE_CALCULATOR) << "learning host is not set. using default";
+        dbgWarning(D_WAAP_SERIALIZE) << "learning host is not set. using default";
     }
     return defaultLearningHost;
 }
@@ -919,7 +1048,7 @@ SerializeToLocalAndRemoteSyncBase::getSharedStorageHost()
             m_shared_storage_host = string(sharedStorageHost);
             return sharedStorageHost;
         }
-        dbgWarning(D_WAAP_CONFIDENCE_CALCULATOR) << "shared storage host is not set. using default";
+        dbgWarning(D_WAAP_SERIALIZE) << "shared storage host is not set. using default";
     }
     return defaultSharedStorageHost;
 }

components/security_apps/waap/waap_clib/Signatures.cc

@@ -116,7 +116,7 @@ Signatures::Signatures(const std::string& filepath) :
     ),
     allowed_text_re(sigsSource["allowed_text_re"].get<std::string>(), error, "allowed_text_re"),
     pipe_split_re(
-        "([\\w\\=\\-\\_\\.\\,\\(\\)\\[\\]\\/\\%\\s]+?)\\||([\\w\\=\\-\\_\\.\\,\\(\\)\\[\\]\\/\\%\\s]+)|\\|()",
+        "([^|]*)\\||([^|]+)|\\|()",
         error,
         "pipe_decode"),
     semicolon_split_re("([\\w\\=\\-\\_\\.\\,\\(\\)\\%]+?);|([\\w\\=\\-\\_\\.\\,\\(\\)\\%]+)|;()", error, "sem_decode"),

components/security_apps/waap/waap_clib/SingleDecision.cc

@@ -19,7 +19,10 @@ USE_DEBUG_FLAG(D_WAAP);
 SingleDecision::SingleDecision(DecisionType type):
     m_type(type),
     m_log(false),
-    m_block(false)
+    m_block(false),
+    m_ForceLog(false),
+    m_forceAllow(false),
+    m_forceBlock(false)
 {}
 
 SingleDecision::~SingleDecision()
@@ -35,11 +38,28 @@ bool SingleDecision::shouldLog() const
     return m_log;
 }
 
+bool SingleDecision::shouldForceLog() const
+{
+    return m_ForceLog;
+}
+
 bool SingleDecision::shouldBlock() const
 {
     return m_block;
 }
 
+bool SingleDecision::shouldForceAllow() const
+{
+    dbgTrace(D_WAAP) << "should force allow: " << m_forceAllow;
+    return m_forceAllow;
+}
+
+bool SingleDecision::shouldForceBlock() const
+{
+    dbgTrace(D_WAAP) << "should force block: " << m_forceBlock;
+    return m_forceBlock;
+}
+
 void SingleDecision::setLog(bool log)
 {
     dbgTrace(D_WAAP) << "Decision " << getTypeStr() << " changes should log from " << m_log << " to " << log;
@@ -51,3 +71,22 @@ void SingleDecision::setBlock(bool block)
     dbgTrace(D_WAAP) << "Decision " << getTypeStr() << " changes should block  from " << m_block << " to " << block;
     m_block = block;
 }
+
+void SingleDecision::setForceLog(bool overridesLog)
+{
+    dbgTrace(D_WAAP) << "Decision "<< getTypeStr() <<
+        " changes overrides log from " << m_ForceLog << " to " << overridesLog;
+    m_ForceLog = overridesLog;
+}
+
+void SingleDecision::setForceAllow(bool allow)
+{
+    dbgTrace(D_WAAP) << "Decision " << getTypeStr() << " changes force allow from " << m_forceAllow << " to " << allow;
+    m_forceAllow = allow;
+}
+
+void SingleDecision::setForceBlock(bool block)
+{
+    dbgTrace(D_WAAP) << "Decision " << getTypeStr() << " changes force block from " << m_forceBlock << " to " << block;
+    m_forceBlock = block;
+}

components/security_apps/waap/waap_clib/SingleDecision.h

@@ -25,15 +25,24 @@ public:
 
     void setLog(bool log);
     void setBlock(bool block);
+    void setForceLog(bool overridesLog);
+    void setForceAllow(bool allow);
+    void setForceBlock(bool block);
     DecisionType getType() const;
     bool shouldLog() const;
     bool shouldBlock() const;
+    bool shouldForceLog() const;
+    bool shouldForceAllow() const;
+    bool shouldForceBlock() const;
     virtual std::string getTypeStr() const = 0;
 
 protected:
     DecisionType m_type;
     bool m_log;
     bool m_block;
+    bool m_ForceLog;
+    bool m_forceAllow;
+    bool m_forceBlock;
 };
 
 #endif

components/security_apps/waap/waap_clib/Telemetry.cc

@@ -76,6 +76,7 @@ WaapTelemetrics::initMetrics()
     waf_blocked.report(0);
     force_and_block_exceptions.report(0);
 }
+
 void
 WaapTelemetrics::updateMetrics(const string &asset_id, const DecisionTelemetryData &data)
 {
@@ -243,6 +244,46 @@ WaapAttackTypesMetrics::updateMetrics(const string &asset_id, const DecisionTele
     }
 }
 
+void
+WaapAdditionalTrafficTelemetrics::initMetrics()
+{
+    requests.report(0);
+    sources.report(0);
+    blocked.report(0);
+    temperature_count.report(0);
+    sources_seen.clear();
+}
+
+void
+WaapAdditionalTrafficTelemetrics::updateMetrics(const string &asset_id, const DecisionTelemetryData &data)
+{
+    initMetrics();
+
+    auto is_keep_alive_ctx = Singleton::Consume<I_Environment>::by<GenericMetric>()->get<bool>(
+        "keep_alive_request_ctx"
+    );
+    if (!is_keep_alive_ctx.ok() || !*is_keep_alive_ctx) {
+        requests.report(1);
+    } else {
+        dbgTrace(D_WAAP) << "Not increasing the number of requests due to keep alive";
+    }
+
+    if (!data.source.empty()) {
+        if (sources_seen.find(data.source) == sources_seen.end()) {
+            sources_seen.insert(data.source);
+            sources.report(1);
+        }
+    }
+
+    if (data.blockType == WAF_BLOCK) {
+        blocked.report(1);
+    }
+
+    if (data.temperatureDetected) {
+        temperature_count.report(1);
+    }
+}
+
 void
 WaapMetricWrapper::upon(const WaapTelemetryEvent &event)
 {
@@ -268,10 +309,17 @@ WaapMetricWrapper::upon(const WaapTelemetryEvent &event)
         attack_types_telemetries
     );
     initializeTelemetryData<WaapTrafficTelemetrics>(asset_id, data, "WAAP traffic telemetry", traffic_telemetries);
+    initializeTelemetryData<WaapAdditionalTrafficTelemetrics>(
+        asset_id,
+        data,
+        "WAAP Additional Traffic Telemetry",
+        additional_traffic_telemetries
+    );
 
     telemetries[asset_id]->updateMetrics(asset_id, data);
     attack_types_telemetries[asset_id]->updateMetrics(asset_id, data);
     traffic_telemetries[asset_id]->updateMetrics(asset_id, data);
+    additional_traffic_telemetries[asset_id]->updateMetrics(asset_id, data);
 
     auto agent_mode = Singleton::Consume<I_AgentDetails>::by<WaapMetricWrapper>()->getOrchestrationMode();
     string tenant_id = Singleton::Consume<I_AgentDetails>::by<WaapMetricWrapper>()->getTenantId();

components/security_apps/waap/waap_clib/TuningDecisions.h

@@ -60,6 +60,7 @@ private:
             MessageMetadata req_md(getSharedStorageHost(), 80);
             req_md.insertHeader("X-Tenant-Id", agentDetails->getTenantId());
             req_md.setConnectioFlag(MessageConnectionConfig::UNSECURE_CONN);
+            req_md.setConnectioFlag(MessageConnectionConfig::ONE_TIME_CONN);
             auto req_status = messaging->sendSyncMessage(
                 method,
                 uri,
@@ -69,11 +70,14 @@ private:
             );
             return req_status.ok();
         }
+        MessageMetadata req_md;
+        req_md.setConnectioFlag(MessageConnectionConfig::ONE_TIME_FOG_CONN);
         auto req_status = messaging->sendSyncMessage(
             method,
             uri,
             obj,
-            MessageCategory::GENERIC
+            MessageCategory::GENERIC,
+            req_md
         );
         return req_status.ok();
     }

components/security_apps/waap/waap_clib/TypeIndicatorsFilter.cc

@@ -128,6 +128,8 @@ void TypeIndicatorFilter::loadParams(std::shared_ptr<Waap::Parameters::WaapParam
         std::to_string(TYPE_FILTER_CONFIDENCE_THRESHOLD)));
     std::string learnPermanentlyStr = pParams->getParamVal("typeIndicators.learnPermanently", "true");
     params.learnPermanently = !boost::iequals(learnPermanentlyStr, "false");
+    params.maxMemoryUsage = std::stoul(pParams->getParamVal("typeIndicators.maxMemoryUsage",
+        std::to_string(TYPE_FILTER_CONFIDENCE_MAX_MEMORY_USAGE)));
 
     std::string remoteSyncStr = pParams->getParamVal("remoteSync", "true");
     bool syncEnabled = !boost::iequals(remoteSyncStr, "false");

components/security_apps/waap/waap_clib/TypeIndicatorsFilter.h

@@ -24,6 +24,7 @@
 #define TYPE_FILTER_CONFIDENCE_MIN_INTERVALS 5
 #define TYPE_FILTER_CONFIDENCE_THRESHOLD 0.8
 #define TYPE_FILTER_INTERVAL_DURATION std::chrono::minutes(60)
+#define TYPE_FILTER_CONFIDENCE_MAX_MEMORY_USAGE (40 * 1024 * 1024) // 40MB
 
 class TypeIndicatorFilter : public IndicatorFilterBase
 {

components/security_apps/waap/waap_clib/WaapAssetState.cc

@@ -446,13 +446,14 @@ WaapAssetState::WaapAssetState(std::shared_ptr<Signatures> signatures,
         // update orig_size and orig_capacity after string copy
         orig_size = text.size();
         orig_capacity = text.capacity();
-        dbgTrace(D_WAAP_SAMPLE_PREPROCESS) << "unescape: (1) (after filterUTF7) '" << text << "'";
+        dbgTrace(D_WAAP_SAMPLE_PREPROCESS) << "unescape: (1) (after filterUTF7) '" << text <<
+            "' size: " << text.size();
 
         // 2. Replace %xx sequences by their single-character equivalents.
         // Also replaces '+' symbol by space character.
         // Python equivalent: text = urllib.unquote_plus(text)
         text.erase(unquote_plus(text.begin(), text.end()), text.end());
-        dbgTrace(D_WAAP_SAMPLE_PREPROCESS) << "unescape: (2) '" << text << "'";
+        dbgTrace(D_WAAP_SAMPLE_PREPROCESS) << "unescape: (2) '" << text << "' size: " << text.size();
 
         fixBreakingSpace(text);
 
@@ -460,38 +461,38 @@ WaapAssetState::WaapAssetState(std::shared_ptr<Signatures> signatures,
         // remove all characters whose ASCII code is >=128.
         // Python equivalent: text.encode('ascii',errors='ignore')
         filterUnicode(text);
-        dbgTrace(D_WAAP_SAMPLE_PREPROCESS) << "unescape: (3) '" << text << "'";
+        dbgTrace(D_WAAP_SAMPLE_PREPROCESS) << "unescape: (3) '" << text << "' size: " << text.size();
 
         // 4. oh shi?... should I handle unicode html entities (python's htmlentitydefs module)???
         // Python equivalent: text = HTMLParser.HTMLParser().unescape(text)
         text.erase(escape_html(text.begin(), text.end()), text.end());
-        dbgTrace(D_WAAP_SAMPLE_PREPROCESS) << "unescape: (4) '" << text << "'";
+        dbgTrace(D_WAAP_SAMPLE_PREPROCESS) << "unescape: (4) '" << text << "' size: " << text.size();
 
         // 5. Apply backslash escaping (like in C)
         // Python equivalent: text = text.decode('string_escape')
         text.erase(escape_backslashes(text.begin(), text.end()), text.end());
-        dbgTrace(D_WAAP_SAMPLE_PREPROCESS) << "unescape: (5) '" << text << "'";
+        dbgTrace(D_WAAP_SAMPLE_PREPROCESS) << "unescape: (5) '" << text << "' size: " << text.size();
 
         // 6. remove all unicode characters from string. Basically,
         // remove all characters whose ASCII code is >=128.
         // Python equivalent: text.encode('ascii',errors='ignore')
         filterUnicode(text);
-        dbgTrace(D_WAAP_SAMPLE_PREPROCESS) << "unescape: (6) '" << text << "'";
+        dbgTrace(D_WAAP_SAMPLE_PREPROCESS) << "unescape: (6) '" << text << "' size: " << text.size();
 
         // 7. Replace %xx sequences by their single-character equivalents.
         // Also replaces '+' symbol by space character.
         // Python equivalent: text = urllib.unquote_plus(text)
         text.erase(unquote_plus(text.begin(), text.end()), text.end());
-        dbgTrace(D_WAAP_SAMPLE_PREPROCESS) << "unescape: (7) '" << text << "'";
+        dbgTrace(D_WAAP_SAMPLE_PREPROCESS) << "unescape: (7) '" << text << "' size: " << text.size();
 
         unescapeUnicode(text);
-        dbgTrace(D_WAAP_SAMPLE_PREPROCESS) << "after unescapeUnicode '" << text << "'";
+        dbgTrace(D_WAAP_SAMPLE_PREPROCESS) << "after unescapeUnicode '" << text << "' size: " << text.size();
 
         // 8. remove all unicode characters from string. Basically,
         // remove all characters whose ASCII code is >=128.
         // Python equivalent: text.encode('ascii',errors='ignore')
         filterUnicode(text);
-        dbgTrace(D_WAAP_SAMPLE_PREPROCESS) << "unescape: (8) '" << text << "'";
+        dbgTrace(D_WAAP_SAMPLE_PREPROCESS) << "unescape: (8) '" << text << "' size: " << text.size();
 
         // 9. ???
         //
@@ -530,7 +531,7 @@ WaapAssetState::WaapAssetState(std::shared_ptr<Signatures> signatures,
             << AlertInfo(AlertTeam::CORE, "WAAP sample processing")
             << "unescape: original size=" << orig_size << " capacity=" << orig_capacity
             << " new size=" << text.size() << " capacity=" << text.capacity()
-            << " text='" << text << "'";
+            << " text='" << text << "'" << " orig text='" << s << "'";
 
         return text;
     }
@@ -1098,6 +1099,9 @@ WaapAssetState::apply(
     // Detect long text spans, and also any-length spans that end with file extensions such as ".jpg"
     bool longTextFound = m_Signatures->longtext_re.hasMatch(res.unescaped_line);
 
+    // When this flag remains false until the last evasion handling, a second unescape is performed
+    bool evasion_detected = false;
+
     if (longTextFound) {
         dbgTrace(D_WAAP_SAMPLE_SCAN) << "longtext found";
     }
@@ -1297,6 +1301,7 @@ WaapAssetState::apply(
         }
 
         if (kwCount != res.keyword_matches.size() && !binaryDataFound) {
+            evasion_detected = true;
             // Recalculate repetition and/or probing indicators
             unsigned int newWordsCount = 0;
             calcRepetitionAndProbing(res, ignored_keywords, unescaped, detectedRepetition, detectedProbing,
@@ -1326,6 +1331,7 @@ WaapAssetState::apply(
         }
 
         if (kwCount != res.keyword_matches.size() && !binaryDataFound) {
+            evasion_detected = true;
             // Recalculate repetition and/or probing indicators
             unsigned int newWordsCount = 0;
             calcRepetitionAndProbing(res, ignored_keywords, unescaped, detectedRepetition, detectedProbing,
@@ -1409,6 +1415,42 @@ WaapAssetState::apply(
         dbgTrace(D_WAAP_EVASIONS) << "status after evasion checking " << nicePrint(res);
     }
 
+    bool tab_evasion = (res.unescaped_line.find('\t') != std::string::npos);
+
+    if (tab_evasion) {
+        dbgTrace(D_WAAP_EVASIONS) << "Tab character evasion detected";
+
+        // Create normalized version with tabs removed
+        std::string unescaped = res.unescaped_line;
+
+        // Remove all tab characters to normalize the string
+        std::string::iterator end_pos = std::remove(unescaped.begin(), unescaped.end(), '\t');
+        unescaped.erase(end_pos, unescaped.end());
+
+        size_t kwCount = res.keyword_matches.size();
+
+        if (res.unescaped_line != unescaped) {
+            SampleValue unescapedSample(unescaped, m_Signatures->m_regexPreconditions);
+            checkRegex(unescapedSample, m_Signatures->specific_acuracy_keywords_regex, res.keyword_matches,
+                res.found_patterns, longTextFound, binaryDataFound);
+            checkRegex(unescapedSample, m_Signatures->words_regex, res.keyword_matches, res.found_patterns,
+                longTextFound, binaryDataFound);
+            checkRegex(unescapedSample, m_Signatures->pattern_regex, res.regex_matches, res.found_patterns,
+                longTextFound, binaryDataFound);
+        }
+
+        if (kwCount != res.keyword_matches.size() && !binaryDataFound) {
+            // If new keywords were found, add a specific indication
+            res.keyword_matches.push_back("tab_character_evasion");
+
+            // Recalculate repetition and/or probing indicators
+            unsigned int newWordsCount = 0;
+            calcRepetitionAndProbing(res, ignored_keywords, unescaped, detectedRepetition, detectedProbing,
+                newWordsCount);
+            // Take minimal words count because empirically it means evasion was probably successfully decoded
+            wordsCount = std::min(wordsCount, newWordsCount);
+        }
+    }
 
     bool quoutes_space_evasion = Waap::Util::find_in_map_of_stringlists_keys(
         "quotes_space_ev_fast_reg",
@@ -1471,6 +1513,7 @@ WaapAssetState::apply(
         }
 
         if (kwCount != res.keyword_matches.size() && !binaryDataFound) {
+            evasion_detected = true;
             // Recalculate repetition and/or probing indicators
             unsigned int newWordsCount = 0;
             calcRepetitionAndProbing(res, ignored_keywords, unescaped, detectedRepetition, detectedProbing,
@@ -1499,7 +1542,7 @@ WaapAssetState::apply(
                     longTextFound, binaryDataFound);
         }
 
-
+        evasion_detected = true;
         // Recalculate repetition and/or probing indicators
         unsigned int newWordsCount = 0;
         calcRepetitionAndProbing(res, ignored_keywords, unescaped, detectedRepetition, detectedProbing,
@@ -1613,6 +1656,7 @@ WaapAssetState::apply(
         }
 
         if (kwCount != res.keyword_matches.size() && !binaryDataFound) {
+            evasion_detected = true;
             // Recalculate repetition and/or probing indicators
             unsigned int newWordsCount = 0;
             calcRepetitionAndProbing(res, ignored_keywords, unescaped, detectedRepetition, detectedProbing,
@@ -1644,6 +1688,7 @@ WaapAssetState::apply(
         }
 
         if (kwCount != res.keyword_matches.size() && !binaryDataFound) {
+            evasion_detected = true;
             // Recalculate repetition and/or probing indicators
             unsigned int newWordsCount = 0;
             calcRepetitionAndProbing(res, ignored_keywords, unescaped, detectedRepetition, detectedProbing,
@@ -1675,6 +1720,7 @@ WaapAssetState::apply(
         }
 
         if (kwCount != res.keyword_matches.size() && !binaryDataFound) {
+            evasion_detected = true;
             // Recalculate repetition and/or probing indicators
             unsigned int newWordsCount = 0;
             calcRepetitionAndProbing(res, ignored_keywords, unescaped, detectedRepetition, detectedProbing,
@@ -1706,6 +1752,7 @@ WaapAssetState::apply(
         }
 
         if (kwCount != res.keyword_matches.size() && !binaryDataFound) {
+            evasion_detected = true;
             // Recalculate repetition and/or probing indicators
             unsigned int newWordsCount = 0;
             calcRepetitionAndProbing(res, ignored_keywords, unescaped, detectedRepetition, detectedProbing,
@@ -1715,6 +1762,74 @@ WaapAssetState::apply(
         }
     }
 
+    // Detect evasion for command injection vulnerability using encoded characters (such as ^).
+    // example: "c^m^d /c d^i^r" -> "cmd /c dir"
+    // if '^' is found in the line, remove it and rescan
+    // search in scan result for any keyword with "os", "cmd", "exec" "command" or "shell" in it
+    // if not found, we don't want this evasion
+    // if found, we want to use its results
+    // Note: this is not a perfect solution, but it is better than nothing
+    if (line.find('^') != std::string::npos) {
+        dbgTrace(D_WAAP_EVASIONS) << "Windows command injection evasion suspected";
+
+        std::string unescaped = line;
+        bool is_win_cmd_evasion = false;
+        // remove all occurances of '^' character
+        unescaped.erase(std::remove(unescaped.begin(), unescaped.end(), '^'), unescaped.end());
+        dbgTrace(D_WAAP_EVASIONS) << "unescaped == '" << unescaped << "'";
+        if (!unescaped.empty()) {
+            std::vector<std::string> evKeywordMatches(res.keyword_matches);
+            std::vector<std::string> evRegexMatches(res.regex_matches);
+            Waap::Util::map_of_stringlists_t evFoundPatterns(res.found_patterns);
+            bool evLongTextFound = longTextFound;
+            bool evBinaryDataFound = binaryDataFound;
+
+            if (line != unescaped) {
+                SampleValue unescapedSample(unescaped, m_Signatures->m_regexPreconditions);
+                checkRegex(unescapedSample, m_Signatures->specific_acuracy_keywords_regex, evKeywordMatches,
+                    res.found_patterns, evLongTextFound, evBinaryDataFound);
+                checkRegex(unescapedSample, m_Signatures->words_regex, evKeywordMatches, evFoundPatterns,
+                    evLongTextFound, evBinaryDataFound);
+                checkRegex(unescapedSample, m_Signatures->pattern_regex, evRegexMatches, evFoundPatterns,
+                    evLongTextFound, evBinaryDataFound);
+            }
+
+            if (evKeywordMatches.size() != res.keyword_matches.size()) {
+                if (Waap::Util::find_in_map_of_stringlists_keys("os_commands", evFoundPatterns)) {
+                    is_win_cmd_evasion = true;
+                } else {
+                    for (const auto &kw : evKeywordMatches) {
+                        if (kw.size() < 2 ||
+                                str_contains(kw, "cmd") ||
+                                str_contains(kw, "command") ||
+                                str_contains(kw, "os_") ||
+                                str_contains(kw, "exec")) {
+                            is_win_cmd_evasion = true;
+                            break;
+                        }
+                    }
+                }
+            }
+
+            if (is_win_cmd_evasion) {
+                dbgTrace(D_WAAP_EVASIONS) << "Evasion and relevant matches found, updating results";
+                // found relevant keywords after unescaping, set to new matches
+                res.keyword_matches = evKeywordMatches;
+                res.regex_matches = evRegexMatches;
+                res.found_patterns = evFoundPatterns;
+                longTextFound = evLongTextFound;
+                binaryDataFound = evBinaryDataFound;
+                // Recalculate repetition and/or probing indicators
+                unsigned int newWordsCount = 0;
+                calcRepetitionAndProbing(res, ignored_keywords, unescaped, detectedRepetition, detectedProbing,
+                    newWordsCount);
+                // Take minimal words count because empirically it means evasion was probably successfully decoded
+                wordsCount = std::min(wordsCount, newWordsCount);
+            } else {
+                dbgTrace(D_WAAP_EVASIONS) << "Evasion not relevant, will not apply to results";
+            }
+        }
+    }
 
     // python: escape ='hi_acur_fast_reg_evasion' in found_patterns
     bool escape = Waap::Util::find_in_map_of_stringlists_keys("evasion", res.found_patterns);
@@ -1757,6 +1872,7 @@ WaapAssetState::apply(
             escape = false;
         }
         else if (!binaryDataFound) {
+            evasion_detected = true;
             // Recalculate repetition and/or probing indicators
             unsigned int newWordsCount = 0;
             calcRepetitionAndProbing(res, ignored_keywords, unescaped, detectedRepetition, detectedProbing,
@@ -1833,6 +1949,7 @@ WaapAssetState::apply(
             escape = false;
         }
         else if (!binaryDataFound) {
+            evasion_detected = true;
             // Recalculate repetition and/or probing indicators
             unsigned int newWordsCount = 0;
             calcRepetitionAndProbing(res, ignored_keywords, unescaped, detectedRepetition, detectedProbing,
@@ -1841,6 +1958,42 @@ WaapAssetState::apply(
             wordsCount = std::min(wordsCount, newWordsCount);
         }
     }
+
+    // Handle second URL encoding evasion
+    if (!evasion_detected &&
+            !binaryDataFound &&
+            !longTextFound &&
+            Waap::Util::containsPercentEncoding(res.unescaped_line)) {
+        dbgTrace(D_WAAP_EVASIONS) << "Second URL encoding evasion detected";
+
+        std::string unescaped = unescape(res.unescaped_line);
+        size_t kwCount = res.keyword_matches.size();
+
+        if (res.unescaped_line != unescaped) {
+            SampleValue unescapedSample(unescaped, m_Signatures->m_regexPreconditions);
+            checkRegex(unescapedSample, m_Signatures->specific_acuracy_keywords_regex, res.keyword_matches,
+                res.found_patterns, longTextFound, binaryDataFound);
+            checkRegex(unescapedSample, m_Signatures->words_regex, res.keyword_matches, res.found_patterns,
+                longTextFound, binaryDataFound);
+            checkRegex(unescapedSample, m_Signatures->pattern_regex, res.regex_matches, res.found_patterns,
+                longTextFound, binaryDataFound);
+        }
+
+        if (kwCount == res.keyword_matches.size()) {
+            // Remove the evasion keyword if no real evasion found
+            keywordsToRemove.push_back("evasion");
+        }
+        else if (!binaryDataFound) {
+            evasion_detected = true;
+            // Recalculate repetition and/or probing indicators
+            unsigned int newWordsCount = 0;
+            calcRepetitionAndProbing(res, ignored_keywords, unescaped, detectedRepetition, detectedProbing,
+                newWordsCount);
+            // Take minimal words count because empirically it means evasion was probably succesfully decoded
+            wordsCount = std::min(wordsCount, newWordsCount);
+        }
+    }
+
     dbgTrace(D_WAAP_SAMPLE_SCAN) << "after evasions..." << nicePrint(res);
     // Remove evasion keywords that should not be reported because there's no real evasion found
     if (!keywordsToRemove.empty()) {

components/security_apps/waap/waap_clib/WaapConfigApplication.cc

@@ -83,7 +83,6 @@ void WaapConfigApplication::load(cereal::JSONInputArchive& ar)
     assets_ids_aggregation.insert(m_assetId);
 }
 
-
 bool WaapConfigApplication::operator==(const WaapConfigApplication& other) const
 {
     const WaapConfigBase* configBase = this;

components/security_apps/waap/waap_clib/WaapConfigBase.cc

@@ -132,7 +132,7 @@ void WaapConfigBase::loadOverridePolicy(cereal::JSONInputArchive& ar)
     try {
         m_overridePolicy = std::make_shared<Waap::Override::Policy>(ar);
     }
-    catch (std::runtime_error& e) {
+    catch (const cereal::Exception &e) {
         ar.setNextName(nullptr);
         dbgWarning(D_WAAP) << failMessage << e.what();
         m_overridePolicy = nullptr;
@@ -329,37 +329,38 @@ const std::string& WaapConfigBase::get_AssetName() const
     return m_assetName;
 }
 
-const std::string& WaapConfigBase::get_PracticeIdByPactice(DecisionType practiceType) const
+const string &
+WaapConfigBase::get_PracticeIdByPactice(DecisionType practiceType) const
 {
-
+    dbgTrace(D_WAAP) << "get practice ID of decision type: " << practiceType;
     switch (practiceType)
     {
-    case DecisionType::AUTONOMOUS_SECURITY_DECISION:
-        return m_practiceId;
+    case DecisionType::AUTONOMOUS_SECURITY_DECISION: break;
     default:
-        dbgError(D_WAAP)
-        << "Can't find practice type for practice ID by practice: "
-        << practiceType
-        << ", return web app practice ID";
-        return m_practiceId;
+        dbgDebug(D_WAAP)
+            << "Can't find practice type for practice ID by practice: "
+            << practiceType
+            << ", return web app practice ID";
     }
 
+    return m_practiceId;
 }
 
-const std::string& WaapConfigBase::get_PracticeNameByPactice(DecisionType practiceType) const
+const string &
+WaapConfigBase::get_PracticeNameByPactice(DecisionType practiceType) const
 {
+    dbgTrace(D_WAAP) << "get practice name of decision type: " << practiceType;
     switch (practiceType)
     {
-    case DecisionType::AUTONOMOUS_SECURITY_DECISION:
-        return m_practiceName;
+    case DecisionType::AUTONOMOUS_SECURITY_DECISION: break;
     default:
-        dbgError(D_WAAP)
-        << "Can't find practice type for practice name by practice: "
-        << practiceType
-        << ", return web app practice name";
-        return m_practiceName;
+        dbgDebug(D_WAAP)
+            << "Can't find practice type for practice name by practice: "
+            << practiceType
+            << ", return web app practice name";
     }
 
+    return m_practiceName;
 }
 
 const std::string& WaapConfigBase::get_RuleId() const

components/security_apps/waap/waap_clib/WaapOverride.cc

@@ -82,11 +82,48 @@ State::State() :
     forceBlockIds(),
     bForceException(false),
     forceExceptionIds(),
-    bIgnoreLog(false),
+    bSupressLog(false),
     bSourceIdentifierOverride(false),
     sSourceIdentifierMatch("")
 {
 }
 
+bool ExceptionsByPractice::operator==(const ExceptionsByPractice &other) const
+{
+    return m_web_app_ids == other.m_web_app_ids &&
+        m_api_protect_ids == other.m_api_protect_ids &&
+        m_anti_bot_ids == other.m_anti_bot_ids;
+}
+
+const std::vector<std::string>& ExceptionsByPractice::getExceptionsOfPractice(DecisionType practiceType) const
+{
+    switch (practiceType)
+    {
+
+    case DecisionType::AUTONOMOUS_SECURITY_DECISION:
+        return m_web_app_ids;
+    default:
+        dbgError(D_WAAP) <<
+        "Can't find practice type for exceptions by practice: " <<
+        practiceType <<
+        ", return web app exceptions";
+        return m_web_app_ids;
+    }
+}
+
+const std::set<std::string>& ExceptionsByPractice::getAllExceptions() const
+{
+    return m_all_ids;
+}
+
+bool ExceptionsByPractice::isIDInWebApp(const std::string &id) const
+{
+    auto it = std::find(m_web_app_ids.begin(), m_web_app_ids.end(), id);
+    if (it != m_web_app_ids.end()) {
+        dbgTrace(D_WAAP) << "rule id is in web application exceptions by practice: " << id;
+        return true;
+    }
+    return false;
+}
 }
 }

components/security_apps/waap/waap_clib/WaapOverride.h

@@ -21,6 +21,7 @@
 #include <memory>
 #include "debug.h"
 #include "CidrMatch.h"
+#include "DecisionType.h"
 #include "RegexComparator.h"
 
 USE_DEBUG_FLAG(D_WAAP_OVERRIDE);
@@ -264,6 +265,15 @@ public:
 
     template <typename _A>
     void serialize(_A &ar) {
+        try {
+            ar(cereal::make_nvp("parsedMatch", m_match));
+        }
+        catch(const cereal::Exception &e)
+        {
+            dbgDebug(D_WAAP_OVERRIDE) << "An override rule was not loaded, parsedMatch error:" << e.what();
+            isValid = false;
+        }
+
         try {
             ar(cereal::make_nvp("id", m_id));
         }
@@ -272,7 +282,6 @@ public:
             dbgDebug(D_WAAP_OVERRIDE) << "An override rule has no id.";
             m_id.clear();
         }
-        ar(cereal::make_nvp("parsedMatch", m_match));
         if (!m_match.isValidMatch()) {
             dbgDebug(D_WAAP_OVERRIDE) << "An override rule was not load";
             isValid = false;
@@ -342,14 +351,50 @@ private:
     bool isValid;
 };
 
+class ExceptionsByPractice
+{
+public:
+    template <typename _A>
+    void serialize(_A& ar)
+    {
+        ar(
+            cereal::make_nvp("WebApplicationExceptions", m_web_app_ids),
+            cereal::make_nvp("APIProtectionExceptions", m_api_protect_ids),
+            cereal::make_nvp("AntiBotExceptions", m_anti_bot_ids)
+        );
+        m_all_ids.insert(m_web_app_ids.begin(), m_web_app_ids.end());
+        m_all_ids.insert(m_api_protect_ids.begin(), m_api_protect_ids.end());
+        m_all_ids.insert(m_anti_bot_ids.begin(), m_anti_bot_ids.end());
+    }
+
+    bool operator==(const ExceptionsByPractice &other) const;
+    const std::vector<std::string>& getExceptionsOfPractice(DecisionType practiceType) const;
+    const std::set<std::string>& getAllExceptions() const;
+    bool isIDInWebApp(const std::string &id) const;
+private:
+    std::vector<std::string> m_web_app_ids;
+    std::vector<std::string> m_api_protect_ids;
+    std::vector<std::string> m_anti_bot_ids;
+    std::set<std::string> m_all_ids;
+};
+
 class Policy {
 public:
     template <typename _A>
     Policy(_A &ar) {
+        try {
+            ar(
+                cereal::make_nvp("exceptionsPerPractice", m_exceptionsByPractice)
+            );
+        }
+        catch (std::runtime_error & e) {
+            ar.setNextName(nullptr);
+            dbgInfo(D_WAAP_OVERRIDE) << "Failed to load exceptions per practice, error: ", e.what();
+            m_exceptionsByPractice = ExceptionsByPractice();
+        }
         std::vector<Waap::Override::Rule> rules;
         ar(cereal::make_nvp("overrides", rules));
         m_isOverrideResponse = false;
-
         for (std::vector<Waap::Override::Rule>::const_iterator it = rules.begin(); it != rules.end(); ++it) {
             const Waap::Override::Rule& rule = *it;
             if (!rule.isValidRule()) {
@@ -379,6 +424,14 @@ public:
         const std::vector<Waap::Override::Rule>& rules = requestOverrides ? m_RequestOverrides : m_ResponseOverrides;
         dbgTrace(D_WAAP_OVERRIDE) << "Start matching override rules ...";
         for (const Waap::Override::Rule &rule : rules) {
+            if (m_exceptionsByPractice.getAllExceptions().size() > 0 &&
+                !m_exceptionsByPractice.isIDInWebApp(rule.getId())
+            ) {
+                dbgInfo(D_WAAP_OVERRIDE)
+                << "match rule id is not in web application exceptions by practice: "
+                << rule.getId();
+                continue;
+            }
             dbgTrace(D_WAAP_OVERRIDE) << "Matching override rule ...";
             rule.match(testFunctor, matchedBehaviors, matchedOverrideIds);
         }
@@ -389,9 +442,17 @@ public:
         return m_isOverrideResponse;
     }
 
+    bool isValidRules() {
+        return !m_RequestOverrides.empty() || !m_ResponseOverrides.empty();
+    }
+
+    const ExceptionsByPractice& getExceptionsByPractice() const {
+        return m_exceptionsByPractice;
+    }
 private:
     std::vector<Waap::Override::Rule> m_RequestOverrides; //overrides that change request data
     std::vector<Waap::Override::Rule> m_ResponseOverrides; //overrides that change response/log data
+    ExceptionsByPractice m_exceptionsByPractice;
     bool m_isOverrideResponse;
 };
 
@@ -403,7 +464,7 @@ struct State {
     bool bForceException;
     std::set<std::string> forceExceptionIds;
     // overrides decision in case log should be ignored
-    bool bIgnoreLog;
+    bool bSupressLog;
     // user identfier override to be applied
     bool bSourceIdentifierOverride;
     std::string sSourceIdentifierMatch;
@@ -437,8 +498,8 @@ struct State {
 
             if (matchedBehavior.getLog() == "ignore")
             {
-                dbgTrace(D_WAAP_OVERRIDE) << "applyOverride(): setting bIgnoreLog due to override behavior.";
-                bIgnoreLog = true;
+                dbgTrace(D_WAAP_OVERRIDE) << "applyOverride(): setting bSupressLog due to override behavior.";
+                bSupressLog = true;
             }
 
             sSourceIdentifierMatch = matchedBehavior.getSourceIdentifier();

components/security_apps/waap/waap_clib/WaapResponseInspectReasons.cc

@@ -26,8 +26,7 @@ namespace Waap {
     errorLimiter(false),
     rateLimiting(false),
     collectResponseForLog(false),
-    applyOverride(false),
-    triggerReport(false)
+    applyOverride(false)
     {
     }
 
@@ -40,12 +39,11 @@ namespace Waap {
             " RateLimiting=" << rateLimiting <<
             " ErrorLimiter=" << errorLimiter <<
             " collectResponseForLog=" << collectResponseForLog <<
-            " applyOverride=" << applyOverride <<
-            " triggerReport=" << triggerReport;
+            " applyOverride=" << applyOverride;
 
         return
             openRedirect || errorDisclosure || rateLimiting || errorLimiter ||
-            collectResponseForLog || applyOverride || triggerReport;
+            collectResponseForLog || applyOverride;
     }
 
     void
@@ -93,14 +91,6 @@ namespace Waap {
         applyOverride = flag;
     }
 
-    void
-    ResponseInspectReasons::setTriggerReport(bool flag)
-    {
-        dbgTrace(D_WAAP) << "Change ResponseInspectReasons(setTriggerReport) " << triggerReport << " to " <<
-            flag;
-        triggerReport = flag;
-    }
-
     bool
     ResponseInspectReasons::getApplyOverride(void)
     {

components/security_apps/waap/waap_clib/WaapResponseInspectReasons.h

@@ -25,7 +25,6 @@ public:
     void setErrorLimiter(bool flag);
     void setCollectResponseForLog(bool flag);
     void setApplyOverride(bool flag);
-    void setTriggerReport(bool flag);
 
     bool getApplyOverride(void);
 private:
@@ -35,7 +34,6 @@ private:
     bool rateLimiting;
     bool collectResponseForLog;
     bool applyOverride;
-    bool triggerReport;
 };
 
 }

components/security_apps/waap/waap_clib/WaapTrigger.cc

@@ -75,5 +75,53 @@ bool Policy::operator==(const Policy &other) const
     return triggers == other.triggers;
 }
 
+bool TriggersByPractice::operator==(const TriggersByPractice &other) const
+{
+    return m_web_app_ids == other.m_web_app_ids &&
+        m_api_protect_ids == other.m_api_protect_ids &&
+        m_anti_bot_ids == other.m_anti_bot_ids;
+}
+
+const std::vector<std::string>& TriggersByPractice::getTriggersByPractice(DecisionType practiceType) const
+{
+    switch (practiceType)
+    {
+    case DecisionType::AUTONOMOUS_SECURITY_DECISION:
+        return m_web_app_ids;
+    default:
+        dbgError(D_WAAP) <<
+        "Can't find practice type for triggers by practice: " <<
+        practiceType <<
+        ", return web app triggers";
+        return m_web_app_ids;
+    }
+}
+
+const std::vector<std::string>& TriggersByPractice::getAllTriggers() const
+{
+    return m_all_ids;
+}
+
+bool WebUserResponseByPractice::operator==(const WebUserResponseByPractice &other) const
+{
+    return m_web_app_ids == other.m_web_app_ids &&
+        m_api_protect_ids == other.m_api_protect_ids &&
+        m_anti_bot_ids == other.m_anti_bot_ids;
+}
+
+const std::vector<std::string>& WebUserResponseByPractice::getResponseByPractice(DecisionType practiceType) const
+{
+    switch (practiceType)
+    {
+    case DecisionType::AUTONOMOUS_SECURITY_DECISION:
+        return m_web_app_ids;
+    default:
+        dbgDebug(D_WAAP)
+            << "Can't find practice type for triggers by practice: "
+            << practiceType
+            << ", return web app triggers";
+        return m_web_app_ids;
+    }
+}
 }
 }

components/security_apps/waap/waap_clib/WaapTrigger.h

@@ -19,6 +19,7 @@
 #include <string>
 #include <memory>
 #include "debug.h"
+#include "DecisionType.h"
 
 USE_DEBUG_FLAG(D_WAAP);
 
@@ -143,15 +144,86 @@ struct Trigger {
     std::shared_ptr<Log> log;
 };
 
+class TriggersByPractice
+{
+public:
+    template <typename _A>
+    void serialize(_A& ar)
+    {
+        ar(
+            cereal::make_nvp("WebApplicationTriggers", m_web_app_ids),
+            cereal::make_nvp("APIProtectionTriggers", m_api_protect_ids),
+            cereal::make_nvp("AntiBotTriggers", m_anti_bot_ids)
+        );
+        m_all_ids.insert(m_all_ids.end(), m_web_app_ids.begin(), m_web_app_ids.end());
+        m_all_ids.insert(m_all_ids.end(), m_api_protect_ids.begin(), m_api_protect_ids.end());
+        m_all_ids.insert(m_all_ids.end(), m_anti_bot_ids.begin(), m_anti_bot_ids.end());
+    }
+
+    bool operator==(const TriggersByPractice &other) const;
+    const std::vector<std::string>& getTriggersByPractice(DecisionType practiceType) const;
+    const std::vector<std::string>& getAllTriggers() const;
+private:
+    std::vector<std::string> m_web_app_ids;
+    std::vector<std::string> m_api_protect_ids;
+    std::vector<std::string> m_anti_bot_ids;
+    std::vector<std::string> m_all_ids;
+};
+
+class WebUserResponseByPractice
+{
+public:
+    template <typename _A>
+    void serialize(_A& ar)
+    {
+        ar(
+            cereal::make_nvp("WebApplicationResponse", m_web_app_ids),
+            cereal::make_nvp("APIProtectionResponse", m_api_protect_ids),
+            cereal::make_nvp("AntiBotResponse", m_anti_bot_ids)
+        );
+    }
+
+    bool operator==(const WebUserResponseByPractice &other) const;
+    const std::vector<std::string>& getResponseByPractice(DecisionType practiceType) const;
+private:
+    std::vector<std::string> m_web_app_ids;
+    std::vector<std::string> m_api_protect_ids;
+    std::vector<std::string> m_anti_bot_ids;
+};
+
 struct Policy {
     template <typename _A>
     Policy(_A &ar) {
-        ar(cereal::make_nvp("triggers", triggers));
+        try {
+            ar(
+                cereal::make_nvp("triggersPerPractice", triggersByPractice)
+            );
+        }
+        catch (std::runtime_error &e) {
+            ar.setNextName(nullptr);
+            dbgInfo(D_WAAP) << "Failed to load triggers per practice, error: " << e.what();
+            triggersByPractice = TriggersByPractice();
+        }
+        try {
+            ar(
+                cereal::make_nvp("webUserResponsePerPractice", responseByPractice)
+            );
+        }
+        catch (std::runtime_error &e) {
+            ar.setNextName(nullptr);
+            dbgInfo(D_WAAP) << "Failed to load web user response per practice, error: " << e.what();
+            responseByPractice = WebUserResponseByPractice();
+        }
+        ar(
+            cereal::make_nvp("triggers", triggers)
+        );
     }
 
     bool operator==(const Policy &other) const;
 
     std::vector<Waap::Trigger::Trigger> triggers;
+    TriggersByPractice triggersByPractice;
+    WebUserResponseByPractice responseByPractice;
 };
 
 }