Edge Security
  • Overview
  • DDoS Protection
    • DDoS Protection Overview
    • Exclusive DDoS Protection Usage
    • Configuration of Exclusive DDoS protection Rules
      • Increase DDoS Protection Level
      • Exclusive DDoS Traffic Alarm
      • Configuration IP blocklist/allowlist
      • Configuration Region Blocking Rule
      • Configuration Port Filtering
      • Configuration Features Filtering
      • Configuration Protocol Blocking Rule
      • Configuration Connections Attack Protection
      • Related References
        • Action
        • Related Concepts Introduction
  • Web Protection
    • Overview
    • Configuring Web Protection Policy
    • Managed rules
    • CC attack defense
    • Bandwidth Abuse Protection
    • Custom rule
    • Custom Rate Limiting Rules
    • Exception Rules
    • Managed Custom Rules
    • Web security monitoring alarm
    • Refer
      • Web Protection Request Processing Order
      • Action
      • Match Condition
  • Bot Management
    • Overview
    • Bot Intelligent analysis
    • Bot Basic Feature Management
    • Client Reputation
    • Active Detection
    • Custom Bot Rule
    • Bot Exception Rule
    • Related References
      • Action
  • Rules Template
  • IP and IP Segment Grouping
  • Origin Protection
  • Custom Response Page
  • Alarm Notification
  • SSL/TLS
    • Overview
    • Deploying/Updating SSL Certificate for A Domain Name
    • Configuring A Free Certificate for A Domain Name
    • Mutual Authentication
    • HTTPS Configuration
      • Forced HTTPS Access
      • Enabling HSTS
      • SSL/TLS Security Configuration
        • Configuring SSL/TLS Security
        • TLS Versions and Cipher Suites
      • Enabling OCSP Stapling
    • Refer
      • Using OpenSSL to Generate Self-Signed Certificates
      • Certificate Format Requirements
    • Using Keyless Certificate

Bot Intelligent analysis

Overview

following business scenarios:
Business faces threats such as Bot attacks, promotion fraud, and traffic theft. Unable to complete precise Bot security policy configuration and optimization in a short time, it requires quick suppression of non-human user requests.
Business scenarios that require quick deployment, identify and analyze website traffic mode, and understand overall visitor situation.
Bot Intelligent Analysis is based on an intelligent engine with clustering analysis algorithm and big data model. It comprehensively evaluates multiple factors of requests and classifies them into normal human requests, normal Bot requests, suspected Bot requests, and malicious Bot requests. Based on the analysis result, you can configure appropriate handling methods for different types of requests, such as block, JavaScript challenge, monitor, and allow.
Note:
Bot Intelligent Analysis integrates the request characteristics in Bot Basic Feature Management and Client Reputation functions and combines dynamic clustering analysis to form request risk tags. Bot Intelligent Analysis can help you understand the overall visitor situation and quickly deploy Bot management strategies. If you have very clear policy requirements for request features (for example, allowing specific search engine requests, intercepting Web development tool requests, etc.), you can further use Bot Basic Feature Management, Client reputation, and Custom bot rules for policy adjustment.

Policy Mechanism

Bot Intelligent Analysis implements a rule-based policy mechanism, which consists of static rules and dynamic rules:
Static rules are based on predefined feature libraries, including blocklist characteristics of some automation tools (such as cURL and Python), like user agent (User-Agent) and fingerprint. Static rules take effect when triggered per request and are permanently valid.
Dynamic rules are automatically generated based on real-time traffic characteristics and request behavior. For example, when domain requests exceed a certain threshold (about 100 times) within a short time frame (around 5 minutes), the system will automatically trigger dynamic rule generation and deployment. Dynamic rules remain valid for 20 minutes and can capture unidentified Bot behavior.
The Bot Intelligent Analysis policy takes effect at the domain level, and traffic from different domains does not affect each other. For the same domain, multiple static and dynamic rules may take effect simultaneously. When a request hits multiple rules at the same time, the system will tag the request accordingly (such as malicious bot, suspected bot, or friendly bot) based on the weighted sum and confidence degree of the rules, and process the request according to the customer's preconfigured handling actions.
By combining static and dynamic rules, Bot Intelligent Analysis provides comprehensive coverage against known and unknown bot threats while reducing rule maintenance complexity, enabling customers to obtain effective bot protection with minimum time cost.

Directions

For example, the e-commerce site shop.example.com found that the product display page had a sudden increase in access volume, and it was judged that it might have suffered a large number of bot visits. The Bot Intelligent Analysis strategy can quickly enable Bot management functions to intercept bot tools. You can follow the steps below:
1. Log in to the Tencent Cloud EdgeOne console, enter Service Overview in the left menu bar, and click the site to be configured under Website Security Acceleration.
2. Click Security > Web Security . By default, it is a site-level security policy. Click the Domain-level security policy tab and then click the target domain name such as shop.example.com, to enter the configuration page for the security policy of the target domain name.
3. Locate the Bot Management tab and click Edit under Bot intelligence to enter the configuration page.
4. Configure the corresponding action for each bot analysis tag. In this example scenario, you can configure the action to JavaScript Challenge for malicious bot requests, and maintain it as Monitor for suspected bot requests and friendly bot requests.

5. Click Save to complete the configuration.

View Traffic Hitting Bot Intelligent Analysis

To view traffic hitting Bot intelligent analysis rules:
1. Log in to the EdgeOne console, then click Web security analysis in the left menu bar.
2. Click Add Filter, select Bot Management - Bot Intelligent Analysis, and filter based on rule ID, Bot tag, Bot intelligent analysis features.
3. Click OK to apply the filtering conditions, and observe the protection effectiveness in Statistical Trend and Statistical Detail.

Usage Limits and Handling Solution

Bot Intelligent Analysis aims to distinguish requests from human users and automated processes. Usage limits may exist in specific business scenarios:
Legitimate automated request access: For businesses that allow automated request access, such as periodic dial tests used in performance monitoring, API calls via third-party tools or components, Bot Intelligent Analysis may misjudge them as malicious bot requests. In such cases, it is recommended to configure protection exception rules to bypass the Bot Intelligent Analysis protection module and avoid false blocking. Meanwhile, combine custom bot rules to enhance protection capability.
Multi-CDN or proxy scenarios: If other CDNs or proxy services are nested before EdgeOne, the actual source IP information of requests may be lost, impacting the accuracy of Bot Intelligent Analysis. In such cases, it is advisable to carefully assess the existing business architecture and adjust EdgeOne's deployment location as needed to ensure accurate request source information can be obtained.
APP scenario: For mobile app business, since usage environment and behavior vary greatly among different users, requests from built-in development components or clients (such as gaming hot update feature) may be misidentified as Bot. In such cases, it is recommended to configure protection exception rules and add specific app requests to allowlist to avoid affecting legitimate user experience.
Advanced Bot warfare scenarios: In Web scenarios, for some advanced bots, such as scripting tools or headless browsers that have already spoofed JA3 fingerprints, relying solely on traffic feature analysis may not be able to effectively identify them. In such cases, it is recommended to enable the proactive feature recognition feature to achieve more accurate Bot authentication.
High-confrontation scenarios: For extreme situations involving fingerprint browsers, real-device operations, screen recording, etc., it is advisable to combine custom Bot rules with a managed challenge handling method. Deploying Captcha increases the cost for malicious attackers, ensuring business security.

Related References

Request Bot Tags

Bot Intelligent Analysis classifies requests into the following types based on the analysis results:
Malicious bot requests: Highly suspected requests from malicious automated processes with relatively high risk.Suggested to be configured as interception or challenge actions.
Suspicious bot requests: Suspected request from an automated process with certain risk. Recommended at least configured as observation or challenge disposal action.
Normal bot requests: Request comes from legitimate bots (such as search engine crawler).
Normal requests: Client requests without obvious bot features, only support release action.

Factors Affecting Bot Intelligent Analysis Judgment

The Bot intelligent analysis engine performs comprehensive evaluation and scoring of risks for each element based on dimensional features and behavior analysis. Primary influencing factors are as follows:
1. Request rate: Based on network-wide traffic audit, the IP traffic baseline performs anomaly detection on the access frequency of specific IPs. High or abnormal request frequency features will be assigned a relatively high risk score.
2. IP Intelligence Library: The engine will refer to a comprehensive IP Intelligence Library, assess the historical behavior, reputation, and potential risk of the client IP, and identify known malicious behavior records or blocklist information.
3. Search engine features: Combining public crawler IP lists and corresponding User-Agent features from search engine manufacturers, the system can accurately identify and verify crawler requests from legitimate search engines such as Google and Baidu. Requests meeting legitimate bot features will be assigned a normal request tag.
4. Access URL sequence: Analyze the visitor's URL access sequence and mode to identify abnormal browsing behavior, locating actions similar to spider pool crawlers or automated scan tools. Behavior sequences with malicious features will be assigned a high risk score.
5. JA3 fingerprintNote 1: The system leverages JA3 fingerprint technology to identify the characteristics of client TLS connections and detect unconventional access tools such as Python scripts, curl, and headless browsers. Requests from abnormal clients or specific tools will be assigned a high risk score.
6. Client consistency check: The system verifies whether traffic characteristics match the browser and operating system information declared in the User-Agent. Inconsistent features usually indicate client spoofing, and such requests will be assigned a high risk score.
7. Proxy IP identification: Detect high-frequency IP switch behavior based on VPN, proxy, or local dialing technologies in real time through traffic characteristics analysis. Requests compliant with second-dial IP features will be assigned a relatively high risk score.
8. ClientfingerprintNote 2: Identify malicious crawler requests from botnets by analyzing HTTP header characteristics/features and comparing them with known high-risk client fingerprints.
Note:

Note 1:
JA3 is a method for generating fingerprints based on characteristics during the TLS client handshake process (specific algorithm reference: JA3 fingerprint). By collecting information provided by clients during the TLS handshake process (such as supported encryption suites, extensions, etc.), a unique hash value is generated as a fingerprint. JA3 fingerprints can help us identify clients that initiate requests using specific tools or libraries, such as requests initiated using Python libraries. By comparing the client's JA3 fingerprint with the fingerprints of known malicious tools or libraries, we can more accurately identify potential malicious bot behavior.

Note 2
: BotnetID is an identification method based on bot network behavior characteristics. Bot networks (Botnets) are usually composed of multiple controlled malicious devices, which may be used to launch attacks or perform other malicious activities. By analyzing client behavior characteristics and their similarity to known bot networks, a BotnetID can be generated. By comparing the client's BotnetID with known malicious bot network IDs, we can more accurately identify potential malicious bot behavior.

edgeone Logo
Copyright © 2013-2025 Tencent Cloud. All Rights Reserved.