Bot Intelligent analysis
Overview
following business scenarios:
Business faces threats such as Bot attacks, promotion fraud, and traffic theft. Unable to complete precise Bot security policy configuration and optimization in a short time, it requires quick suppression of non-human user requests.
Business scenarios that require quick deployment, identify and analyze website traffic mode, and understand overall visitor situation.
Bot Intelligent Analysis is based on an intelligent engine with clustering analysis algorithm and big data model. It comprehensively evaluates multiple factors of requests and classifies them into normal human requests, normal Bot requests, suspected Bot requests, and malicious Bot requests. Based on the analysis result, you can configure appropriate handling methods for different types of requests, such as block, JavaScript challenge, monitor, and allow.
Note:
Bot Intelligent Analysis integrates the request characteristics in Bot Basic Feature Management and Client Reputation functions and combines dynamic clustering analysis to form request risk tags. Bot Intelligent Analysis can help you understand the overall visitor situation and quickly deploy Bot management strategies. If you have very clear policy requirements for request features (for example, allowing specific search engine requests, intercepting Web development tool requests, etc.), you can further use Bot Basic Feature Management, Client reputation, and Custom bot rules for policy adjustment.
Policy Mechanism
Bot Intelligent Analysis implements a rule-based policy mechanism, which consists of static rules and dynamic rules:
Static rules are based on predefined feature libraries, including blocklist characteristics of some automation tools (such as cURL and Python), like user agent (User-Agent) and fingerprint. Static rules take effect when triggered per request and are permanently valid.
Dynamic rules are automatically generated based on real-time traffic characteristics and request behavior. For example, when domain requests exceed a certain threshold (about 100 times) within a short time frame (around 5 minutes), the system will automatically trigger dynamic rule generation and deployment. Dynamic rules remain valid for 20 minutes and can capture unidentified Bot behavior.
The Bot Intelligent Analysis policy takes effect at the domain level, and traffic from different domains does not affect each other. For the same domain, multiple static and dynamic rules may take effect simultaneously. When a request hits multiple rules at the same time, the system will tag the request accordingly (such as malicious bot, suspected bot, or friendly bot) based on the weighted sum and confidence degree of the rules, and process the request according to the customer's preconfigured handling actions.
By combining static and dynamic rules, Bot Intelligent Analysis provides comprehensive coverage against known and unknown bot threats while reducing rule maintenance complexity, enabling customers to obtain effective bot protection with minimum time cost.
Directions
For example, the e-commerce site
shop.example.com found that the product display page had a sudden increase in access volume, and it was judged that it might have suffered a large number of bot visits. The Bot Intelligent Analysis strategy can quickly enable Bot management functions to intercept bot tools. You can follow the steps below:1. Log in to the Tencent Cloud EdgeOne console, enter Service Overview in the left menu bar, and click the site to be configured under Website Security Acceleration.
2. Click Security > Web Security . By default, it is a site-level security policy. Click the Domain-level security policy tab and then click the target domain name such as
shop.example.com, to enter the configuration page for the security policy of the target domain name.3. Locate the Bot Management tab and click Edit under Bot intelligence to enter the configuration page.
4. Configure the corresponding action for each bot analysis tag. In this example scenario, you can configure the action to JavaScript Challenge for malicious bot requests, and maintain it as Monitor for suspected bot requests and friendly bot requests.
5. Click Save to complete the configuration.
View Traffic Hitting Bot Intelligent Analysis
To view traffic hitting Bot intelligent analysis rules:
1. Log in to the EdgeOne console, then click Web security analysis in the left menu bar.
2. Click Add Filter, select Bot Management - Bot Intelligent Analysis, and filter based on rule ID, Bot tag, Bot intelligent analysis features.
3. Click OK to apply the filtering conditions, and observe the protection effectiveness in Statistical Trend and Statistical Detail.
Usage Limits and Handling Solution
Bot Intelligent Analysis aims to distinguish requests from human users and automated processes. Usage limits may exist in specific business scenarios:
Legitimate automated request access: For businesses that allow automated request access, such as periodic dial tests used in performance monitoring, API calls via third-party tools or components, Bot Intelligent Analysis may misjudge them as malicious bot requests. In such cases, it is recommended to configure protection exception rules to bypass the Bot Intelligent Analysis protection module and avoid false blocking. Meanwhile, combine custom bot rules to enhance protection capability.
Multi-CDN or proxy scenarios: If other CDNs or proxy services are nested before EdgeOne, the actual source IP information of requests may be lost, impacting the accuracy of Bot Intelligent Analysis. In such cases, it is advisable to carefully assess the existing business architecture and adjust EdgeOne's deployment location as needed to ensure accurate request source information can be obtained.
APP scenario: For mobile app business, since usage environment and behavior vary greatly among different users, requests from built-in development components or clients (such as gaming hot update feature) may be misidentified as Bot. In such cases, it is recommended to configure protection exception rules and add specific app requests to allowlist to avoid affecting legitimate user experience.
Advanced Bot warfare scenarios: In Web scenarios, for some advanced bots, such as scripting tools or headless browsers that have already spoofed JA3 fingerprints, relying solely on traffic feature analysis may not be able to effectively identify them. In such cases, it is recommended to enable the proactive feature recognition feature to achieve more accurate Bot authentication.
High-confrontation scenarios: For extreme situations involving fingerprint browsers, real-device operations, screen recording, etc., it is advisable to combine custom Bot rules with a managed challenge handling method. Deploying Captcha increases the cost for malicious attackers, ensuring business security.
Related References
Request Bot Tags
Bot Intelligent Analysis classifies requests into the following types based on the analysis results:
Malicious bot requests: Highly suspected requests from malicious automated processes with relatively high risk.Suggested to be configured as interception or challenge actions.
Suspicious bot requests: Suspected request from an automated process with certain risk. Recommended at least configured as observation or challenge disposal action.
Normal bot requests: Request comes from legitimate bots (such as search engine crawler).
Normal requests: Client requests without obvious bot features, only support release action.
Factors Affecting Bot Intelligent Analysis Judgment
The Bot intelligent analysis engine performs comprehensive evaluation and scoring of risks for each element based on dimensional features and behavior analysis. Primary influencing factors are as follows:
1. Request rate: Based on network-wide traffic audit, the IP traffic baseline performs anomaly detection on the access frequency of specific IPs. High or abnormal request frequency features will be assigned a relatively high risk score.
2. IP Intelligence Library: The engine will refer to a comprehensive IP Intelligence Library, assess the historical behavior, reputation, and potential risk of the client IP, and identify known malicious behavior records or blocklist information.
3. Search engine features: Combining public crawler IP lists and corresponding User-Agent features from search engine manufacturers, the system can accurately identify and verify crawler requests from legitimate search engines such as Google and Baidu. Requests meeting legitimate bot features will be assigned a normal request tag.
4. Access URL sequence: Analyze the visitor's URL access sequence and mode to identify abnormal browsing behavior, locating actions similar to spider pool crawlers or automated scan tools. Behavior sequences with malicious features will be assigned a high risk score.
5. JA3 fingerprintNote 1: The system leverages JA3 fingerprint technology to identify the characteristics of client TLS connections and detect unconventional access tools such as Python scripts, curl, and headless browsers. Requests from abnormal clients or specific tools will be assigned a high risk score.
6. Client consistency check: The system verifies whether traffic characteristics match the browser and operating system information declared in the User-Agent. Inconsistent features usually indicate client spoofing, and such requests will be assigned a high risk score.
7. Proxy IP identification: Detect high-frequency IP switch behavior based on VPN, proxy, or local dialing technologies in real time through traffic characteristics analysis. Requests compliant with second-dial IP features will be assigned a relatively high risk score.
8. ClientfingerprintNote 2: Identify malicious crawler requests from botnets by analyzing HTTP header characteristics/features and comparing them with known high-risk client fingerprints.
Note:
Note 1:
JA3 is a method for generating fingerprints based on characteristics during the TLS client handshake process (specific algorithm reference: JA3 fingerprint). By collecting information provided by clients during the TLS handshake process (such as supported encryption suites, extensions, etc.), a unique hash value is generated as a fingerprint. JA3 fingerprints can help us identify clients that initiate requests using specific tools or libraries, such as requests initiated using Python libraries. By comparing the client's JA3 fingerprint with the fingerprints of known malicious tools or libraries, we can more accurately identify potential malicious bot behavior.
Note 2
: BotnetID is an identification method based on bot network behavior characteristics. Bot networks (Botnets) are usually composed of multiple controlled malicious devices, which may be used to launch attacks or perform other malicious activities. By analyzing client behavior characteristics and their similarity to known bot networks, a BotnetID can be generated. By comparing the client's BotnetID with known malicious bot network IDs, we can more accurately identify potential malicious bot behavior.