Bot Intelligent analysis
Overview
Bot Intelligent Analysis is suitable for situations where rapid deployment, identification, and analysis of website traffic patterns are needed. Bot Intelligent Analysis is based on a clustering analysis algorithm and a big data model intelligent engine, aiming to help you comprehensively judge the risk of requests from multiple perspectives and more conveniently use Bot management to quickly identify and deal with known or unknown bots, avoiding fixed single strategies being bypassed. Bot Intelligent Analysis will comprehensively analyze multiple factors and classify requests into normal requests, normal bot requests, suspicious bot requests, and malicious bot requests, and support the configuration of corresponding action methods for different types of requests.
Note:
Bot Intelligent Analysis integrates the request characteristics in Bot Basic Management and Client Reputation Analysis functions and combines dynamic clustering analysis to form request risk tags. Bot Intelligent Analysis can help you understand the overall visitor situation and quickly deploy Bot management strategies. If you have very clear policy requirements for request features (for example, allowing specific search engine requests, intercepting Web development tool requests, etc.), you can further use Bot basic management, Client reputation, and Custom bot rules for policy adjustment.
Directions
For example, the e-commerce site
shop.example.com found that the product display page had a sudden increase in access volume, and it was judged that it might have suffered a large number of bot visits. The Bot Intelligent Analysis strategy can quickly enable Bot management functions to intercept bot tools. You can follow the steps below:1. Log in to the EdgeOne console, click the site list in the left menu bar, click the site to be configured in the site list, and enter the site details page.
2. Click Security > Web Security . By default, it is a site-level security policy. Click the Domain-level security policy tab and then click the target domain name such as
shop.example.com, to enter the configuration page for the security policy of the target domain name.3. Locate the Bot Management tab and click Edit under Bot intelligence to enter the configuration page.
4. Configure the corresponding action for each bot analysis tag. In this example scenario, you can configure the action to JavaScript Challenge for malicious bot requests, and maintain it as Monitor for suspected bot requests and friendly bot requests.
5. Click Save to complete the configuration.
Related References
Request Bot Tags
Bot Intelligent Analysis classifies requests into the following types based on the analysis results:
Malicious bot requests: Requests from bots with higher risks, suggested to be configured as interception or challenge actions.
Suspicious bot requests: Requests from bot clients with certain risks, suggested to be configured as at least observation or challenge actions.
Normal bot requests: Valid crawler requests, including requests from search engine crawlers.
Normal requests: Client requests without obvious bot features, only support release action.
Factors Affecting Bot Intelligent Analysis Judgment
The Bot intelligence engine will comprehensively evaluate requests based on the following main factors:
1. Request rate: The request rate will affect the identification of bots, and too high request rate may indicate malicious bot behavior.
2. IP Intelligence Library: The engine will refer to IP intelligence library to identify whether there are malicious behavior records or blocklist information.
3. Search Engine Features: Based on whether the source IPs match valid search engine crawlers, such as Google, Baidu, etc.
4. Access URL sequence: Analyze the sequence and pattern of accessed URLs to evaluate whether the request is similar to normal user behavior or normal bot behavior.
5. JA3 FingerprintNote 1: Use JA3 fingerprint technology to identify the features of client TLS connections, such as identifying non-browser clients like Python tools.
6. BotnetID FingerprintNote 2: By analyzing the BotnetID fingerprint and comparing it with known malicious BotnetIDs, malicious crawler behavior from botnets can be identified.
Note:
Note 1: JA3 is a fingerprint generation method for features in the TLS handshake process of clients. By collecting information provided by clients during the TLS handshake process (such as supported encryption suites, extensions, etc.), a unique hash value is generated as a fingerprint. JA3 fingerprints can help us identify clients that initiate requests using specific tools or libraries, such as requests initiated using Python libraries. By comparing the client's JA3 fingerprint with the fingerprints of known malicious tools or libraries, we can more accurately identify potential malicious bot behavior.
Note 2: BotnetID is an identification method based on bot network behavior characteristics. Bot networks (Botnets) are usually composed of multiple controlled malicious devices, which may be used to launch attacks or perform other malicious activities. By analyzing client behavior characteristics and their similarity to known bot networks, a BotnetID can be generated. By comparing the client's BotnetID with known malicious bot network IDs, we can more accurately identify potential malicious bot behavior.