Custom Rate Limiting Rules
Overview
In site operation, problems such as malicious resource occupation, business abuse, and brute force cracking often occur. If these problems are ignored, they will lead to a decline in service quality, generate high-cost bills, and may even cause sensitive data leakage. To effectively manage these risks, client access frequency is an important indicator. Malicious clients usually access at a higher frequency to quickly achieve the purpose of cracking login, occupying resources, and crawling content. Using appropriate threshold limits for client access frequency can effectively distinguish between normal clients and malicious clients, thereby mitigating the risks of resource occupation and abuse.
Note:
When managing and combating crawlers, the effect of using only rate limiting strategy is limited. Please combine Bot management function to formulate a complete crawler management strategy.
Typical Scenarios and Usage
Rate limiting is commonly used to distinguish between normal client access and malicious access. By selecting appropriate statistical methods, limit thresholds, and disposal methods, rate limiting can help you mitigate security risks. Rate limiting configuration is divided into the following types:
Precise rate limiting: User-defined access frequency control strategy. Supports multiple condition combinations to match requests, limit the request rate of each request source, and is suitable for most scenarios to distinguish between normal user access and malicious high-frequency access.
Managed custom policies: Policies customized by Tencent security experts, which do not support console adjustment of policies. For details, please refer to Managed Custom Rules.
Accurate Matching Rules
Example Scenario 1: Limit the access frequency of the login API interface to mitigate credential stuffing and brute force cracking attacks
In the face of credential stuffing and brute force cracking attacks, attackers often frequently use access to the login API interface to try to obtain or crack information. By limiting the request frequency of the login interface, we can significantly mitigate the attacker's cracking attempts, effectively defend against such attacks, and protect sensitive information from being leaked.
For example: The domain name
www.example.com provides an external interface /api/UpdateConfig, the allowed access call frequency is 100 times/minute, and when the frequency limit is exceeded, the IP will be blocked for 10 minutes. The operation steps are as follows:1. Log in to the EdgeOne console and click Site List in the left sidebar. In the site list, click the target Site.
2. Click Security > Web Security . By default, it is a site-level security policy . Click the Domain-level security policy tab and then click the target domain name such as
www.example.com , to enter the configuration page for the security policy of the target domain name.3. Locate the Rate Limiting tab and click Add rule under Rate limit.
4. Enter the rule adding page, select the Write interface rate limit rule template, and click Add.
Note:
The configuration fields and example values in the rule template are for illustrative purposes only. You should conduct an evaluation and adjust the judgment conditions, rate thresholds, and actions according to your specific business requirements and normal traffic levels.
5. Configure the judgment conditions, rate thresholds, and actions. In this example scenario, you can configure the matching fields as Request method equals
POST HEAD and Request path equals /api/UpdateConfig , adjust the statistical method to Based on Responses (origin to EdgeOne) , and configure the rate threshold as the count exceeding 100 times in a counting cycle of 1 minute to trigger an action. Adjust the triggered action to Block with a duration of 10 minutes .6. Click Save and publish . The rule will be deployed and take effect.
Example Scenario 2: Limit the request rate causing 404 status code to mitigate random resource scanning
When malicious clients randomly scan site image resources and try to crawl content, they often cause the origin server to respond with a 404 error due to non-existent access paths. By limiting the request rate that causes the origin server's 404 status code, EdgeOne can prevent malicious attackers from scanning and requesting static resources on a large scale, thereby reducing the origin server's error response, alleviating server pressure, and improving the security and stability of static resource sites. For example: For the domain name
www.example.com's image static resources .jpg``.jpeg``.webp``.png``.svg, when the resource does not exist and responds with a 404, if the access exceeds 200 times within 10 seconds, the corresponding client IP request will be directly blocked for 60 seconds. The operation steps are as follows:1. Log in to the EdgeOne console and click Site List in the left sidebar. In the site list, click the target Site.
2. Click Security > Web Security . By default, it is a site-level security policy . Click the Domain-level security policy tab and then click the target domain name such as
www.example.com , to enter the configuration page for the security policy of the target domain name.3. Locate the Rate Limiting tab and click Add rule under Rate limit.
4. Enter the rule adding page, select the Random scans rule template, and click Add.
Note:
The configuration fields and example values in the rule template are for illustrative purposes only. You should conduct an evaluation and adjust the judgment conditions, rate thresholds, and actions according to your specific business requirements and normal traffic levels.
5. Configure the judgment conditions, rate threshold, and actions. In this example scenario, you can configure the matching fields as HTTP status code (supported by the Enterprise plan) equals
404 and Request path with the file extension including .jpg, .jpeg, .webp, .png, and .svg . Adjust the statistical method to Based on Responses (origin to EdgeOne) , and configure the rate threshold as the count exceeding 200 times within a counting cycle of 1 minute to trigger an action. Adjust the triggered action to Block with a duration of 1 minute .
6. Click Save and publish . The rule will be deployed and take effect.
Example Scenario Three: Restricting High-Concurrency Search Engine Crawlers Access to Web Sites to Mitigate Impact on Regular Operations
A certain Y search engine provider employs a large-scale distributed crawler architecture, which lacks restrictions on access behavior. This leads to aggressive crawling activities, generating substantial traffic in a short period, potentially impacting normal operations and consuming significant resources. Therefore, rate limiting is used to identify and restrict such crawler access, mitigating its effects. For instance, the site
www.example.com is affected by high-frequency visits from the Y search engine crawler. Through web security analysis, it is found that the distributed architecture used by the Y search engine crawler clusters in JA3 fingerprint and User-Agent characteristics. Hence, rate limiting rules are configured. When the number of access requests with the same JA3 fingerprint and User-Agent exceeds 60 within a 30-second statistical window, requests with identical JA3 fingerprint and User-Agent characteristics are intercepted, with the interception lasting for 10 minutes. The operational steps are as follows:1. Log in to the Edgeone console and click Site List in the left sidebar. In the Site List, select the Site that requires configuration to proceed to the Site Details page.
2. Click Security > Web Security . By default, it is a site-level security policy . Click the Domain-level security policy tab and then click the target domain name such as
www.example.com , to enter the configuration page for the security policy of the target domain name.3. Locate the Rate Limiting tab and click Add rule under Rate limit .
4. On the rule adding page, select creating a blank rule, enter the rule name, and click Add .
5. Configure the judgment conditions, rate thresholds, and actions. In this example scenario, you can configure the matching field as Application layer protocol equals
HTTPS and the statistical method as Requests (client to EdgeOne) . Select the request features as JA3 fingerprint and User-Agent in the HTTP header , and configure the rate threshold as the count exceeding 60 times within a counting cycle of 30 seconds to trigger an action. Configure the triggered action to Block with a duration of 10 minutes .
6. Click Save and publish . The rule will be deployed and take effect.
Related References
When establishing rate limit rules, it is necessary to configure the rule specify scope, triggering method, and action. The explanations for each configuration item are as follows:
Note:
If your current rate rule needs to be matched based on a specific known value of the HTTP header, you can configure judgment conditions > specified matching fields for matching the specified parameter value of HTTP header .
If your current rate rule needs to be matched based on a type of HTTP headers that possibly contain the same value, you can configure rate thresholds > request features for matching the HTTP header of specified name .
Specify Scope
Based on the origin of the request, header characteristics, response status codes, and other factors, a combination of matching conditions 1 is established. The rate limit rule is only applied to manage the operations that meet these conditions. For more information of the matching conditions and the level of support provided by different packages, please refer to Matching Conditions.
Trigger Method
Note:
1. If the rate limit threshold is not reached, the request will not be handled or logged.
2. Specific configuration options and the configurable range vary with different subscription plans. For details, see Comparison of EdgeOne Plans.
3. In the counting cycle options, the 1 second option only supports the request features Client IP and Client IP (prioritizing XFF header) .
The rule will based on the statistical rules configured in the trigger method. When the cumulative number of requests within the counting cycle exceeds the threshold, the rule is activated and executes the corresponding limiting action2. The tally is based on the technical cycle and statistical method, counting the number of requests for different feature values under the specified feature dimension (such as client IP) 1. You can define the following parameters for the trigger method:
Counting Cycle: The length of the rolling time window used for counting. It supports a minimum of 1 second and a maximum of 1 hour.
Statistical Method: The method of distinguishing request sources, where the rate limit is to limit the request rate for each source. Refer to the statistical dimension for details.
Rate Threshold: The number of requests allowed per source (such as client IP) within the counting cycle.
Trigger State Retention Duration: After the rule is triggered, the duration for which requests matching the conditions of this source are continuously limited3. It supports a minimum of 1 second and a maximum of 30 days.
Request Features
Supports statistical analysis based on one or more request characteristics. When the request features within the statistical dimension reach the rate threshold set in the trigger method, the rate limit rule is activated. You may specify the following statistical dimensions1:
Client IP: Requests originating from the same source IP will be accounted for in a singular counter. Upon exceeding the threshold, the rule's disposition action is triggered.
Client IP (prioritizing XFF header): Requests originating from the same client IP will be accounted for in a single counter, triggering the rule's disposition action upon exceeding the threshold. When the X-Forwarded-For header is present and contains a valid IP list, the first IP in the X-Forwarded-For header will be prioritized for statistics.
Designated Cookie Name: Extracts the value of the specified cookie name from the request header. Requests with identical cookie values are counted in the same counter. When the threshold is exceeded, the rule's disposition action is triggered.
For instance, when a site employs a cookie labeled
user-session to mark visitation sessions, you can configure the value of the cookie named user-session as a statistical dimension, thereby tracking the request rate of each session. If the request rate within a single session surpass the threshold, the disposal action configured in the rule will be triggered.Designated Name HTTP Header: Extracts the value of the specified name in the request header, with requests bearing identical header values being accounted for in the same counter. When this threshold is surpassed, the rule's disposition action is triggered. For instance, you may specify the Origin header to limit the access frequency from each external domain. When the access frequency from a particular external domain exceeds the threshold, the disposition action configured by the rule is initiated.
Specified Name URL Query Parameter: Extracts the value of the specified name parameter from the request URL query parameters. Requests with the same query parameter value are counted in the same counter, triggering the rule's disposition action when exceeding the threshold.
For instance, when a site uses a query parameter named
user-session to mark access sessions, you can configure the specified name user-session as a statistical dimension, tallying the request rate for each session. When the request rate within a single session surpasses the threshold, it triggers the disposition action configured by the rule.Request JA3 Fingerprint4: Compute the JA3 fingerprint for each request, tallying the count of requests with identical JA3 fingerprints, and triggering the rule's disposition action when the threshold is exceeded. Each request corresponds to a unique JA3 fingerprint value, with no key-value model present, thus eliminating the need for specified parameter input. Considering the characteristics of JA3, it is recommended that you configure it at the same time as the User-Agent header statistics dimension to better distinguish clients.
Access Path of Request : Extracts the access path (URL path, excluding the URL query parameters) from the request. Requests with the same access path will be counted in the same counter. When the threshold is exceeded, the action of the rule is triggered.
Notes:
1: Depending on the package you subscribe to, the configurable matching conditions, statistical dimensions, and action options may vary. For more details, please refer to the Package Options Comparison.
2: If multiple rate limit rules exist, a single request can match multiple rule contents simultaneously, and the decision to trigger the rule will be based on the statistical methods of different rules. Once a rule is triggered and blocked, the remaining rules will not be triggered. When multiple rules are triggered simultaneously, they are executed in the order of priority of the triggered rules, with the rules with smaller priority values matching first. For more information, see the Web Protection Request Processing Order.
3: Once a rule is triggered, it only applies to requests that match the current rule.
4: A JA3 fingerprint is identification information formed based on the client's TLS information, which can effectively distinguish requests from different Bot networks. When a request is initiated based on a non-SSL HTTP protocol, the JA3 fingerprint of the request is empty. If you need to use a JA3 fingerprint, please ensure that the Bot management function has been enabled for your current domain.
5: If you need to perform statistics on requests with the same characteristics through a combination of multiple statistical dimensions, you need to subscribe to the EdgeOne Enterprise Edition package.
Action
When requests exceed the established threshold, corresponding restrictive actions are implemented. These include block, monitor, JSChallenge, redirect and ReturnCustomPage1. For more information, please refer to the section on Disposal Methods.
- Overview
- Typical Scenarios and Usage
- Accurate Matching Rules
- Example Scenario 1: Limit the access frequency of the login API interface to mitigate credential stuffing and brute force cracking attacks
- Example Scenario 2: Limit the request rate causing 404 status code to mitigate random resource scanning
- Example Scenario Three: Restricting High-Concurrency Search Engine Crawlers Access to Web Sites to Mitigate Impact on Regular Operations
- Related References