Custom Rate Limiting Rules
Overview
In site operations, issues such as malicious resource occupation, service abuse, and brute-force attacks frequently occur. If overlooked, these problems can lead to degraded service quality, high-cost bills, and even potential leakage of sensitive data. To effectively manage these risks, client access frequency serves as a critical metric. Malicious clients typically access at higher frequencies to rapidly achieve goals like credential cracking, resource occupation, and content scraping. When an appropriate threshold is set to restrict client access frequency, legitimate clients can be effectively distinguished from malicious ones, thereby mitigating risks of resource occupation and abuse.
Note:
1. When crawlers are managed and combated, relying solely on rate limiting policies has limited effectiveness. Please combine the Bot Management feature to formulate a comprehensive crawler management policy.
2. Rate limiting rules are counted separately per edge access server IP address, which can be used to combat HTTP DDoS attacks and ensure origin server availability to some extent. These rules cannot precisely control origin-pulling QPS. For origin-pulling QPS control requirements, see Origin-pull Rate Limit Policy.
Typical Scenarios and Configuration Methods
Rate limiting is commonly used to distinguish legitimate client access from malicious access. By selecting appropriate statistical methods, restriction thresholds, and handling measures, rate limiting can help you mitigate security risks. Rate limiting configurations are categorised into the following types:
Precise Rate Limiting: A user-defined policy for controlling access frequency. It supports combining multiple conditions to match requests and restricts the request rate per source. This is suitable for distinguishing legitimate user access from malicious high-frequency access in most scenarios.
Managed Custom Policy: A policy customised by Tencent security experts that cannot be adjusted in the console. For details, see Managed Custom Rules.
Precise Rate Limiting
Scenario 1: Restricting access rate to login APIs to mitigate credential stuffing and brute-force attacks
In scenarios involving credential stuffing and brute-force attacks, attackers typically attempt to obtain or crack information by frequently accessing the login API. By limiting the request frequency to sensitive APIs, we can significantly mitigate attackers' cracking attempts, thereby effectively defending against such attacks and protecting sensitive information from leakage. For example: the site domain
www.example.com provides a public API /api/UpdateConfig. This API allows a call frequency of 50 times/10 seconds. When the frequency limit is exceeded, the IP address will be blocked for 2 minutes. The operation steps are as follows:1. Log in to the Tencent Cloud EdgeOne console, enter Service Overview in the left menu bar, and click the site to be configured under Website Security Acceleration.
2. Click Security Protection > Web Protection. By default, the site-level protection policy is displayed. Click the Domain-level Protection Policy Tab. On the Domain-level Protection Policy page, click the target domain name to go to its protection policy configuration page, for example:
www.example.com.3. Click Add Rule under Rate Limiting Rules.
4. Go to the Add Rule page and select the Sensitive Interface Protection rule template.
5. Configure the matching conditions, rate threshold, and enforcement actions, then click Add. For the example scenario, you can configure the matching field as Request Method (Method) equals
POST HEAD and Request Path (Path) equals /api/UpdateConfig. Adjust the rate threshold to trigger when the count exceeds 50 times within a counting period of 10 seconds. Set the enforcement action upon trigger to Block with a duration of 2 minutes.Note:
The configuration fields and sample values in the rule templates are for illustrative purposes only. Please evaluate based on your specific business requirements and normal traffic levels, and adjust the trigger conditions, rate thresholds, and enforcement actions for protection accordingly.
6. Click Deploy, and the rule will take effect.
Example scenario 2: Limit the request rate causing 404 status codes to mitigate random resource scanning
When malicious clients randomly scan a site's image resources in an attempt to crawl content, the origin server often responds with a
404 error because the requested path does not exist. By limiting the frequency of requests that result in a 404 status code from the origin, EdgeOne can prevent malicious attackers from scanning and requesting static resources on a large scale. This reduces error responses from the origin, alleviates server pressure, and enhances the security and stability of static resource sites. For example: for image static resources (.jpg .jpeg .webp .png .svg) under the site domainwww.example.com, when a resource does not exist and a 404 response is triggered, if the access exceeds 1 time within 10 seconds, the corresponding client IP request is directly blocked for 10 seconds. The operation steps are as follows:1. Log in to the Tencent Cloud EdgeOne console, enter Service Overview in the left menu bar, and click the site to be configured under Website Security Acceleration.
2. Click Security Protection > Web Protection. By default, the site-level protection policy is displayed. Click the Domain-level Protection Policy Tab. On the Domain-level Protection Policy page, click the target domain name to go to its protection policy configuration page, for example:
www.example.com.3. Click Add Rule under Rate Limiting Rules.
4. Go to the Add Rule page and select the Random Scans rule template.
5. Configure the matching conditions, rate threshold, and enforcement actions, then click Add. For the example scenario, you can configure the matching field as HTTP Status Code (supported in Enterprise plans) equals
401 or 403 or 404, and the request's Request Path (Path) has a File Extension Match content of .jpg .jpeg .webp .png .svg. Adjust the rate threshold to trigger when the count exceeds 1 time within a counting period of 10 seconds. Set the enforcement action upon trigger to Block with a duration of 10 seconds.Note:
The configuration fields and sample values in the rule templates are for illustrative purposes only. Please evaluate based on your specific business requirements and normal traffic levels, and adjust the trigger conditions, rate thresholds, and enforcement actions for protection accordingly.
6. Click Deploy, and the rule will take effect.
Example scenario 3: Restrict high-concurrency search engine crawlers from accessing Web sites to mitigate impacts on normal business operations
A certain Y search engine provider uses a large-scale distributed crawler architecture with insufficient restrictions on access behaviour. This leads to aggressive crawler activity, generating a high volume of access in a short time, which may impact normal business operations and consume significant resources. Therefore, rate limiting can be used to identify and restrict such crawler access, mitigating its impact. For example: the site
www.example.com experienced impact on normal business due to high-frequency access from the Y search engine crawler. Through Web Security Analysis, it was found that the distributed architecture used by the Y search engine's crawler clusters in JA3 fingerprint and User-Agent characteristics. Therefore, a rate limiting rule can be configured to block requests with the same JA3 fingerprint and User-Agent when they exceed 60 times within a 30-second statistical window. The blocking lasts for 10 minutes. The operation steps are as follows:1. Log in to the Tencent Cloud EdgeOne console, enter Service Overview in the left menu bar, and click the site to be configured under Website Security Acceleration.
2. Click Security Protection > Web Protection. By default, the site-level protection policy is displayed. Click the Domain-level Protection Policy Tab. On the Domain-level Protection Policy page, click the target domain name to go to its protection policy configuration page, for example:
www.example.com.3. Click Add Rule under Rate Limiting Rules.
4. Go to the Add Rule page, select Blank Rule, fill in the rule name, and click Add.
5. Configure the matching conditions, rate threshold, and enforcement actions. For the example scenario, you can configure the matching field as Application Layer Protocol equals
HTTPS for clients with identical JA3 fingerprint and User-Agent in the HTTP header characteristics. Set the rate threshold to trigger when the count exceeds 60 times within a counting period of 30 seconds. After triggering, the enforcement action is Block with a duration of 10 minutes.
6. andClick Deploy, and the rule will take effect.
Reference
When creating a rate limiting rule, you need to configure the rule matching objects, trigger conditions, and enforcement actions. The configuration items are described as follows:
Note:
If your current rate rule needs to match based on a known HTTP header value, configure Match Condition > Specified Match Field as Specified HTTP Header Parameter Value.
If your current rate rule needs to match based on a category of HTTP headers that may have identical values, you can configure Rate Threshold> Request Characteristic as a specified HTTP header name to perform the matching.
Match Object
Configure a combination of matching conditions based on request source, header characteristics, response status codes, and so on.1. The rate limiting rule only controls traffic that matches the conditions. For details on matching condition descriptions and support levels across different plans, see Matching Conditions.
Trigger Method
Note:
1. When the request rate does not reach the threshold, no action will be taken and the requests will not be logged.
2. The specific configuration options and configurable ranges vary depending on the subscription plan. For details, see Plan Comparison.
3. In the counting cycle options, the 1 second option only supports the Client IP and Client IP (preferentially matches XFF header) request characteristics.
The rule will count requests based on the statistical rules configured in the trigger conditions. When the accumulated request count exceeds the threshold within the counting cycle, the rule triggers and executes the corresponding restrictive action2. Statistics are based on the counting cycle and counting method, where requests are counted per distinct feature value (such as client IP) under specified feature dimensions1. You can configure the following parameters for trigger conditions:
Counting cycle: The length of the rolling time window used for counting. Supported range is from 1 second to 1 hour.
Counting method: The method of distinguishing request sources. Rate limiting restricts the request rate per source. For details, see Counting Dimensions.
Rate threshold: The number of allowed requests per source (such as Client IP address) within a counting cycle.
Duration of Triggered State: The period during which requests from the source matching the conditions continue to be restricted3 after the rule is triggered. Supported range is from 1 second to 30 days.
Request Characteristics
The system supports statistics based on one or more request characteristics. When requests within a statistical dimension meet the rate threshold set in the trigger conditions, the rate limiting rule is triggered. You can specify the following counting dimensions1:
Client IP address: Requests from the same source IP address will be counted in the same counter. When the threshold is exceeded, the rule's action will be triggered.
Client IP (preferentially matches XFF header): Requests from the same client IP address will be counted in the same counter. When the threshold is exceeded, the rule's action will be triggered. If the X-Forwarded-For header is present and contains a valid list of IP addresses, the first IP address in the X-Forwarded-For header will be used for counting.
Specified Cookie Name: Extract the value of the specified Cookie from the request header. Requests with the same Cookie value are counted in the same counter. When the threshold is exceeded, the rule's action will be triggered.
For example, when a website uses a Cookie named user-session to track access sessions, you can configure the Cookie value with the specified name user-session as the counting dimension to track request rates per session. When the request rate in a single session exceeds the threshold, the action configured in the rule will be triggered.
Specified HTTP Header Name: Extract the value of the specified header from the request header. Requests with the same header value are counted in the same counter. When the threshold is exceeded, the rule's action will be triggered. For example: You can specify the Origin header to limit the access frequency from each external domain. When the access frequency from a particular external domain exceeds the threshold, the action configured in the rule will be triggered.
Specified URL Query Parameter: Extract the value of the specified parameter from the request URL query. Requests with the same query parameter value are counted in the same counter. When the threshold is exceeded, the rule's action will be triggered.
For example, when a website uses a query parameter named
user-session to track access sessions, you can configure the query parameter with the specified name user-session as the counting dimension to track request rates per session. When the request rate in a single session exceeds the threshold, the action configured in the rule will be triggered.JA3 Fingerprint of Request4: Calculate the JA3 fingerprint for each request. Requests with identical JA3 fingerprints are counted collectively. When the threshold is exceeded, the rule's configured action will be triggered. Each request corresponds to a unique JA3 fingerprint value, and no key-value model is involved, so no specific parameters need to be entered. Considering the characteristics of JA3, it is recommended to configure it alongside the User-Agent header as a counting dimension to effectively distinguish between clients.
JA4 Fingerprint of Request: Calculate the JA4 fingerprint for each request. Requests with identical JA4 fingerprints are counted collectively. When the threshold is exceeded, the rule's configured action will be triggered. Each request corresponds to a unique JA4 fingerprint value, and no key-value model is involved, so no specific parameters need to be entered.
Access Path of Request: Extract the access path of the request (URL Path, excluding URL query parameters). Requests with the same access path are counted in the same counter. When the threshold is exceeded, the rule's action will be triggered.
Note:
Note 1
: The configurable matching conditions, statistical dimensions, and action options may vary depending on your subscription plan. For details, see Plan Comparison.Note 2:
If multiple rate limiting rules exist, a single request can match multiple rules simultaneously. The system determines whether to trigger each rule based on its respective counting method. Once a rule is triggered and the request is blocked, the remaining rules will not be triggered. When multiple rules are triggered at the same time, they are executed according to the priority order of the triggered rules, with the rule having the smaller priority value being matched first. For details, see Web Protection Request Processing Order.Note 3
: When a rule is triggered, it will only take effect on requests that match the current rule.Note 4
: JA3 fingerprint is identification information formed based on the client's TLS information, which can effectively distinguish requests from different Bot networks. When a request is initiated via non-SSL HTTP protocol, its JA3 fingerprint will be empty. To use JA3 fingerprint, ensure that Bot Management feature is enabled for your current domain.Note 5: To count the number of requests with identical characteristics by combining multiple statistical dimensions, you need to subscribe to the EdgeOne Enterprise Edition plan.
Action
When a request exceeds the limit threshold, the corresponding action will be taken. Supported actions include blocking, observation, JavaScript challenge, redirecting to a URL, and responding with a custom page1. For detailed descriptions of these actions, see Actions.
- Overview
- Typical Scenarios and Configuration Methods
- Precise Rate Limiting
- Scenario 1: Restricting access rate to login APIs to mitigate credential stuffing and brute-force attacks
- Example scenario 2: Limit the request rate causing 404 status codes to mitigate random resource scanning
- Example scenario 3: Restrict high-concurrency search engine crawlers from accessing Web sites to mitigate impacts on normal business operations
- Reference