EdgeOne Resource Abuse Prevention
It may take you 20 minutes to read this document, which helps you:
1. Understand what is Content Delivery Network (CDN) resource abuse and its common types and harms.
2. Understand how to configure traffic alarms and usage cap policies on the EdgeOne platform, enable real-time log push, and prevent CDN resource abuse.
3. Understand how to recognize and locate CDN resource abuse attacks by using the EdgeOne traffic analysis and log analysis features.
4. Understand the configuration recommendations in the practical tutorial for EdgeOne resource abuse prevention, which is provided for small and medium websites and enterprise-level business platforms.
What Is CDN Resource Abuse?
CDN resource abuse indicates the behavior of unauthorized users to obtain a large amount of website resources and consume the website bandwidth and server resources by illegal means. Compared to DDoS attacks, which directly affect the website availability, CDN resource abuse primarily consumes the computing resources of websites such as bandwidth and generates a sudden high bandwidth or large traffic, leading to bills higher than daily consumptions and significantly increasing the operational costs of websites. Common methods of CDN resource abuse include:
Sending a large volume of false requests through automation tools, proxy servers, or botnets;
Continuously downloading large files or transferring a large amount of data through automation tools;
Sending a large volume of concurrent requests through load testing tools to perform overload testing on the server.
Note:
If you use Tencent Cloud CDN currently, we recommend upgrading it to EdgeOne and configuring corresponding protection policies on EdgeOne. Reasonable configuration of protection measures can effectively reduce the effects of CDN resource abuse, guarantee normal business operation, and prevent unexpected high bills. For CDN service migration methods, see Guide to Using the EdgeOne Tool for Migrating Content Delivery Network (CDN) Related Services.
Preventive Measures
Configuring Usage Cap Policies
To prevent high bills caused by resource abuse attacks, it is an effective control means to add usage cap policies for key website metrics (such as bandwidth, traffic, and request volume) and configure reasonable usage caps and alarm thresholds. If an alarm occurs, you should promptly check whether the real-time requests are normal according to the Investigation Measures, and then handle the issue according to the Countermeasures.
Note:
It takes a delay of about 10 minutes for a usage cap policy to take effect. The consumption generated during this period will be billed normally.
Under all usage cap policies, the usage is calculated by subdomain name. When the effective range is selected as entire site or all subdomain names, it indicates that all subdomain names under the site will share a single cap policy.
When the same domain name has policies for two or more of the metrics such as traffic, bandwidth, and request volume, the domain name will deactivate the service if any one of these metrics reaches its threshold.
Currently, the usage cap policies can be configured only for L7 (application layer) traffic/bandwidth and HTTP/HTTPS requests, but cannot be configured for L4 (transport layer TCP/UDP application) traffic and other value-added services such as QUIC and BOT.
Configuration Sample
For detailed operations of configuring a usage cap policy, see Usage Cap Policy. In the Add capping policy window, select a site for the effective range and configure a cap policy based on the following suggestions:
Configuration Item | Configuration Options | Corresponding Suggestions | Use Cases |
Statistical period | 5 minutes (recommended) | Set a lower threshold to quickly detect and respond to abnormal traffic or requests. | Enables timely detection of short-term abnormal traffic or request peaks and allows you to quickly take protection measures. It is suitable for real-time monitoring and immediate response needs. |
| Hour | Set a medium threshold in combination with the data from normal business peak periods to avoid false triggering of the cap during short-term traffic surges. | Captures short-term traffic fluctuation trends and provides some response time for protection adjustment. |
| Day (24 hours) | Set a higher threshold based on 2-3 times the normal daily business traffic to ensure recognition of long-term abnormal traffic. | Provides a global perspective to identify abnormal traffic or request patterns throughout the day. It is suitable for formulating long-term protection policies and resource plans. |
Cap configuration | L7 traffic (recommended) | Set a traffic threshold based on 2-3 times the normal business traffic to handle traffic surges and avoid false triggering of the cap due to short-term normal traffic growth. | Effectively prevents attackers from consuming bandwidth resources through massive downloads of large files. |
| HTTP/HTTPS request volume | Set a threshold based on 2-3 times the normal request volume to avoid false triggering of the cap during normal business peak periods. | Effectively prevents request flooding attacks that consume resources through a large volume of false requests. |
| L7 bandwidth | Set a bandwidth threshold based on 2-3 times the normal bandwidth usage to handle bandwidth usage surges. | Effectively prevents excessive bandwidth consumption and avoids resource waste caused by large-traffic download attacks. |
Exceeding the threshold | Disable the service, which should be enabled again in the domain list. | | |
Alarm threshold | 50% (recommended). An alarm message is sent when the usage reaches 50% of the configured alarm threshold. | | |
Note:
When the alarm threshold is enabled and the short-term usage surge is large, since the scan interval is 5 minutes, the previous scan may not trigger the alarm threshold but the next scan directly reaches the access threshold. In this case, both a percentage alarm message and an access threshold alarm message are sent at the same time.
Enabling Real-Time Log Push
For more elaborate protection measures, it is recommended to enable the Real-time Log Push feature. This feature enables shipping access request logs to your specified destination with a low latency and supports configuration through the console or API. The latency from initiating a request to receiving the request at the destination is within 5 minutes. It is suitable for real-time monitoring and rapid investigation, such as CDN resource abuse prevention. Through real-time analysis on access behaviors, you can timely identify and analyze the characteristics of resource abuse attacks, and then configure a corresponding policy for precise blocking.
The ranges of requests recorded by different types of logs are described as follows:
Site Acceleration Logs: Record the domain name access requests, including all L7 requests passing through CDN. By default, only the allowed requests are recorded, while the blocked requests are not recorded. These logs can provide comprehensive access information and help identify abnormal high-frequency requests, abnormal traffic, and potential resource abuse behaviors.
Note:
The feature of Site Acceleration Logs in Real-Time Logs to record full L7 requests (including requests blocked by L7 protection) is currently in beta test. If you need to use it, contact us.
Rate Limiting and CC Attack Defense Logs: Record only the requests that match the security rules of the Rate Limiting and CC Attack Defense Module for L7 Protection, no matter whether the requests are blocked or not. They can help identify resource abuse behaviors through high-frequency requests.
Managed Rule Logs: Record only the requests that match the security rules of the Managed Rules Module for L7 Protection, no matter whether the requests are blocked or not. They can help detect the protection status based on managed rules and identify potential attacks and resource abuse behaviors.
Custom Rule Logs: Record only the requests that match the security rules of the Custom Rules Module for L7 Protection, no matter whether the requests are blocked or not. They can help identify abnormal requests that match custom rules and prevent specific types of resource abuse behaviors.
Bot Management Logs: Record only the requests that match the security rules of the Bot Management Module for L7 Protection, no matter whether the requests are blocked or not. They can help identify resource abuse behaviors triggered by automated scripts or malicious Bots.
Note:
Bot Management Logs are supported only after the site domain name has enabled Bot Management. For pricing details of Bot Management enabled, see VAU Fee (Pay-as-You-Go).
If you need to push specific field values in HTTP request headers, HTTP response headers, or Cookies, you can use the Custom Log Push Fields feature to accurately record such information in logs.
Investigation Measures
After configuring the preventive measures as mentioned above, if you receive an alarm and judge that the usage surge is significant, you should conduct thorough investigation at the next step. This section mainly describes how to perform multi-dimensional characteristic analysis and locating on suspected resource abuse by using the EdgeOne traffic analysis and log analysis features.
Traffic Analysis
Metric Analysis is a powerful data analysis service offered by EdgeOne, aimed to help users gain deep insights into the business operation and security status. By real-time monitoring and analysis on key metrics, you can quickly identify issues, optimize the resource allocation, and enhance the business stability and security. In scenarios of resource abuse attacks, it is recommended to focus on the following data through data filtering and selection and in combination with TOP rankings:
Referer distribution: High-frequency occurrence of a blank Referer or an invalid Referer often indicates credential stuffing, crawlers, or other malicious requests.
Changes in the access volumes for URLs or resource types: If the request volume for a small number of URLs or resource types surges and far exceeds that of other resources, this may indicate targeted resource abuse.
TOP client IP addresses: Observe whether most of requests come from a small number of IP addresses, and evaluate the feasibility of IP-based request frequency control.
Directions
1. Log in to the EdgeOne console and click Metric Analysis in the left sidebar.
2. On the metric analysis page, click Add filter to add sites with usage alarms into the filter.
3. Select a date range during which the suspected resource abuse attack occurs.
4. On the L7 access traffic page, scroll down to view the rankings in the following dimensions:
Hosts: Subdomain names requested by the client.
URLs: Specific URLs of resources requested by the client.
Resource types: Types of resources requested by the client, such as
.png, .json, etc.Client IP addresses: Specific source IP address of the client request.
Referers: Referer information of the client request.
Client device types:
Device type: Type of the hardware device used by the client for requests. Valid values include:
TV: Television.
Tablet: Tablet computer.
Mobile: Mobile phone.
Desktop: Computer.
Other: Others.
Browser: Type of the browser used by the client for requests.
Operating system: Type of the operating system used by the client for requests.
5. Click Add filter to add the following suggested filter criteria, which focus on abnormal traffic. Then click OK.
Referer: Locates requests with an empty Referer;
URL: Includes TOP 5 URLs to locate suspicious hotspot resources;
Resource types: Includes TOP 5 resource types to locate the type distribution of hotspot resources;
Client device/browser/operating system: Equals
Other; Empty to locate suspicious abnormal clients.
6. Observe the distribution of each metric after filtering, identify data that obviously deviates from normal levels, and analyze its correlation with resource abuse.
Offline Log Analysis
To further discover more characteristics of resource abuse requests, you should perform in-depth analysis on the offline logs in the alarm occurrence period. Through comprehensive field analysis, we can depict the profile of resource abuse requests from multiple dimensions such as source IP, URL path, request parameter, User-Agent, and Referer source, laying the data foundation for formulating precise countermeasures at the next step. The following describes the log fields that need special attention in offline log analysis for resource abuse investigation:
Field Name | Data Type | Description | Supported by Offline Logs or Not | Supported by Real-Time Logs or Not |
RequestUrl | String | URL path of the client request, excluding query parameters. This field is a key analysis dimension for resource abuse attacks. | ✓ | ✓ |
RequestUrlQueryString | String | Query parameter in the URL of the client request. If the query parameter of requests for resource abuse is fixed or has obvious characteristics, you can set a blocklist for the source IP of such requests or for requests matching this parameter. | ✓ | ✓ |
RequestUA | String | User-Agent information of the client request. Simple resource abuse tools often use the same User-Agent. If a large volume of access requests use a specific and uncommon User-Agent, you can consider blocking such requests. | ✓ | ✓ |
RequestReferer | String | Referer information of the client request. The Referer of a normal request is usually the URL of another page on the site or a search engine URL, but command line tools such as curl may forge the Referer. If the URL of the abused page is not actually referenced by other sites but appears in the Referer, it can be deemed abnormal. You can block such requests by configuring Referer Hotlink Protection. | ✓ | ✓ |
ClientIP | String | Client IP connected to the EdgeOne node, namely the source IP of the request. If a small number of IP addresses far exceed other IP addresses in access volume, you can consider blocking them. | ✓ | ✓ |
EdgeResponseBodyBytes | Integer | Size of the response body returned by the node to the client, in bytes. Malicious resource abuse often includes repeated downloading of large files, so analyzing the statistical results of EdgeResponseBodyBytes is a critical step in resource abuse analysis. | ✓ | ✓ |
Countermeasures
In complex and changeable attack scenarios of website resource abuse, there is no one-size-fits-all solution. EdgeOne offers a comprehensive suite of protection features including access control and rate limiting, which can be flexibly combined. You should select an optimal protection configuration combination based on the factors such as attack characteristics and actual business situations. The following provides a detailed practical tutorial for EdgeOne resource abuse prevention from different perspectives of personal site operators and online business sites.
Small and Medium Website Platforms
Scenario 1: Rapid Blocking of Abnormal Source IP Addresses Based on Metric Analysis
Scenario Example
During the suspected resource abuse period, the proportion of access to a 5 MB file was found abnormally high through analysis on the resource type ranking metric of L7 access traffic. Further investigation revealed that the file path was
/test/installer.apk, and the requests primarily came from client IP addresses of the 1.1.1.0/24 IP range. Based on this clue, you can quickly create an IP blocklist policy to block this malicious IP range and curb potential resource abuse behaviors.Recommended Configuration
It is recommended to configure protection policies by using the custom rules of the EdgeOne Web protection feature. For detailed operations, see Custom Rules.
For the Personal plan, users can configure the rule type as Client IP Control in Basic Access Control, and select the matching method as Client IP equals
1.11.32.0/24 and the action as Block.
For the Basic plan and above, users can configure the matching fields as Client IP matches
1.1.1.0/24 and Request path includes /test/installer.apk, and select the action as JavaScript Challenge in Precise Matching Rules.
Scenario 2: Rapid Blocking of Abnormal User-Agents Based on Log Analysis
Scenario Example
Real-time logs show that within a certain period, the distribution of RequestUA was abnormally concentrated. Further analysis revealed that
python-requests/2.22.0 was accessed most frequently, and a large volume of requests used the unique User-Agent identifier of Python scripts containing python-requests/. Since these requests significantly deviated from the User-Agent characteristics of regular browsers, they can be identified as automated requests or even malicious crawlers. Based on this, you can configure a User-Agent blocklist rule to precisely block suspicious requests containing specific User-Agent identifiers.Recommended Configuration
It is recommended to configure protection policies by using the custom rules of the EdgeOne Web protection feature. For detailed operations, see Custom Rules.
For the Personal plan, users can configure the rule type as User-Agent Control in Basic Access Control, and select the matching method as User-Agent matches wildcard pattern, the matching content as
python-requests/2.22.0, and the action as Block.
For the Basic plan and above, users can configure the matching field as User-Agent includes
python-requests and the action as JavaScript Challenge in Precise Matching Rules.
Scenario 3: Prevention and Blocking Based on Known Malicious User-Agents
Scenario Example
For known common resource abuse tools, you can configure their characteristic User-Agent strings in your custom rules in advance. Preventively enabling this rule in the global site or key paths can minimize the risk of resource abuse by such tools. Common User-Agents for resource abuse include
empty User-Agent; curl/xx.xx; Wget/xx.xx; ApacheBench/xx.xx; python-requests/xx.xx.Recommended Configuration
It is recommended to configure protection policies by using the custom rules of the EdgeOne Web protection feature. For detailed operations, see Custom Rules.
For the Personal plan, users can configure the following two rules in Basic Access Control:
Rule 1: Configure the rule type as User-Agent control, the matching method as User-Agent is empty, and the action as Block.
Rule 2: Configure the rule type as User-Agent control, the matching method as User-Agent matches wildcard pattern, the matching content as
curl/; Wget/; ApacheBench/; python-requests/, and the action as Block.
For the Basic plan and above, users can configure the following two rules in Precise Matching Rules:
Rule 1: Configure the matching field as User-Agent includes
curl/; Wget/; ApacheBench/; python-requests/ and the action as JavaScript Challenge.
Rule 2: Configure the matching field as User-Agent is empty and the action as JavaScript Challenge.
Scenario 4: Allowing Only Common User-Agents (Temporary High Defense)
In case of large-scale, dispersed User-Agent resource abuse, if it is difficult to identify each malicious User-Agent characteristic, you can use the reverse logic to allow only valid User-Agent access requests from common normal browsers and applications. This method can filter a large volume of suspicious requests at once. However, due to the strictness of the rules, there is a risk of misjudgment, so it should be used cautiously in combination with other dimensional characteristics.
Recommended Configuration
It is recommended to configure protection policies by using the custom rules of the EdgeOne Web protection feature. For detailed operations, see Custom Rules.
For the Personal plan, users can configure the rule type as User-Agent control in Basic Access Control, and select the matching method as User-Agent does not match wildcard pattern, the matching content as
*Linux*; *Macintosh*; *Android*; *iPhone*; *iPad*; *Windows*, and the action as Block.
For the Basic plan and above, users can configure the matching field as User-Agent does not match wildcard pattern, the matching content as
Android, iPhone, iPad, Mac, Windows, Linux, and the action as JavaScript Challenge in Precise Matching Rules.
Note:
This policy is not required for application scenarios when the User-Agent is empty in normal business.
If the User-Agent value is an application name, you should add the application name of normal business in the User-Agent into the matching content.
It should be configured cautiously due to high strictness. To avoid false blocking, you should perform joint judgment in combination with other dimensional characteristics.
Scenario 5: Configuring the Single-IP High-Frequency Access Limit for CC Attacks (Temporary High Defense)
CC Attack Defense supports identifying and handling CC attacks through rate baseline learning, header feature statistical analysis, and client IP intelligence. EdgeOne offers three preset CC attack defense policies:
Adaptive frequency control: It is used to defend against CC attacks that occupy server resources through high-frequency and a large volume of concurrent connection requests, and can limit the access frequency based on a single IP source.
Slow attack defense: It is used to defend against CC attacks that occupy server resources through a large volume of slow connection requests, and can limit the minimum access connection rate based on a single session and eliminate slow connection clients.
Intelligent client filtering: It integrates rate baseline learning, header feature statistical analysis, and client IP intelligence, to dynamically generate protection rules in real time. It also supports human-machine verification for requests from high-risk clients or those with high-risk header features. Intelligent client filtering is enabled by default and conducts a JavaScript Challenge to clients that meet the rules.
In case of suspected website resource abuse attacks or abnormal usage alarms, it is recommended to temporarily set Adaptive Frequency Control to the Adaptive - Emergency level and the action to JavaScript Challenge. This measure can efficiently block a large volume of requests from malicious IP addresses and effectively prevent resource abuse and other attacks. For detailed operations, see CC Attack Defense.
Note:
After handling the resource abuse attack, you should promptly restore Adaptive Frequency Control to the recommended configuration, Adaptive - Loose, to ensure smooth access with normal business traffic. For details of various limitation levels, see CC Attack Defense.
Scenario 6: Personalized Frequency Control Based on Business Levels
Different from strong DDoS attacks, resource abuse tends to be more covert. You should perform judgment based on specific business scenarios and formulate personalized frequency control policies to avoid false blocking of valid users. No matter whether the blocking policy is specific to the IP or User-Agent, it belongs to precise blocking. However, in actual attacks, the attack characteristics may not be obvious, especially when the volume of requests from a source IP address is as high as hundreds of thousands.
Defense policies combined with the business scenario first require website administrators to evaluate the normal access mode of the business and determine the business traffic baseline. For example, in application download or upgrade scenarios, most IP addresses are generally used for only one or two downloads. In rare cases, there might be multiple retries due to failures, but the number of retries is usually within a reasonable frequency range. Abnormal high-frequency access possibly indicates attacks or malicious resource abuse.
When a website suffers from resource abuse attacks, the domain name bandwidth significantly increases. To address this issue, you can use rate limiting of the EdgeOne Web protection feature to set thresholds based on normal business levels and configure rate limiting policies, or monitor and adjust the policies through Real-Time Logs. For detailed operations, see Rate Limiting.
Note:
During configuration, the frequency control rules should be dynamically adjusted based on the actual defense effect. Initially, you can set frequency thresholds based on empirical values to quickly achieve defense. If the effect is poor, the rules can be gradually tightened; conversely, if the rules affect normal business, they should be appropriately loosened.
Game Package Download Frequency Limiting Based on Business Baselines
Scenario Example
A game platform offers download services for installation and update packages of multiple games through EdgeOne acceleration. The download URLs of game packages have a fixed pattern. Examples are as follows:
Game A installation package:
https://cdn.example.com/games/A/installer_v1.0.zipGame A update package:
https://cdn.example.com/games/A/patch_v1.1.zipGame B installation package:
https://cdn.example.com/games/B/installer_v2.0.exeGame B update package:
https://cdn.example.com/games/B/patch_v2.1.exeOn the day when a game version was released, the number of downloads per single IP address was generally 1 and the number of download retries due to individual network issues did not exceed 3. However, the installation and update packages were downloaded frequently to some IP addresses after the version was released, far exceeding the frequency of normal user behaviors, so a usage alarm was triggered. The possible causes may be that pirate websites or sharing communities were capturing game packages, or attackers intended to consume bandwidth. You can configure the Rate Limiting rules to promptly block these malicious requests.
Recommended Configuration
In Precise Rate Limiting, configure the matching object as Custom protection object, the matching field as Request URL includes
games/; installer/; patch/ and Request method equals GET. In the Rate limit section, select Requests (client to EdgeOne) with the request feature as Client IP, configure the trigger as Count in 10 minutes exceeds 3 times, and select the action as JavaScript Challenge with the action duration as 1 hour**. For detailed operations, see Rate Limiting.
Abnormal User-Agent Frequency Limiting Based on Log Analysis
Scenario Example
A website was accessed frequently by attackers, triggering a usage alarm. Through real-time log analysis, it was found that the attacking IP addresses were dispersed, suggesting distributed attacks, but the User-Agents were abnormally uniform.
Most of requests came from a single User-Agent string,
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0). This case deviated significantly from normal business, which typically exhibited a diverse range of User-Agents representing various browsers and devices. Under normal circumstances, the volume of access requests with this suspicious User-Agent is very low, but at this time, the request volume surged and occupied most of the traffic. It can be basically determined as CDN resource abuse attack. You can configure the Rate Limiting rules to promptly block these malicious requests.Recommended Configuration
In Precise Rate Limiting Rules, configure the matching object as Custom protection object and the matching field as User-Agent equals
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0). In the Rate limit section, select Requests (client to EdgeOne) with the request feature as Client IP, configure the trigger as Count in 1 minute exceeds 400 times, and select the action as JavaScript Challenge with the action duration as 30 minutes. For detailed operations, see Rate Limiting.
Note:
You should adjust the threshold for triggering protection and the action duration based on your own normal business level and the attacker characteristics and attack frequency obtained from real-time log analysis.
EdgeOne Hotlink Protection Practical Tutorial
In addition to direct protection measures against resource abuse, you should pay attention to protection of the website resources and adopt active defense. Hotlink protection is an important means for preventing unauthorized use of website resources.
Hotlink refers to the behavior of illegally referencing and using resources (such as images, videos, software packages, etc.) from the original site on other sites and consuming the bandwidth and resources of the original site, without the permission of the website owner. It not only infringes the legal rights of the original site but may also cause adverse SEO impacts on the original site. Therefore, it is necessary to actively implement hotlink protection measures.
EdgeOne offers a perfect hotlink protection solution to control hotlink behaviors by Referer hotlink protection, Token hotlink protection, remote authentication, etc. This protects your content from unauthorized hotlink access and enhances the security of acceleration services. For details, see EdgeOne Hotlink Protection Practical Tutorial.
Enterprise-Level Business Platform
For online business sites facing resource abuse threats, in addition to using general protection measures commonly adopted for personal sites, it is recommended to select the EdgeOne Standard or Enterprise plan and enable the Bot management feature. With its built-in artificial intelligence engine and extensive behavioral feature analysis, you will gain a smarter, more effortless Bot management experience, to effectively address various resource abuse attacks.
The Bot Intelligence Analysis module employs advanced machine learning algorithms to form a threat identification model through massive data training. This model extracts key behavioral characteristics from multiple dimensions such as request rate, IP intelligence, URL sequences, and SSL/TLS fingerprints, and adopts techniques such as cluster analysis and similarity comparisons to accurately determine whether a request comes from an automated program and has malicious intents. It can reduce false blocking of valid requests with a comprehensive and multi-dimensional analytical approach.
Additionally, the EdgeOne Enterprise plan supports JA3 fingerprint characteristics. Website administrators can preset fingerprint conditions for high-risk Bots based on their business scenarios, to achieve precise blocking of specific attack tools. For example, you can incorporate fingerprints of Python libraries and headless browsers commonly used by malicious crawlers into the resource abuse defense rules, to automatically block related traffic and achieve more proactive and efficient protection.
Note:
The Bot management feature is supported only after the site domain name has enabled Bot Management. For pricing details of Bot Management enabled, see VAU Fee (Pay-as-You-Go).