What is HTTP Cache-Control: Mastering the Principles and Functionality

HTTP cache control is a mechanism in the HTTP protocol that allows web browsers and servers to manage the storage and retrieval of web resources. It improves web application performance, reduces server load, and saves bandwidth. This is achieved through cache control headers in HTTP requests and responses, which instruct how and when to cache resources. Understanding HTTP Cache-Control is crucial for building efficient web applications.

What is the HTTP Protocol?

The HTTP protocol stands for Hypertext Transfer Protocol, which is used to transfer hypertext from the World Wide Web (WWW) server to the local browser.

HTTP is a protocol that uses TCP/IP communication to transfer data (HTML files, image files, query results, etc.).

The HTTP protocol works on a client-server architecture. The browser, as an HTTP client, sends all requests to the HTTP server, or the WEB server, through a URL. The Web server, upon receiving the request, sends a response back to the client.

The history of HTTP protocol development

The initial HTTP 0.9 was extremely simple, with requests consisting of a single line of instruction. By the time HTTP 2.0 was introduced, it had better performance, and more data was transmitted through HTTP requests, resulting in significant optimization of network efficiency.

HTTP 3.0 is currently in the drafting and testing phase and is the future of the HTTP protocol. HTTP 3.0 runs on top of the QUIC protocol, which provides reliable transmission based on UDP. Using UDP will avoid the head-of-line blocking problem of TCP and speed up network transmission, but it also needs to implement a mechanism for reliable transmission. HTTP 3.0 is not an extension of HTTP 2.0; it will be an entirely new protocol.

Understanding HTTP Request and Caching

An HTTP request consists of three parts: the request line, message headers, and request body.

The caching mechanism we are going to introduce today is closely related to the message headers, as they determine the browser's caching behavior. Using the browser's "Inspect" feature (F12), we can easily view the HTTP protocol's transmission process:

Open the transmission details of any CSS file at random to see the details of the HTTP transmission process:

As you can see, the request headers are composed of key-value pairs, with each header corresponding to different behaviors and meanings.

Why do We Need Caching?

As we mentioned earlier, the HTTP protocol is a Client-Server working model, which means that all content and resources browsed by the browser are downloaded from the server.

• If we need to fetch resources from the server every time we refresh a web page, and if the resource files are particularly large, it will result in a long request time and a poor surfing experience!
• At the same time, if all large file resources need to be fetched from the server, it will also cause increased server pressure and higher bandwidth costs.

To solve this problem, browsers have introduced a resource caching mechanism – when the browser requests resources, it negotiates with the server. Cacheable static resources will be cached locally in the browser for use during subsequent visits.

The Principles of Cache-Control Mechanisms

Rules for caching

    1. Freshness (expiration mechanism): This refers to the validity period of a cached copy. A cache copy is considered valid and fresh by the browser if it meets the following conditions:

• It contains complete expiration time control header information (HTTP protocol header) and is still within the validity period.
• The browser has already used this cached copy and has checked its freshness within a session.

    2. Validation value (validation mechanism): When the server returns a resource, it sometimes includes an entity tag (such as ETag) in the control header information. This can be used as a validation identifier during the browser's subsequent request process. If the validation identifier does not match, it indicates that the resource has been modified or expired, and the browser needs to reload the resource content.

    3. Auxiliary: Vary header. This is used to differentiate between different cache copies, for example: when the "request header" contains Vary: Accept-Encoding, the browser can cache different content copies based on different Content-Encoding in the "response header".

Two Stages of HTTP Caching

Browser caching is generally divided into two categories: strong caching (also known as local caching) and negotiated caching (also known as weak caching).

Local Cache:

Before the browser sends a request, it will first check the cache to see if it hits the strong cache. If it does, the resource is read directly from the cache, and no request is sent to the server. Otherwise, proceed to the next step.

Negotiated Cache:

When the strong cache is not hit, the browser will definitely send a request to the server. The server will determine whether the negotiated cache is hit based on some fields in the Request Header. If it is hit, the server will return a 304 response without carrying any response entity, just telling the browser that it can directly fetch the resource from the browser cache. If neither the local cache nor the negotiated cache is hit, the resource is loaded directly from the server.

Freshness

As mentioned earlier, freshness refers to the validity period of a cached copy. So, how exactly does the browser specify the freshness of a cache?

In theory, once a resource is cached, it should be able to be stored in the cache permanently. However, since the cache has limited space for storing resource copies, it will periodically delete some copies in a process called cache eviction.

On the other hand, when a resource on the server is updated, the corresponding resource in the cache should also be updated. Since HTTP is a client-server protocol, the server cannot directly notify the client to update the cache when a resource is updated. Therefore, both parties must agree on an expiration time for the resource. Before that expiration time, the resource (cache copy) is considered fresh. After the expiration time, the resource (cache copy) becomes stale.

The eviction algorithm is used to replace stale resources (cache copies) with fresh ones. Note that a stale resource (cache copy) will not be directly cleared or ignored. When the client initiates a request, if the cache retrieves a corresponding stale resource (cache copy), the cache will first attach an If-None-Match header to the request and send it to the target server to check if the resource copy is still considered fresh. If the server returns a 304 (Not Modified) response (which does not carry any entity information), it indicates that the resource copy is fresh, which can save some bandwidth.

If the server determines that the resource has expired through If-None-Match or If-Modified-Since, it will return the response with the resource's entity content.

In the HTTP protocol, the commonly used headers for controlling cache freshness include:

Expires（HTTP/1.0）

Expires is a cache time control header introduced in HTTP 1.0, which means how long the cached content will expire. The Expires response header contains a date/time, after which the response expires. An invalid date, such as 0, represents a past date, indicating that the resource has already expired.

If the "max-age" or "s-max-age" directive is set in the Cache-Control response header, the Expires header will be ignored. Syntax:

Expires: <http-date>

Pragma（HTTP/1.0）

Pragma is a general header introduced in HTTP 1.0, mainly used to specify whether the content should not be cached. Its effect is similar to Cache-Control: no-cache. It is used for backward compatibility with servers that only support the HTTP 1.0 protocol when the Cache-Control in HTTP 1.1 was not yet available. Syntax:

Pragma: no-cache

Cache-Control（HTTP/1.1）

Cache-Control is a header introduced in HTTP 1.1, used to implement caching mechanisms in HTTP requests and responses by specifying directives. Cache-Control plays a significant and important role in cache control and is the essential header for controlling cache in mainstream browsers.

Some Cache-Control directives that can be used in requests:

Cache-Control: max-age=<seconds>
Cache-Control: max-stale[=<seconds>]
Cache-Control: min-fresh=<seconds>
Cache-Control: no-cache
Cache-Control: no-store
Cache-Control: no-transform
Cache-Control: only-if-cached

Some Cache-Control directives that can be used in responses:

Cache-Control: must-revalidate
Cache-Control: no-cache
Cache-Control: no-store
Cache-Control: no-transform
Cache-Control: public
Cache-Control: private
Cache-Control: proxy-revalidate
Cache-Control: max-age=<seconds>
Cache-Control: s-maxage=<seconds>

We can classify Cache-Control directives into three categories: cacheability, expiration, and others.

Cacheability:

• public: Indicates that the response can be cached by any object (including the client sending the request, proxy servers, etc.), even for content that is usually not cacheable. (For example: 1. The response does not have a max-age directive or Expires header; 2. The request method corresponding to the response is POST.)
• private: Indicates that the response can only be cached by a single user and cannot be used as a shared cache (i.e., proxy servers cannot cache it). Private caches can cache response content, such as the corresponding user's local browser.
• no-cache: Forces the cache to submit the request to the original server for validation (negotiated cache validation) before releasing the cache copy.
• no-store: The cache should not store any content related to the client request or server response, i.e., not use any cache.

Expiration:

• max-age=: Sets the maximum period for cache storage, after which the cache is considered expired (in seconds). Unlike Expires, the time is relative to the request time.
• s-maxage=: Overrides the max-age or Expires header, but only applies to shared caches (such as proxies); private caches will ignore it.
• max-stale[=]: Indicates that the client is willing to accept an expired resource. An optional number of seconds can be set, indicating that the response should not have been stale for more than the given time.
• min-fresh=: Indicates that the client wants to receive a response that will remain fresh for the specified number of seconds.
• stale-while-revalidate=: Indicates that the client is willing to accept a stale response while asynchronously checking for a new response in the background. The number of seconds indicates the time the client is willing to accept a stale response.
• stale-if-error=: Indicates that if a new check fails, the client is willing to accept a stale response. The number of seconds represents the time the client is willing to accept a stale response after the initial expiration.
• must-revalidate: Once a resource expires (e.g., exceeds max-age), the cache must not use the resource to respond to subsequent requests until it has successfully validated it with the original server.
• proxy-revalidate: Similar to must-revalidate, but it only applies to shared caches (e.g., proxies) and is ignored by private caches.
• immutable: Indicates that the response body will not change over time. The resource (if not expired) does not change on the server, so the client should not send revalidation request headers (such as If-None-Match or If-Modified-Since) to check for updates, even if the user explicitly refreshes the page. In Firefox, immutable can only be used in https:// transactions. 

Others:

• no-transform: The resource must not be transformed or altered. HTTP headers such as Content-Encoding, Content-Range, and Content-Type cannot be modified by proxies. For example, non-transparent proxies or services like Google's Light Mode may convert image formats to save cache space or reduce traffic on slow links. The no-transform directive does not allow this.
• only-if-cached: Indicates that the client only accepts cached responses and does not check the original server for an updated copy.

Validation Value (Validation Mechanism)

When a cache content is stored, and the local browser does not know whether it has expired and needs validation, the client (browser) will initiate a request with specific cache validation headers back to the server. If the server validates that the content has not been updated, it will return an HTTP status code 304, telling the client that the content has not been updated, and the client can continue to use the local cache for service.

When the cached content expires, cache validation or resource re-acquisition is required. Validation only occurs when the server returns a strong validator or a weak validator.

Let's look at an example:

Add an acceleration domain on Tencent EdgeOne and configure the corresponding resolution. We use the curl command to simulate client requests and access service resources on the CDN.

First, we normally retrieve the content resource:

curl 'loadblance.qcdntest.cn' -v

The server normally outputs the content and returns a 200 status.

Here we notice that the server also returned a header to us:

Etag: "665555e6-fe4"

If we follow the definition in the HTTP 1.1 protocol and include the If-None-Match header in the next request to validate the cache content's validity, let's see what happens.

We perform another curl request to simulate the client, but this time we will include the If-None-Match header, and the header value will be the Etag value that the server gave us last time:

curl 'loadblance.qcdntest.cn' -v -H "If-None-Match: "665555e6-fe4""

We can see that the server returned a 304 and did not return any content. At this point, the client can know that our cached content has not been updated, so we can safely continue to use the previously cached content.

As you can see, cache validation and negotiation are also an important part of caching, which can save a lot of bandwidth overhead for our servers.

Common cache validation headers:

Last-Modified & If-Modified（HTTP/1.0）

Used to validate whether resources have expired; Last-Modified and If-Modified-Since are commonly used together, corresponding to the response phase and request phase.

When a browser requests a URL for the first time, the server's return status will be 200, and the content is the resource you requested. At the same time, there is a Last-Modified property that marks the last time this file was modified on the server, in a format like this:

Last-Modified: Wed, 21 May 2024 07:28:00 GMT

When the client requests this URL for the second time, according to the provisions of the HTTP protocol, the browser will send the If-Modified-Since header to the server, asking whether the file has been modified after this time:

If-Modified-Since: Wed, 21 May 2024 07:28:00 GMT

If the resource on the server has not changed, it automatically returns the HTTP 304 (Not Changed.) status code, and the content is empty, which saves the amount of data transferred. When the server-side code changes or the server is restarted, the resource is reissued, similar to the first request. This ensures that resources are not repeatedly issued to the client and that the client can get the latest resources when the server changes.

ETag & If-None-Match（HTTP/1.1）

The HTTP protocol specification defines ETag as "the entity value of the requested variable". Another way to put it is that ETag is a token that can be associated with a web resource. A typical web resource can be a web page, but it can also be a JSON or XML document. The server is solely responsible for determining what the token is and its meaning and sending it to the client in the HTTP response header. Here is the format returned by the server:

ETag: "665555e6-fe4"

The client's query update format is as follows:

If-None-Match: "665555e6-fe4"

If the ETag has not changed, it returns status 304 and then does not return, which is also the same as Last-Modified.

Practical Operation on Tencent EdgeOne

Let the browser cache the content for 10 minutes

In the Tencent EdgeOne, configure the Browser Cache TTL for the domain through the feature-rich and powerful rule engines.

The response header contains Cache-Control: max-age=600, and the browser will cache the content for 10 minutes according to the header in the response.

Make CDN follow the cache control of the origin server

Make the CDN cache server follow the cache control behavior of the origin server. If the Cache-Control field exists in the HTTP Response Header of the origin server, then:

• If the Cache-Control field is max-age, the CDN node caches the resource according to the max-age value.
• If the Cache-Control field is no-cache/no-store/private, the CDN node does not cache the resource.
• If there is no Cache-Control field in the HTTP Response Header of the origin server, then: The CDN node does not cache the resource.

The cache control mechanism in the HTTP protocol is a key content of our study of the HTTP protocol. Understanding the principles behind it is of great help to our understanding of the browser's cache control mechanism. At the same time, CDN is an important participant in internet cache servers. Studying the cache mechanism of the HTTP protocol can also help us understand some cache control mechanisms and principles behind CDN. If you are interested in experiencing these benefits firsthand, feel free to contact us to learn more about Tencent EdgeOne and its features. You can also click here to get started free and experience its benefits firsthand.