Observability
  • Log Service
    • Overview
    • Real-time Logs
      • Real-time Logs Overview
      • Push to Tencent Cloud CLS
      • Push to AWS S3-Compatible COS
      • Push to HTTP Server
    • Offline Logs
    • Related References
      • Field description
        • L7 Access Logs
        • L4 Proxy Logs
      • Real-Time Log Push Filter Conditions
      • Custom Log Push Fields
  • Data Analysis
    • Overview
    • Traffic Analysis
    • Cache Analysis
    • Security Analysis
      • Site Security Overview
      • Web Security Analysis
    • L4 Proxy
    • DNS Resolution
    • Related References
      • How to use filter condition
      • How to Modify Query Time Range
      • How to Export Statistical Data and Reports

Push to HTTP Server

EdgeOne Real-time Log Push supports pushing logs to a custom API address. You can configure it through the console or API. EdgeOne can use an HTTP POST request to call the backend API address you provide, and transfer logs in the HTTP body to a server you specify.

Directions

1. Log in to the EdgeOne console and click Site List in the left sidebar. Then click on the site to be configured in the site list, to enter the site details page.
2. On the site details page, click Log Service > Real-time Logs.
3. On the real-time logs page, click Create Push Task.
4. On the log source selection page, enter a task name, select a log type, service area, and domain name/L4 proxy instance requiring log push, and click Next.
5. On the push content definition page:
(Required) Check the log fields to be pushed from the predefined field list.
(Optional) Add a custom log field, which supports extracting specified field names from the request headers, response headers, and Cookie headers.
(Optional) Configure the log push filter conditions. Full logs are pushed by default.
(Optional) In advanced configuration, set the sampling ratio. By default, sampling is not enabled and 100% of logs are pushed to the destination.
(Optional) In advanced configuration, set the log output format. The default format is JSON Lines.
6. On the destination selection page, select HTTP (POST) and click Next.
7. On the destination information page, enter the related destination and parameter information.
Parameter Name
Description
API Address
Enter your log receiving API address, for example: https://www.example.com/edgeone-logs
Content Compression
To reduce the size of log content and save the traffic costs, you can enable content compression by checking Compress log files with gzip. EdgeOne will compress the logs in gzip format and then transmit them. Meanwhile, it will add an HTTP header Content-Encoding: gzip to indicate the compression format.
Origin Authentication
When encrypted authentication is selected, the pushed logs will carry authentication information for verification by the origin server, to ensure the security of the data source identity. For the authentication algorithm, see Authentication Algorithm Reference.
Custom HTTP Header
Add the HTTP headers to be carried when EdgeOne initiates a request. For example:
Add the header log-source: EdgeOne, to identify the log source as EdgeOne.
Add the header BatchSize: ${batchSize}, to obtain the number of log entries pushed in each POST request.
Note
If the header name you enter is a default header carried for EdgeOne log push, such as Content-Type, the header value you enter will override the default value.
8. Click Push, confirm the related cost tips in the pop-up window, and click Confirm Creation.
9. During the configuration phase of the real-time log push task, test data will be sent to the API address to verify the API connectivity. The data format is as follows:
{ "ClientState": "CH-AH", "EdgeResponseTime": 366, "RequestID": "13515444256055847385", "ClientRegion": "CN", "RemotePort": 443, "RequestHost": "www.tencent.com", "RequestMethod": "GET", "RequestUrlQueryString": "-", "RequestUrl": "/en-us/about.html", "RequestProtocol": "HTTP/2.0", "EdgeServerID": "336d5ebc5436534e61d16e63ddfca327-d41d8cd98f00b204e9800998ecf8427e", "RequestTime": "2022-07-01T02:37:13Z", "EdgeCacheStatus": "-", "EdgeResponseBytes": 39430, "EdgeResponseStatusCode": 200, "ClientIP": "0.0.0.0", "RequestReferer": "https://www.tencent.com/", "RequestUA": "Mozilla/5.0 (iPhone; CPU iPhone OS 15_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.5 Mobile/15E148 Safari/604.1", "EdgeServerIP": "0.0.0.0", "RequestRange": "0-100/200", "EdgeInternalTime": 334, "RequestBytes": 237 }

Related References

Code Example for Server-side Log Parsing

When origin server authentication is not enabled, you can refer to the following Python code for parsing the log content in the request body on the server side.
# Import modules from the Python standard library.
import time # Used to get the current time.
import gzip # Used to handle data compressed with gzip.
# Import HTTPServer and BaseHTTPRequestHandler classes from the http.server module.
from http.server import HTTPServer, BaseHTTPRequestHandler
import json # Used to handle JSON data.
# Define a class inheriting from BaseHTTPRequestHandler, used to handle HTTP requests.
class Resquest(BaseHTTPRequestHandler):
# Override the do_POST method, which is called when the server receives a POST request.
def do_POST(self):
# Print the request header information.
print(str(self.headers))
# Print the HTTP request command (e.g., POST).
print(self.command)
# Read the request body content. The reading length is determined according to the Content-Length field in the request header.
req_datas = self.rfile.read(int(self.headers['content-length']))
try:
# Attempt to decode the request body content and print it.
print(req_datas.decode())
except Exception as e:
# If an exception occurs during decoding, print the exception information.
print(e)
# Check whether the request header contains Content-Encoding: gzip. If yes, decompress the request body.
if self.headers['Content-Encoding'] == 'gzip':
data = gzip.decompress(req_datas)
# Print the decompressed gzip content
print('---------------decompress gzip content-------------------------')
print(data.decode())
# Check whether the request path is '/edgeone-logs'. If not, return a 404 error.
if self.path != '/edgeone-logs':
self.send_error(404, "Page not Found!")
return
# If the request path is correct, prepare the response data.
data = {
'result_code': '1',
'result_desc': 'Success',
'timestamp': int(time.time()) # Respond with the current timestamp
}
# Send an HTTP response status code 200, indicating the request succeeded.
self.send_response(200)
# Set the response header Content-type to application/json.
self.send_header('Content-type', 'application/json')
# End the sending of response headers.
self.end_headers()
# Write the response data in JSON format to the response body.
self.wfile.write(json.dumps(data).encode('utf-8'))
# Check whether the current script is running as the main program.
if __name__ == '__main__':
# Define the server listening address and port. You can replace 9002 with your own port.
host = ('', 9002)
# Create an HTTPServer object, passing in the listening address and port, and the request handler class.
server = HTTPServer(host, Resquest)
# Print the server startup information.
print("Starting server, listen at: %s:%s" % host)
# Start the server and keep it running until externally interrupted.
server.serve_forever()

Request Authentication Algorithm

If you select Encrypted Signature for origin server authentication in the push destination information, you can enter a custom SecretId and SecretKey. EdgeOne will add auth_key and access_key of signature to the request URL. The details of the signature algorithm are as follows:
1. Request URL Composition
As shown below, the request URL carries auth_key and access_key after ?.
http://DomainName[:port]/[uri]?auth_key=timestamp-rand-md5hash&access_key=SecretId
Parameter description:
timestamp: Current request time, in the format of Unix 10-digit timestamp in seconds.
rand: Random number.
access_key: Custom SecretId, used to identify the identity of the API requester.
SecretKey: Custom SecretKey, with a fixed length of 32 characters.
uri: Resource identifier, for example: /access_log/post.
md5hash: md5hash = md5sum(string_to_sign), where string_to_sign = "uri-timestamp-rand-SecretKey". It is a verification string calculated through the MD5 algorithm, consisting of digits 0-9 and lowercase letters a-z, with a fixed length of 32 characters.
2. Calculation Example
Assume the parameters entered are as follows: API Address: https://www.example.com/access_log/post SecretId = YourID SecretKey = YourKey uri = /access_log/post timestamp = 1571587200 rand = 0
string_to_sign = "/access_log/post-1571587200-0-YourKey"
By calculation based on this string, the following result is obtained:
md5hash=md5sum("/access_log/post-1571587200-0-YourKey")=1f7ffa7bff8f06bbfbe2ace0f14b7e16
The request URL in final push is:
https://www.example.com/cdnlog/post?auth_key=1571587200-0-1f7ffa7bff8f06bbfbe2ace0f14b7e16&access_key=YourID
After the server receives the push request, it extracts and splits the value of auth_key to obtain timestamp, rand, and md5hash. The server can first check whether the timestamp has expired. The recommended validity time is 300s. Then it assembles a string to be encrypted with the SecretKey based on the aforementioned rules. After encryption, the string is compared with the md5hash value in auth_key. If consistent, it indicates authentication succeeded.
3. Code Example for Server-Side Authentication Request Parsing
Python
Golang
import hashlib

from flask import Flask, request

app = Flask(__name__)


def get_rsp(msg, result={}, code=0):
return {
"respCode": code,
"respMsg": msg,
"result": result
}


def get_secret_key(access_key):
return "secret_key"


@app.route("/access_log/post", methods=['POST'])
def access_log():
if request.method == 'POST':
if request.content_type.startswith('application/json'):
current_time_ts, rand_num, md5hash = request.args.get("auth_key").split("-")
# Judge whether the request time is within the validity period.
if time.time() - int(current_time_ts) > 300:
return get_rsp(msg="The request is out of time", code=-1)

access_key = request.args.get("access_key")
# Get the secret_key using access_key (SecretId).
secret_key = get_secret_key(access_key)
raw_str = "%s-%s-%s-%s" % (request.path, current_time_ts, rand_num, secret_key)
auth_md5hash = hashlib.md5(raw_str.encode("utf-8")).hexdigest()
if auth_md5hash == md5hash:
# Authentication succeeded.
if request.headers['content-encoding'] == 'gzip':
# Data decompression
pass
# Data handling
return get_rsp("ok")
return get_rsp(msg="Please use content_type by application/json", code=-1)
return get_rsp(msg="The request method not find, method == %s" % request.method, code=-1)

if __name__ == '__main__':
app.run(host='0.0.0.0', port=8888, debug=True)

package main

import (
"context"
"crypto/md5"
"fmt"
"log"
"net/http"
"os"
"os/signal"
"strings"
"syscall"
)

func main() {
mux := http.NewServeMux()
mux.Handle("/access_log/post", &logHandler{})

server := &http.Server{
Addr: ":5000",
Handler: mux,
}

// Create a system signal receiver.
done := make(chan os.Signal)
signal.Notify(done, os.Interrupt, syscall.SIGINT, syscall.SIGTERM)
go func() {
<-done

if err := server.Shutdown(context.Background()); err != nil {
log.Fatal("Shutdown server:", err)
}
}()

err := server.ListenAndServe()
if err != nil {
if err == http.ErrServerClosed {
log.Print("Server closed under request")
} else {
log.Fatal("Server closed unexpected")
}
}
}

type logHandler struct{}

func (*logHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
if r.Method == "POST" {
query := r.URL.Query()
authKey := query.Get("auth_key")
accessKey := query.Get("access_key") // access_key is the SecretId you provide.
authKeys := strings.Split(authKey, "-")
if len(authKeys) == 3 {
currentTimeTs := authKeys[0]
// Judge whether the timestamp is within the validity period.
RandNum := authKeys[1]
md5Hash := authKeys[2]
secretKey := getSecretKey(accessKey)
authStr := fmt.Sprintf("%s-%s-%s-%s", "/access_log/post", currentTimeTs, RandNum, secretKey)
data := []byte(authStr)
has := md5.Sum(data)
authMd5 := fmt.Sprintf("%x", has) // Converted to a string for comparison.
if authMd5 == md5Hash {
// TODO authentication succeeded.
if r.Header.Get("Content-Encoding") == "gzip" {
// Data decompression
}
// Data handling
}
} else {
// Exception handling
}
}
}

// Get SecretKey.
func getSecretKey(accessKey string) string {
if accessKey != "" {
// Get Secret_Key using Access_key (SecretId).
return "secret_key"
}
return ""
}