Obtaining Real TCP Client IPs via TOA
You can use this document to learn how to get the TCP client IP via TOA when using L4 proxy.
Use Cases
Using L4 acceleration for data packets will have the source IP and port modified, so the origin does not get the original information. To enable the origin to get the real client IP and port, you can pass the information using TOA when creating an acceleration channel. In this way, the real client IP and port are passed into the TCP option field. Meanwhile, you need to install TOA on the origin to get that information.
Note:
The L4 proxy is only available with the Enterprise Edition package.
Directions
Step 1: Pass the client IP via TOA
To get the TCP client IP via TOA, set Pass client IP to TOA in L4 proxy forwarding rules in the console. For details on how to modify rules, see: Modifying an L4 Proxy Instance.
Step 2: Load TOA on backend server
You can load the TOA module using either of the following methods:
Method 1 (recommended): Based on the Linux version the origin uses, download the complied toa.ko file and load it directly.
Method 2: If you cannot find the appropriate Linux version, download the TOA source code file and compile and load it yourself. The source code only supports the x86_64 version. If you need support for the arm64 version, please contact us.
Note:
Due to the differences in installation environments, if you encounter issues during the loading process using Method 1, please try Method 2 and install the compilation environment manually.
1. Download and decompress the TOA package corresponding to the version of Linux OS on Tencent Cloud.
centos
TencentOS
debian
suse linux
ubuntu
2. After decompression is complete, run the
cd
command to access the decompressed folder. Then load the module as follows:/bin/bash -c "$(curl -fsSL https://edgeone-document-file-1258344699.cos.ap-guangzhou.myqcloud.com/TOA/install_toa.sh)"
When it is loaded successfully, you will see the following information:
# Decompress the tar package.tar -zxvf CentOS-7.2-x86_64.tar.gz# Enter the directory of the decompressed package.cd CentOS-7.2-x86_64# Load the TOA module.insmod toa.ko# Copy the TOA module to the kernel module directory.cp toa.ko /lib/modules/`uname -r`/kernel/net/netfilter/ipvs/toa.ko# Configure the TOA module to load automatically at system startup.echo "insmod /lib/modules/`uname -r`/kernel/net/netfilter/ipvs/toa.ko" >> /etc/rc.local
Run the following command to check whether the loading is successful:
lsmod | grep toa
If you see "TOA" in the message, the module is loaded successfully:
1. Install the compilation environment.
1.1 Make sure kernel-devel and kernel-headers are installed and consistent with the kernel version.
1.2 Make sure the gcc and make dependencies are installed.
1.3 If these environmental dependencies are not installed, run the installation command:
yum install -y gccyum install -y makeyum install -y kernel-headers kernel-devel
apt-get install -y gccapt-get install -y makeapt-get install -y linux-headers-$(uname -r)
2. After the compilation environment is installed, download, compile and load the source code.
/bin/bash -c "$(curl -fsSL https://edgeone-document-file-1258344699.cos.ap-guangzhou.myqcloud.com/TOA/compile_install_toa.sh)"
# Create a compilation directory and enter it.mkdir toa_compile && cd toa_compile# Download the source code (tar.gz)curl -o toa.tar.gz https://edgeone-document-file-1258344699.cos.ap-guangzhou.myqcloud.com/TOA/toa.tar.gz# Decompress the tar packagetar -zxvf toa.tar.gz# Compile the toa.ko file. After the compilation is successful, the file will be generated in the current directory.make# Load the TOA module.insmod toa.ko# Copy the TOA module to the kernel module directory.cp toa.ko /lib/modules/`uname -r`/kernel/net/netfilter/ipvs/toa.ko# Configure the TOA module to load automatically at system startupecho "insmod /lib/modules/`uname -r`/kernel/net/netfilter/ipvs/toa.ko" >> /etc/rc.local
3. Run the following command to check whether the loading is successful:
lsmod | grep toa
If you see "TOA" in the message, the module is loaded successfully:
Step 3: Verify the configuration
You can verify the configuration by building a TCP server to receive client requests from another server. See the sample:
1. On the current server, create an HTTP server in Python to act as a TCP server:
# Use python2python2 -m SimpleHTTPServer 10000# Use python3python3 -m http.server 10000
2. Make another server work as a client to send requests:
# Use curl to initiate an HTTP request, where the hostname and forwarding port of the L4 proxy is used.curl -i "http://a8b7f59fc8d7e6c9.example.com.edgeonedy1.com:10000/"
3. If TOA is loaded, the real client address can be seen on the server:
You can also get either the IPv4 or IPv6 address of the client by following the steps above.
For origin IPv4 addresses, get the client IPv4 address.
For origin IPv6 addresses, get the client IPv6 address.
If you need to get both IPv4 and IPv6 addresses, modify the origin's business code while loading the TOA module as instructed here.
Getting Both IPv4 and IPv6 Addresses
Note:
This section provide guidance on how to get both IPv4 and IPv6 addresses by modifying the business code of the origin.
The origin can listen on requests in either of the following methods:
1. Use the structure
struct sockaddr_in
to listen on IPv4 addresses.2. Use the structure
struct sockaddr_in6
to listen on IPv6 addresses.Sample code
#include <sys/socket.h>#include <stdio.h>#include <unistd.h>#include <netinet/in.h>#include <memory.h>#include <arpa/inet.h>int main(int argc, char** argv) {int l_sockfd;// The server address is an IPv4 address.struct sockaddr_in serveraddr;// In this case, the client address must adopt the IPv6 structure.struct sockaddr_in6 clientAddr;int server_port = 10000;memset(&serveraddr, 0, sizeof(serveraddr));// Create a socket.l_sockfd = socket(AF_INET, SOCK_STREAM, 0);if (l_sockfd == -1){printf("Failed to create socket.\n");return -1;}// Initialize the server.memset(&serveraddr, 0, sizeof(struct sockaddr_in));serveraddr.sin_family = AF_INET;serveraddr.sin_port = htons(server_port);serveraddr.sin_addr.s_addr = htonl(INADDR_ANY);int isReuse = 1;setsockopt(l_sockfd, SOL_SOCKET,SO_REUSEADDR,(const char*)&isReuse,sizeof(isReuse));// Associate the socket and server address.int nRet = bind(l_sockfd,(struct sockaddr*)&serveraddr, sizeof(serveraddr));if(-1 == nRet){printf("bind error\n");return -1;}// Listen on the socket.listen(l_sockfd, 5);int clientAddrLen = sizeof(clientAddr);memset(&clientAddr, 0, sizeof(clientAddr));// Accept connections from the client.int linkFd = accept(l_sockfd, (struct sockaddr*)&clientAddr, &clientAddrLen);if(-1 == linkFd){printf("accept error\n");return -1;}// Modifications to make: Decide whether the client is an IPv4 or IPv6 address based on sin6_family.// AF_INET indicates that the client adopts IPv4. In this case, convert the client address pointer to struct sockaddr_in* and get the IPv4 address.// AF_INET6 indicates that the client adopts IPv6. In this case, use struct sockaddr_in6* to get the IPv6 address.if (clientAddr.sin6_family == AF_INET) {printf("AF_INET accept getpeername %s : %d successful\n",inet_ntoa(((struct sockaddr_in*)&clientAddr)->sin_addr),ntohs(((struct sockaddr_in*)&clientAddr)->sin_port));}else if (clientAddr.sin6_family == AF_INET6){char addr_p[128] = {0};inet_ntop(AF_INET6, (void *)&((struct sockaddr_in6*)&clientAddr)->sin6_addr, addr_p, (socklen_t )sizeof(addr_p));printf("AF_INET6 accept getpeername %s : %d successful\n",addr_p,ntohs(((struct sockaddr_in6*)&clientAddr)->sin6_port));}else{printf("unknow sin_family:%d \n", clientAddr.sin6_family);}close(l_sockfd);return 0;}
import java.io.IOException;import java.io.InputStream;import java.io.OutputStream;import java.net.InetAddress;import java.net.InetSocketAddress;import java.net.ServerSocket;import java.net.Socket;import java.net.SocketAddress;public class ServerDemo {/** If using the IPv4 address structure to build the service, use IPV4_HOST */public static final String IPV4_HOST = "0.0.0.0";/** If using the IPv6 address structure to build the service, use IPV6_HOST */public static final String IPV6_HOST = "::";public static void main(String[] args) {int serverPort = 10000;try (ServerSocket serverSocket = new ServerSocket()) {// Setting address reuseserverSocket.setReuseAddress(true);// Bound server address and port, using IPv4 hereserverSocket.bind(new InetSocketAddress(InetAddress.getByName(IPV4_HOST), serverPort));System.out.println("Server is listening on port " + serverPort);while (true) {// Accepting Client connectionsSocket clientSocket = serverSocket.accept();System.out.println("New client connected: " + clientSocket.getRemoteSocketAddress());// Processing Client requestshandleClientRequest(clientSocket);}} catch (IOException e) {System.err.println("Failed to create server socket: " + e.getMessage());}}/*** Processing Function, site business implement, here is just an example* The purpose of this Function is to Return the Client's input verbatim to the Client*/private static void handleClientRequest(Socket clientSocket) {try (InputStream inputStream = clientSocket.getInputStream();OutputStream outputStream = clientSocket.getOutputStream()) {// Reading the Data received from the Clientbyte[] buffer = new byte[1024];int bytesRead;while ((bytesRead = inputStream.read(buffer)) != -1) {// Reply the received Data to the Client as it isoutputStream.write(buffer, 0, bytesRead);}} catch (IOException e) {// When the Client disconnectsSystem.err.println("Failed to handle client request: " + e.getMessage());} finally {try {clientSocket.close();} catch (IOException e) {System.err.println("Failed to close client socket: " + e.getMessage());}}}}
#include <sys/socket.h>#include <stdio.h>#include <unistd.h>#include <netinet/in.h>#include <memory.h>#include <arpa/inet.h>int main(int argc, char **argv){int l_sockfd;// The server address is an IPv6 address.struct sockaddr_in6 serveraddr;// The client address is an IPv6 address.struct sockaddr_in6 clientAddr;int server_port = 10000;memset(&serveraddr, 0, sizeof(serveraddr));// Create a socket.l_sockfd = socket(AF_INET6, SOCK_STREAM, 0);if (l_sockfd == -1){printf("Failed to create socket.\n");return -1;}// Set the server address.memset(&serveraddr, 0, sizeof(struct sockaddr_in6));serveraddr.sin6_family = AF_INET6;serveraddr.sin6_port = htons(server_port);serveraddr.sin6_addr = in6addr_any;int isReuse = 1;setsockopt(l_sockfd, SOL_SOCKET,SO_REUSEADDR,(const char*)&isReuse,sizeof(isReuse));// Associate the socket and server address.int nRet = bind(l_sockfd,(struct sockaddr*)&serveraddr, sizeof(serveraddr));if(-1 == nRet){printf("bind error\n");return -1;}// Listen on the socket.listen(l_sockfd, 5);int clientAddrLen = sizeof(clientAddr);memset(&clientAddr, 0, sizeof(clientAddr));// Accept connection requests from the client.int linkFd = accept(l_sockfd, (struct sockaddr*)&clientAddr, &clientAddrLen);if(-1 == linkFd){printf("accept error\n");return -1;}// The client addresses received here are all stored in the IPv6 structure.// The IPv4 addresses are mapped to IPv6 addresses, for example, "::ffff:119.29.1.1".char addr_p[128] = {0};inet_ntop(AF_INET6, (void *)&clientAddr.sin6_addr, addr_p, (socklen_t )sizeof(addr_p));printf("accept %s : %d successful\n", addr_p, ntohs(clientAddr.sin6_port));// Modifications to make: Use the macro definition IN6_IS_ADDR_V4MAPPED to decide whether the client IP is an IPv4-mapped IPv6 address.if(IN6_IS_ADDR_V4MAPPED(&clientAddr.sin6_addr)) {struct sockaddr_in real_v4_sin;memset (&real_v4_sin, 0, sizeof (struct sockaddr_in));real_v4_sin.sin_family = AF_INET;real_v4_sin.sin_port = clientAddr.sin6_port;// The last four bytes represent the IPv4 address of the client.memcpy (&real_v4_sin.sin_addr, ((char *)&clientAddr.sin6_addr) + 12, 4);printf("connect %s successful\n", inet_ntoa(real_v4_sin.sin_addr));}close(l_sockfd);return 0;}
import java.io.IOException;import java.io.InputStream;import java.io.OutputStream;import java.net.InetAddress;import java.net.InetSocketAddress;import java.net.ServerSocket;import java.net.Socket;import java.net.SocketAddress;public class ServerDemo {/** If using the IPv4 address structure to build the service, use IPV4_HOST */public static final String IPV4_HOST = "0.0.0.0";/** If using the IPv6 address structure to build the service, use IPV6_HOST */public static final String IPV6_HOST = "::";public static void main(String[] args) {int serverPort = 10000;try (ServerSocket serverSocket = new ServerSocket()) {// Setting address reuseserverSocket.setReuseAddress(true);// Bound server address and port, using IPv4 hereserverSocket.bind(new InetSocketAddress(InetAddress.getByName(IPV6_HOST), serverPort));System.out.println("Server is listening on port " + serverPort);while (true) {// Accepting Client connectionsSocket clientSocket = serverSocket.accept();System.out.println("New client connected: " + clientSocket.getRemoteSocketAddress());// Processing Client requestshandleClientRequest(clientSocket);}} catch (IOException e) {System.err.println("Failed to create server socket: " + e.getMessage());}}/*** Processing Function, site business implement, here is just an example* The purpose of this Function is to Return the Client's input verbatim to the Client*/private static void handleClientRequest(Socket clientSocket) {try (InputStream inputStream = clientSocket.getInputStream();OutputStream outputStream = clientSocket.getOutputStream()) {// Reading the Data received from the Clientbyte[] buffer = new byte[1024];int bytesRead;while ((bytesRead = inputStream.read(buffer)) != -1) {// Reply the received Data to the Client as it isoutputStream.write(buffer, 0, bytesRead);}} catch (IOException e) {// When the Client disconnectsSystem.err.println("Failed to handle client request: " + e.getMessage());} finally {try {clientSocket.close();} catch (IOException e) {System.err.println("Failed to close client socket: " + e.getMessage());}}}}
console output result
Server is listening on port 10000New client connected: /127.0.0.1:50680New client connected: /0:0:0:0:0:0:0:1:51124New client connected: /127.0.0.1:51136
References
Monitoring TOA Running Status
To ensure execution stability, this kernel module allows you to monitor status. After inserting the
toa.ko
kernel module, you can monitor the TOA working status in either of the following ways.cat /proc/net/toa_stats
This figure shows the TOA running status:
The monitoring metrics are described as follows:
Metric | Description |
syn_recv_sock_toa | Receives connections with TOA information. |
syn_recv_sock_no_toa | Receives connections without TOA information. |
getname_toa_ok | This count increases when you call getsockopt and obtain the source IP address successfully or when you call accept to receive client requests. |
getname_toa_mismatch | This count increases when you call getsockopt and obtain a source IP address that does not match the required type. For example, if a client connection contains a source IPv4 address whereas you obtain an IPv6 address, the count will increase. |
getname_toa_empty | This count increases when the getsockopt function is called in a client file descriptor that does not contain TOA. |
ip6_address_alloc audio/video proxy | Allocates space to store the information when TOA obtains the source IP address and source port saved in the TCP data packet. |
ip6_address_free audio/video proxy | When the connection is released, TOA will release the memory previously used to save the source IP and source port. If all connections are closed, the total count of ip6_address_alloc for each CPU should be equal to the count of this metric. |