Overview
Menu

Obtaining Real TCP Client IPs via TOA

You can use this document to learn how to get the TCP client IP via TOA when using L4 proxy.

Use Cases

Using L4 acceleration for data packets will have the source IP and port modified, so the origin does not get the original information. To enable the origin to get the real client IP and port, you can pass the information using TOA when creating an acceleration channel. In this way, the real client IP and port are passed into the TCP option field. Meanwhile, you need to install TOA on the origin to get that information.

Directions

Step 1: Pass the client IP via TOA

To get the TCP client IP via TOA, set Pass client IP to TOA in L4 proxy forwarding rules in the console. For details on how to modify rules, see: Modifying an L4 Proxy Instance.




Step 2: Load TOA on backend server

You can load the TOA module using either of the following methods:
Method 1 (recommended): Based on the Linux version the origin uses, download the complied toa.ko file and load it directly.
Method 2: If you cannot find the appropriate Linux version, download the TOA source code file and compile and load it yourself.
Note:
Due to the differences in installation environments, if you encounter issues during the loading process using Method 1, please try Method 2 and install the compilation environment manually.
Method 1: Download and load the compiled TOA module
Method 2: Compile and load the TOA module
1. Download and decompress the TOA package corresponding to the version of Linux OS on Tencent Cloud.
centos
debian
suse linux
ubuntu
2. After decompression is complete, run the cd command to access the decompressed folder. Then load the module as follows:
Load TOA with a script
Manually load TOA
/bin/bash -c "$(curl -fsSL https://edgeone-document-file-1258344699.cos.ap-guangzhou.myqcloud.com/TOA/install_toa.sh)"
When it is loaded successfully, you will see the following information:


# Decompress the tar package.
tar -zxvf CentOS-7.2-x86_64.tar.gz
# Enter the directory of the decompressed package.
cd CentOS-7.2-x86_64
# Load the TOA module.
insmod toa.ko
# Copy the TOA module to the kernel module directory.
cp toa.ko /lib/modules/`uname -r`/kernel/net/netfilter/ipvs/toa.ko
# Configure the TOA module to load automatically at system startup.
echo "insmod /lib/modules/`uname -r`/kernel/net/netfilter/ipvs/toa.ko" >> /etc/rc.local
Run the following command to check whether the loading is successful:
lsmod | grep toa
If you see "TOA" in the message, the module is loaded successfully:




1. Install the compilation environment.
1.1 Make sure kernel-devel and kernel-headers are installed and consistent with the kernel version.
1.2 Make sure the gcc and make dependencies are installed.
1.3 If these environmental dependencies are not installed, run the installation command:
CentOS
Ubuntu/Debian
yum install -y gcc
yum install -y make
yum install -y kernel-headers kernel-devel
apt-get install -y gcc
apt-get install -y make
apt-get install -y linux-headers-$(uname -r)

2. After the compilation environment is installed, download, compile and load the source code.
Compile and load TOA with a script
Manually compile and load TOA
/bin/bash -c "$(curl -fsSL https://edgeone-document-file-1258344699.cos.ap-guangzhou.myqcloud.com/TOA/compile_install_toa.sh)"
# Create a compilation directory and enter it.
mkdir toa_compile && cd toa_compile
# Download the source code (tar.gz)
curl -o toa.tar.gz https://edgeone-document-file-1258344699.cos.ap-guangzhou.myqcloud.com/TOA/toa.tar.gz
# Decompress the tar package
tar -zxvf toa.tar.gz
# Compile the toa.ko file. After the compilation is successful, the file will be generated in the current directory.
make
# Load the TOA module.
insmod toa.ko
# Copy the TOA module to the kernel module directory.
cp toa.ko /lib/modules/`uname -r`/kernel/net/netfilter/ipvs/toa.ko
# Configure the TOA module to load automatically at system startup
echo "insmod /lib/modules/`uname -r`/kernel/net/netfilter/ipvs/toa.ko" >> /etc/rc.local
3. Run the following command to check whether the loading is successful:
lsmod | grep toa
If you see "TOA" in the message, the module is loaded successfully:


Step 3: Verify the configuration

You can verify the configuration by building a TCP server to receive client requests from another server. See the sample:
1. On the current server, create an HTTP server in Python to act as a TCP server:
# Use python2
python2 -m SimpleHTTPServer 10000

# Use python3
python3 -m http.server 10000
2. Make another server work as a client to send requests:
# Use curl to initiate an HTTP request, where the hostname and forwarding port of the L4 proxy is used.
curl -i "http://a8b7f59fc8d7e6c9.example.com.edgeonedy1.com:10000/"
3. If TOA is loaded, the real client address can be seen on the server:


You can also get either the IPv4 or IPv6 address of the client by following the steps above.
For origin IPv4 addresses, get the client IPv4 address.
For origin IPv6 addresses, get the client IPv6 address.
If you need to get both IPv4 and IPv6 addresses, modify the origin's business code while loading the TOA module as instructed here.

Getting Both IPv4 and IPv6 Addresses

Note:
This section provide guidance on how to get both IPv4 and IPv6 addresses by modifying the business code of the origin.
The origin can listen on requests in either of the following methods:
1. Use the structure struct sockaddr_in to listen on IPv4 addresses.
2. Use the structure struct sockaddr_in6 to listen on IPv6 addresses.

Sample code

Listen on IPv4 addresses
Listen on IPv6 addresses

C
Java
#include <sys/socket.h>
#include <stdio.h>
#include <unistd.h>
#include <netinet/in.h>
#include <memory.h>
#include <arpa/inet.h>

int main(int argc, char** argv) {
int l_sockfd;
// The server address is an IPv4 address.
struct sockaddr_in serveraddr;
// In this case, the client address must adopt the IPv6 structure.
struct sockaddr_in6 clientAddr;
int server_port = 10000;

memset(&serveraddr, 0, sizeof(serveraddr));
// Create a socket.
l_sockfd = socket(AF_INET, SOCK_STREAM, 0);
if (l_sockfd == -1){
printf("Failed to create socket.\n");
return -1;
}
// Initialize the server.
memset(&serveraddr, 0, sizeof(struct sockaddr_in));
serveraddr.sin_family = AF_INET;
serveraddr.sin_port = htons(server_port);
serveraddr.sin_addr.s_addr = htonl(INADDR_ANY);

int isReuse = 1;
setsockopt(l_sockfd, SOL_SOCKET,SO_REUSEADDR,(const char*)&isReuse,sizeof(isReuse));
// Associate the socket and server address.
int nRet = bind(l_sockfd,(struct sockaddr*)&serveraddr, sizeof(serveraddr));
if(-1 == nRet)
{
printf("bind error\n");
return -1;
}
// Listen on the socket.
listen(l_sockfd, 5);

int clientAddrLen = sizeof(clientAddr);
memset(&clientAddr, 0, sizeof(clientAddr));
// Accept connections from the client.
int linkFd = accept(l_sockfd, (struct sockaddr*)&clientAddr, &clientAddrLen);
if(-1 == linkFd)
{
printf("accept error\n");
return -1;
}
// Modifications to make: Decide whether the client is an IPv4 or IPv6 address based on sin6_family.
// AF_INET indicates that the client adopts IPv4. In this case, convert the client address pointer to struct sockaddr_in* and get the IPv4 address.
// AF_INET6 indicates that the client adopts IPv6. In this case, use struct sockaddr_in6* to get the IPv6 address.
if (clientAddr.sin6_family == AF_INET) {
printf("AF_INET accept getpeername %s : %d successful\n",
inet_ntoa(((struct sockaddr_in*)&clientAddr)->sin_addr),
ntohs(((struct sockaddr_in*)&clientAddr)->sin_port));
}else if (clientAddr.sin6_family == AF_INET6){
char addr_p[128] = {0};
inet_ntop(AF_INET6, (void *)&((struct sockaddr_in6*)&clientAddr)->sin6_addr, addr_p, (socklen_t )sizeof(addr_p));
printf("AF_INET6 accept getpeername %s : %d successful\n",
addr_p,
ntohs(((struct sockaddr_in6*)&clientAddr)->sin6_port));
}else{
printf("unknow sin_family:%d \n", clientAddr.sin6_family);
}
close(l_sockfd);
return 0;
}
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.net.ServerSocket;
import java.net.Socket;
import java.net.SocketAddress;


public class ServerDemo {


/** If using the IPv4 address structure to build the service, use IPV4_HOST */
public static final String IPV4_HOST = "0.0.0.0";


/** If using the IPv6 address structure to build the service, use IPV6_HOST */
public static final String IPV6_HOST = "::";


public static void main(String[] args) {
int serverPort = 10000;
try (ServerSocket serverSocket = new ServerSocket()) {
// Setting address reuse
serverSocket.setReuseAddress(true);
// Bound server address and port, using IPv4 here
serverSocket.bind(new InetSocketAddress(InetAddress.getByName(IPV4_HOST), serverPort));
System.out.println("Server is listening on port " + serverPort);


while (true) {
// Accepting Client connections
Socket clientSocket = serverSocket.accept();
System.out.println("New client connected: " + clientSocket.getRemoteSocketAddress());


// Processing Client requests
handleClientRequest(clientSocket);
}
} catch (IOException e) {
System.err.println("Failed to create server socket: " + e.getMessage());
}
}


/**
* Processing Function, site business implement, here is just an example
* The purpose of this Function is to Return the Client's input verbatim to the Client
*/
private static void handleClientRequest(Socket clientSocket) {
try (InputStream inputStream = clientSocket.getInputStream();
OutputStream outputStream = clientSocket.getOutputStream()) {


// Reading the Data received from the Client
byte[] buffer = new byte[1024];
int bytesRead;
while ((bytesRead = inputStream.read(buffer)) != -1) {
// Reply the received Data to the Client as it is
outputStream.write(buffer, 0, bytesRead);
}


} catch (IOException e) {
// When the Client disconnects
System.err.println("Failed to handle client request: " + e.getMessage());
} finally {
try {
clientSocket.close();
} catch (IOException e) {
System.err.println("Failed to close client socket: " + e.getMessage());
}
}
}
}

C
Java
#include <sys/socket.h>
#include <stdio.h>
#include <unistd.h>
#include <netinet/in.h>
#include <memory.h>
#include <arpa/inet.h>

int main(int argc, char **argv)
{
int l_sockfd;
// The server address is an IPv6 address.
struct sockaddr_in6 serveraddr;
// The client address is an IPv6 address.
struct sockaddr_in6 clientAddr;
int server_port = 10000;

memset(&serveraddr, 0, sizeof(serveraddr));
// Create a socket.
l_sockfd = socket(AF_INET6, SOCK_STREAM, 0);
if (l_sockfd == -1){
printf("Failed to create socket.\n");
return -1;
}
// Set the server address.
memset(&serveraddr, 0, sizeof(struct sockaddr_in6));
serveraddr.sin6_family = AF_INET6;
serveraddr.sin6_port = htons(server_port);
serveraddr.sin6_addr = in6addr_any;

int isReuse = 1;
setsockopt(l_sockfd, SOL_SOCKET,SO_REUSEADDR,(const char*)&isReuse,sizeof(isReuse));
// Associate the socket and server address.
int nRet = bind(l_sockfd,(struct sockaddr*)&serveraddr, sizeof(serveraddr));
if(-1 == nRet)
{
printf("bind error\n");
return -1;
}
// Listen on the socket.
listen(l_sockfd, 5);

int clientAddrLen = sizeof(clientAddr);
memset(&clientAddr, 0, sizeof(clientAddr));
// Accept connection requests from the client.
int linkFd = accept(l_sockfd, (struct sockaddr*)&clientAddr, &clientAddrLen);
if(-1 == linkFd)
{
printf("accept error\n");
return -1;
}
// The client addresses received here are all stored in the IPv6 structure.
// The IPv4 addresses are mapped to IPv6 addresses, for example, "::ffff:119.29.1.1".
char addr_p[128] = {0};
inet_ntop(AF_INET6, (void *)&clientAddr.sin6_addr, addr_p, (socklen_t )sizeof(addr_p));
printf("accept %s : %d successful\n", addr_p, ntohs(clientAddr.sin6_port));
// Modifications to make: Use the macro definition IN6_IS_ADDR_V4MAPPED to decide whether the client IP is an IPv4-mapped IPv6 address.
if(IN6_IS_ADDR_V4MAPPED(&clientAddr.sin6_addr)) {
struct sockaddr_in real_v4_sin;
memset (&real_v4_sin, 0, sizeof (struct sockaddr_in));
real_v4_sin.sin_family = AF_INET;
real_v4_sin.sin_port = clientAddr.sin6_port;
// The last four bytes represent the IPv4 address of the client.
memcpy (&real_v4_sin.sin_addr, ((char *)&clientAddr.sin6_addr) + 12, 4);
printf("connect %s successful\n", inet_ntoa(real_v4_sin.sin_addr));
}
close(l_sockfd);
return 0;
}
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.net.ServerSocket;
import java.net.Socket;
import java.net.SocketAddress;

public class ServerDemo {

/** If using the IPv4 address structure to build the service, use IPV4_HOST */
public static final String IPV4_HOST = "0.0.0.0";

/** If using the IPv6 address structure to build the service, use IPV6_HOST */
public static final String IPV6_HOST = "::";

public static void main(String[] args) {
int serverPort = 10000;
try (ServerSocket serverSocket = new ServerSocket()) {
// Setting address reuse
serverSocket.setReuseAddress(true);
// Bound server address and port, using IPv4 here
serverSocket.bind(new InetSocketAddress(InetAddress.getByName(IPV6_HOST), serverPort));
System.out.println("Server is listening on port " + serverPort);

while (true) {
// Accepting Client connections
Socket clientSocket = serverSocket.accept();
System.out.println("New client connected: " + clientSocket.getRemoteSocketAddress());

// Processing Client requests
handleClientRequest(clientSocket);
}
} catch (IOException e) {
System.err.println("Failed to create server socket: " + e.getMessage());
}
}

/**
* Processing Function, site business implement, here is just an example
* The purpose of this Function is to Return the Client's input verbatim to the Client
*/
private static void handleClientRequest(Socket clientSocket) {
try (InputStream inputStream = clientSocket.getInputStream();
OutputStream outputStream = clientSocket.getOutputStream()) {

// Reading the Data received from the Client
byte[] buffer = new byte[1024];
int bytesRead;
while ((bytesRead = inputStream.read(buffer)) != -1) {
// Reply the received Data to the Client as it is
outputStream.write(buffer, 0, bytesRead);
}

} catch (IOException e) {
// When the Client disconnects
System.err.println("Failed to handle client request: " + e.getMessage());
} finally {
try {
clientSocket.close();
} catch (IOException e) {
System.err.println("Failed to close client socket: " + e.getMessage());
}
}
}
}

console output result

Server is listening on port 10000
New client connected: /127.0.0.1:50680
New client connected: /0:0:0:0:0:0:0:1:51124
New client connected: /127.0.0.1:51136

References

Monitoring TOA Running Status

To ensure execution stability, this kernel module allows you to monitor status. After inserting the toa.ko kernel module, you can monitor the TOA working status in either of the following ways.
cat /proc/net/toa_stats
This figure shows the TOA running status:

The monitoring metrics are described as follows:
Metric
Description
syn_recv_sock_toa
Receives connections with TOA information.
syn_recv_sock_no_toa
Receives connections without TOA information.
getname_toa_ok
This count increases when you call getsockopt and obtain the source IP address successfully or when you call accept to receive client requests.
getname_toa_mismatch
This count increases when you call getsockopt and obtain a source IP address that does not match the required type. For example, if a client connection contains a source IPv4 address whereas you obtain an IPv6 address, the count will increase.
getname_toa_empty
This count increases when the getsockopt function is called in a client file descriptor that does not contain TOA.
ip6_address_alloc audio/video proxy
Allocates space to store the information when TOA obtains the source IP address and source port saved in the TCP data packet.
ip6_address_free audio/video proxy
When the connection is released, TOA will release the memory previously used to save the source IP and source port. If all connections are closed, the total count of ip6_address_alloc for each CPU should be equal to the count of this metric.