Socket Options & Go: Multiple Listeners, One Port

Photo by Sven Brandsma on Unsplash

The net package in the Go standard library abstracts away many low-level networking details, making it easy for Go developers to create complex applications. However, the standard library can confuse developers who want to explore more advanced networking features like setting socket options on listeners.

Today’s article is going to focus on exploring advanced networking options by enabling two processes to bind the same port.

What are socket options?

A socket option is a value that can be set on a socket (UDP or TCP) to modify the default behavior. Socket options are used throughout the Go standard library to provide the network behaviors that we all expect.

A great example of this is Nagle’s algorithm. By default, with Go, every new connection has Nagle’s algorithm disabled; this is disabled because, with every new connection, Go will set the TCP_NODELAY socket option.

Beyond the defaults, however, Go also provides convenient methods to enable socket options that can improve TCP/IP connectivity, like setting SO_KEEPALIVE with the tcp.Conn.SetKeepAlive() method.

Many socket options are available, with each Operating System platform implementing them differently. In this article, we will explore the SO_REUSEADDR and SO_REUSEPORT socket options.

A Basic TCP Listener

First, we must create an example TCP server application to showcase how to modify listener behavior with socket options. The below example is simple and lacks much of the error handling a production-ready application would need.

package main

import (
 "log"
 "net"
)

func main() {
 // Start Listener
 l, err := net.Listen("tcp", "0.0.0.0:9000")
 if err != nil {
  log.Printf("Could not start TCP listener: %s", err)
  return
 }

 // Wait for new connections
 for {
  // Accept new connections
  c, err := l.Accept()
  if err != nil {
   log.Printf("Listener returned: %s", err)
   break
  }

  // Kickoff a Goroutine to handle the new connection
  go func() {
   defer c.Close()
   log.Printf("New connection created")

   // Write a hello world and close the session
   _, err := c.Write([]byte("Hello World"))
   if err != nil {
    log.Printf("Unable to write on connection: %s", err)
   }
  }()
 }
}

Before going too far, I want to explain how the above application works.

The code above first starts with creating a net.Listener using the net.Listen() method. This method is straightforward and can create TCP or Unix Socket listeners by passing the desired “network” type. In this case, we want to create a TCP service, so we will specify tcp followed by the IP (0.0.0.0) and Port (9000) to listen on.

As a side note, the IP address 0.0.0.0 is a special IP that, when used in this way, will allow our application to listen to all available IPs on the server.

After creating our listener, we have a for-loop that continuously runs the net.Listener.Accept() method. This method accepts and returns net.Conn types for every new connection to our listener.

Once a new connection is created, a goroutine is spawned to handle the new connection. The reason the connection handler is within a goroutine is to allow the process to quickly return to the net.Listener.Accept() method in case new connections are established while handling the previous connection.

In the example above, our connection handler is basic and should finish writing “Hello World” to the client quickly.

Running our Listener

With the example application written, we can now run the process and connect to it using the curl command.

$ curl -v telnet://localhost:9000
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 9000 (#0)

* Closing connection 0
Hello World%         

Everything seems to be working fine with a single instance, but what happens if we launch a second instance of our application?

$ go run main.go
2023/03/26 10:47:53 Could not start TCP listener: listen tcp 0.0.0.0:9000: bind: address already in use

As expected, our second instance fails to start the TCP listener. Why? Because, by default, only one process can use an IP and Port combination at a time. Since we already have a process listening on the 0.0.0.0:9000 address, we received an error.

With socket options, we can change this behavior.

Using Socket Options

To enable our TCP listener to bind the same IP and Port across multiple processes, we will need to use theSO_REUSEADDR and SO_REUSEPORT socket options.

Socket options behavior can vary across different operating systems. For instance, enabling the SO_REUSEADDR option on Windows systems is sufficient for two processes to bind the same IP and Port. However, on Linux, it’s best to set both SO_REUSEADDR and SO_REUSEPORT.

To enable our process to run multiple instances, we must rewrite how we initiate the listener and set these socket options.

  // Create Listener Config
  lc := net.ListenConfig{
    Control: func(network, address string, c syscall.RawConn) error {
      return c.Control(func(fd uintptr) {
        // Enable SO_REUSEADDR
        err := unix.SetsockoptInt(int(fd), unix.SOL_SOCKET, unix.SO_REUSEADDR, 1)
        if err != nil {
          log.Printf("Could not set SO_REUSEADDR socket option: %s", err)
        }

        // Enable SO_REUSEPORT
        err = unix.SetsockoptInt(int(fd), unix.SOL_SOCKET, unix.SO_REUSEPORT, 1)
        if err != nil {
          log.Printf("Could not set SO_REUSEPORT socket option: %s", err)
        }
      })
    },
  }

  // Start Listener
  l, err := lc.Listen(context.Background(), "tcp", "0.0.0.0:9000")
  if err != nil {
    log.Printf("Could not start TCP listener: %s", err)
    return
  }

While there is quite a bit of difference in how we start the listener, the rest of the application remains unchanged. A revised version of the full code will be available at the end of this article.

Using ListenConfig

In the code above, a new instance of net.ListenConfig is created with a function defined under the Control field.

// Create Listener Config
  lc := net.ListenConfig{
    Control: func(network, address string, c syscall.RawConn) error {
      return c.Control(func(fd uintptr) {
        // do stuff
      })
    },
  }

This function is executed on a listener's socket before attempting to bind the socket. The defined function executes the syscall.RawConn.Control() method on the listening socket.

Within the syscall.RawConn.Control() method, another function is passed. This function executes the steps necessary to enable socket options. Specifically, the unix.SetsockoptInt() function.

  // Create Listener Config
  lc := net.ListenConfig{
    Control: func(network, address string, c syscall.RawConn) error {
      return c.Control(func(fd uintptr) {
        // Enable SO_REUSEADDR
        err := unix.SetsockoptInt(int(fd), unix.SOL_SOCKET, unix.SO_REUSEADDR, 1)
        if err != nil {
          log.Printf("Could not set SO_REUSEADDR socket option: %s", err)
        }

        // Enable SO_REUSEPORT
        err = unix.SetsockoptInt(int(fd), unix.SOL_SOCKET, unix.SO_REUSEPORT, 1)
        if err != nil {
          log.Printf("Could not set SO_REUSEPORT socket option: %s", err)
        }
      })
    },
  }

In the code above, we can see the unix.SetsockoptInt() function being used to set unix.SO_REUSEADDR to true (1) on the file descriptor (represented as fd) on the socket. This process is repeated for unix.SO_REUSEPORT as well.

Starting the Listener

With the net.ListenConfig's Control field defined to execute unix.SetsockoptInt() we can now start the listener.

  // Start Listener
  l, err := lc.Listen(context.Background(), "tcp", "0.0.0.0:9000")
  if err != nil {
    log.Printf("Could not start TCP listener: %s", err)
    return
  }

This code has changed slightly from the original example in that rather than using the net.Listen() function, we are using the net.ListenConfig.Listen() method. This method requires a context.Context to be passed but otherwise accepts the same parameters as net.Listen().

With our socket options now set, we can spawn as many instances of our application as desired.

Full Example Code

The code snippet below contains the full working example code from above. This code can be copied, pasted, and executed.

package main

import (
 "context"
 "golang.org/x/sys/unix"
 "log"
 "net"
 "syscall"
)

func main() {
 // Create Listener Config
 lc := net.ListenConfig{
  Control: func(network, address string, c syscall.RawConn) error {
   return c.Control(func(fd uintptr) {
    // Enable SO_REUSEADDR
    err := unix.SetsockoptInt(int(fd), unix.SOL_SOCKET, unix.SO_REUSEADDR, 1)
    if err != nil {
     log.Printf("Could not set SO_REUSEADDR socket option: %s", err)
    }

    // Enable SO_REUSEPORT
    err = unix.SetsockoptInt(int(fd), unix.SOL_SOCKET, unix.SO_REUSEPORT, 1)
    if err != nil {
     log.Printf("Could not set SO_REUSEPORT socket option: %s", err)
    }
   })
  },
 }

 // Start Listener
 l, err := lc.Listen(context.Background(), "tcp", "0.0.0.0:9000")
 if err != nil {
  log.Printf("Could not start TCP listener: %s", err)
  return
 }

 // Wait for new connections
 for {
  // Accept new connections
  c, err := l.Accept()
  if err != nil {
   log.Printf("Listener returned: %s", err)
   break
  }

  // Kickoff a Goroutine to handle the new connection
  go func() {
   defer c.Close()
   log.Printf("New connection created")

   // Write a hello world and close the session
   _, err := c.Write([]byte("Hello World"))
   if err != nil {
    log.Printf("Unable to write on connection: %s", err)
   }
  }()
 }
}

Summary

In this post, we explored how to use net.ListenConfig to set socket options and enable SO_REUSEADDR and SO_REUSEPORT on a listener’s socket. This allows multiple processes to listen on the same IP and Port, which can be useful for load distribution or handling application failover and upgrades.

With these options, some Operating Systems may distribute new connections across all listeners. However, some may require additional socket options, such as FreeBSD, which requires setting the SO_REUSEPORT_LB option.

Socket Options on a TCP Connection

In addition to the method shown for Listeners, the Go standard library allows users access to the underlying syscall.RawConn object of TCP connections using the net.TCPConn.SyscallConn() method. Which would allow us to set socket options on a TCP Connection. But that’s an article for another day.