Table of Contents

I wrote a library to manage TCP connections as part of my Golearn project. Source code is available from my Gitlab repository.

The TCPServer Library #

The TCPServer library enables communications between an application (end user code) and one or more TCP clients. This is done by providing data and control (Go) channels for the application to interact seamlessly with the clients. This is illustrated in the following figure:

graph LR; A[Application] ---|Control| B(TCPServer Library) B ---|Data| A B -.->|Socket| D[Client 1] B -.->|Socket| E[Client 2]

The TCPServer library abstracts the client connections into a set of two control and two data channels. The control channels are used to notify the application about new TCP client connections and to allow the application to close client connections when desired. The data channels are used to transfer messages in and out between the application and the clients.

A sample application using this library is the multi-echo server described in my post A Multi-Echo Chat Server Written In Go.

The App’s Code Structure #

Let’s discuss how to organize the application code. First, the application must create and initialize the TCPServer providing the host interface and port to listen on. It then must start the server as a Go routine, as illustrated below:

server := new(tcpserver.TCPServer)
server.Address = "localhost:9090"
channels, err := server.Init()
if err != nil {
    log.Fatal(err)
}

// get ready to receive control and data messages

go server.Start(ctx)

The server’s Init() function returns a pointer to a channels structure that includes the two control and two data channels mentioned above; more on these later. What follows is the core of the application, a loop listening and writing on the control and data channels, as illustrated in the following pseudo code:

for {
    select {
    case ctrl := <- channels.CtrlIn:
        // the application received a control message from the TCP server
        <do something>
        // send a control message to the TCP server
        channels.CtrlOut <- ctrlOut

    case data := <- channels.DataIn:
        // the application received data from one of the TCP clients
        <do something>
        // send data to a client
        channels.DataOut <- someData
    }
}

The control messages are always associated with a client. They are of type ClientCtrl, a structure that includes the following fields:

ID: The client’s IP address and port. This is a unique string created automatically by the TCPServer.
Enable: A boolean to indicate the status of the connection, open or closed.

The TCPServer notifies the application about new clients by sending a control message with Enable set to true on the CtrlIn channel. It notifies also when a connection closes by sending a message with Enable set to false on the same channel.

The application can request that a client connection be closed by sending a control message with Enable set to false on the CtrlOut channel.

Data messages are also always associated with a client. They are of type ClientData, a structure that includes the following fields:

ID: The client’s IP address and port.
Data: An array of bytes.

The TCPServer sends client’s incoming data to the application by writing a data message to the DataIn channel. The application sends data to clients by writing a data message to the DataOut channel.

The TCPServer Library #

The following figure illustrates the overall structure of the library. Logical connections are illustrated on the left side and data structures are depicted on the right.

The visible element of the library, visible by the application code, is the TCPServer. Fundamentally, the TCPServer implements a client multiplexer. On the north-bound side the TCPServer manages the control and data channels with the application. On the south-bound side it manages control and data channels with the connection managers in charge of existing connections. Finally, the TCPServer handles new connections by communicating with an accept-connections Go routine via a dedicated channel.

The overall workflow of the TCPServer starts by spawning a dedicated new connections Go routine which has as its sole purpose to accept new connections. This routine notifies the TCPServer about new connections by sending them over the newConn channel.

Once the TCPServer receives the details of a new connection, it notifies the application by sending a control message on the CtrlIn channel. It then spawns a dedicated connection manager to manage the new connection; there is one connection manager for each TCP client.

The TCPServer Interface #

The TCPServer interface defines two functions, Init() and Start(). The Init() function returns an error if it fails to initialize the server using the specified IP address and port. Otherwise, it returns a pointer to an AppChannels structure containing the four application channels described above.

The Start() function starts the TCP server. It takes a Go context.Context (with Cancel) variable as a parameter. The TCPServer uses this context in two ways:

It looks for a context value with a key of type ServerCtx set to "wg". This context value must be a Go sync.WaitGroup variable that the TCPServer uses to mark the completion of the job.
It listens on the cancellation channel of the context to initiate a managed shutdown.

The Server’s Code Structure #

The basic structure of the TCPServer is the loop illustrated below:

func (serv *TCPServer) Start(ctx context.Context) {
    key := ServerCtx("wg")
    wg := ctx.Value(key).(*sync.WaitGroup)
    defer wg.Done()

    for {
        select {
        case conn := <-newConn:
            // new connection notification
            // notify the application
            // create connection manager

        case ctrl := <- CtrlOut:
            // the server received a client control message from the application
            // notify the corresponding connection manager if the connection has to end

        case data := <- DataOut
            // the server received data from the application destined to some client
            // send data to the corresponding connection manager

        case ctrl := <-connMgrsCh:
            // the server received a control message from a connection manager
            // the client connection likeley closed, tell the application about it

        case <- ctx.Done():
            // shutdown the server
        }
    }
}

The Connection Managers #

The TCPServer creates a connection manager for each new TCP connection it receives via the newConn channel. Each connection manager is scheduled as a Go routine dedicated entirely to the control and data I/O of the associated connection.

On the north-bound side, each connection manager has dedicated control and data channels for incoming messages from the TCPServer. The latter uses the CtrlIn channel to disable (close) the connection, usually because the application wants the client connection to close, and the DataOut channel to send the application outgoing messages to the client.

Still on the north-bound side, the connection managers use the single shared CtrlOut channel to notify the TCPServer when the corresponding client closes the connection. They also use the single shared DataIn channel to push incoming data messages from the client directly to the application.

The only difference in the payloads between the dedicated and the shared channels is that the latter carry the unique ID of the client so that the TCPServer (for control messages) and the application (for data messages) can identify them.

The Connection Manager Interface #

The connection manager interface defines the single start(context.Context) function, which must be invoked as a Go routine.

When the TCPServer receives a new connection via the newConn channel it:

Notifies the application about the incoming connection by providing it with the connection ID.
Creates a new connMgr object and initializes it with its ID, TCP connection pointer, and a pointer to a connMgrChannels structure describing the north-bound connection channels to use.
Calls the connection manager’s Start(ctx) function as a Go routine.

The connection manager uses the context variable in the Start() function in the same manner that the TCPServer uses its own context. It expects to find a sync.WaitGroup variable as a context value, which it uses to signal the TCPServer when the connection closes.

The Connection Manager Code Structure #

The basic structure of a connection manager is the loop illustrated below:

for {
    select {
    case ctrl := <-ctrlIn:
        // the CM received a control message from the TCPServer,
        // likely requesting the connection to close.

    case msg := <-errCh:
        // the CM received an error message from the connection IO

    case data := <-dataIn:
        // the CM received a data message from the client that has to be
        // sent to the application

    case data := <-connDataOut:
        // the CM received a data message from the application that has to be
        // sent to the client
        dataOut <- data

    case <-ctx.Done():
        // shutdown the CM
    }
}

The errCh, dataIn and dataOut channels in the loop are the south-bound channels that connect the connection manager to the connection IO driver described below.

The Connection IO Driver #

Each connection manger creates and initializes a connIO driver which is in charge of the low-level read and write socket operations. The initialization provides the driver with the the TCP connection details and the list of channels to use. The connIO driver uses these channels as follows:

errCh: Used to notify any errors. The payload in this channel includes a boolean flag to indicate whether it is a read or write error.
dataIn: Used to push data received from the client up to the connection manager.
dataOut: Used to write data received from the connection manager to the client.

The connIO driver interface defines read(context.Context) and write(context.Context) functions which the connection manager invokes as Go routines. These functions use the context’s cancel channel to determine when to stop processing data.

The basic structure of the connIO reader is the loop illustrated below:

for {
    select {
    case <-ctx.Done():
        // stop
    default:
        // read data
        // send it to the connection manager
        dataIn <- data
    }
}

Note that the socket read operation is blocking. Therefore there is no guarantee that the reading operation terminates as soon as the connection manager invokes the connIO driver’s context cancel operation. In most cases, the reading operation terminates when the Go run-time exits.

The basic structure of the connIO writer is the loop illustrated below:

for {
    select {
    case <-ctx.Done():
        // stop
    case data := <-dataOut:
        // write data to client
    }
}

Summary #

In this post I discuss my work building a TCP library that abstracts the complexities of low-level socket communications into a set of control and data channels that simplify the application code substantially. The library makes extensive use of Go routines and uses Go channels to manage internal communications and the interactions with the application code.

There are no restrictions on the type of applications that can use the library. It is all left to the imagination on the application developer.