F# async on the server side

Last time I talked about cool ways to use F# asyncs to help write client GUIs.  Today I’ll talk about the killer app for F# asyncs, namely, non-blocking I/O on the server side.  Once again, this is material I lifted from a recent talk that Matthew Podwysocki and I gave at TechReady.  (So if today’s blog entry sounds like I’m presenting a powerpoint deck and a demo to an audience in a room, that’s why!)

A server example: stock-quote server

Most typical server applications have a number of things in common:

• Many clients
• Streams of data to/from clients
• I/O to carry out the guts/machinery of the service

There are various kinds of file & application servers: web servers, file servers, chat servers… As an example for today, I’ll show a mocked-up stock-quote server:

• Clients connect with info about stocker ticker they’re interested in (MSFT, GOOG, whatever), and the server will send back a continual stream of stock price updates, maybe about 1 per second. 
• We want to be able to handle thousands of concurrent clients.

This small sample application demonstrates some essential scaling features found in many server scenarios.

Server commonalities

At a very high level, servers often have this basic pseudocode structure:

• Server code often has the form

while (true) {
    client = socket.AcceptClient()
    BeginServicingClient(client)
}

• Where BeginServicingClient has the form

try {
while (true) {
     SendOrReceiveData()  // local & remote I/O
}
} catch (Exception e) { Handle(e) }

There’s a main loop that listens for new client connections, and each connection spawns a loop that is dedicated to servicing that client, and typically involves I/O (talking to the file system, a database, a web server, whatever).

• In the case of our stock-quote server, we’ll have a main loop that accepts new clients, and then for each client, we will send updated price data every second until the connection ends.
• We need to be able to handle multiple clients concurrently; the simplest (but naïve) approach is to use threads – e.g. BeginServicing() creates a new thread, one thread per client.  This is very easy to code up and lets us continue to use easy ‘sync’ APIs (e.g. “Stream.Write”) rather than difficult ‘async’ APIs (BeginWrite/EndWrite).
• I’ve coded an example of this strategy in C#.  Rather than using any real stock quote data, the server just sends dummy packets back to its clients, but otherwise this is similar to what you might find in practice.  Let’s have a look at the C# code.

(The entirety of the code can be found at the end of this blog entry.)

A synchronous solution in C#, using threads

I’ve coded the server as a short C# program, only about 100 lines of code.  I’ll just call out a couple noteworthy bits:

static void SyncMain(string[] args)
{
...
        while (true)
        {
            var client = socket.AcceptTcpClient();
...
            var t1 = new Thread(new ThreadStart(() =>
...
                        ServiceClient(client);

In the main loop we accept new clients (AcceptTcpClient), and for each one we start a new thread (ThreadStart) that calls ServiceClient(), which is this little function that is a lot like the pseudocode I mentioned earlier:

static void ServiceClient(TcpClient client)
{
    using (var stream = client.GetStream())
    {
        stream.Write(quote, 0, 1);  // write header
        while (true)
        {
            WriteStockQuote(stream);
        }
    }
}

We write an initial header byte, and then just sit in a loop sending stock quotes back to the client.  To keep the example simple, this just goes forever, until e.g. the client disconnects (which would cause an exception and get out of the loop).

Ok, let’s try running it.  I’ll start the server, and I’ll fire up Task Manager so we can look at CPU usage.  To simulate clients, I’ve got a short F# script I can send to the F# Interactive to add clients on the fly.  So I’ll highlight this F# code and “Send to Interactive”, and now I have 1000 clients hitting the server.  (You can try this yourself with the code attached at the end of this blog entry.  Or just follow along with my narration and imagine you’re seeing on the screen what I’m talking about.)

(Incidentally, this shows a nice use of F#’s Interactive REPL: you can easily write little snippets of test code to test some library or app, whether it be C#, or F#, or whatever.)

Ok, so there’s no flashy UI, but you see the server is handling 1000 quotes per second and we’re hardly using any CPU, maybe just 3-5%.  So everything looks great.  But let’s try adding another 1000 clients.  (I highlight some F# script code and “Send to Interactive” again.)  BOOM, on the screen we get “CSSyncServer has stopped working… Unhandled exception: System.OutOfMemoryException”.  Our server crashed! 

Threads don’t scale well!

Using all those threads ran us out of memory.  Threads are expensive.  We created one thread per client, but our threads were mostly wasted waiting on blocking operations – despite all these threads, we saw the CPU utilization was next to nothing.  After a little more than a thousand clients (and thus a thousand threads, since we had one thread per client), the app falls down, running out of memory since each thread typically needs a megabyte of memory for its call stack.

• One thread per client is not the right way to write a scalable .NET server.
• The real solution is to use non-blocking I/O.  This requires switching to Begin/End APIs.  Then we can use the .NET ThreadPool to use only a few threads to service many clients.

So let’s do it right.  It turns out I’ve already written this solution with C#, so let’s have a look at how it performs.

An asynchronous solution in C#, using Begin/End methods

I can switch the “startup project” in the solution to the “CSAsyncServer” project, where I’ve written the same server code in a non-blocking fashion.  That is, I’m using the asynchronous APIs like BeginWrite/EndWrite, so as not to block and to utilize the .NET ThreadPool for callbacks.  If I run that code, I can start the server, and hit it with 1000 clients. Ok, everything looks fine like before, now let’s add another 1000 clients.  (Server keeps running.)  Another 1000.  (Keeps running, starting to show a little CPU usage, in the 10-20% range…) Another 1000 clients.  You get the idea, it scales now.  Hurrah – problem solved?

• Ok, I’ve convinced you that one-thread-per-client is bad, and that async non-blocking I/O is good…
• But I did it all with C#.  This is an F# talk, right?  So what’s the story?

The story is that in C#, the async programming model is awful.  Let’s have a look at the C# code written using Begin/End APIs to do async I/O.  Recall back in the synchronous version, our ServiceClient() was this little method we saw before:

static void ServiceClient(TcpClient client)
{
    using (var stream = client.GetStream())
    {
        stream.Write(quote, 0, 1);  // write header
        while (true)
        {
            WriteStockQuote(stream);
        }
    }
}

When I change to using the Begin/End pattern to make it non-blocking, it becomes this monstrosity:

static IAsyncResult BeginServiceClient(TcpClient client, AsyncCallback cb, object state)
{
    var stream = client.GetStream();
    var ar = new ServiceClientAsyncResult(stream, cb, state);

    try
    {
        stream.Write(quote, 0, 1);  // write header
    }
    catch
    {
        stream.Dispose();
        throw;
    }
    var everWentAsync = false;
    Action<Action> wrap = (a) =>
    {
        try { a(); }
        catch (Exception e)
        {
            ar.Complete(!everWentAsync, e);
        }
    };
    Action loop = null;
    Action<IAsyncResult> end = (iar) =>
    {
        wrap(() => EndWriteStockQuote(iar));
    };
    AsyncCallback callback = (iar) =>
    {
        if (iar.CompletedSynchronously)
            return;
        end(iar);
        loop();
    };
    loop = () =>
    {
        wrap(() =>
        {
            while (true)
            {
                var iar = BeginWriteStockQuote(stream, callback, state);
                if (!iar.CompletedSynchronously)
                {
                    everWentAsync = true;
                    break;
                }
                end(iar);
            }
        });
    };
    loop();
    return ar;
}

static void EndServiceClient(IAsyncResult iar)
{
    try
    {
        AsyncResult.End<ServiceClientAsyncResult>(iar);
    }
    finally
    {
        (iar as ServiceClientAsyncResult).Dispose();
    }
}

Gaaaah! You end up duplicating the exception handling for Begin/End calls, putting state in the IAsyncResult object, dealing with CompletedSynchronously (see e.g. the “stack dive” section of this blog by Michael Marucheck for an explanation), … I’m not going to go into all the little details.  You get the point – writing async code in C# is a difficult mess.

So it’s possible to ‘do it right’ in C# to utilize your server hardware and the .NET ThreadPool, but it takes a lot of crazy difficult code.  So now you see why I was showing the C# code in an F# talk: now we can have F# come to the rescue!  Let’s look at the F# code!

F# solutions – sync and async

Here’s the F# synchronous code for serviceClient, it looks very much like the C#:

let serviceClient (client: TcpClient) =
    use stream = client.GetStream()
    stream.Write(quote, 0, 1)  // write header
    while true do
        writeStockQuote(stream)

Now here’s the F# async code:

let asyncServiceClient (client: TcpClient) = async {
    use stream = client.GetStream()
    do! stream.AsyncWrite(quote, 0, 1)  // write header
    while true do
        do! asyncWriteStockQuote(stream) }

See how similar they are?  We just had to wrap the code in an ‘async{…}’ block, and then change “Write” to “AsyncWrite” with a “do!”, and “WriteStockQuote” to “AsyncWriteStockQuote” with a “do!”.  That’s it!  Using F# async, we can write non-blocking I/O, but our code structure stays exactly the same as if we’d written things synchronously.  We can evolve our naïve code or proof of concept ideas smoothly and easily into production quality code.

(Once again I do the demo, this time with the F# async server scaling to thousands of clients.  Cue applause.)

Using F# async has other benefits too.  For example, we get cancellation for free; all F# asyncs use the new .NET 4.0 APIs for cancelation (CancelationTokenSource & CancelationToken) and check for cancelation at every “let!” or “do!”  If I were to add the same checks to the C# code, it would take even more effort there.

The final score

Let’s try to sum this all up.  This slide summarizes what I just demonstrated:

 

First off, look at column 1: regardless of what language you use, using asynchronous non-blocking I/O is a big win for scaling; you can handle far more concurrent clients this way than by using dedicated threads and synchronous APIs.

Second, looking at the lines of code (LoC) columns objectively, a solution with the traditional .NET Asynchronous Programming Model (Begin/End methods) is much longer and more complex than the sync version (by about three times in this example).  Whereas with F# async, the solutions are about the same length.

Finally, looking at the ‘coding’ columns, just anecdotally, the APM is much harder to reason about and debug.  These numbers describe how long it took me to code and debug these solutions: though it was easy to write the “sync” versions in both C# and F#, it took me about 3 hours to get the C# async example working correctly, compared to about just 10 more minutes for the F# version.  I admit I’m a little out of practice with C# async, but I don’t lack experience: before working on F#, I was a developer on WCF (Windows Communication Foundation), and the entire runtime there is built out of tons of Begin/End calls.  This stuff is just intrinsically difficult to write and debug in C#.

The two take-home messages here are clear:

• If you are writing .NET server code, you need to use async non-blocking I/O calls to scale.  Otherwise you’ll be throwing money away on extra servers since your software doesn’t scale well to utilize them.
• F# makes it much easier to write, debug, and reason about async code with non-blocking I/O.

So there you go.  Who would have guessed that a programming language could help save customers money on server hardware?  :)

Source code

You can find a Visual Studio 2010 solution with all the source code here.