libevent and so on

Since we need a HTTP server to provide efficient service to all kinds of clients, I started to look into some lightweight open source solutions. The first item that jumps into my eyes is libevent, because I happen to read a blog of a facebook developer's, stating that facebook is using libevent as a HTTP server in their haystack photo storage service.

Libevent is a lightweight event driven library wildly used in many applications, such as memcached and tor. Libevent has simple but efficient HTTP support. Here is a sample code building a simple HTTP server with evhttp:

#include <sys/queue.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netdb.h>

#include <string.h>
#include <stdio.h>

#include <evhttp.h>

void ocr_handler(struct evhttp_request *req, void *arg)
{
 struct evbuffer *buf;
 struct evkeyval *header;

 TAILQ_FOREACH(header, req->input_headers, next) {
  fprintf(stdout, "key:%s\tvalue:%s\n", header->key, header->value);
 }
}

int main(int argc, char **argv)
{
 int err;
 struct evhttp *httpd;
 struct event_base *evbase;
 int port = 1234;

 evbase = event_init();

 fprintf(stderr, "event method: %s\n", event_base_get_method(evbase));

 httpd = evhttp_new(evbase);
 if ((err = evhttp_bind_socket(httpd, NULL, port)) < 0) {
  fprintf(stderr, "error binding http server to port %d\n", port);
  return -1;
 }

 /* set callback for "/" */
 evhttp_set_cb(httpd, "/", ocr_handler, NULL);

 /* Set a send callback for all other requests. */
 evhttp_set_gencb(httpd, NULL, NULL);

 event_base_dispatch(evbase);

 evhttp_free(httpd);
 event_base_free(evbase);
 return 0;
}

However, as I looked into the library in details and wrote some test programs, it turns out that life is not that easy. I tried to dynamically create threads to serve incoming HTTP requests, but the code didn't work as I thought. After searching for a while, I found the problem:

Steven Grimm:

What libevent doesn't support is sharing a libevent instance across threads. It works just fine to use libevent in a multithreaded process where only one thread is making libevent calls.

Therefore, to use libevent in a multi-threaded program, we should create each thread a event base when initialising the program and call ev_set_base() after ev_set() but before ev_add(). Then we will have a thread pool to serve HTTP requests. There will a main thread listening to all incoming HTTP requests. When a request comes, it passes the request to some thread from the thread pool and wakes it up to handle the request.

Although this will work, we somehow end up with a master/worker thread architecture, where the main thread handles all reads from netwrok. This will certainly be a bottleneck when there are thousands of clients(think the C10K problem). I don't know how the facebook guys deal with this problem(maybe they patched libevent?:), But IMO, using an evhttp dispacher in a multi-threaded process, we'll end up this way.

So, currently, I'm planning to look at other solutions like lighttpd before making any decision on the server architecture.