Tuesday, May 12, 2009

libevent and so on

Since we need a HTTP server to provide efficient service to all kinds of clients, I started to look into some lightweight open source solutions. The first item that jumps into my eyes is libevent, because I happen to read a blog of a facebook developer's, stating that facebook is using libevent as a HTTP server in their haystack photo storage service.

Libevent is a lightweight event driven library wildly used in many applications, such as memcached and tor. Libevent has simple but efficient HTTP support. Here is a sample code building a simple HTTP server with evhttp:
#include <sys/queue.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netdb.h>

#include <string.h>
#include <stdio.h>

#include <evhttp.h>

void ocr_handler(struct evhttp_request *req, void *arg)
{
struct evbuffer *buf;
struct evkeyval *header;

TAILQ_FOREACH(header, req->input_headers, next) {
fprintf(stdout, "key:%s\tvalue:%s\n", header->key, header->value);
}
}

int main(int argc, char **argv)
{
int err;
struct evhttp *httpd;
struct event_base *evbase;
int port = 1234;

evbase = event_init();

fprintf(stderr, "event method: %s\n", event_base_get_method(evbase));

httpd = evhttp_new(evbase);
if ((err = evhttp_bind_socket(httpd, NULL, port)) < 0) {
fprintf(stderr, "error binding http server to port %d\n", port);
return -1;
}

/* set callback for "/" */
evhttp_set_cb(httpd, "/", ocr_handler, NULL);

/* Set a send callback for all other requests. */
evhttp_set_gencb(httpd, NULL, NULL);

event_base_dispatch(evbase);

evhttp_free(httpd);
event_base_free(evbase);
return 0;
}



However, as I looked into the library in details and wrote some test programs, it turns out that life is not that easy. I tried to dynamically create threads to serve incoming HTTP requests, but the code didn't work as I thought. After searching for a while, I found the problem:

Steven Grimm:
What libevent doesn't support is sharing a libevent instance across threads. It works just fine to use libevent in a multithreaded process where only one thread is making libevent calls.


Therefore, to use libevent in a multi-threaded program, we should create each thread a event base when initialising the program and call ev_set_base() after ev_set() but before ev_add(). Then we will have a thread pool to serve HTTP requests. There will a main thread listening to all incoming HTTP requests. When a request comes, it passes the request to some thread from the thread pool and wakes it up to handle the request.

Although this will work, we somehow end up with a master/worker thread architecture, where the main thread handles all reads from netwrok. This will certainly be a bottleneck when there are thousands of clients(think the C10K problem). I don't know how the facebook guys deal with this problem(maybe they patched libevent?:), But IMO, using an evhttp dispacher in a multi-threaded process, we'll end up this way.

So, currently, I'm planning to look at other solutions like lighttpd before making any decision on the server architecture.

5 comments:

  1. Bergwolf,

    i think ngynx its a good source of hacking.. and i didnt go too far , but what i saw was a main event, handling requests.. scalar..
    k
    and then there's a pool of threads.. waiting for this main event loop dispatch them for do some work...
    since im doing my homework either.. i would keep my eye on a good lockless queue.. and use the hot workstealing mechanism for the pool..

    im trying to get the lockfree queue right now..

    good luck and i hope it helps!

    ReplyDelete
    Replies
    1. Fabio,

      Sorry for the late reply. I don't have notification for comments...

      Yeah, ngynx is has very good event handling implementation. However it is not a library so that other programs can not just make use of it, unless building their applications alike factcgi. That said, if the main purpose of the service is to support http requests, ngynx is actually an excellent option.

      Delete
  2. Replies
    1. I haven't looked at libevent since then. But I *guess* the conclusion about unsharable event base still holds.

      Delete
  3. I would advise that you pick the #1 Forex broker - AvaTrade.

    ReplyDelete