lighttpd is used by many well-known sites. The typical scenario is using lighttpd as off-load server to push out static content and leave to complex work to another server.

• Lighttpd enters the Top5
• Lighttpd powers 5 Alexa Top 250 sites

One example is YouTube. They have a farm of servers which push out the thumbnails you see before you see the movies:

$ curl -I http://sjl-static16.sjl.youtube.com/vi/TgF_eRkfqEY/2.jpg
HTTP/1.1 200 OK
Content-Type: image/jpeg
ETag: "983726135810477085"
Accept-Ranges: bytes
Last-Modified: Fri, 03 Feb 2006 04:32:53 GMT
Content-Length: 3495
Date: Wed, 04 Apr 2007 06:49:51 GMT
Server: lighttpd-aio/1.4.11.8

As you see in the name-scheme, there are some more of those servers pushing out content.

On wikipedia they run at least 2 servers with lighttpd:

• upload.wikimedia.org
• download.wikimedia.org

While download is used to distribute the SQL dumps of the database when tend to grow above the magic 4GByte border, upload is used to push out all the images and resize them when necessary.

$ curl -I https://upload.wikimedia.org/wikipedia/commons/thumb/2/21/Mandel_zoom_00_mandelbrot_set.jpg/250px-Mandel_zoom_00_mandelbrot_set.jpg
HTTP/1.0 200 OK
Content-Type: image/jpeg
ETag: "6460328581220324712"
Accept-Ranges: bytes
Last-Modified: Mon, 04 Dec 2006 22:24:53 GMT
Content-Length: 5973
Date: Wed, 14 Mar 2007 11:16:19 GMT
Server: lighttpd/1.4.13
X-Cache: HIT from sq13.wikimedia.org
X-Cache-Lookup: HIT from sq13.wikimedia.org:3128
X-Cache: HIT from knsq12.knams.wikimedia.org
X-Cache-Lookup: HIT from knsq12.knams.wikimedia.org:3128
Age: 6265
X-Cache: HIT from knsq10.knams.wikimedia.org
X-Cache-Lookup: HIT from knsq10.knams.wikimedia.org:80
Via: 1.0 sq13.wikimedia.org:3128 (squid/2.6.STABLE12), 1.0 knsq12.knams.wikimedia.org:3128 (squid/2.6.STABLE12), 1.0 knsq10.knams.wikimedia.org:80 (squid/2.6.STABLE12)
Connection: close

Security, speed, compliance, and flexibility–all of these describe LightTPD which is rapidly redefining efficiency of a webserver; as it is designed and optimized for high performance environments. With a small memory footprint compared to other web-servers, effective management of the cpu-load, and advanced feature set (FastCGI, CGI, Auth, Output-Compression, URL-Rewriting and many more) LightTPD is the perfect solution for every server that is suffering load problems. And best of all it’s Open Source licensed under the revised BSD license.

Web 2.0

lighttpd powers several popular Web 2.0 sites like YouTube, wikipedia and meebo. Its high speed io-infrastructure allows them to scale several times better with the same hardware than with alternative webservers.

This fast web server and its development team create a webserver with the needs of the future web in mind:

• Faster FastCGI
• COMET meets mod_mailbox
• Async IO

Its event-driven architecure is optimized for a large number of parallel connections (keep-alive) which is important for high performant AJAX applications.

Back in the days of February 2003 I (Jan) was finishing my thesis. Writing a thesis is as far as I can say the most frustrating job you can do.

While my subject was great (‘Development of a handheld system to monitor and control CAN-bus systems in a non-automative environment’) and the hard- and software development was a real challenge I had to write the thesis which documented the design-decision, the preparations and so on.

For a developer this is the real hard part: writing the documentation. Especially as the date when you have to finish the thesis gets nearer every day. To motivate yourself you need a distraction.

For me it was a proof-of-concept of the c10k problem written by Dan Kegel. How to handle 10000 connections in parallel on one server. I already had seen apaches killing systems because they ran out of memory into swap with only 100 parallel connections.

In the first weeks it was only a challenge how to write something fast, optimized. In the ChangeLog you can still see my comments how to achieve this. Cache as much as you can. Why regenerated the timestamp for the ‘Date:’ header 1000 times per second if it is the same all the time ?

As I needed PHP support for my own purposes it was one of the first features added. The ChangeLog says I had it working two weeks are the rest was done. Including the load-balancing to distribute the load from one webserver to multiple fastcgi-backends.

At one point I asked myself: This is just a proof-of-concept. Where are we now ? How do we compare to the other servers ?

The first opponent was thttpd, the big single-threaded webserver. Especially on large files we outperformed it 2 to 5 times. Next were boa and mathopd, both with problems and slower. Zeus was the first real challenge and they proof that Zeus is a great webserver. If you want to spend money on a webserver (next to asking me to develop something for you) is buying licenses from Zeus. (no, I’m not payed by them in any way).

Different optimisations were added: new event-handlers like epoll and kqueue, new network backends like sendfile().

So, I look back now on 2.5 years of development see the numbers of installations rising every month. https://news.netcraft.com/ is telling as every month that the numbers are still increasing and I send a mail to the mailinglist, so everyone know and keeps up the joy that this little proof-of-concept went into a well working webserver.