Can Apache handle 10K redirect entries?
Published: dec 22, 2009
Category: Scribbles
Tags: apache, http, performance
Language: [English]
page views: 1390

The question

How does Apache handle 10K redirects in a virtual host. How does having so many redirects affect the performance of Apache, even when traffic doesn't use those redirections?

Environment

Not quite up to date anymore. But average of what can be found laying about.

• Server: AMD Athlon II with 2 cores, running Linux amd64 2.6.28 kernel (Ubuntu jaunty), 2G of memory.
• Client: AMD Athlon 64, running Linux amd64 2.6.31 kernel (Ubuntu karmic), 1.5G of memory.
• Network: 100mb/s LAN over with 2 consumer grade switches between server and client. Network was also handling other traffic.

Client is also running as desktop, with browsers and mp3s playing so.. :)

software:

• Webserver: apache 2.2.11
• Client software: siege 2.68

The tests

For all the tests, I used an empty virtualhost (test.local) in apache that i changed for the needs. This apache instance also had other virtualhosts defined, but their traffic is minimal. Apache was configured to do logging to disk. All tests ran for 120 seconds. With 10 concurrent requests.

<VirtualHost *:80>
        ServerName test.local

        DocumentRoot /var/www/test.local
        <Directory /var/www/test.local>
        </Directory>

        ErrorLog /var/log/apache2/test.local-error.log
        LogLevel warn
        CustomLog /var/log/apache2/test.local-access.log combined
</VirtualHost>

One: Maximum requests per second

Just as a baseline, how many requests per second can this apache handle. Let's create a 10K file with random content and get that as fast as we can:

1. server: $ dd if=/dev/urandom of=/var/www/test.local/f10k.bin bs=1k count=10
2. client: $ siege -d0 -c10 -t120s http://test.local/f10k.bin

Siege's output

Transactions:		      124427 hits
Availability:		      100.00 %
Elapsed time:		      119.84 secs
Data transferred:	     1215.11 MB
Response time:		        0.01 secs
Transaction rate:	     1038.28 trans/sec
Throughput:		       10.14 MB/sec
Concurrency:		        9.98
Successful transactions:      124427
Failed transactions:	           0
Longest transaction:	        0.05
Shortest transaction:	        0.00

1038 transactions per sec... not that great... but wait... Throughput is at 10.14MB/s. This is only a 100mb/s connect. So I just filled the pipe, not stressed the server. A 10KB is just too big for these tests. Lets try with a 1KB. That also explains why the load was only around 0.15.

• server: $ dd if=/dev/urandom of=/var/www/test.local/f1k.bin bs=1k count=1
• client: $ siege -d0 -c10 -t120s http://test.local/f1k.bin

Transactions:		      396585 hits
Availability:		      100.00 %
Elapsed time:		      119.34 secs
Data transferred:	      387.29 MB
Response time:		        0.00 secs
Transaction rate:	     3323.15 trans/sec
Throughput:		        3.25 MB/sec
Concurrency:		        9.94
Successful transactions:      396585
Failed transactions:	           0
Longest transaction:	        0.03
Shortest transaction:	        0.00

3323 transactions per second... An expected number for this hardware and Apache. The Throughput was just over 30% of the network speed. Meaning that ever with TCP/IP overhead we are not hitting the network's limits anymore. Load on the Apache side was hanging around 1.70 So 3323 req/sec is the maximum available number on this setup.

Two: Following the redirection

Let's see how slow following a redirect is. Redirects mean having to do two requests to the server for getting one file. One the server we add a redirect to the Directory section so that it looks like:

        <Directory /var/www/test.local>
                Redirect /redir.bin http://test.local/f1k.bin permanent
        </Directory>

• client: $ siege -d0 -c10 -t120s http://test.local/redir.bin

Transactions:		      365142 hits
Availability:		      100.00 %
Elapsed time:		      119.94 secs
Data transferred:	      227.22 MB
Response time:		        0.00 secs
Transaction rate:	     3044.37 trans/sec
Throughput:		        1.89 MB/sec
Concurrency:		        9.87
Successful transactions:      365146
Failed transactions:	           0
Longest transaction:	        0.30
Shortest transaction:	        0.00

Unsurprisingly the transaction rate dropped to 3044 transactions/sec. Important to note here is that Siege measures transactions, not requests. So getting a redirect and following it is seen as one transaction, not two requests. The load on the Server was around 4. Since each transaction is now two requests this number reflects the extra load of double the requests being handled.

Three: Building a set of files

Hitting just one URL is just not realistic. In fact it's so artificial that it can distort our investigation. Lets start by creating 10.000 1KB files and see how fast Apache can dish up their contents. On the server we have to create those 10K of files. I get file content from /dev/urandom to make sure the content is unique and doesn't compress well. Then get the md5 hash from this file to generate a nice and random filename. Luckily bash is our friend for all of this.

$ for ((c=1; c<=10000; c++)) do dd if=/dev/urandom of=tmp.bin bs=1k count=1; mv -v tmp.bin $(md5sum tmp.bin | cut -c -32).bin; done;
$ for c in *.bin; do echo "http://test.local/$c"; done > /tmp/urllist.txt

We copy the /tmp/urllist.txt to the client. Siege will randomly select a url to request. The Siege command is now: $ siege -d0 -c10 -t120s -i -f urllist.txt

Transactions:		      402691 hits
Availability:		      100.00 %
Elapsed time:		      119.10 secs
Data transferred:	      393.25 MB
Response time:		        0.00 secs
Transaction rate:	     3381.12 trans/sec
Throughput:		        3.30 MB/sec
Concurrency:		        9.93
Successful transactions:      402691
Failed transactions:	           0
Longest transaction:	        0.05
Shortest transaction:	        0.00

I had to run Siege two times, since it's was noticeable that Apache is waiting for files to come from disk (and I do not want to measure disk performace right now). Once the files were in memory the load settled around 2. Conclusion: Requesting 1 file or randomly requesting 10K files doesn't make any noticeable difference in performance (once the files are cached in memory).

Four: Apache URL redirector, mod_alias

Now that the warming up has been done. Let's get into the heart of the matter. What is the impact of having 10K redirects in the configuration? First we test it with the simpler mod_alias. Notice, that I'm not going to request those redirects, I'm only going to declare them in the configuration and see how that impacts the performance of requesting URLs that have files for them. I use a similar trick for getting the URLs that will be redirected that i did for generating the files. On the server I run:

$ for ((c=1; c<=10000; c++)) do dd if=/dev/urandom of=/tmp/tmp.bin bs=1k count=1 2> /dev/null; echo $(md5sum /tmp/tmp.bin | cut -c -32).bin >> /tmp/redir-names-list.txt; done;

The redir-names-list.txt contains a list of random names that are similar to the real files. From those URLs we are building the redirects for mod_alias via: $ for c in $(cat /tmp/redir-names-list.txt); do echo "redirect /$c http://test.local/does_not_exist.bin permanent"; done > /tmp/redirs.conf So in /tmp/redirs.conf there are 10.000 lines like:

redirect /c9bb8ba0a88d3323e1b88f6d980a0836.bin http://test.local/does_not_exist.bin permanent
redirect /07723beafa05944e2992dcb29a6c59da.bin http://test.local/does_not_exist.bin permanent
redirect /ca37c0c6c879cbd120ad497b61844e2c.bin http://test.local/does_not_exist.bin permanent
...

And I change the apache Directory config like so:

        <Directory /var/www/test.local>
                Include /tmp/redirs.conf
        </Directory>

And reload the server, of course. Running Siege as in part three, $ siege -d0 -c10 -t120s -i -f urllist.txt The results were:

Transactions:		      279216 hits
Availability:		      100.00 %
Elapsed time:		      119.47 secs
Data transferred:	      272.67 MB
Response time:		        0.00 secs
Transaction rate:	     2337.12 trans/sec
Throughput:		        2.28 MB/sec
Concurrency:		        9.95
Successful transactions:      279216
Failed transactions:	           0
Longest transaction:	        0.66
Shortest transaction:	        0.00

So, adding the 10K redirects made the number of transactions drop from 3381 to 2337. That's 70% of the performance without the redirects. Or to put it in an other way, with 10.000 redirects added via mod_alias, the server spends 30% of the time it takes to handle sending a file, figuring out if to send a redirect or a file. The server load, it went to around 50, and was still responsive. Next up, lets try mod_rewrite

Five: Apache URL redirector, mod_rewrite

First, add mod_rewrite to the apache server, since i do not have it enabled normally: $ sudo a2enmod rewrite with that done build the RewriteRules: $ for c in $(cat /tmp/redir-names-list.txt); do echo "RewriteRule /$c http://test.local/does_not_exist.bin [R=301,L]"; done > /tmp/rewrite.conf Now in /tmp/rewrite.conf there are 10.000 statements like:

RewriteRule /c9bb8ba0a88d3323e1b88f6d980a0836.bin http://test.local/does_not_exist.bin [R=301,L]
RewriteRule /07723beafa05944e2992dcb29a6c59da.bin http://test.local/does_not_exist.bin [R=301,L]
RewriteRule /ca37c0c6c879cbd120ad497b61844e2c.bin http://test.local/does_not_exist.bin [R=301,L]
...

Change the webserver configuration to

        <Directory /var/www/test.local>
		RewriteEngine ON
                Include /tmp/rewrite.conf
        </Directory>

Restart the webserver and run Siege $ siege -d0 -c10 -t120s -i -f urllist.txt

Transactions:		       60188 hits
Availability:		      100.00 %
Elapsed time:		      119.38 secs
Data transferred:	       58.78 MB
Response time:		        0.02 secs
Transaction rate:	      504.17 trans/sec
Throughput:		        0.49 MB/sec
Concurrency:		        9.97
Successful transactions:       60188
Failed transactions:	           0
Longest transaction:	        2.46
Shortest transaction:	        0.00

With mod_rewrite there are only 504 transactions per second. That's only about 14% of the performance without redirects. The server load started to stabilize around 95 to 100.

Conclusions

As said before. This is a simplified case, and in most real world situations, you wont need 10K redirects. Completely excluded here were regular expressions which normally handle many URLs of similar structure in one redirect rule. But besides the limitations, a clear conclusion can be drawn. If you have many redirects, it's faster to use mod_alias then the more powerfull, but much slower mod_rewrite. But to put all these numbers in a context, having a sustained load of 500 requests per second (that's 43million hits per day or 1.3 billion per month) will bring down most any server that's serving something more interesting then just static files.