Varnish command line tools
Varnish comes with several very useful command line tools that can be a bit hard to get the grasp of. The list below is by no means meant to be exhaustive, but give an introduction to some tools and use cases.
varnishhist
Easily seen as a geeky graph with little information, varnishhist is actually extremely useful to get an overview of the overall status of your backend servers and varnish.
The pipes (|) are requests served from the cache whereas the hash-signs (#) are requests to the backend. The X axis is a logaritmic scale for request times. So the histogram above shows that we have a good amount of cache hits that are served really fast whereas roughly half of the backend requests takes a bit more than 0,1s. Like most of the other command line applications you can filter out the data you need with regex, only show backend requests or only cache hits. See https://www.varnish-cache.org/docs/trunk/reference/varnishhist.html for a complete list of parameters.
varnishstat
varnishstat can be used in two modes, equally useful. If run with only "varnishstat" you will get a continously updated list of the counters that fit on your screen. If you want all the values for indepth analysis you can however use "varnishstat -1" for the current counter values.
Now a couple of important figures from this image:
- The very first row is the uptime of varnish. This instance had been restarted 1 day and 2h 44 mins before screenshot
- The counters below that are the average hitrate of varnish. The first row is the timeframe and the second is the average hitrate. If varnishstat is kept open for longer the second timeframe will go up to 100 seconds and the third to 1000 seconds
- As for the list of variables below the values correspons to total value, followed by current value and finally the average value. Some rows that are apparently interesting would be
- cache_hit/cache_miss to see if you have monumental miss storms
- Relationship between client_conn and client_req to see if connections are being reused. In this case there's only API traffic where very few connections are kept open. So the almost 1:1 ratio is to be seen as normal.
- Also relationship between s_hdrbytes and s_bodybytes is rather interesting as you can see how much of your bandwith is actually being used by the headers. So if s_hdrbytes are a high percentage of your s_bodybytes you might want to consider if all your headers are actually necesary and useful.
varnishtop
Varnishtop is a very handy tool to get filtered information about your traffic. Especially since alot of high-traffic varnish sites do not have access_logs on their backend servers - this can be of great use.
tx are always requests to backends, whereas rx are requests from clients to varnish. The examples below should clarify what I mean.
Some handy examples to work from:
See what requests are most common to the backend servers.
varnishtop -i txurl
See what useragents are the most common from the clients
varnishtop -i RxHeader -C -I ^User-Agent
See what user agents are commonly accessing the backend servers, compare to the previous one to find clients that are commonly causing misses.
varnishtop -i TxHeader -C -I ^User-Agent
See what cookies values are the most commonly sent to varnish.
varnishtop -i RxHeader -I Cookie
See what hosts are being accessed through varnish. Will of course only give you useful information if there are several hosts behind your varnish instance.
varnishtop -i RxHeader -I '^Host:'
See what accept-charsets are used by clients
varnishtop -i RxHeader -I '^Accept-Charset'
varnishlog
varnishlog is yet another powerful tool to log the requests you want to analyze. It's also very useful without parameters to develop your vcl and see the exact results of your changes in all it's verbosity. See https://www.varnish-cache.org/docs/trunk/tutorial/logging.html for the manual and a few examples. You will find it very similar to varnishstop in it's syntax.
One useful example for listing all details about requests resulting in a 500 status:
varnishlog -b -m "RxStatus:500"
varnishncsa
varnishncsa is a handy tool for producing apache/ncsa formatted logs. This is very useful if you want to log the requests to varnish and analyze them with one of the many availalable log analyzers that reads such logs, for instance awstats.
Send data to include files with dwoo
Sometimes it is wanted to send some data, like title or an id to an included template with Dwoo. Even though described in the manual I did not first look there because I mostly don't find what I look for there. So in case anyone else have the same bother; this will solve it:
{include(file='elements/google-like-button.tpl' url='http://www.eldefors.com')}
This will allow you to use $url in the included template just as if it was a local variable.
More features that one might miss or look too long for in the docs:
Scope $__.var will always fetch 'var' variable from template root. This is useful for loops where scope is changed.
Loop names Adding name="loopname" lets you access it's variables. inside a nested construct by accessing $.foreach.loopname.var (change foreach with your loop element).
Default view for empty foreach By adding a {else} before your end tag you can output data for empty variables passed without an extra element.
What is Dwoo some might wonder. It is a template system with similar syntax to smarty. It has however been rewritten alot and is in my experience working great both performance-wise and feature-wise. It is very easy to extend and the by far most inexpandable feature to me is that of template inheritance. This lacks in most PHP templating systems but with Dwoo you can apply the same thinking as with normal class inheritance. Since many elements on your page are the same on all pages and even more in the same section; you can define blocks which you override (think of it as class methods) and for instance create a general section template that inherits the base layout template and then let every section page template inherit this.
Huge server architectures
I enjoy to read about the architecture about some of the bigger internet related systems around. Be it about database sharding, Hadoop usage, choice of languages or development methods. I will continuously try to post some numbers from these adventures. Here is a start together with links that can be followed for more details.
From Data Center Knowledge quoting a talk at Structure 2010 by Facebook’s Jonathan Heiliger.
- 400 million users.
- 16 billion minutes spend on Facebook each day.
- 3 billion new photos per month
- More than a million photos viewed per second.
- Probably over 60,000 servers at date (Data Center Knowledge)
- Thousands of memcache servers. Most likely the biggest memcache cluster (Pingdom).
Akamai
- 65,000 servers spanning 70 countries and 1000 networks.
- Hundreds of billions "Internet interactions" per day
- Traffic peaks at 2 Terabits per second
Source: Akamai
- Server number estimated at 450,000 at 2006 (High Scalability)
- Server number estimated at 1 million by Gartner (Pandia)
A first glance at hiphop
Facebook recently released their PHP on steroids named HipHop as open source. I listened to their presentation a while before at the FOSDEM conference in Brussels and was as many others impressed - but not as entusiastic as many others seem now.
Some say it's nothing new because there has been a small amount of PHP compilers before or because there are op-code caches already. What HipHop does however is not only to compile the code base to C++ but also process the code in several stages - to use as specific data type possible for instance. Facebook engineers are saying that they see 30-50% performance improvement over PHP that is already boosted by APC. Indeed that is a huge deal given the amount of application servers they use.
On the other side a lot of attention it has been gotten is almost the same as that of APC. It´s seen as a general purpose performance booster. But as with APC results for most people will be disappointing for the reason that most of the application time is not spent in the PHP code with most websites.
Facebook is indeed special compared to most websites. For instance they generally do no joins of data at the database level. That results in alot more data in the application as well as more basic application logic.
The reason they often chose to totally exclude joins are several. Amongst others it´s performance draining for the database servers which are generally harder to scale. It´s also very hard to do when you need to query a whole lot of servers (that can also be sharded by different factors) for each and every type of data you want to join.
HipHop is made for the giants by a giant. The huge sites with a lot of traffic that have big amounts of data to do queries against. Smaller sites will have much less benefit from the performance boost of it as most of their time is spent in databases, caches, reading from disk etc. The results will also vary greatly depending on how much of the code base of the website is mundane (basic constructs like loops, processing with non-dynamic variables etc). Basically everything that can be rewritten with static functions and variables are the most welcome targets to the HipHop optimizations.
Even with some 10 commodity application servers I would say that Hip Hop should give little enough performance boost to justify the time that needs to be spent to learn, test and maintain the framework. For Facebook though, I can certainly see how it's very welcome even with several man-years of development costs as they have thousands (?) of application servers and have good reasons not to drop PHP in most heavy parts of it.