Server Topology for Web Applications 101
Server topology is unique in every situation and in some cases does not matter much at all. There are, however, common elements and good practices that can be put into place in order to prepare for the bitter-sweet surprise of unexpected growth. Topology can affect performance, availability, maintainability, troubleshooting and security. A well planned topology can make operating system and software upgrades a pleasure while a poorly planned topology can leave system administrators and developers near paralyzed when it comes to the maintenance of several machines.
The topology of an ISP or webhosting company is going to be different than that of a popular web application. This article will focus on the web application. We will cover a bit of theory and save its implementation for another post. The purpose is to demonstrate the advantages and disadvantages that subtle changes in topology can bring.
It should go without saying that every development environment should consist of at least three tier’s: development, staging and production. Our focus will be on the production.
The typical web application consists of load balancers, application servers, a database cluster and shared storage. We will discuss the advantages and disadvantages of the following configurations:
‘Configuration A’ is typical in most organizations. It is the naturally occurring topology for web applications that start small. The private address space provides a physical layer of security. I am assuming the private address space is located on a separate physical switch. The load balancers are connected to the Internet via one interface and the private switch via a second. Sometimes this segregation is not physical and is accomplished via NAT (managers can call it a firewall). The load balancers are configured round-robin in DNS or something like FreeBSD’s CARP arpbalance if the NAT hides them from the public facing net. It is important to note that CARP also provides fail-over between load balancers. Each load balancer proxies web content using any of the following: apache mod_proxy_balancer, nginx, pound, haproxy, etc.
‘Configuration B’ looks very similar to ‘Configuration A’ except for a few nuances. The private address space still hides the application servers, database and shared storage from the public facing world. All of the internal servers can still speak to one another. The difference is that the load balancers are now clustered with specific application servers. The application servers are also named accordingly.
The advantages of ‘Configuration B’ are not initially clear in an example where there are only six application servers, but using your imagination consider the implications of each topology with 20+ AppServers. Note: ‘Configuration B’ can easily transform into ‘Configuration A’ if needed, it’s a matter of shifting to a second set of configuration files on the load balancers.
Both configurations can take advantage of a build server which contains packages for quick deployment of various types of servers. The build server can also make upgrading easy by providing the appropriate pre-compiled worlds and kernels. Both configurations allow for a mechanism to push updates from the build server to the production servers automatically.
There are cases where an upgrade passes all tests on the staging server but does not go smoothly in production. When upgrading on ‘Configuration A’ we can either go through the time-consuming process of upgrading each server one at a time or automate them all at once. Upgrading 20+ servers by hand can take days and can pose inconsistencies in user experience. Automating the upgrade process of ‘Configuration A’ would require synchronously editing the config files on the proxy to avoid hitting the application servers being upgraded. Even if that process is automated it can be a daunting procedure, especially when the servers are being upgraded sequentially and you notice something wrong mid-way.
‘Configuration B’ has the advantage of taking entire segments down for batch upgrades without modifying load balancer settings. The single command `ifconfig carp0 down` on LoadBalancerA can shift all traffic to segment B. Segment A can then be upgraded asynchronously and tested offline. When segment A is confirmed to be in good working order it can trade places with segment B. If anything goes wrong it is already quarantined and can be fixed before additional segments are serviced.
The organizational structure of ‘Configuration B’ aids in debugging as well. If one of the application servers are malfunctioning it’s easy to single out the segment and further drill down to the specific server. Additionally, entire segments can even be transformed into staging environments or used to test bugs that surface under specific conditions without threatening the rest of the production environment. I’ve experienced the usefulness of this in applications where upgrading the database schema needed to be manually synchronized with an update to the application. Scheduled downtime can be eliminated or converted to reduced functionality if segmented off.
There are no clear cut rules to topology. ‘Configuration B’ can be overkill or it can save you from potential headaches in larger implementations.