Nathan Evans' Nemesis of the Moment

Deploying HA-Proxy + Keepalived with Mercurial for distributed config

Posted in Automation, Source Control, Unix Environment by Nathan B. Evans on February 27, 2011

Something I have learnt (and re-learnt) too many times to count is that one of the strange wonders of working for a startup company is that the most bizarre tasks can land on your lap seemingly with no warning.

We’ve recently been doing a big revamp of our data centre environment, including two shiny new Hyper-V hosts, a Sonicwall firewall and the decommissioning of lots of legacy hardware that doesn’t support virtualisation. As part of all this work we needed to put in place several capabilities for routing application requests on our SaaS platform:

  1. Expose HTTP/80 and HTTPS/443 endpoints on the public web and route incoming requests based upon URL to specific (and possibly many) private internal servers.
  2. Expose a separate and “special” TCP 443 endpoint (on public web) that isn’t really HTTPS at all but will be used for tunnelling of our TCP application protocol. We intend to use this when we acquire pilot programme customers that don’t want the “hassle” of modifying anything on their network firewalls/proxies. Yes, really. Even worse, it will inspect the source IP address and, from that, determine what customer it is and then route it to the appropriate private internal server and port number.
  3. Expose various other TCP ports on public web and map these (in as traditional “port map” style as possible) directly to one specific private internal server.
  4. Be easy to change the configuration and be scriptable, so we can tick off the “continuous deployment” check box.
  5. Configuration changes must never tamper with existing connections.
  6. Optional bonus, be source controllable.

My first suggestion was that we would write some PowerShell scripts to access the Sonicwall firewall through SSH and control its firewall tables directly. This was the plan for several months in fact, whilst everything was getting put in place inside the data centre. I knew full well it wouldn’t be easy. First there was some political issues inside the company with regard to a developer (me) having access to a central firewall. Second, I knew that creation and testing of the scripts would be difficult and that the whole CLI on the Sonicwall would surely not be as good as a Cisco.

I knew I could achieve #1 and #3 easily on a Sonicwall, like with any router really. But #2 was a little bit of an unknown as, frankly, I doubted if a Sonicwall could do it without jumping through a ton of usability hoops. #4 and #6 were the greatest unknown. I know you can export a Sonicwall’s configuration from the web interface. But it comes down as a binary file; which sort of made me doubt whether the CLI could do it properly as some form of text file. And of course if you can’t get the configuration as a text file then it’s not really going to be truly source controllable either, so that’s #6 out.

Fortunately an alternative (and better!) solution presented itself in the form of HA-Proxy. I’ve been hearing more and more positive things about this over the past couple years: most notably from the Stack Exchange. And having recently finally shed my long-time slight phobia of Linux, I decided to have a go at setting it up this weekend on a virtual machine.

The only downside was that as soon as you move some of your routing decisions away from your core firewall then you start to get a bit worrisome about server failure. So naturally we had to ensure that whatever we came up with involving HA-Proxy can be deployed as a clustered master-master or master-slave style solution. That would mean that if our VM host “A” had a failure then Mr Backup over there, “B”, could immediately take up the load.

It seems that Stack Exchange chose to use the Linux-HA Heartbeat system for providing their master-slave cluster behaviour. In the end we opted for Keepalived instead. It is more or less the same thing except that it’s apparently more geared towards load balancers and proxies such as HA-Proxy. Whereas Heartbeat is designed more for situations where you only ever want one active server (i.e. master-slave(s)). Keepalived just seems more flexible in the event that we decide to switch to a master-master style cluster in the future.

HA-Proxy Configuration

Here’s the basic /etc/haproxy/haproxy.conf that I came up with to meet requirements #1, #2 and #3.

# Global settings for HA-Proxy.
	maxconn 8192

# Default settings for all sections, unless overridden.
	mode http

	# Known-good TCP timeouts.
	timeout connect 5000ms
	timeout client 20000ms
	timeout server 20000ms

	# Prevents zombie connections hanging around holding resources.
	option nolinger

# Host HA-Proxy's web stats on Port 81.
listen HAProxy-Statistics *:81
	mode http
	stats enable
	stats uri /haproxy?stats
	stats refresh 20s
	stats show-node
	stats show-legends
	stats auth admin:letmein

# Front-ends
	# Public HTTP/80 endpoint.
	frontend Public-HTTP
		mode http
		bind *:80
		default_backend Web-Farm

	# Public HTTPS/443 endpoint.
	frontend Public-HTTPS
		mode tcp
		default_backend Web-Farm-SSL

	# A "fake" HTTPS endpoint that is used for tunnelling some customers based on the source IP address.
	# Note: At no point is this a true TLS/SSL connection!
	# Note 2: This only works if the customer network allows TCP 443 outbound without passing through an internal proxy (... which most of ours do).
	frontend Public-AppTunnel
		mode tcp

		# Bind to a different interface so as not to conflict with Public-HTTPS (above).

		# Pilot Customer 2 (testing)
		acl IsFrom_PilotCustomer2 src
		use_backend App-PilotCustomer2 if IsFrom_PilotCustomer2

# Back-ends
# General
	# IIS 7.5 web servers.
	backend Web-Farm
		mode http
		balance roundrobin
		option httpchk
		server Web0 check
		server Web1 check

	# IIS 7.5 web servers, that expose HTTPS/443.
	# Note: This is probably not the best way, but it works for now. Need to investigate using the stunnel solution.
	backend Web-Farm-SSL
		mode tcp
		balance roundrobin
		server Web0 check
		server Web1 check

# Back-ends
# Application Servers (TCP bespoke protocol)
	# Customer 1
	listen App-Customer1
		mode tcp
		bind *:35007
		server AppLive0 check

	# Pilot Customer 2 (testing)
	listen App-PilotCustomer2
		mode tcp
		bind *:35096
		server AppLive0 check

I doubt the file will remain this small for long. It’ll probably be 15x bigger in a week or two 🙂

Keepalived Configuration

And here’s the /etc/keepalived/keepalived.conf file.

vrrp_instance_VI_1 {
	state MASTER
	interface seth0
	virtual_router_id 51
	! this priority (below) should be higher on the master server, than on the slave.
	! a bit of a pain as it makes Mercurial'ising this config more difficult - anyone know a solution?
	priority 200
	advert_int 1
	authentication {
		auth_type PASS
		auth_pass some_secure_password_goes_here
	virtual_ipaddress {

It is rather straight forward as far as other Keepalived configurations go. It is effectively no different to a Windows Server Network Load Balancing (NLB) deployment, with the right options to give the master-slave behaviour. Note the only reason I’ve specified two virtual IP addresses is because I need to use the TCP port 443 twice (for different purposes). These will be port mapped on the Sonicwall to different public IP addresses, of course.

Mercurial, auto-propagation script for haproxy.conf

cd /etc/haproxy/

# Check whether remote repo contains new changesets.
# Otherwise we have no work to do and can abort.
if hg incoming; then
  echo "The HA-Proxy remote repo contains new changesets. Pulling changesets..."
  hg pull

  # Update to the working directory to latest revision.
  echo "Updating HA-Proxy configuration to latest revision..."
  hg update -C

  # Re-initialize the HA-Proxy by informing the running instance
  # to close its listen sockets and then load a new instance to
  # recapture those sockets. This ensures that no active
  # connections are dropped like a full restart would cause.
  echo "Reloading HA-Proxy with new configuration..."
  /etc/init.d/haproxy reload

  echo "The HA-Proxy local repo is already up to date."

I turned the whole /etc/haproxy/ directory into a Mercurial repository. The script above was also included in this directory (to gain free version control!), called I cloned this repository onto our central Mercurial master server.

It is then just a case of setting up a basic “* * * * * /etc/haproxy/” cronjob so that the script above gets executed every minute (don’t worry it’s not exactly going to generate much load).

This is very cool because we can use the slave HA-Proxy server as a sort of testing ground of sorts. We can modify the config on that server quite a lot and test against it (by connecting directly to it’s IP rather than the clustered/virtual IP provided by Keepalived). Then once we’ve got the config just right we can commit it to the Mercurial repository and then push the changeset(s) to the master server. Within 60 seconds then the other server (or servers, in your case possibly!) will then run the synchronisation script.

One very neat thing about the newer versions of HA-Proxy (I deployed version 1.4.11) is that they have an /etc/init.d script that already includes everything you need for doing configuration file rebinds/reloads. This is great because what actually happens is that HA-Proxy will send a special signal to the old process so that it stops listening on the front-end sockets. Then it will attempt to start the new instance based upon the new configuration. If this fails it will send another signal to the “old”, but now resurrected process, that it can resume listening. Otherwise the old process will eventually exit once all its existing client connections have ended. This is brilliant because it meets and rather elegantly exceeds exceeds our expectations for requirement #5.

The fact that our HA-Proxy’s will contain far more meticulous configuration details than even our Sonicwall, I think that this solution based upon Mercurial is simply brilliant. We have what is effectively a test and slave server all-in-one, and a hg revert or hg rollback command is of course only a click away.

It’s still a work in progress but so far I’m very pleased with the progress with HA-Proxy.


Lightweight shelving of your work-in-progress, with Mercurial

Posted in Automation, Source Control by Nathan B. Evans on February 22, 2011

Before we switched to Mercurial in January 2011, we were a TFS shop. I hated TFS with some level of passion after using it for 4 years. It was slow, cranky and, well… merging was frequently a nightmare or sometimes actually impossible.

But one neat thing that TFS did provide was the concept of “shelve-sets”. For those that don’t know: these are basically uncommitted (or unchecked-in) changes, effectively your work-in-progress.

This feature was great because it let me:

  • develop on multiple PCs, and be able to easily migrate unfinished code changes between those PCs.
  • an immediate requirement came in that perhaps evaporated before the changes could be finished – but I suspected the requirement might spark up again a week or month later.
  • perform zero-risk code reviews without getting stuck in all the mud of the wider TFS system (this point clearly doesn’t apply to Mercurial nor any of the popular DVCS).

I’m not a big fan of the “Continuous Check-ins” movement that somebody seems to have started. It feels wrong to be committing stuff in unlogically grouped and unatomic units. And hell, the work-in-progress that you want to “shelve” may not even compile yet! Do I really want to be committing stuff that I might throw away later anyway? Maybe, if I was using Git which allows history rewriting far more easily. But Mercurial has a stricter policy around history and maintains a level of discipline to ensure an immutable history.

There is already an extension for Mercurial that performs a type of shelving. But this was not suitable for me. It maintains the storage folder locally on the PC and doesn’t provide any obviously supported means of copying that between PCs.

Mercurial provides (almost) everything you need out-of-the-box to be able to do a lightweight form of shelving, and you don’t even need to enable any extensions. What you need are the hg diff and hg import commands. Coupled with two shell scripts, and you will have a complete solution.

First setup your Mercurial.ini (or .hgrc) with the following useful aliases:

gitdiff = diff --git
exportwip = gitdiff
importwip = import --no-commit --force

The reason we use the extended diff format from Git is that, primarily, this supports binary files whereas a vanilla diff does not. And the reason for the --no-commit option on the importwip alias is to prevent the imported work-in-progress from being immediately committed to the repository. You definitely don’t want that!

Now from a shell prompt you can do:

$ hg exportwip > my-work-in-progress-for-xyz-customer-22-02-2011.diff

Then, at a later date, or immediately from another PC (once you’ve copied the diff file across or e-mailed it to yourself):

$ hg importwip my-work-in-progress-for-xyz-customer-22-02-2011.diff

Note: There is no need for piping the file input on the importwip alias because the real command already requires that as an input argument.

Not-so powerful, uh, PowerShell…

If you’re a PowerShell user (like myself) there is a rather large annoyance in the way it performs binary file output when piping. If you use the straight “>” output piping operator in PowerShell then it will default to use Unicode encoding. This sounds fine on the surface, except that Mercurial’s diff engine does not support Unicode. It outputs in UTF8 and expects any input it is given to also be in UTF8. And when I say it does not support Unicode – I mean it really doesn’t. It will give a vague error of:

abort: no diffs found

Additionally, if you try to workaround this by changing the encoding used by PowerShell, as follows:

PS $ hg exportwip | out-file -filepath "my-work-in-progress-for-xyz-customer-22-02-2011.diff" -encoding OEM -force

Then what you get is a file that looks suitable. And in fact it should be suitable. The only difference appears to be that the line endings have been normalised from \n (Mercurial’s raw output when producing diffs) to \r\n (Windows environment style). Unfortunately there is a bug, as of at least Mercurial 1.7.5, where it cannot handle diffs that use the \r\n style of line endings (despite having the eol extension enabled). What you will get is an ugly exception like the following:

** unknown exception encountered, please report by visiting
** Python 2.6.4 (r264:75708, Oct 26 2009, 08:23:19) [MSC v.1500 32 bit (Intel)]
** Mercurial Distributed SCM (version 1.7.5)
** Extensions loaded: fixfrozenexts, graphlog, eol, fetch, transplant, rebase, purge, churn, mq
Traceback (most recent call last):
  File "hg", line 36, in
  File "mercurial\dispatch.pyo", line 16, in run
  File "mercurial\dispatch.pyo", line 36, in dispatch
  File "mercurial\dispatch.pyo", line 58, in _runcatch
  File "mercurial\dispatch.pyo", line 590, in _dispatch
  File "mercurial\dispatch.pyo", line 401, in runcommand
  File "mercurial\dispatch.pyo", line 641, in _runcommand
  File "mercurial\dispatch.pyo", line 595, in checkargs
  File "mercurial\dispatch.pyo", line 588, in
  File "mercurial\util.pyo", line 426, in check
  File "mercurial\dispatch.pyo", line 297, in __call__
  File "mercurial\util.pyo", line 426, in check
  File "mercurial\extensions.pyo", line 130, in wrap
  File "mercurial\util.pyo", line 426, in check
  File "hgext\mq.pyo", line 2988, in mqcommand
  File "mercurial\util.pyo", line 426, in check
  File "mercurial\extensions.pyo", line 130, in wrap
  File "mercurial\util.pyo", line 426, in check
  File "hgext\mq.pyo", line 2960, in mqimport
  File "mercurial\util.pyo", line 426, in check
  File "mercurial\commands.pyo", line 2371, in import_
  File "mercurial\commands.pyo", line 2328, in tryone
  File "mercurial\patch.pyo", line 1259, in patch
  File "mercurial\patch.pyo", line 1231, in internalpatch
  File "mercurial\patch.pyo", line 1110, in applydiff
  File "mercurial\patch.pyo", line 1129, in _applydiff
  File "mercurial\patch.pyo", line 1028, in iterhunks
KeyError: 'mycompany.snk\r'

The guys in the Mercurial chat room are aware of this issue as we were all scratching our collective heads on it for an hour or two the other day. Here’s hoping for a bug fix soon! Alternatively, the whole issue could be bypassed if the hg diff command had an option such as “--out <filename>” to write the diff to a file directly. That would remove the dependency upon the shell for file output piping.

Tagged with: ,

CentOS 5.5 losing time synchronisation on Hyper-V R2

Posted in Unix Environment, Windows Environment by Nathan B. Evans on February 21, 2011

Recently I deployed a CentOS 5.5 x64 guest on a Server 2008 R2 Hyper-V host for running a basic mail server. However I quickly noticed that the guest was losing time sync very rapidly, in the order of several positive minutes per hour. This was a surprise as I had already installed the Hyper-V Linux Integration components from Microsoft which I had assumed would just take care of everything for me. Not so, apparently.

I might also add that Dovecot (an IMAP/POP3 server) kept crashing due to the periodic NTP sync being so far out that it resulted in the guest’s clock actually going BACK in time! This appears to be an acknowledged bug in Dovecot but it is understandable why they don’t feel a pressing need to fix it. Though I understand the 2.0 release has. For the record, I was running the release.

Anyway, it turned out that the solution was very simple.

Modify the /boot/grub/grub.conf as follows:

Editing the /boot/grub/grub.conf file.

Essentially you need to modify the lines that start with the word “kernel” and add two extra options onto the end:

  • clock=pit

    This sets the clock source to use the Programmable Interrupt Timer (PIT). This is a fairly low level way for the kernel to track time and it works best with Hyper-V and Linux.

  • notsc

    This is included more as belt-and-braces than anything. Because setting the PIT clock source (above) should already imply this setting really. But I include it for pure expressiveness 🙂

  • divider=10

    This adjusts the PIT frequency resolution to be accurate to 10 milliseconds (which is perfectly sufficient for most applications). This isn’t strictly required but it will reduce some CPU load caused by the VM. If the VM will be running time sensitive calculations a lot (such as say a VoIP server or gaming server) then you probably shouldn’t include this option.

Once you’ve done this, save the file and reboot the box.

The time should now be synchronized precisely with the Hyper-V host!

PS: I’ve not tested whether this solution will work without the Hyper-V Linux Integration Components installed, but I believe that it will. As it operates independently of Hyper-V’s clock synchronisation mechanism and relies purely on the virtualised Programmable Interrupt Timer that is exposed by Hyper-V. And the PIT clock source will remain unchanged whether you have the Linux Integration Components installed or not.

PPS: The VMware knowledge base has a great article on this subject: Although take it with a slight pinch of salt (because the subject is Hyper-V) but it certainly gives several more options and ideas to try for different Linux distributions and 32-bit vs 64-bit etc environments.

Tagged with: ,