Consider this:

public DateTime Timestamp {
    get {
        return Settings.Default.Timestamp;
    }
    set {
        Settings.Default.Timestamp = value;
        Settings.Default.Save();
    }
}

Would you agree that it looks cleaner like the following?

public DateTime Timestamp {
    get => Settings.Default.Timestamp;
    set => {
        Settings.Default.Timestamp = value;
        Settings.Default.Save();
    }
}

The biggest seller for it, for me at least, is that it removes a largely superfluous “return” statement on the getter.

C# properties are all about the syntactic sugar, so why not go the last mile?

You’ll log on to them one day and this dialog is sitting there waiting for you.

In this case, it is a customer’s server; so not really my problem (yet!). But still.

What is also interesting is that the server had only just came back up a few hours before from a batch of Windows Updates. So why didn’t AVG piggyback its updates onto that reboot?

Microsoft really needs to clamp down on all this seemingly legalised crapware. How does it even pass Windows Logo requirements?

Buoyed with enthusiasm of WebSockets, I set about implementing a simple test harness of a WebSockets server in C#.NET using System.Net.HttpListener. Unfortunately, things did not go well. It turns out that HttpListener (and indeed, the underlying HTTP Server API a.k.a. http.sys) cannot be used at all to develop a WebSockets server on current versions of Windows. The http.sys is simply too strict with its policing of what it believes to be correct HTTP protocol.

In an IETF discussion thread, a Microsoft fellow called Stefen Shackow was quoted as saying the following:

The current technical issue for our stack is that the low-level Windows HTTP driver that handles incoming HTTP request (http.sys) does not recognize the current websockets format as having a valid entity body. As you have noted, the lack of a content length header means that http.sys does not make the nonce bytes in the entity available to any upstack callers. That’s part of the work we will need to do to build websockets support into http.sys. Basically we need to tweak http.sys to recognize what is really a non-HTTP request, as an HTTP request.

Implementation-wise this boils down to how strictly a server-side HTTP listener interprets incoming requests as HTTP. For example a server stack that instead treats port 80 as a TCP/IP socket as opposed to an HTTP endpoint can readily do whatever it wants with the websockets initiation request.

For our server-side HTTP stack we do plan to make the necessary changes to support websockets since we want IIS and ASP.NET to handle websockets workloads in the future. We have folks keeping an eye on the websockets spec as it progresses and we do plan to make whatever changes are necessary in the future.

This is a damn shame. As it stands right now, Server 2008/R2 boxes cannot host WebSockets. At least, not whilst sharing ports 80 and 443 with IIS web server. Because, sure, you could always write your WebSocket server to bind to those ports with a raw TCP socket and rewrite a ton of boilerplate HTTP code that http.sys can already do, and then put up with the fact that you can’t share the port with IIS on the same box. This is something that most people, me included, do not want to do.

Obviously it isn’t really anyone’s fault because back in the development time frame of Windows 7 and Server 2008/R2 (between 2006 to 2009) they could not have foreseen the WebSockets standard and the impact it might have on the design of APIs for HTTP servers.

The good thing is that Windows 8 Developer Preview seems to have this covered. According to the MSDN documentation, the HTTP Server API’s HttpSendHttpResponse function supports a new special flag called HTTP_SEND_RESPONSE_FLAG_OPAQUE that seems to suggest it will put the HTTP session into a sort of “dumb mode” whereby you can pass-thru pretty much whatever you want and http.sys won’t interfere:

HTTP_SEND_RESPONSE_FLAG_OPAQUE

Specifies that the request/response is not HTTP compliant and all subsequent bytes should be treated as entity-body. Applications specify this flag when it is accepting a Web Socket upgrade request and informing HTTP.sys to treat the connection data as opaque data.

This flag is only allowed when the StatusCode member of pHttpResponse is 101, switching protocols. HttpSendHttpResponse returns ERROR_INVALID_PARAMETER for all other HTTP response types if this flag is used.

Windows Developer Preview and later: This flag is supported.

Aside from the new System.Net.WebSockets namespace in .NET 4.5, there are also clear indications of this behaviour being exposed in the HttpListener of .NET 4.5 through a new HttpListenerContext.AcceptWebSocketAsync() method. The preliminary documentation seems to suggest that this method will support 2008 R2 and Windows 7. But this is almost certainly a misprint because I have inspected these areas of the .NET 4.5 libraries using Reflector and it is very clear that this is not the case:

The HttpListenerContext.AcceptWebSocketAsync() method directly calls into System.Net.WebSockets.WebSocketHelper (static class) which has a corresponding AcceptWebSocketAsync() method of its own. This method will then call a sanity check method tellingly named EnsureHttpSysSupportsWebSockets() which evaluates an expression containing the words “ComNetOS.IsWin8orLater“. I need say no more.

It seems clear now that Microsoft has chosen not to back port this minor HTTP Server API improvement to Server 2008 R2 / Windows 7. So now we must all hope that Windows Server 8 runs a beta program in tandem with the Windows 8 client, and launches within a month of each other. Otherwise Metro app development possibilities are going to be severely limited whilst we all wait for the Windows Server product to host our WebSocket server applications! Even still, it’s a shame that Server 2008 R2 won’t ever be able to host WebSockets.

It will be interesting if Willie Tarreau (of HA-Proxy fame) will come up with some enhancements in his project that might benefit those determined enough to still want to host (albeit, raw TCP-based) WebSockets on Server 2008 R2.

Where will WebSockets be on this InfoQ chart, in three years time?

On 8th December 2011, a little known (but growing in awareness) standard called “WebSockets” was upgraded to W3C Candidate Recommendation status. That is one small step shy of becoming a fully ratified web standard. And just to remove any remaining possible doubt: Next-gen platforms and frameworks such as Windows 8 and .NET 4.5 (at three levels: System.Net, WCF and ASP.NET) already have deeply nested support, and they aren’t even beta  yet!

After reading up about the standard in detail and absorbing the various online discussion around it, it is becoming increasingly clear to me that this standard is going to steal a large chunk of mind share from RESTful web services. What I mean is that there will come a stage in product development where somebody will have to ask the question:

Right guys, shall we use WebSockets or REST for this project?

I expect that WebSockets will, within a year or two, begin stunting the growth of RESTful web services – at least as we know them today.

What are WebSockets?

They are an overdue and rather elegant protocol extension for HTTP 1.1 that allows what is fundamentally a bi-directional TCP data stream to be tunnelled over a HTTP session. They provide a built-in implementation of TCP message framing, so developers don’t need to worry about any boilerplate code stuff like that when designing their application protocol.

Why are WebSockets a threat to RESTful web services?

From the last few years of working on projects that expose RESTful web services, I have noticed a few shortcomings. I should probably make clear that I’m not claiming that WebSockets answers all those shortcomings. I’m merely suggesting that REST is not the silver bullet solution that it is often hyped up to be. What I am saying is that there is definitely space for another player that can still operate at “web scale”. WebSockets have more scope to be a little more like a black box or quick’n'dirty solution than REST which requires a more design up-front approach due to versioning and public visibility concerns. Always use the correct tool for the job, as they say.

Sub-par frameworks

They might claim REST support but they still haven’t truly “groked” it yet, in my opinion. WCF REST is a good example of this. Admittedly, the WCF Web API for .NET is starting to get close to where things should be, but it is not yet production ready.

Perhaps even more serious is the lack of widespread cross-platform RESTful clients that work in the way that Roy prescribed; of presenting an entry point resource that allows the client to automatically discover and autonomously navigate between further nested resources in a sort of state machine fashion. A single client framework that can operate with hundreds of totally different RESTful web services from different organisations. This does not exist yet, today. This is why so many big providers of RESTful web services end up seeding their own open source projects in various programming languages to provide the essential REST client.

Enterprise loves SOAP (and other RPCs)

Third-parties that want to use your web services often prefer SOAP over REST. Many haven’t even heard of REST! WebSockets are a message-based protocol allowing for SOAP-like RPC protocols that enterprise seem to adore so much. Hell there’s nothing stopping actual SOAP envelopes being transferred over a WebSocket!

This might not be the case if you’re operating in an extremely leading edge market such as perhaps cloud computing where everyone is speaking the same awesomesauce.

Complex domain models

Mapping out complex domain models onto REST can be slow and labourious. You’ll find yourself constantly having to work around its architectural constraints. Transactions, for example, can be a particular problem. Of course, this is partly related to the first problem (sub-par frameworks) but one cannot reasonably expect transaction support in a REST framework. What is probably needed is a set of common design patterns for mapping domain models to REST. And then an extension library for the framework that provides reusable implementations of those patterns. But alas, none of this exists yet.

Text-based formats

JSON/XML (for reasons unknown) are commonly used with REST and these are of course text-based formats. This is great for interoperability and cross-platform characteristics. But it is not so great for memory and bandwidth usage. This especially has implications on mobile devices.

You’ll find yourself running into walls if you try to use something that isn’t JSON or XML, at least that is my experience with current frameworks.

Request-response architecture of HTTP

Fundamentally, REST is nothing more than a re-specification of the way HTTP works and a proposal of a design pattern to build applications on top of HTTP. This means it retains the same statelessness and sessionless characteristics of HTTP. It therefore precludes REST from being bi-directional where the server could act as the requester of some resource from the client, or sender of some message to the client. As a result it requires “hacks” to be used to emulate server-side events, and these hacks have bad characteristics such as high latency (round trip time) and are wasteful of battery life.

Public visibility, versioning concerns

Sometimes having everything publicly visible is not what you want. People start using APIs that you don’t want them to use yet. You have to design everything to the nth degree much more. Have a proper versioning strategy in place. It encourages a more discerning approach to software development, that is for sure. Whilst these are usually good things, they can be a hindrance on early stage “lean agile” projects.

What can WebSockets do that is so amazing?

The fact that there will soon be a second player in this space suggests that there will be rebalancing of use-cases. WebSockets will prove to be disruptive for several reasons:

True bi-directional capability and server-side events, no hacks

Comet, push technology, long-polling etc in web apps are slow, inefficient, inelegant and have a higher potential magnitude for unreliability. They often work by requesting a resource from the server, causing the server to block until such a time that an event (or events) need to be transferred back to the client. They can be unreliable because the TCP connection could be teared down by a intermediate router during the time it is waiting for the response. Or worse, a proxy server might deliberately  time out the long-running request. As such, many implementations of this hack will use some kind of self-timeout mechanism so that perhaps every 60 seconds they will reissue the request to the server anyway. This has implications on both bandwidth and battery usage.

The true bi-directional capability offered by WebSockets is a first for any HTTP-borne protocol. It is something that neither SOAP nor REST have. And which Comet/push/long-polling can only emulate, inefficiently. The bi-directional capability is inherently so good that you could tunnel a real-time TCP protocol such as Remote Desktop or VNC over a WebSocket, if you wanted.

Great firewall penetration characteristics

WebSockets can tunnel out of heavily firewalled or proxied environments far easier than many other RPC designs. I’m sure I’m not alone in observing that enterprise environments rarely operate their SOAP services on port 80 or 443.

If you can access the web on port 80 without a proxy, WebSockets will work.

If you can access the web on port 80 with a proxy, WebSockets should work as long as the proxy software isn’t in the 1% that are broken and incompatible.

If  you can access the web on port 443 with or without a proxy, WebSockets will work.

I strongly suspect that there will be a whole raft of new Logmein/Remote Desktop and VPN solutions that are built on top of WebSockets, purely because of the great tunnelling characteristics.

Lightweight application protocols and TCP tunnelling

There is the potential for extremely lightweight application protocols, in respect of performance, bandwidth and battery usage. Like REST, the application schema/protocol isn’t defined by the standard; it is left completely wide open. WebSockets can transfer either text strings or binary data. It is clear that the text string support was included to aid in transferring JSON messages to JavaScript engines which lack the concept of byte arrays. Whilst the binary support will be most useful tunnelling TCP streams or for custom RPC implementations. After a WebSocket session is established, the overhead per message can be as small as just two bytes (!). Compare that to REST which has a huge HTTP header to attach to every single request and response.

How will the use-cases of REST change?

I believe that REST will lose a certain degree of its lustre. Project teams will less eagerly adopt it if they can get away with a bare bones WebSocket implementation. REST will probably remain the default choice for projects that need highly visible and cross-platform interoperable web services.

Projects without those requirements will probably opt for WebSockets instead and either run JSON over it, or use a bespoke wire protocol. They will particularly be used by web and mobile applications for their back-end communications i.e. data retrieval and push-events. Windows 8 “Metro” applications will need to use them extensively.

I suppose you could summarise that as:

• REST will be (and remain) popular for publicly visible interfaces.
• WebSockets will be popular for private, internal or “limited eyes only” interfaces.

Note: By “public” and “private” I am not referring literally to some form of paid/subscription/membership web service. I am referring to the programming API contract and its level of exposure to eyes outside of your development team or company.

Conclusion

Even though they are competing, the good thing is that REST and WebSockets can actually co-exist with one another. In fact, because they are both built upon HTTP fundamentals they will actually complement each other. A RESTful hypermedia resource could actually refer to a WebSocket as though it were another resource through a ws:// URL. This will pave the way for new RESTful design patterns and framework features. It will allow REST to remedy some of its shortcomings, such as with transaction support; because a WebSocket session could act as the transactional unit of work.

The next year is going to be very interesting on the web.

SMS is a very simple protocol; there is only three essential things you need to understand:

1. It is limited to 160 characters per message, if you use the GSM 03.38 7-bit character set.
2. It is limited to 70 characters per message, if you use the UCS-2 (a.k.a. Unicode, UTF-16) character set.
3. Multiple messages can be joined together to form a multi-part message by including a special concatenation header, which eats up 6 or 12 characters (depending on whether you’re using GSM or UCS-2 character set). Most phones these days refer to this concept on their GUI as “pages”.

Unfortunately the protocol is severely handicapped for when it comes to building automated two-way applications, and here’s why:

It does not provide any facility, not even an extension standard or extension point, for performing reply correlation.

What do I mean by “reply correlation”? It is a simple concept. Assume that you send a question to a buddy, and then he responds to you with the answer. One might hope expect that the message containing the answer contains some sort of ID code, token or cookie (hidden away in its header information, of course) that relates it to the original question message. Unfortunately, it does not and this is the problem; SMS does not include any such ID/token/cookie, anywhere. It simply wasn’t included in neither the original standard nor any subsequent revisions or extensions of the standard.

It is not necessarily the creators fault because clearly they couldn’t foresee how ubiquitous SMS would become. But there is evidence that they did recognise and respond to its popularity in the late 1990′s very quickly by publishing new standard extensions that built upon SMS, such as multi-part messages and WAP. So one can only wonder why they didn’t make an extension that would allow replies to be correlated with their original message. And unfortunately the window of opportunity to actually get this sorted out was over a decade ago, so we’re pretty much screwed then and will have to make do with it.

This is a big problem for SMS. It makes the process of building two-way fully automated applications much more difficult. Very very few companies have actually managed the solve the problem, and those that have tend to be very small or operating in niché markets. I don’t understand this at all, because the possibilities and prospects for building two-way SMS applications are absolutely huge, almost endless.

One of my key responsibilities over the last 8 years has been in devising production-ready solutions that work around this problem and this blog post is going to summarise all of them.

An overview of the solution

The key to solving this problem lies with two fields contained within the header information of every SMS message: the source and destination address. Or what I call the “address pairing”.

By using the address pairing in an intelligent way we can find the right compromise for a particular two-way application. Essentially, whenever the application needs to send a question to a mobile phone number it must ensure that no existing question is already outstanding on the same address pairing.

There are several ways that an application can be designed around this basic concept.

Solution #1: My application only has one source address

The application must be designed to “serialise” the transmission of questions. It can use a mutual exclusion mechanism that will prevent itself sending a further question to the same mobile phone number if an outstanding question is still waiting for a response. It can be expected that some characteristics of a “transaction” or “transactional unit of work” would be adopted in the design of the application to model this mutual exclusion concept.

My past implementations of this pattern were based on a database table with a composite primary key between both the “source” and “destination” columns. The application would try to insert the address pairing into this table and, if successful, it continues sending the message. But if the insertion were to fail then it would realise that a question is already outstanding with the mobile phone number, prompting it to give up and retry later. Or rather than retrying later based on some timer mechanism, you might enqueue it as a job somewhere; so that when a response for the outstanding question is received the application can check the queue for further jobs for that address pairing and dequeue/execute the top job.

There is a caveat with this solution however, and it comes as a side affect of “serialising” the questions one after the other. What if it takes days or weeks for the person to respond to the question? The questions that are queued up waiting to acquire a lock on the address pairing are going to get pushed back and back. They could get pushed back so far that the premise of the question has been entirely voided (e.g. an appointment reminder/confirmation).

The solution to this problem is to introduce a further concept of a “timeout” value. This will ensure that any question sent to the mobile phone can only be outstanding for up to a designated time period. You would probably typically set this to around 24-48 hours, but some questions that contain more time sensitive content may use a lower value of between 1-4 hours.

It is important (though not essential) that when implementing the timeout value concept that you use the “Validity Period” field that is available in every outbound SMS message. You should set the validity period to roughly match what your timeout value for that question will be. This will help ensure messaging integrity in the event that, for example, the mobile phone is turned off for a week and when it is turned back on then you don’t want your “expired” questions to be delivered when your back-end application has already timed out the workflow that was running for that question.

Solution #2: My application can have multiple source addresses

The idea is that you would have a relatively large pool of source addresses, perhaps as many as 50 or 100. Your application would, as with Solution #1, maintain some kind of database table or data structure that prevents duplicate address pairings. The application would then have some logic that enables it to “select” a free source address i.e. a source address that is not “in use” for the destination mobile phone number.

It would still be advisable to implement some kind of “timeout” mechanism, as with Solution #1, but the advantage would be that you would be able to have substantially greater timeout periods. Possibly in the order of weeks or months. Really the timeout mechanism here would be acting more as a type of garbage collector, than as a question expedite governor as in Solution #1.

I’ve always considered that this solution is better suited to applications that provide a “shared” or cloud service of some kind. Simply because setting up a large pool of dedicated source addresses for each of your application’s customers is surely going to get painful.

This solution does have the disadvantage that end-users on their mobile phone will be communicating, potentially, with lots of different source addresses even though it is really the same company/application at the other end. It can mess up the user’s normal “texting” experience, it would rob them of their iPhone’s “bubble chat” GUI style of presentation and the ability of perhaps creating a Contact list entry for a regular contact. Obviously there are things you can do to try to minimise this risk, such as always trying to select the source address with the lowest index. But really I think that will just make things worse. At some point you WILL want to send multiple questions to a mobile phone number, and there’s no getting around that fact. If you’ve got a large pool of source addresses then you’re going to want to use them.

Solution #3: My application only has one source address, but I need to send concurrent questions to the same mobile phone

You can’t. Well you can, but I don’t recommend it at all. I tried it once, on an early version of our system, and our customers didn’t like it.

Essentially you combine the concepts detailed in Solution #1 and then rely on some text processing logic in your response handling code. So rather than perhaps phrasing your question like “Are you attending the meeting tomorrow? Reply with A=Yes or B=No.” You’d phrase it as “… Reply with A1=Yes or B1=No”. Notice the “1″ digit in there? That’s the key bit. That digit refers to a transaction code that will be used for correlation. My implementation of this basically went from zero to nine, so you could have a total of 10 concurrent questions open with the same mobile number.

I don’t like this solution for the following reasons:

• Many end-users forget to include the essential digit in their reply. They might reply “A” instead of “A1″. I’ve seen this happen in the wild.
• Accessing digits on mobile phones when typing a SMS message is often an unintuitive process. Even an iPhone needs you to access a sub-keyboard screen. Blackberry’s need you to hit the ALT key.
• It prohibits your application from accepting literal text responses. Many users would simply reply “Yes” rather than “A” or “A1″. If they do this, your application would be screwed because it wouldn’t have the essential digit to correlate the reply with the original question. I’ve seen this happen in the wild.
• It prohibits your application from accepting “freeform” text responses. You might want to send a question like “What is your full name?”. There’s no way you can tag on the end of that a list of options. It simply doesn’t make sense.
• It reveals implementation details onto the user interface of your application. Not good.
• It compromises messaging integrity. An end-user might inadvertently reply (or possibly even deliberately!) with an incorrect digit.
• It requires both the “reply analysis/text processing” and “reply correlation” concerns of your application to be interdependent on each other, when really they should not be – at least not to perform something so simple.

Experimental solutions

On the last bullet point of Solution #3 I suggested that your application’s “reply analysis” and “reply correlation” concerns shouldn’t be linked together. This I believe is true for something as simple as what was described in that solution. However, there is plenty of mileage to be explored in adopting this approach for more advanced designs.

When you send a question with a constrained set of response options such as “Yes, No, Maybe”, you might want to record these as part of your address pairing in the database or data structure (as described in Solution #1). Then if you need to send a further question (to the same mobile phone, whilst the first question is still outstanding) you can check if the set of response options are different. This question might be looking for a “Good, Bad, Ugly” response. In which case there is no conflict, is there? So a lock on the address pairing, based upon those expected response options, can be allowed to be acquired. Obviously this wouldn’t be possible (or at least would have ramifications on your overall design) if you were expecting a “freeform” response.

Another possible avenue to be explored is an area of computer science called “natural language processing“. The idea is that when you ask a question like “What is your name?” then you would prime your NLP engine to be expecting a reply that looks like somebody’s name. Anything that arrives from that mobile phone that doesn’t look like a person’s name can be assumed to not be related to the outstanding question. Obviously if you want to ask a concurrent question like “What is your wife’s name?” then you’re back to square one. Because that would be a conflict and you’d need to serialise the questions as described in Solution #1. This (NLP and SMS applications) is an active area of research for me, so I may blog about it in more detail at a later time.

Conclusion

Solution #1 is the best, for now. It strikes the right level of compromise without sacrificing neither messaging integrity nor user friendliness. If you desperately need to send multiple concurrent questions to a mobile phone then I would suggest that you should rethink your approach. Perhaps logically separating your business departments and/or workflow concerns onto different source addresses would be a solution in this case. That way you can send out an urgent question, perhaps relating a missed bill payment, on a source address that is dedicated for that purpose.

Solution #2 is usable, and I can think of several use-cases. But I feel it is not as good for frequent one-to-one contact between a company and their customers. It has serious disadvantages in user friendliness. It is best suited to a hosted cloud service of some kind, where everyone shares the same pool of source addresses and where contact is expected to be infrequent.

As it happens the “trigger” for me this time around was not a new project but more as a result of our switch from TFS to Mercurial for our source control. It took some time for Mercurial to “bed in”, but it definitely has now. So then you reach that point where you start asking “Okay, so what else can we do with this VCS to improve the way we work?”.

Our first pass at automated versioning was to simply copy the status quo that worked with TFS. This was rather crude at best. Basically we had a MSBuild task that would increment (based on our own strategy) the AssemblyFileVersionAttribute contained inside the GlobalAssemblyInfo.cs. The usual hoo-har really, involving a simple regular expression etc. This was fine. However we did not really like it because it was, effectively, storing versioning information inside of a file held inside the repository. Separation of concerns and all that. It also caused a small amount of workflow overhead involving merging of named branches – with the occasional albeit easily resolvable conflict. Not a major issue, but not ideal either. Of course, not all projects use named branches. But we do; as they’re totally awesome for maintaining many concurrently supported backward releases.

Versioning strategies

The way I see it, there is only a small number of routes you can go down with project versioning:

1. Some sort of yyyy.mm.dd timestamp strategy.
   This is great for “forward only” projects that don’t need to maintain supported backward releases. So for web applications, cloud apps etc – this is, I suspect, quite popular. For other projects, it simply doesn’t make sense. Because if you want to release a minor bug fix for a release from over a year ago it wouldn’t make any sense for the version to suddenly jump to the current date. How would you differentiate between major product releases?
2. The typical major.minor.build.revision strategy.
   The Major and Minor would be “fixed” for each production branch in the repository. And then you’d increment the build and/or revision parts based on some additional strategy.
3. A DVCS-only strategy where you use the global changeset hash.
   Unfortunately this is of limited use today on .NET projects because both the AssemblyVersionAttribute and AssemblyFileVersionAttribute won’t accept neither a string nor a byte array. Of course there is nothing stopping you coming up with your own Attribute (we called ours DvcsIdentificationAttribute) and including/updating that in your GlobalAssemblyInfo.cs (or equivilent) whenever you run a build. But it is of zero use to the .NET framework itself.
4. Some sort of hybrid between #1 and #2 (and possibly even #3!).
   This is what we do. We use a major.minor.yymm.revision strategy, to be precise.

We like our #4 hybrid strategy because it brings us the following useful characteristics:

• It has “fixed” Major and Minor parts. Great for projects with multiple concurrent versions.
• It contains a cursory Year and Month that can be read from a glance. When chasing down a problem on a customer environment it is simple things like this that can speed up diagnosis times.
• An incremental Revision part that ensures each build in the same month has a unique index.

So then, how did we implement this strategy on the Microsoft stack and with Mercurial?

Implementation

The key to implementing this strategy is first and foremost with retrieving from the repository the “most recent” tag for the current branch. Originally I had big plans here to go write some .NET library to walk the Mercurial revlog file structure. It would have been a cool project to learn some of the nitty gritty details of how Mercurial works under the hood. Unfortunately, I soon discovered that Mercurial has a template command available that already does what I need. It’s called the “latesttag” template. It’s really simple to use as well, for example:

$ hg parents --template {latesttag}
> v5.4.1105.5-build

There is also a related and potentially useful template called “latesttagdistance“. This will count, as it walks the revlog tree, the number of changesets that it walks past in search for the latest tag. It is possible that you could use this value as the incrementation extent for the Revision part in the strategy.

At this point most of the Mercurial fan base will go off and write a Bash script or Python script to do the job. Unfortunately in .NET land it’s not quite that simple, as we all know. I could have written a, erm, “posh” Powershell script to do it, for sure. But then I have to wire that in to the MSBuild script – which I suspect would be a bit difficult and have all sort of gotchas involved.

So I wrote a couple MSBuild tasks to do it, with the interesting one aptly named as MercurialAssemblyFileVersionUpdate:

public class MercurialAssemblyFileVersionUpdate : AssemblyFileVersionUpdate {

    private Version _latest;

    public override bool Execute() {
        var cmd = new MercurialCommand {
            Repository = Path.GetDirectoryName(BuildEngine.ProjectFileOfTaskNode),
            Arguments = "parents --template {latesttag}"
        };

        if (!cmd.Execute()) {
            Log.LogMessagesFromStream(cmd.StandardOutput, MessageImportance.High);
            Log.LogError("The Mercurial Execution task has encountered an error.");
            return false;
        }

        _latest = ParseOutput(cmd.StandardOutput.ReadToEnd());

        return base.Execute();
    }

    protected override Version GetAssemblyFileVersion() {
        return _latest;
    }

    private Version ParseOutput(string value) {
        return string.IsNullOrEmpty(value) || value.Equals("null", StringComparison.InvariantCultureIgnoreCase)
                   ? base.GetAssemblyFileVersion()
                   : new Version(ParseVersionNumber(value));
    }

    private string ParseVersionNumber(string value) {
        var ver_trim = new Regex(@"(\d+\.\d+\.\d+\.\d+)", RegexOptions.Singleline | RegexOptions.CultureInvariant);

        var m = ver_trim.Match(value);
        if (m.Success)
            return m.Groups[0].Value;

        throw new InvalidOperationException(
            string.Format("The latest tag in the repository ({0}) is not a parsable version number.", value));
    }
}

Click here to view the full source, including a couple dependency classes that you’ll need for the full solution.

With that done, it was just a case of updating our MSBuild script to use the new bits:

<Target Name="incr-version">
  <MercurialExec Arguments='revert --no-backup "$(GlobalAssemblyInfoCsFileName)"' />
  <Message Text="Updating '$(GlobalAssemblyInfoCsFileName)' with new version number..." />
  <MercurialAssemblyFileVersionUpdate FileName="$(GlobalAssemblyInfoCsFileName)">
    <Output TaskParameter="VersionNumber" PropertyName="VersionNumber" />
    <Output TaskParameter="MajorNumber" PropertyName="MajorNumber" />
    <Output TaskParameter="MinorNumber" PropertyName="MinorNumber" />
    <Output TaskParameter="BuildNumber" PropertyName="BuildNumber" />
    <Output TaskParameter="RevisionNumber" PropertyName="RevisionNumber" />
  </MercurialAssemblyFileVersionUpdate></strong>
  <Message Text="Done update to '$(GlobalAssemblyInfoCsFileName)'." />
  <Message Text="Tagging current changeset in local repo." />
  <MercurialExec Arguments="tag --force v$(VersionNumber)-build" />
  <Message Text="Pushing commit to master server." />
  <MercurialExec Arguments='push' />
  <Message Text="All done." />
</Target>

Of course, don’t forget to include your tasks into the script, ala:

<UsingTask AssemblyFile="Build Tools\AcmeCorp.MsBuildTasks.dll"
           TaskName="AcmeCorp.MsBuildTasks.MercurialAssemblyFileVersionUpdate" />

<UsingTask AssemblyFile="Build Tools\AcmeCorp.MsBuildTasks.dll"
           TaskName="AcmeCorp.MsBuildTasks.MercurialExec" />

You’ll notice that the parsing implementation is quite forgiving. It is a regular expression that will extract anything that looks like a parsable System.Version string. This is cool because it means the tags themselves don’t have to be exactly pure as System.Version would need them. You can leave a “v” in front, or add a suffix like “-build”. Whatever your convention is, it just makes things a bit more convenient.

Remaining notes

The implementation above will generate global tags that are revision controlled in the repository. I did consider using “local tags” as those wouldn’t need to be held in the repository at all and could just sit on the automated build server only. However unfortunately the “latesttag” template does not work with local tags, it only appears to work with global tags. It would also of course mean that developers wouldn’t benefit from the tags at all, which would be a shame.

Mercurial stores global tags inside a .hgtags file in the root working directory. The file is revision controlled but only for auditing purposes. It does not matter which version you have in your working directory at any given time.

You may still require a workflow to perform merges between your branches after running a build, in order to propagate the .hgtags file immediately (and not leave it to someone else!). If any merge conflicts arise you should take the changes from both sides as the .hgtags file can be considered “cumulative”.

Related reading

http://kiln.stackexchange.com/questions/2194/best-practice-generating-build-numbers
http://stackoverflow.com/questions/4726257/most-recent-tag-before-tip-in-mercurial
http://www.jaharmi.com/2010/03/24/generate_version_numbers_for_mac_os_x_package_installers_with_mercurial_and_semantic_vers
http://mercurial.selenic.com/wiki/NearestExtension

After playing around a lot with all the lifetime services stuff like the ISponsor interface to do sponsorship with nasty hacky timeout values etc (which would have had side-affects for our product, but which we were about to resign ourselves too!) we came across a much better alternative solution in the form of RemotingServices.Disconnect(). Hoorah. Is it just me or does the whole remoting story in .NET need a damn good overhaul? Cross-AppDomain communications deserves something better.

With this discovery, I came up with a useful class that improves MarshalByRefObject  by adding deterministic disposal of both itself and any such nested objects (which is more an implementation detail for us but I’m sure could be useful for anyone).

/// <summary>
/// Enables access to objects across application domain boundaries.
/// This type differs from <see cref="MarshalByRefObject"/> by ensuring that the
/// service lifetime is managed deterministically by the consumer.
/// </summary>
public abstract class CrossAppDomainObject : MarshalByRefObject, IDisposable {

    private bool _disposed; 

    /// <summary>
    /// Gets an enumeration of nested <see cref="MarshalByRefObject"/> objects.
    /// </summary>
    protected virtual IEnumerable<MarshalByRefObject> NestedMarshalByRefObjects {
        get { yield break; }
    }

    ~CrossAppDomainObject() {
        Dispose(false);
    }

    /// <summary>
    /// Disconnects the remoting channel(s) of this object and all nested objects.
    /// </summary>
    private void Disconnect() {
        RemotingServices.Disconnect(this);

        foreach (var tmp in NestedMarshalByRefObjects)
            RemotingServices.Disconnect(tmp);
    }

    public sealed override object InitializeLifetimeService() {
        //
        // Returning null designates an infinite non-expiring lease.
        // We must therefore ensure that RemotingServices.Disconnect() is called when
        // it's no longer needed otherwise there will be a memory leak.
        //
        return null;
    }

    public void Dispose() {
        GC.SuppressFinalize(this);
        Dispose(true);
    }

    protected virtual void Dispose(bool disposing) {
        if (_disposed)
            return;

        Disconnect();
        _disposed = true;
    }

}

It was then just a case of modifying a few of our classes to derive from this instead of MarshalByRefObject and then update a couple other locations in our codebase by ensuring that Dispose() was called during clean-up.

Hyper-V's VMConnect tool displaying the disconnection error message

The full description of the error was as follows:

The connection to the virtual machine was lost. This can occur when a virtual machine stops unexpectedly, or when network problems occur. Try to connect again. If the problem persists, contact your system administator.

Would you like to try to reconnnect?

This was really annoying because we were connecting to local VMs that were present on the exact same VM host from which we were connecting. So presumably there wouldn’t be any packets hitting the network, and thus ruling out any of the new hardware and network changes we had just made.

After racking my brains on it for a bit (which included firing up Wireshark to perform a sanity check), I loaded up TCPView. This is a really great little tool from Mark Russinovich‘s stable called Windows Sysinternals. With this tool running I then retried the VMConnect, so that I could see what socket activities it was performing.

TCPView shows the VMConnect appearing to use TCP/IP V6 even for localhost

What this showed is that even when connecting to the local VM host using “localhost” or “127.0.0.1” as the address (i.e. IPv4) the VMConnect tool was seemingly transforming this into a IPV6 address and then forming a TCPV6 connection. This was interesting.

I immediately went to check whether IPV6 was actually enabled on the VM host’s network adapters. Low and behold, it was not. Turns out that when we flicked over the Gateway IP to point to the new firewall, we also subconciously turned off the IPV6 protocol on the list! A fairly innocuous thing to do, one would think, especially on an internal LAN!

Hyper-V's virtual network adapter TCP/IP settings

So there you have it. If you come across this problem with Hyper-V, I would recommend you immediately check to ensure that you have not inadvertently disabled the IPV6 protocol on your virtual network adapter for Hyper-V.

The very moment we re-enabled IPV6, the problem with VMConnect constantly disconnecting every few seconds totally went away!

Not many problems get more obscure than this.

The option nolinger is considered harmful.

I read somewhere that this option should be enabled because it frees up socket resources quicker and doesn’t leave them lying around when blatently dead. I enabled it and thought nothing more of it. Having forgot I had done so, I then started noticing strange behaviours. Most tellingly was that HA-Proxy’s webstats UI would truncate abruptly before completing. Fortunately, Willy Tarreau (the author/maintainer) was very quick to respond to my pestering e-mails and after seeing my Wireshark trace he immediately had a few ideas of what could be causing it. After following his suggestion to avoid using the “no linger” option, I removed it from my configuration and the problem went away.

Therefore: “option nolinger considered harmful.” You’ve be warned!

Webstats UI has “hidden” administrative functions

While reading the infamous “wall of text” that is the HA-Proxy documentation, I came across a neat option called “stats admin“. It enables a single piece of extra functionality (at least it does in v1.4.11) that will let you flag servers as being online or offline. This is useful if you’re planning to take one or more servers out of a backend’s pool, for maintenance possibly. I would wager that Willy intends to add more administrative features in the future so adding this one to your config now could save you some time in the future.

Of course, it is not likely that you will want such a sensitive function to be exposed to everyone that uses webstats. So it is fortunate then that this option supports a condition expression. I set mine up like the following:

userlist UsersFor_HAProxyStatistics
  group admin users admin
  user admin insecure-password godwouldntbeupthislate
  user stats insecure-password letmein

listen HAProxy-Statistics *:81
  mode http
  stats enable
  stats uri /haproxy?stats
  stats refresh 60s
  stats show-node
  stats show-legends
  acl AuthOkay_ReadOnly http_auth(UsersFor_HAProxyStatistics)
  acl AuthOkay_Admin http_auth_group(UsersFor_HAProxyStatistics) admin
  stats http-request auth realm HAProxy-Statistics unless AuthOkay_ReadOnly
  stats admin if AuthOkay_Admin

Request/response rewriting is mutually exclusive of keep-alive connections

At least in current versions, HA-Proxy doesn’t seem to be able to perform rewriting on connections that have been kept alive. It is limited to analysing only the first request and response. Any further requests that occur on that connection will go unanalysed. So if you are doing request or response rewriting, it is imperative that you set a special option to ensure that a connection can only be used once.

In my case, I just added the following to my frontend definition.

option http-server-close

Identifying your frontend from your backend

I was creating some rules to ensure that a particular URL could only be accessed through my HTTPS frontend. I wanted to prevent unencrypted HTTP access to this URL because it was using HTTP Basic authentication which uses clear text passwords across the wire.

Fortunately, HA-Proxy supports a fairly neat way of doing this by the means of tagging your frontend with a unique identifier which can then be matched against by the backend.

First of all, I setup my frontends like the following:

frontend Public-HTTP
  id 80
  mode http
  bind *:80
  option http-server-close
  default_backend Web-Farm

frontend Public-HTTPS
  id 8443
  mode http
  # Note: Port 8443 because the true 443 is being terminated by Stunnel, which then forwards to this 8433.
  bind *:8443
  option http-server-close
  default_backend Web-Farm

Then in my backend I cleared a space for defining “reusable” ACLs and then added the protective rule for the URL in question:

backend Web-Farm
  mode http
  balance roundrobin
  option httpchk
  server Web0 172.16.61.181:80 check
  server Web1 172.16.61.182:80 check

  # Common/useful ACLs
  acl ViaFrontend_PublicHttp fe_id 80
  acl ViaFrontend_PublicHttps fe_id 8443

  # Application security for: /MyWebPage/
  acl PathIs_MyWebPage path_beg -i /mywebpage
  http-request deny if PathIs_MyWebPage !ViaFrontend_PublicHttps

The piece of magic that makes this all work is the fe_id ACL criterion. Note that the “fe” stands for “frontend”.

Note the http-request deny rule is comprised of two ACLs, by boolean AND’ing them. HA-Proxy defaults to AND’ing. If you want to OR just type “or” or “||“. Negation is done in the normal C way by using an exclamation symbol, as shown in the above example. I seem to like avoiding the use of the “unless” statement as I prefer the explicitness of using “if” and then using negation. But that’s just my personal preference as a long-time coder

Now if a user tries to visit http://.../MyWebPage they will get a big fat ugly 403 Forbidden error.

HTTP Basic authentication is finally very basic to do!

I came across a stumbling block this week. I assumed that Microsoft IIS, one of the best web servers available, could do HTTP Basic authentication i.e. clear text passwords over the wire and then validating against some sort of clear text password file or database. Turns out that while IIS does support HTTP Basic auth’, it doesn’t support any form of simple backend. You have to validate against either the web servers local Windows user accounts, or against Active Directory. Great. The web page in question was just a little hacky thing we knocked up to get a customer of ours out of a hole. We didn’t want to be creating maintenance headaches for ourselves by creating a local user account on each web server in the farm, nor did we fancy creating them an AD account. They don’t even belong to our company!

Fortunately (that word again), and despite how poorly documented it is, HA-Proxy *does* support this!

First of all you need to create a userlist that will contain your users/groups that you will authenticate against:

userlist UsersFor_AcmeCorp
  user joebloggs insecure-password letmein

Then in your backend, you need to create an ACL that uses the http_auth criterion. And lastly, create an http-request auth rule that will cause the appropriate 401 Unauthorized and WWW-Authenticate: Basic response to be generated if the authentication has failed.

backend HttpServers
  .. normal backend stuff goes here as usual ..
  acl AuthOkay_AcmeCorp http_auth(UsersFor_AcmeCorp)
  http-request auth realm AcmeCorp if !AuthOkay_AcmeCorp

Remove sensitive IIS / ASP.NET response headers

Security unconscious folk need not apply.

It’s a slight security risk to be leaking your precise IIS and ASP.NET version numbers. Whilst these can be turned off in IIS configuration, it is more a concern for your frontend load balancer i.e. HA-Proxy. The reason I believe this is because the headers can be useful debugging on the internal LAN/VPN inside your company. Only when the headers are about to touch the WAN does it become dangerous. Therefore:

frontend Public-HTTP
  # Remove headers that expose security-sensitive information.
  rspidel ^Server:.*$
  rspidel ^X-Powered-By:.*$
  rspidel ^X-AspNet-Version:.*$

HTTPS and separation of concerns

I don’t know about Apache, but IIS 7.5 can have some annoying (but arguably expected) behaviours when HA-Proxy is passing traffic where the client believes it has an end-to-end HTTPS connection with the web server. My setup involves Stunnel terminating the SSL connection and then from that point on it is just standard HTTP traffic to the backend servers. This means the backend servers don’t actually need to be listening on HTTPS/443 at all. However when GET requests come in to them using the https:/ scheme they can get a bit confused (or argumentative, I’m undecided). IIS seems to like sending back a 302 Moved Permanently response, with a Location header that uses the http:/ scheme. So then of course the web browser will follow the redirect to either a URL that doesn’t exist or one which does exist but is already merely a redirect to the https:/ scheme! Infinitely loop anyone?

The way to solve this is request rewriting, through some clever use of regular expressions.

frontend Public-HTTPS
  id 8443
  mode http
  bind *:8443
  option http-server-close
  default_backend Web-Farm

  # Rewrite requests so that they are passed to the backend as http:/ schemed requests.
  # This may be required if the backend web servers don't like handling https schemed requests over non-https transport.
  # I didn't use this in the end - but it might come in handy in the future so I left it commented out.
  # reqirep ^(\w+\ )https:/(/.*)$ \1http:/\2

  # Rewrite responses containing a Location header with HTTP scheme using the relative path.
  # We could alternatively just rewrite the http:/ to be https:/ but then it could break off-site redirects.
  rspirep ^Location:\s*http://.*?\.acmecorp.co.tld(/.*)$ Location:\ \1
  rspirep ^Location:(.*\?\w+=)http(%3a%2f%2f.*?\.acmecorp.co.tld%2f.*)$ Location:\ \1https\2

The first rspirep in the above example is the most important. The second is something more specific to a particular web application we’re hosting that uses a ?Redirect=http://yada.yada style query string in certain places.

The rsprep / rspirep rule (the i means case-insensitive matching) is very powerful. The only downside is that you do need to be fairly fluent with regular expressions. It requires only two parameters, the first is your regular expression and the second is your string replacement.

The string replacement that occurs in the second parameter supports expansion based upon indexed capture groups from the regular expression that was matched. This is useful for merging very specific pieces from the match back into the replacement string, as I am doing in the example above. They take the form of \1 or \2 etc. Where the number indicates the capture group index number. And capture groups are denoted in the regular expression by using parenthesis, if you didn’t know.

Truly “live” updates on the Webstats UI

One of the first things I noticed in the hours after deploying HA-Proxy is that the webstat counters that are held for each frontend, listen and backend are not actually updated as frequently as they perhaps ought to be. Indeed, the counters for any given connection are not accumulated until that connection has ended. This is bad if your application(s) tend to hold open long-duration connections. It reduces your usability of HA-Proxy’s reporting. I’m sure there are very good performance reasons that Willy did this, as that is what is alluded to in the documentation. Fortunately there is a very simple workaround for this in the form of the contstats option.

Simply add the following to your proxy and benefit from higher accuracy webstats:

option contstats

Until next time…

During testing we were coming across an issue where the HA-Proxy instance running on the slave member of our cluster would fail to bind some of its frontend proxies:

Starting haproxy: [ALERT] : Starting proxy Public-HTTPS: cannot bind socket

After some head scratching I noticed that the problem was only arising on those proxies that explicitly defined the IP address of a virtual interface that was being managed by Keepalived (or maybe Heartbeat for you).

This is because both of these High-Availability clustering systems use a rather simplistic design whereby the “shared” virtual IP is only installed on the active node in the cluster. While the nodes that are in a dormant state (i.e. the slaves) do not actually have those virtual IPs assigned to them during that state. It’s a sort of “IP address hot-swapping” design. I learnt this by executing a simple a command, first from the master server:

$ ip a
<snipped stuff for brevity>
2: seth0:  mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:15:5d:28:7d:19 brd ff:ff:ff:ff:ff:ff
    inet 172.16.61.151/24 brd 172.16.61.255 scope global seth0
    inet 172.16.61.150/24 brd 172.16.61.255 scope global secondary seth0:0
    inet 172.16.61.159/24 brd 172.16.61.255 scope global secondary seth0:1
    inet6 fe80::215:5dff:fe28:7d19/64 scope link
       valid_lft forever preferred_lft forever
<snipped trailing stuff for brevity>

Then again, from the slave server:

$ ip a
<snipped stuff for brevity>
2: seth0:  mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:15:5d:2d:9c:11 brd ff:ff:ff:ff:ff:ff
    inet 172.16.61.152/24 brd 172.16.61.255 scope global seth0
    inet6 fe80::215:5dff:fe2d:9c11/64 scope link
       valid_lft forever preferred_lft forever
<snipped trailing stuff for brevity>

Unfortunately this behaviour can cause problems for programs like HA-Proxy which have been configured to expect the existence of specific network interfaces on the server. I was considering working around it by writing some scripts that hook events within the HA cluster to handle stopping and starting the HA-Proxy when needed. But this approach seemed clunky and unintuitive. So I dug a little deeper and came across a bit of a gem hidden away in the depths of the Linux networking stack. It is a simple boolean setting called “net.ipv4.ip_nonlocal_bind” and it allows a program like HA-Proxy to create listening sockets on network interfaces that do not actually exist on the server. It was created specially for this situation.

So in the end the fix was as simple as adding/updating the /etc/sysctl.conf file to include the following key/value pair:

net.ipv4.ip_nonlocal_bind=1

My previous experience of setting up these low-level High-Availability clusters was with Windows Server’s feature called Network Load Balancing (NLB). This works quite different from Keepalived and Heartbeat. It relies upon some low level ARP hacking/trickery and some sort of distributed time splicing algorithm. But it does ensure that each node in the cluster (whether in a master or slave position) will remain allocated with the virtual IP address(es) at all times. I suppose there is always more than one way to crack an egg…