Nathan Evans' Nemesis of the Moment

Really simple way to split a F# sequence into chunks / partitions

Posted in .NET Framework, F# by Nathan B. Evans on March 13, 2014

I needed a simple function to split a (potentially infinite) sequence into chunks, suitable for processing. My exact use-case for this was actually in optimising my Azure blob storage uploads. I would split a sequence of 1,000s of items into batches of 60 or so items and then upload them concurrently across 60 connections to the Azure blob store. The performance benefits from this (after also messing around with ServicePointManager’s stupid connection limits and Nagle algorithm stuff) were simply staggering but that’s kind of another story.

I searched high and low for a suitable F# function to do this, but there was nothing. And all the samples I found on the web had design flaws or were overly complex. The design flaws were usually that it would seek the sequence more than once which is highly inefficient and could even cause side affects depending upon the source of the sequence.

I got frustrated and quickly wrote my own, though I will warn you that it uses mutable state. But as a result is very fast…

/// Returns a sequence that yields chunks of length n.
/// Each chunk is returned as an array.
let toChunks n (s:seq<'t>) = seq {
    let pos = ref 0
    let buffer = Array.zeroCreate<'t> n

    for x in s do
        buffer.[!pos] <- x
        if !pos = n - 1 then
            yield buffer |> Array.copy
            pos := 0
            incr pos

    if !pos > 0 then
        yield Array.sub buffer 0 !pos

// Ridiculously imperative, but it works and is performant; won't seek the sequence more than once.
// If you're using in a forward-only manner and won't be holding references to the returned chunks
// then you can get rid of the Array.copy to gain some extra perf and reduce GC.

Here’s the Gist if that’s better for you:

Oh, this is MIT licensed so you know that I won’t come calling.


Tagged with: , ,

Quick and dirty literal port of my PBKDF2 password hash function from C# to F#

Posted in .NET Framework, F# by Nathan B. Evans on March 13, 2014

Now that I’m fully on board the F# bandwagon I’ve found myself wanting to refactor some of my old utility functions that I’ve had for years in C# land. Sure, I could just reference my C# assemblies, and probably should have. But there’s something nice about porting some code over to your shiny new language, if only just as a learning exercise.

module Crypto.Pbkdf2
open System
open System.Security.Cryptography

let private subkeyLength = 32
let private saltSize = 16

/// Hashes a password by a specified number of iterations using the PBKDF2 crypto function.
let hash password iterations =
    use algo = new Rfc2898DeriveBytes(password, saltSize, iterations)
    let salt = algo.Salt
    let bytes = algo.GetBytes(subkeyLength)

    let iters = if BitConverter.IsLittleEndian then BitConverter.GetBytes(iterations) else BitConverter.GetBytes(iterations) |> Array.rev

    let parts = Array.zeroCreate<byte> 54
    Buffer.BlockCopy(salt, 0, parts, 1, saltSize)
    Buffer.BlockCopy(bytes, 0, parts, 17, subkeyLength)
    Buffer.BlockCopy(iters, 0, parts, 50, sizeof<int>)


/// Hashes a password using 10,000 iterations of the PBKDF2 crypto function.
let fastHash password = hash password 10000

/// Hashes a password using 100,000 iterations of the PBKDF2 crypto function.
let strongHash password = hash password 100000

/// Hashes a password using 300,000 iterations of the PBKDF2 crypto function.
let uberHash password = hash password 300000

/// Verifies a PBKDF2 hashed password with a candidate password.
/// Returns true if the candidate password is correct.
/// The hashed password must have been originally generated by one of the hash functions within this module.
let verify hashedPassword (password:string) =
    let parts = Convert.FromBase64String(hashedPassword)
    if parts.Length <> 54 || parts.[0] <> byte 0 then
        let salt = Array.zeroCreate<byte> saltSize
        Buffer.BlockCopy(parts, 1, salt, 0, saltSize)

        let bytes = Array.zeroCreate<byte> subkeyLength
        Buffer.BlockCopy(parts, 17, bytes, 0, subkeyLength)

        let iters = Array.zeroCreate<byte> sizeof<int>
        Buffer.BlockCopy(parts, 50, iters, 0, sizeof<int>);

        let iters = if BitConverter.IsLittleEndian then iters else iters |> Array.rev

        let iterations = BitConverter.ToInt32(iters, 0)

        use algo = new Rfc2898DeriveBytes(password, salt, iterations)
        let challengeBytes = algo.GetBytes(32)

        match Seq.compareWith (fun a b -> if a = b then 0 else 1) bytes challengeBytes with
        | v when v = 0 -> true
        | _ -> false

Here’s the Gist, if it’s better for you:

Sure the code is quite imperative in style, but it is just a utility function and I literally did a “one pass” refactor from the C# code. It’s not really worth giving a second pass just for the sake of making it more pure functional.

This is MIT licensed by the way. I ain’t going to come calling.


Tagged with: , , ,

Point-to-site (P2S) Azure VPN

Posted in Azure, Windows Environment by Nathan B. Evans on March 1, 2014

It seems there’s still some bugs to be worked out by the Azure guys with this point-to-site Azure VPN feature.

I have been wanting a secure way to access my Azure virtual machines for some time and I only just noticed they added this feature (still in Preview) a few months back. So I went about setting it up.

I had seemingly followed all the official guides. Got past all the the hurdles for the certificate creation stuff using the makecert tool (why doesn’t Azure offer to do all this for you? It’s not like everybody that uses Azure has a Visual Studio command shell installed on their PC!)

I then downloaded the 64-bit VPN Package for my Virtual Network. Installed it. Tried to connect and it was throwing this bizarre error:

A certificate chain processed, but terminated in a root certificate which is not trusted by the trust provider.

After some playing around it turned out that the VPN Package installer has a serious bug in it whereby it simply doesn’t install the Azure Gateway’s certificate into your certificate store. Luckily I had a copy of WinRAR on my machine and so I extracted the installer to take a peek inside and sure enough it contains a .cer certificate file. So I did WinKey+R, mmc, and added a Certificates snap-in for the Local Machine (not Current User!). Then navigated to the Trusted Root Certificates Authorities, right-click and choose the Import task. Find the .cer file you extracted from the VPN Package installer and install it.

Now retry the Azure VPN Connection and the error should go away and you’ll log straight in!

Tagged with: ,