Symbolic Forest

Sometimes, given that I often work with people who are twenty years or so younger than me, I feel old. I mean, the archives of this blog go back over twenty years now: these are serious, intelligent colleagues, and when I started writing my first blog posts they were likely still toddlers.

Sometimes, though, that has an advantage. I was thinking of this when debugging some code a colleague had written, which worked fine up to a point, but failed if its input file was more than, say, a few tens of megabytes. When the input reached that size, the whole thing crashed with OutOfMemoryException even on a computer with multiple gigabytes of memory, a hundred times more memory than the hundred-megabyte example file the client had sent.

When I was younger, you see, that would have seemed a ridiculous amount of data, unimaginable to fit in one file. Even when I had my first PC, the thought of a file too big to fit on even a superfloppy like a Zip disk was a little bit mindblowing, even though the PC seemed massive compared to what I’d experienced before.

Back when I was at school, I’d tried to teach myself how to code on an Amstrad CPC, a mid-1980s 8-bit machine with a 64k address space and a floppy disk drive of 180k capacity. It was the second-generation of 8-bit home machine really, more powerful than a C64 or a Sinclair Spectrum despite sharing the same CPU as the latter. Unlike those, it had a fully-bitmapped screen with individual pixels all fully addressable; however, that took up 16k of the 64k address space, so the actual code on it had to be pretty damn tight to fit. The programmers’ Firmware Manual—what we’d now call the API reference documentation—is of course scanned and online; one of the reasons I was never very succssful coding on the machine itself^* was that in the 1980s and 90s copies of it were almost impossible to find once Amstrad’s print run was exhausted. On the CPC, every byte you used counted; a lot of software development houses ended up cross-assembling their code purely because for a large program it was difficult to fit the source code itself onto the machine.^** That’s the background I came from, and it makes me wary still nowadays not to waste too much memory or resources. I’m the sort of developer who will pass an expected size parameter to the List<T> constructor if it’s known, to avoid unnecessary reallocations, who doesn’t add ToList() automatically by reflex to the end of every LINQ operation—which is a good idea in any case, as long as you know when you do need to.

Returning to the present: what had my team member done, then, that he was provoking a machine into running out of memory when in theory he had plenty to play with? Well, there were two problems at work.

Firstly, yes, we’re talking about someone who has never tried building code on a tiny tiny environment. The purpose of this particular code was to take an input zip file, open it, modify some of its content, recompress it, and send it off to an API elsewhere. Moreover, this had been done re-using existing internal code, some of which wanted to operate on a Stream and some of which, for whatever reason I don’t know, wanted to operate on a byte[]. We had ended up with code that received the data in a MemoryStream, unzipped it in memory, and copied the contents out into more MemoryStream objects. Each of those was being copied into a byte array which was being passed to a routine that immediately copied its input into a new MemoryStream, before deserializing…well, you get the idea. The whole thing ended up with many, many copies of the input data in memory, either in essentially its original format, or in a slightly modified form, and all of these copies were still in memory at the end of the process.

Secondly, there was another issue that was not quite so much the developer’s responsibility. This .NET code was being combined in “Portable” form, and the server was, again for reasons best known to itself, deciding that it should run it with the 32-bit runtime. Therefore, although there should have been 16Gb of memory on the server instance, we were working with a 2Gb memory ceiling.

I did dig in and rewrite as much of the code as I thought I needed to. Some of the copying could be elided altogether; and as this wasn’t a time-critical piece of code, I changed a lot of the rest to use a temporary file instead of memory. The second issue had an easy, lazy fix: compile the thing as 64-bit only, so the server would have no choice of runtime. As a result I never did get to the bottom of why it was preferring the 32-bit runtime, but I had working, shippable, code at the end of the day, and that’s what mattered here.

What I couldn’t help thinking, though, was that the rewriting might not have been needed to begin with. A young developer—who’s never worked on a genuinely small system—has spent so much time, though, never worrying about working anywhere near the boundaries of what their virtual servers can cope with, that when they do hit those boundaries, it comes as a nasty, sudden shock. They have no idea at all what to do, or even where to start: an OutOfMemoryException may as well be an act of the gods. Maybe when I’m helping train people up, I should give them all an Amstrad CPC emulator and see what the result is.

* My high point was successfully cloning Minesweeper, but with keyboard controls.

** Some software was shipped on 16k ROMs, to go along with third-party ROM socket boxes that attached to the expansion bus; this kept the assembler and editor code out of the main address space, but it could still be difficult to fit the source code and the assembler output in memory at the same time. The ROMs were scanned on boot and each declared named entrypoints which could then be accessed as BASIC commands. At least one game I can remember—The Bard’s Tale—crashed if too many ROMs were attached, because each ROM could reserve an area of RAM for its own bookkeeping, and the game found itself without enough memory available.

Keyword noise: software, software development, resources, coding.

One of my flaws is that as soon as I’m familiar with something, I assume it must be common knowledge. I love tutoring and mentoring people, but I’m bad at pitching exactly where their level might be, and in working out what they might not have come across before. Particularly, in my career, software development is one of those skills where beyond a certain base level nearly all your knowledge is picked up through osmosis and experience, rather than through formal training. Sometimes, when I’m reviewing my team’s code I come across things that surprise me a little. That’s where this post comes from, really: a few months back I spotted something in a review and realised it wouldn’t work.

This post is about C#, so apologies to anyone with no interest in coding in general or C# in particular; I’ll try to explain this at a straightforward level, so that even if you don’t know the language you can work out what’s going on. First, though, I have to explain a few basics. That’s because there’s one particular thing in C# (in .NET, in fact) that you can’t do, that people learn very on that you can’t do, and you have to find workarounds for. This post is about a very similar situation, which doesn’t work for the same reason, but that isn’t necessarily immediately obvious even to an experienced coder. In order for you to understand that, I’m going to explain the well-known case first.

Since its first version over twenty years ago, C# has had the concept of “enumerables” and “enumerators”. An enumerable is essentially something that consists of a set of items, all of the same type, that you can process or handle one-by-one. An enumerator is a thing that lets you do this. In other words, you can go to an enumerable and say “can I have an enumerator, please”, and you should get an enumerator that’s linked to your enumerable. You can then keep saying to the enumerator: “can I have the next thing from the enumerable?” until the enumerator tells you there’s none left.

This is all expressed in the methods IEnumerable<T>.GetEnumerator()^* and IEnumerator<T>.MoveNext(), not to mention the IEnumerator<T>.Current property, which nobody ever actually uses. In fact, the documentation explicity recommends you don’t use them, because they have easier wrappers. For example, the foreach statement.

List<string> someWords = new List<string>() { "one", "two", "three" };
foreach (string word in someWords)
{
    Process(word);
}

Under the hood, this is equivalent^** to:

List<string> someWords = new List<string>() { "one", "two", "three" };
IEnumerator<string> wordEnumerator = someWords.GetEnumerator();
while (wordEnumerator.MoveNext())
{
    string word = wordEnumerator.Current;
    Process(word);
}

The foreach statement is essentially using a hidden enumerator that the programmer doesn’t need to worry about.

The thing that developers generally learn very early on is that you can’t modify the contents of an enumerable whilst it’s being enumerated. Well, you can, but your enumerator will be rendered unusable. On your next call to the enumerator, it will throw an exception.

// This code won't work
List<string> someWords = new List<string>() { "one", "two", "three" };
foreach (string word in someWords)
{
    if (word.Contains('e'));
    {
        someWords.Remove(word);
    }
}

This makes sense, if you think about it: it’s reasonable for an enumerator to be able to expect that it’s working on solid ground, so to speak. If you try to jiggle the carpet underneath it, it falls over, because it might not know where to step next. If you want to do this using a foreach, you will need to do it some other way, such as by making a copy of the list.

List<string> someWords = new List<string>() { "one", "two", "three" };
List<string> copy = someWords.ToList();
foreach (string word in copy)
{
    if (word.Contains('e'));
    {
        someWords.Remove(word);
    }
}

So, one of my colleagues was in this situation, and came up with what seemed like a nice, clean way to handle this. They were going to use the LINQ API to both make the copy and do the filtering, in one go. LINQ is a very helpful API that gives you filtering, projection and aggregate methods on enumerables. It’s a “fluent API”, which means it’s designed for you to be able to chain calls together. In their code, they used the Where() method, which takes an enumerable and returns an enumerable containing the items from the first enumerable which matched a given condition.

// Can you see where the bug is?
List<string> someWords = new List<string>() { "one", "two", "three" };
IEnumerable<string> filteredWords = someWords.Where(w => w.Contains('e'));
foreach (string word in filteredWords)
{
    someWords.Remove(word);
}

This should work, right? We’re not iterating over the enumerable we’re modifying, we’re iterating over the new, filtered enumerable. So why does this crash with the same exception as the previous example?

The answer is that LINQ methods—strictly speaking, here, we’re using “LINQ-To-Objects”—don’t return the same type of thing as their parameter. They return an IEnunerable<T>, but they don’t guarantee exactly what implementation of IEnumerable<T> they might return. Moreover, in general, LINQ prefers “lazy evaluation”. This means that Where() doesn’t actually do the filtering when it’s called—that would be a very inefficient strategy on a large dataset, because you’d potentially be creating a second copy of the dataset in memory. Instead, it returns a wrapper object, which doesn’t actually evaulate its filter until something tries to enumerate it.

In other words, when the foreach loop iterates over filteredWords, filteredWords isn’t a list of words itself. It’s an object that, at that point, goes to its source data and thinks: “does that match? OK, pass it through.” And the next time: “does that match? No, next. Does that match? Yes, pass it through.” So the foreach loop is still, ultimately, triggering one or more enumerations of someWords each time we go around the loop, even though it doesn’t immediately appear to be used.

What’s the best way to fix this? Well, in this toy example, you really could just do this:

someWords = someWords.Where(w => !w.Contains('e')).ToList();

which gets rid of the loop completely. If you can’t do that for some reason—and I can’t remember why we couldn’t do that in the real-world code this is loosely based on—you can add a ToList() call onto the line creating filteredWords, forcing evaluation of the filter at that point. Or, you could avoid a foreach loop a different way by converting it to a for loop, which are a bit more flexible than a foreach and in this case would save memory slightly; the downside is a bit more typing and that your code becomes prone to subtle off-by-one errors if you don’t think it through thoroughly. There’s nearly always more than one way to do something like this, and they all have their own upsides and downsides.

I said at the start, I spotted the issue here straightaway just by reading the code, not by trying to run it. If I hadn’t spotted it inside somebody else’s code, I wouldn’t even have thought to write a blog post on something like this. There are always going to be people, though, who didn’t realise that the code would behave like this because they hadn’t really thought about how LINQ works; just as there are always developers who go the other way and slap a ToList() on the end of the LINQ chain because they don’t understand how LINQ works but have come across this problem before and know that ToList() fixed it. Hopefully, some of the people who read this post will now have learned something they didn’t know before; and if you didn’t, I hope at least you found it interesting.

* Note. for clarity I’m only going to use the generic interface in this post. There is also a non-generic interface, but as only the very first versions of C# didn’t support generics, we really don’t need to worry about that. If you write your own enumerable you’re still required to support the non-generic interface, but you can usually do so with one line of boilerplate: public IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();

** In recent versions of C#, at any rate. In earlier versions, the equivalence was slightly different. The change was a subtle but potentially breaking one, causing a change of behaviour in cases where the loop variable was captured by a lambda expression.

Keyword noise: coding, software, .NET, .NET Framework, computing, C#, development.

The previous post in this series is here.

Spending some more time going through the things The Parents should arguably have thrown out decades ago, I came across a leather bag, which seemed to have belonged to my father. Specifically, he seemed to have used it for going to college, in the 1970s. Him being him, he’d never properly cleaned it out, so it had accumulated all manner of things from all across the decade. There were “please explain your non-attendance” slips from 1972; an unread railway society magazine from 1977; and the most recent thing with a date on was an Open University exam paper from 1983. It was about relational database design, and to be honest some of the questions wouldn’t be out of place in a modern exam paper if you asked for the answers in SQL DDL rather than in CODASYL DDL, so I might come back to that and give it its own post. What he scored on the exam, I don’t know. There were coloured pencils, and an unopened packet of gum.

It seems to be from before the invention of the Best Before date, but the RRP printed on the side is £0.04.

Slightly more expensive: a rather nice slide rule. Look, it has a Standard Deviation scale and all. Naturally, my dad being my dad, it was still in its case and with the original instruction book, which will be useful if I ever try to work out how to use it.

And finally (for today) I spotted what appeared to be a slip of paper at the bottom of the bag with “NEWTON’S METHOD” written on it in small capitals, in fountain-pen ink. Had he been cheating in his exams? Had he written a crib to the Newton-Raphson method down and slipped it into the bottom of the bag? I pulled it out and…I was wrong.

It was a rolled-up 8-bit paper tape! Presumably with his attempt at a program to numerically solve a particular class of equation using Newton’s method.

I don’t know what type of machine it would have been written for, but I could see that it was likely binary data or text in some unfamiliar encoding, as whichever way around you look at it a good proportion of the high bits would be set so it was unlikely to be ASCII. Assuming I’m holding the tape the right way round, this is a transcription of the first thirty-two bytes…

0A 8D 44 4E C5 A0 35 B8 0A 8D 22 30 A0 59 42 A0 47 4E C9 44 C9 56 C9 44 22 A0 D4 4E C9 D2 50 A0

That’s clearly not ASCII. In fact, I think I know what it might: an 8080/Z80 binary. I recognise those repeated C9 bytes: that’s the opcode for the ret instruction, which has survived all the way through to the modern-day x64 instruction set. If I try to hand-disassemble those few bytes assuming it’s Z80 code we get:

ld a,(bc)
adc a,l
ld b,h
ld c,(hl)
push bc
and b
dec (hl)
cp b
ld a,(bc)
adc a,l

This isn’t the place to go into Z80 assembler syntax—that might be a topic for the future—other than to say that it reads left-to-right and brackets are a pointer dereference, so ld c,(hl) means “put the value in register c into the memory location whose address is in register hl. As valid code it doesn’t look too promising to my eyes—I didn’t even realise dec (hl) was something you could do—but I’ve never been any sort of assembly language expert. The “code” clearly does start off making assumptions about the state of the registers, but on some operating systems that would make sense. This disassembly only takes us as far as the repeated 0A8D, though: maybe that’s some sort of marker separating segments of the file, and the actual code is yet to come. The disassembly continues…

ld (&a030),hl
ld e,c
ld b,d
and b
ld b,a
ld c,(hl)
ret
ld b,h
ret
ld d,(hl)
ret
ld b,h
ld (&a0d4),hl
ld c,(hl)
ret
jp nc,(&a050)

Well, that sort of makes some sort of sense. The instructions that reference fixed addresses all appear to point to a consistent place in the address space. It also implies code and data is in the same address space, in the block starting around &a000 which means you’d expect that some of the binary wouldn’t make sense when decompiled. If this was some other arbitrary data, I’d expect references like that to be scattered around at random locations. As the label says this is an implementation of Newton’s method, we can probably assume that this is a college program that includes an implementation of some mathematical function, an implementation of its first derivative, and the Newton’s method code that calls the first two repeatedly to find a solution for the first. I wouldn’t expect it to be so sophisticated as to be able to operate on any arbitrary function, or to work out the derivative function itself.

If I could find jumps or calls pointing to the instructions after those ret opcodes, I’d be happier. Maybe, if I ever have too much time on my hands, I’ll try to decompile the whole thing.

The next post in this series is here

Keyword noise: computers, coding, programming, history, chewing gum, slide rule, mathematics, calculations, Newton's Method, Newton-Raphson Method.

One thing I have been doing over the past few weeks is: finally, finally, taking the hard drive out of my last desktop computer—last used about 8 years ago at a guess—and actually copying all the documents off it. It also had stuff preserved from pretty much every desktop machine I’d had before that, so there was a whole treasure-chest of photographs I hadn’t seen in years, things I’d written, and various incomplete coding projects.

Some of the photos will no doubt get posted on here over the coming weeks, but this post isn’t about those. Because, by pure coincidence, I was browsing my Twitter feed this morning and saw this tweet from @ireneista:

we were trying to help a friend get up to speed on how to make a Unix process into a daemon, which is something we found plenty of guides on in the 90s but it’s largely forgotten knowledge

Hang on a minute, I think. Haven’t I just been pulling old incomplete coding projects off my old hard disk and saving them into Github repositories instead? And don’t some of those have exactly that code in? A daemon, on Unix, is roughly the equivalent of a “Service” on Windows. It’s a program that runs all the time in the background on a computer, doing important work.^* Many servers don’t even run anything else to speak of. On both Unix and Windows systems, there are special steps you have to take to properly “detach” your code and let it run in the background as part of the system, and if you don’t do all those steps properly you will either produce something that is liable to break and stop running that it’s not supposed to, or write something that fills up your system’s process table with so-called “zombie” entries for processes that have stopped running but still need some bookkeeping information kept about them.

Is this forgotten knowledge? Well, it’s certainly not something I would be able to do, off the top of my head, without a lot of recourse to documentation. For a start all the past projects I’m talking about were written in C, for Linux systems, and I haven’t touched the language nor the operating system much for a number of years now.

None of the projects I’m talking about ever approached completion or were properly tested, so there’s not that much point releasing their full source code to the world. However, clearly, the information about how to set up a daemon has disappeared out of circulation a bit. Moreover, that code was generally stuff that I pulled wholesale from Usenet FAQs myself, tidying it up and adding extra logging as I needed, so compared to the rest of the projects, it’s probably much more reliable. The tweet thread above links to some CIA documentation released by Wikileaks which is nice and explanatory, but doesn’t actually include some of the things I always did when starting up a daemon. You could, of course, argue they’re not always needed. So, here is some daemonisation code I have cobbled together by taking an average across the code I was writing about twenty-ish years ago and adding a bit of explanation. Hopefully this will be useful to somebody.

Bear in mind this isn’t real code: it depends on functions and variables that you can assume we’ve declared in headers, or in the parts of the code that have been omitted. As the old saying goes, I accept no responsibility if this code causes loss, damage, or demons flying out of your nose.^**

/* You can look up yourself which headers you'll need to include */

int main(int argc, char **argv)
{
    /* 
     * First you'll want to read config and process command line args,
     * because it might be nice to include an argument to say "dont'
     * run as a daemon!" if you fancy that.
     *
     * This code is also written to use GNU intltools, and the setup for that
     * goes here too.
     */

    /* Assume the daemonise variable was set by processing the config */
    if (daemonise)
    {
        /* First we fork to a new process and exit the original process */
        switch (fork ())
        {
        case -1:
            syslog (LOG_ERR, _("Forking hell, aborting."));
            exit (EXIT_FAILURE);
        default:
            exit (0);
        case 0:
            break;
        }

        /* Then we call setsid() to become a process group leader, making sure we are detached
         * from any terminals */
        if (setsid () == -1)
        {
            syslog (LOG_ERR, _("setsid() failed, aborting."));
            exit (EXIT_FAILURE);
        }

        /* Then we fork again */
        switch (fork ())
        {
        case -1:
            syslog (LOG_ERR, _("Forking hell x2, aborting."));
            exit (EXIT_FAILURE);
        default:
            exit (0);
        case 0:
            break;
        }

        /* Next, a bit of cleanup.  Change our CWD to / so we don't block any umounts, and 
         * redirect our standard streams to taste */
        umask (0022);
        if (chdir ("/"))
            syslog (LOG_WARNING, _("Cannot chdir to root directory"));
        freopen ("/dev/null", "w", stdout);
        freopen ("/dev/null", "r", stdin);
        freopen ("/dev/console", "w", stderr); /* This one in particular might not be what you want */

        /*
         * Listen to some signals.  The second parameters are function pointers which 
         * you'll have to imagine are defined elsewhere.  Reloading config on SIGHUP
         * is a common daemon behaviour you might want.  I can't remember why I thought
         * it important to ignore SIGPIPE
         */
        signal (SIGPIPE, SIG_IGN);
        signal (SIGHUP, warm_restart);
        signal (SIGQUIT, graceful_shutdown);
        signal (SIGTERM, graceful_shutdown);

        /* And now we're done!  Let's go and run the rest of our code */
        run_the_daemon ();
    }
}

The above probably includes some horrible mistake somewhere along the way, but hopefully it’s not too inaccurate, and hopefully would work in the real world. If you try it—or have opinions about it—please do get in touch and let me know.

* NB: this is a simplification for the benefit of the non-technical. Yes, I know I’m generalising and lots of daemons and services don’t run all the time. Please don’t write in with examples.

** “demons flying out of your nose” was a running joke in the comp.lang.c Usenet group, for something it would be considered entirely legitimate for a C compiler to do if you wrote code that was described in the C language standard as having “undefined behaviour”.

Keyword noise: coding, Linux, Unix, computing, C, daemon.

It’s nearly a month now since I started the new job. It’s not been plain sailing all the way, of course, but it feels like it’s going reasonably well.

One aspect of coming in and taking over a team is that naturally I want to put my own flavour on things. I’m not saying I automatically think my ideas are always better than what the team already does, of course, but I do think firstly that new ideas can often trigger new thinking and give new energy to a team, and secondly, that some of the things are familiar to me are inevitably going to be new to my new colleagues, and things they might not have considered before. So, if I can bring in processes and practices that have worked well for me before, they may well work well in the new job too. If anything, as long as I can get the team behind them, they might be worth a try.

“As long as I can get the team behind them” is the key point there, of course; and it’s important to make sure we all feel we are a team and that we can all contribute to making change happen. I started writing down some Basic Good Principles along those lines. “We know we’re not perfect but we’re always improving,” for example. “We all review each other’s code, because everyone gets it wrong sometimes.” — I’m always surprised how many people don’t quite follow that one. “This document is not set in stone and is always changing, just as our processes are always changing and improving.” is another one, because when a particular practice or process turns into dogma is when it starts to impede you rather than work for you.

After a while brainstorming this stuff, writing down these sort of things, though, I started to get a bit of self-doubt. Is writing down these sort of principles really helpful, or were my notes just turning into meaningless platitudes in themselves? Was I at risk of accidentally coming up with the “Live, Laugh, Love” of software development?

Which of course, set me thinking. What is the “Live, Laugh, Love” of software development? Maybe if I deliberately came up with something quite so awful, quite so twee and nauseating, it would reduce the risk of ending up in that zone by accident?

I scribbled a few things down, crossed some out, shuffled some ideas around, and eventually came up with the following. I think it’s pretty good. By which I mean, I feel slightly disgusted with myself.

Code

Create

Care

One of our regular correspondents writes: Yeah, I think you might need a shower after that.

Most definitely. I felt dirty. If you ever find me giving that as advice to my team, please, tell me I need to take a holiday. All the same, if I was to pick a font and put it on a framed print, I wonder how many I could sell?

Keyword noise: work, coding, motivation.

I passed a very minor milestone yesterday. Duolingo, the language-learning app, informed me that I had a “streak” of 1,000 days. In other words, for the past not-quite-three-years, most days, I have fired up the Duolingo app or website and done some sort of language lesson. I say “most days”: in theory the “streak” is supposed to mean I did it every single day, but in practice you can skip days here and there if you know what you’re doing. I’ve mostly been learning Welsh, with a smattering of Dutch, and occasionally revising my tourist-level German.

My Welsh isn’t, I have to admit, at any sort of level where I can actually hold a conversation. I barely dare say “Ga i psygod a sglodion bach, plîs,” in the chip shop when visiting I’m Welsh-speaking Wales, because although I can say that I am wary I wouldn’t be any use at comprehending the response, if they need to ask, for example, exactly what type of fish I want. To be honest, I see this as a big drawback to the whole Duolingo-style learning experience, which seems essentially focused around rote learning of a small number of set phrases in the hope that a broader understanding of grammar and vocabulary will follow. I’ve been using Duolingo much longer than three years—I first used it to start revising my knowledge of German back in 2015. When I last visited Germany, though, I was slightly confused to find that after over a year of Duolingo, if anything, I felt less secure in my command of German, less confident in my ability to use it day-to-day. Exactly why I don’t now, but it helped me realise that I can’t just delegate that sort of learning to a question-and-answer app. If I want to progress with my Welsh, I know I’m going to have to find some sort of conversational class.

Passing the 1,000 days milestone made me start wondering if anyone has produced something along the same lines as Duolingo but for computer languages. In some ways it should be a less difficult problem than for natural language learning, because, after all, any nuances of meaning are less ambiguous. I lose track of the number of times Duolingo marks me down because I enter an English answer which means the same as the accepted answer but uses some other synonym or has a slightly different word order. With a coding language, if you have your requirements and the output meets them, your answer is definitely right. In theory it shouldn’t be too hard to create a Duolingo-alike thing but with this sort of question:

Given a List<Uri> called uris, return a list of the Uris whose hostnames end in .com in alphabetical order.

1. uris.Select(u => u.Host).Where(h => h.EndsWith(".com")).OrderBy();
2. uris.Where(u => u != null && u.Host.EndsWith(".com")).OrderBy(u => u.AbsoluteUri).ToList();
3. uris.SelectMany(u => u.Where(Host.EndsWith(".com"))).ToList().Sort();

The answer, by the way, is 2. Please do write in if I’ve made any mistakes by being brave enough to write this off the top of my head; writing wrong-but-plausible-looking code is harder than you think. Moreover, I know the other two answers contain a host of errors and wouldn’t even compile, just as the wrong answers in Duolingo often contain major errors in grammar and vocabulary.

Clearly, you could do something like this, and you could memorise a whole set of “cheat sheets” of different coding fragments that fit various different circumstances. Would you, though, be able to write decent, efficient, and most importantly well-understood code this way? Would you understand exactly the difference between the OrderBy() call in the correct answer, and the Sort() call in answer three?^* I suspect the answer to these questions is probably no.

Is that necessarily a bad thing, though? It’s possibly the level that junior developers often work at, and we accept that that’s just a necessary phrase of their career. Most developers start their careers knowing a small range of things, and they start out by plugging those things together and then sorting the bugs out. As they learn and grow they learn more, they fit things together better, they start writing more original code and slowly they become fluent in writing efficient, clean and idiomatic code from scratch. It’s a good parallel to the learner of a natural language, learning how to put phrases together, learning the grammar for doing so and the idioms of casual conversation, until finally they are fluent.

I realise Duolingo is only an early low-level step in my language-learning. It’s never going to be the whole thing; I doubt it would even get you to GCSE level on its own. As a foundational step, though, it might be a very helpful one. One day maybe I’ll be fluent in Welsh or German just as it’s taken me a few years to become fully fluent in C#. I know, though, it’s going to take much more than Duolingo to get me there.

* The call in answer 2 is a LINQ method which does not modify its source but instead returns a new enumeration containing the sorted data. The call in answer 3 modifies the list in-place.

Keyword noise: Duolingo, learning, languages, coding, Welsh, Cymraeg.

This week, Microsoft released .NET 5, and it reminded me I’ve been meaning to post a piece of technical advice that has bitten me a few times but which doesn’t seem to be very well-documented or well-described online. It’s a piece of technical advice, though, that will slowly be fading away in relevance because it’s advice on .NET Framework; so I thought I should put it up here whilst it is still helpful to people.

(Note for non-technical readers, who are used to photos of trains and cemeteries and probably won’t find this post very interesting: .NET 5 is the latest version of .NET Core, which is the replacement for .NET Framework, hence Microsoft have dropped “Core” from the name to try to make that clearer. .NET 5 is the successor to .NET Core 3, because there were many very popular versions of .NET Framework 4.x which were and are heavily used for a long time, so Microsoft thought reusing the number 4 would be just Too Confusing. Are you less confused now?)

This problem, too, is pretty much specific to people working in teams. It only happens (well, I’ve only seen it happen) if all of the following apply:

• you’re working in parallel in a team, on a complex system, probably that has solutions containing a relatively large number of projects
• you’re using the MSBuild tool as part of your continuous integration pipeline, deployment process, or similar
• you’re using Git as your version control system

The symptoms of the problem are:

• You can open the solution in Visual Studio and build it with no problems
• When MSBuild tries to build the solution, it immediately errors, claiming that the solution file has a syntax error on line 2.

Spoiler: there is no syntax error on line 2.

Another note for non-technical readers who are still here: what you might think of loosely as a programming project, in any kind of .NET flavour, has a primary file called a “solution” (its name ends in .sln). The solution contains one or more “projects”; each project contains code. Visual Studio can open your solution file and your project files and turn the projects into some sort of output product, such as a program, a website, a code library or whatever. However, you don’t have to use Visual Studio to do this. .NET Framework has a program called MSBuild that does the same thing. If you have automated your build process (which if you’re working in a team you probably should) and you’re using .NET Framework, your build process will probably use MSBuild to do its work. What happens here is one of a range of problems called “well it worked on my machine”. A developer has code that seems to be in a happy, working state, they upload their code to the team’s server, the automated build process runs, and the automated build process falls over and says it doesn’t work.

The cause of the problem is: two people on the team have added different projects to the solution, in parallel. Now, Git is often quite good, when two people change code at the same time, at either working out how to merge the changes together, or at least, asking you to sort the situation out manually. This, though, is a situation where Git does the wrong thing and breaks your solution file—but it breaks it in a way that only MSBuild notices, and that Visual Studio happily ignores.

The reason this happens is down to the syntax of solution files. The part which lists the projects they contain looks a bit like this:

Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Important.Project.Library", "Important.Project.Library\Important.Project.Library.csproj", "{E6FF8E04-A41D-446B-9F8A-CCFAF4B08AD2}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Important.Project", "Important.Project\Important.Project.csproj", "{9A7E2940-50B8-4F3A-A535-AB6220E6CE3A}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Important.Project.Tests", "Important.Project.Tests\Important.Project.Tests.csproj", "{68035DDB-1C24-407C-B6B3-32CEC1D964E5}"
EndProject

Don’t worry too much about what each line says: the important thing to spot is that each project has a pair of lines: a Project(...) = line that contains the important information, and an EndProject line that, er, doesn’t. The projects are in a fairly arbitrary order, too; on your screen in Visual Studio they get sorted alphabetically, but that isn’t reflected in the file, where they are in the order they were added in.

The real cause of the problem is that Git doesn’t know that every Project... has to be followed by an EndProject. So, imagine two people have added new, different projects to the solution file. Git sees this and thinks: Alice has added Project... to line 42, and Bob has added a different Project... to line 42. So I’ll make those into lines 42 and 43. Alice added EndProject to line 43, and so did Bob, so I’ll just pop that in as line 44. So you get this:

Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Alices.Library", "Alices.Library\Alices.Library.csproj", "{0902233A-3857-4E5E-99F4-54F3F5E695E5}"
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bobs.Library", "Bobs.Library\Bobs.Library.csproj", "{56ABE9BB-1373-43D3-B1C5-1526E443AD73}"
EndProject

Visual Studio is quite unpeturbed by this. MSBuild, however, doesn’t like it at all. It reads the file, realises there’s a Project without a matching EndProject, and falls over. For some reason, it always complains that the error is on line 2, even though it isn’t anywhere near line 2.

The fix for this, as you might have guessed, is to open up the solution file in a text editor and manually enter that missing EndProject line after Alice’s project. And that’s it. Or, if you don’t feel comfortable going in and hacking your solution file directly, remember I said that Visual Studio is completely unfazed by this? You can make some sort of small change in Visual Studio that will imply a different change to the solution file: for example, tell it not to build one of the projects in one of the build configurations. Visual Studio doesn’t just change that bit, it will write out the whole file from scratch, so the problem gets silently fixed for you. Which one is less work depends on which one you’re happier doing, to be honest.

That’s the abstruse technical post over for now. Next time I write one, I’ll see if I can find something even more technically obscure.

Keyword noise: programming, coding, .NET Framework, Visual Studio, MSBuild.

Settling down to see what else I should write in the series of posts about how I rebuilt this website, I realised that the main issues now have already been covered. The previous posts in this series have discussed the following:

• What architecture to choose
• Adding extra functionality to Wintersmith
• Writing Pug templates

And throughout the last two, we touched on some other important software engineering topics, such as refactoring your code so that you Don’t Repeat Yourself, and optimising your code when it’s clear that (and where) it is needed.

There are a few other topics to touch on, but I wasn’t sure they warranted a full post each, so this post is on a couple of those issues, and any other odds and ends that spring to mind whilst I am writing it.

Firstly, the old blog was not at all responsive: in web front-end terms, that means it didn’t mould itself to fit the needs of the users and their devices. Rather, it expected the world to all use a monitor the same size as mine, and if they didn’t, then tough. When I wrote the last designs the majority of the traffic the site was receiving was from people on regular computers; nowadays, that has changed completely.

However, the reason that this isn’t a particularly exciting topic to write about is that I didn’t learn any new skills or dive into interesting new programming techniques. I went the straightforward route, installed Bootstrap 4.5, and went from there. Now, I should say, using Bootstrap doesn’t magically mean your site will become responsive overnight; in fact, it’s very straightforward to accidentally write an entirely non-responsive website. Responsiveness needs carefully planning. However, with that careful planning, and some careful use of the Bootstrap CSS layout classes, I achieved the following aims:

• The source code is laid out so that the main content of the page always comes before the sidebars in the code, wherever the sidebars are actually displayed. This didn’t matter so much on this blog, but on the Garden Blog, which on desktop screens has sidebars on both sides of the page, it does need to be specifically coded. Bootstrap’s layout classes, though, allow you to separate the order in which columns appear on a page from the order in which they appear in the code.
• More importantly, the sidebars move about depending on page width. If you view this site on a desktop screen it has a menu sidebar over on the right. On a narrow mobile screen, the sidebar content is down below at the bottom of the page.
• The font resizes based on screen size for easier reading on small screens. You can’t do this with Bootstrap itself; this required @media selectors in the CSS code with breakpoints chosen to match the Bootstrap ones (which, fortunately, are clearly documented).
• Content images (as opposed to what you could call “structural images”) have max-width: 100%; set. Without this, if the image is bigger than the computed pixel size of the content column, your mobile browser will likely rescale things to get the whole image on screen, so the content column (and the text in it) will become too narrow to read.

On the last point, manual intervention is still required on a couple of types of content. Embedded YouTube videos like in this post need to have the embed code manually edited, and very long lines of text without spaces need to have soft hyphens or zero-width spaces inserted, in order to stop the same thing happening. The latter usually occurs in example code, where zero-width spaces are more appropriate than soft hyphens. All in all though, I’ve managed to produce something that is suitably responsive 95% of the time.

The other point that is worth writing about is the build process of the site. As Wintersmith is a static site generator, every change to the site needs the static files to be built and deployed. The files from the previous version need to be deleted (in case any stale ones, that have disappeared completely from the latest iteration of the output, are still lying about) and then Wintersmith is run to generate the new version. You could do this very simply with a one-liner: rm -rf ../../build/* && wintersmith build if you’re using Bash, for example. However, this site actually consists of three separate Wintersmith sites in parallel. The delete step might only have to be done once, but doing the build step three separate times is a pain. Moreover, what if you only want to delete and rebuild one of the three?

As Wintersmith is a JavaScript program, and uses npm (the Node Package Manager) for managing its dependencies, it turns out that there’s an easy solution to this. Every npm package or package-consumer uses a package.json file to control how npm works; and each package.json file can include a scripts section containing arbitrary commands to run. So, in the package.json file for this blog, I inserted the following:

"scripts": {
  "clean": "node ../shared/js/unlink.js ../../build/blog",
  "build": "wintersmith build"
}

You might be wondering what unlink.js is. I said before that “if you’re using Bash” you could use rm -rf ../../build. However, I develop on a Windows machine, and for this site I use VS Code to do most of the writing. Sometimes therefore I want to build the site using PowerShell, because that’s the default VS Code terminal. Sometimes, though, I’ll be using GitBash, because that’s convenient for my source control commands. One day I might want to develop on a Linux machine. One of the big changes between these different environments is how you delete things: del or Remove-Item in PowerShell; rm in Bash and friends. unlink.js is a small script that reproduces some of the functionality of rm using the JavaScript del package, so that I have a command that will work in the same way across any environment.

So, this means that in the main blog’s folder I can type npm run clean && npm run build and it will do just the same thing as the one-liner command above (although note that the clean step only deletes the main blog’s files). In the other Wintersmith site folders, we have a very similar thing. Then, in the folder above, we have a package.json file which contains clean and build commands for each subsite, and a top-level command that runs each of the others in succession.

"scripts": {
  "clean:main": "cd main && npm run clean",
  "clean:misc": "cd misc && npm run clean",
  "clean:garden": "cd garden && npm run clean",
  "clean": "npm run clean:misc && npm run clean:main && npm run clean:garden",
  "build:main": "cd main && npm run build",
  "build:misc": "cd misc && npm run build",
  "build:garden": "cd garden && npm run build",
  "build": "npm run build:misc && npm run build:main && npm run build:garden"
}

And there you have it. By typing npm run clean && npm run build at the top level, it will recurse into each subsite and clean and build all of them. By typing the same command one folder down, it will clean and build that site alone, leaving the others untouched.

When that’s done, all I have to do is upload the changed files; and I have a tool to do that efficiently. Maybe I’ll go through that another day. I also haven’t really touched on my source-control and change management process; and all I have to say there is, it doesn’t matter what process you use, but you will find things a lot more straightforward if you find a process you like and stick to it. Even if you’re just a lone developer, using a sensible source control workflow and release process makes life much easier and makes you less likely to make mistakes; you don’t need anything as rigid as a big commercial organisation, but just having a set process for storing your changes and releasing them to the public means that you are less likely to slip up and make mistakes. This is probably something else I’ll expand into an essay at some point.

Is the site perfect now? No, of course not. There are always more changes to be made, and more features to add; I haven’t even touched on the things I decided not to do right now but might bring in one day. Thoe changes are for the future, though. Right now, for a small spare-time project, I’m quite pleased with what we have.

Keyword noise: blogging, Wintersmith, Bootstrap, web design, responsive design, coding, websites, HTML, CSS, accessibility, JavaScript, npm.

If you want to start reading this series of articles from the start, the first part is here. In the previous part we discussed how I adapted Wintersmith to my purposes, adding extra page generators for different types of archive page, and refactoring them to make sure that I wasn’t repeating the same logic in multiple places, which is always a good process to follow on any sort of coding project. This post is about the templating language that Wintersmith uses, Pug. When I say “that Wintersmith uses”, incidentally, you should always add a “by default” rider, because as we saw previously adding support for something else generally wouldn’t be too hard to do.

In this case, though, I decided to stick with Pug because rather than being a general-purpose templating or macro language, it’s specifically tailored towards outputting HTML. If you’ve ever tried playing around with HTML itself, you’re probably aware that the structure of an HTML document (or the Document Object Model, as it’s known) has to form a tree of elements, but also that the developer is responsible for making sure that it actually is a valid tree by ending elements in the right order. In other words, when you write code that looks like this:

<section><article><h2>Post title</h2><p>Some <em>content</em> here.</p></article></section>

it’s the developer who is responsible for making sure that those </p>, </article> and </section> tags come in the right order; that the code ends its elements in reverse order to how they started. Because HTML doesn’t pay any attention to (most) white space, they have to be supplied. Pug, on the other hand, enforces correct indentation to represent the tree structure of the document, meaning that you can’t accidentally create a document that doesn’t have a valid structure. It might be the wrong structure, if you mess up your indentation, but that’s a separate issue. In Pug, the above line of HTML would look like this:

section
  article
    h2 Post title
    p Some
      em content
      | here.

You specify the content of an element by putting it on the same line or indenting the following line; elements are automatically closed when you reach a line with the same or less indentation. Note that Pug also assumes that the first word on each line will be an opening tag, and how we can suppress this assumption with the | symbol. You can supply attributes in brackets, so an <a href="target"> ... </a> element becomes a(href="target") ..., and Pug also has CSS-selector-style shortcuts for the class and id attributes, because they’re so commonly used. The HTML code

<section class="mainContent"><article id="post-94">...</article></section>

becomes this in Pug:

section.mainContent
  article#post-94 ...

So far so good; and I immediately cracked on with looking at the pages of the old Wordpress blog and converting the HTML of a typical page into Pug. Moreover, Pug also supports inheritance and mixins (a bit like functions), so I could repeat the exercise of refactoring common code into a single location. The vast majority of the template code for each type of page sits in a single layout.pug file, which is inherited by the templates for specific types of page. It defines a mixin called post() which takes the data structure of a single post as its argument and formats it. The template for single posts is reduced to just this:

extends layout
block append vars
  - subHeader = '';
block append title
  | #{ ' : ' + page.title }
block content
  +post(page)

The block keyword is used to either append to or overwrite specific regions of the primary layout.pug template. The content part of the home page template is just as straightforward:

extends layout
block content
  each article in articles
    +post(article)

I’ve omitted the biggest part of the home page template, which inserts the “Newer posts” and “Older posts” links at the bottom of the page; you can see though that for the content block, the only difference is that we iterate over a range of articles—chosen by the page generator function—and call the mixin for each one.

The great thing about Pug, though, is that it lets you drop out into JavaScript at any point to run bits of code, and when doing that, you don’t just have access to the data for the page it’s working on, you can see the full data model of the entire site. So this makes it easy to do things such as output the sidebar menus (I say sidebar; they’re at the bottom if you’re on mobile) with content that includes things like the number of posts in each month and each category. In the case of the tag cloud, it effectively has to put together a histogram of all of the tags on every post, which we can only do if we have sight of the entire model. It’s also really useful to be able to do little bits of data manipulation on the content before we output it, even if it’s effectively little cosmetic things. The mixin for each post contains the following Javascript, to process the post’s categories:

- if (!Array.isArray(thePost.metadata.categories)) thePost.metadata.categories = [ thePost.metadata.categories ]
- thePost.metadata.categories = Array.from(new Set(thePost.metadata.categories))

The - at the start of each line tells Pug that this is JavaScript code to be run, rather than template content; all this code does is tidy up the post’s category data a little, firstly by making sure the categories are an array, and secondly by removing any duplicates.

You can, however, get a bit carried away with the JavaScript you include in the template. My first complete design for the blog, it turned out, took something like 90 minutes to 2 hours to build the site on my puny laptop; not really helpful if you just want to knock off a quick blog post and upload it. That’s because all of the code I had written to generate the tag cloud, the monthly menus and the category menus, was in the template, so it was being re-computed over again for each page. If you assume that the time taken to generate all those menus is roughly proportional to the number of posts on the blog, O(n) in computer science terms (I haven’t really looked into it—it can’t be any better but it may indeed be worse) then the time taken to generate the whole blog becomes O(n²), which translates as “this doesn’t really scale very well”. The garden blog with its sixtyish posts so far was no problem; for this blog (over 750 posts and counting) it wasn’t really workable.

What’s the solution to this? Back to the Wintersmith code. All those menus are (at least with the present design) always going to contain the same data at any given time, so we only ever need to generate them once. So, I created another Wintersmith plugin, cacher.coffee. The JavaScript code I’d put into my layout templates was converted into CoffeeScript code, called from the plugin registration function. It doesn’t generate HTML itself; instead, it generates data structures containing all of the information in the menus. If you were to write it out as JSON it would look something like this:

"monthData": [
  { "url": "2020/10/", "name": "October 2020", "count": 4 },
  { "url": "2020/09/", "name": "September 2020", "count": 9 },
  ...
],
"categoryData": [
  { "name": "Artistic", "longer": "Posts categorised in Artistic", "count": 105 },
  ...
],
"tagData": [
  { "name": "archaeology", "count": 18, "fontSize": "0.83333" },
  { "name": "art", "count": 23, "fontSize": "0.97222" },
  ...
]

And so on; you get the idea. The template then just contains some very simple code that loops through these data structures and turns them into HTML in the appropriate way for each site. Doing this cut the build time down from up to two hours to around five minutes. It’s still not as quick to write a post here as it is with something like Wordpress, but five minutes is a liveable amount of overhead as far as I am concerned.

The Plain People Of The Internet: So, you’re saying you got it all wrong the first time? Wouldn’t it all have been fine from the start if you’d done it that way to begin with?

Well, yes and no. It would have been cleaner code from the start, that’s for certain; the faster code also has a much better logical structure, as it keeps the code that generates the semantic content at arm’s length from the code that handles the visual appearance, using the data structure above as a contract between the two. Loose coupling between components is, from an architectural point of view, nearly always preferable than tight coupling. On the other hand, one of the basic principles of agile development (in all its many and glorious forms) is: don’t write more code than you need. For a small side project like this blog, the best course of action is nearly always to write the simplest thing that will work, be aware than you’re now owing some technical debt, and come back to improve it later. The difficult thing in the commercial world is nearly always making sure that that last part actually happens, but for a site like this all you need is self-discipline and time.

That just about covers, I think, how I learned enough Pug to put together the templates for this site. We still haven’t covered, though, the layout itself, and all the important ancillary stuff you should never gloss over such as the build-deploy process and how it’s actually hosted. We’ll make a start on that in the next post in this series.

The next post in this series, in which we discuss responsive design, and using npm to make the build process more straightforward, is here

Keyword noise: blogging, coding, websites, Wintersmith, Pug, templates, refactoring, performance, optimisation, Agile, HTML, JavaScript.

Previously, I discussed some various possible ways to structure the coding of a website, and why I decided to rebuild this site around the static site generator Wintersmith. Today, it’s time to dive a little deeper into what that actually entailed. Don’t worry if you’re not a technical reader; I’ll try to keep it all fairly straightforward.

To produce this website using Wintersmith, there were essential four particular technologies I knew I’d need to know. Firstly, the basics: HTML and CSS, as if I was writing every single one of the four-thousand-odd HTML files that currently make up this website from scratch. We’ll probably come onto that in a later post. Second-and-thirdly, by default Wintersmith uses Markdown to turn content into HTML, and Pug as the basis for its page templates. Markdown I was fairly familiar with as it’s so widely used; Pug was something new to me. And finally, as I said before, Wintersmith itself is written using CoffeeScript. I was vaguely aware that, out of the box, Wintersmith’s blog template wouldn’t fully replicate all of Wordpress’s features and I’d probably need to extend it. That would involve writing code, and when you’re extending an existing system, it’s always a good idea to try to match that system’s coding style and idioms. However, I’d come across CoffeeScript briefly a few years ago, and if you’ve used JavaScript, CoffeeScript is fairly straightforward to comprehend.

The Plain People Of The Internet: Hang on a minute there, now! You told us up there at the top, you were going to keep all this nice and straightforward for us non-technical Plain People. This isn’t sounding very non-technical to us now.

Ah, but I promised I would try. And look, so far, all I’ve done is listed stuff and told you why I needed to use it.

The Plain People Of The Internet: You’re not going to be enticing people to this Wintersmith malarkey, though, are you? Us Plain People don’t want something that means we need to learn three different languages! We want something nice and simple with a box on-screen we can write words in!

Now, now. I was like you once. I didn’t spring into life fully-formed with a knowledge of JavaScript and an instinctive awareness of how to exit Vim. I, too, thought that life would be much easier with a box I could just enter text into and that would be that. The problem is, I’m a perfectionist and I like the site to look just right, and for that you need to have some knowledge of HTML, CSS and all that side of things anyway. If you want your site to do anything even slightly out-of-the-ordinary, you end up having to learn JavaScript. And once you know all this, and you’re happy you at least know some of it, then why not go the whole hog and start knocking together something with three different programming languages you only learned last week? You’ll never know unless you try.

The Plain People Of The Internet: Right. You’re not convincing me, though.

Well, just stick with it and we’ll see how it goes.

In any case, I had at least come across CoffeeScript before at work, even if I didn’t use it for very much. It went through a phase a few years ago, I think, of almost being the next big language in the front-end space; but unlike TypeScript, it didn’t quite make it, possibly because (also unlike TypeScript) it is just that bit too different to JavaScript and didn’t have quite so much energy behind it. However, it is essentially just a layer on top of JavaScript, and everything in CoffeeScript has a direct JavaScript equivalent, so even if the syntax seems a bit strange at points it’s never going to be conceptually too far away from the way that JavaScript handles something. The official website goes as far as to say:

Underneath that awkward Java-esque patina, JavaScript has always had a gorgeous heart. CoffeeScript is an attempt to expose the good parts of JavaScript in a simple way.

Now if you ask me, that’s going a little bit far; but then, I don’t mind the “Java-esque patina” because the C-derived languages like C# and Java are the ones I’m happiest using anyway. CoffeeScript brings Python-style whitespace-significance to JavaScript: in other words, whereas in JavaScript the empty space and indentation in your code is just there to make it look pretty, in CoffeeScript it’s a significant part of the syntax. My own feeling on this, which might be controversial, is that the syntax of CoffeeScript is harder to read than the equivalent JavaScript. However, despite what some people will tell you, there’s no such thing as an objective viewpoint when it comes to language syntax; and as I said above, as Wintersmith is written in CoffeeScript, the best language to use to change and extend its behaviour is also CoffeeScript.

Wintersmith, indeed, is designed for its behaviour to be changeable and extendable. By default it only has a fairly small set of capabilities. It takes a “content tree”, a particular set of files and folders, and a set of templates. Markdown files in the content tree are converted to HTML, merged with a template, and written to an output file. JSON files are treated in almost the same way, almost as content files without any actual content aside from a block of metadata. Other filetypes, such as images, are copied through to the output unchanged. So, to take this article you’re reading as an example: it started out as a file called articles/we-can-rebuild-it-we-have-the-technology-part-two/index.md. That file starts with this metadata block, which as is normal for Markdown metadata, is in YAML:

---
title: We can rebuild it! We have the technology! (part two)
template: article.pug
date: 2020-09-28 20:09:00
...
---

I’ve configured Wintersmith to use a default output filename based on the date and title in the metadata of each article. This file, therefore, will be merged with the article.pug template and output as 2020/09/28/we-can-rebuild-it-we-have-the-technology-part-two/index.html, so its URI will nicely match the equivalent in Wordpress. So there you go, we have a page for each blog post, almost right out of the box.

That’s fine for individual article pages, but what about the home page of the blog? Well, Wintersmith is designed to use plugins for various things, including page generation; and if you create a new Wintersmith site using its blog template, you will get a file called paginator.coffee added to your site’s plugins folder, plus a reference in the site configuration file config.json to make sure it gets loaded.

"plugins": [
    "./plugins/paginator.coffee"
]

The code in paginator.coffee defines a class called PaginatorPage, which describes a page consisting of a group of articles. It then calls a Wintersmith API function called registerGenerator, to register a generator function. The generator function looks over every article in the content/articles folder, slices them up into blocks of your favoured articles-per-page value, and creates a PaginatorPage object for each block of articles. These are then output as index.html, page/2/index.html, page/3/index.html and so on. There, essentially, is the basis of a blog.

If you’ve used something like Wordpress, or if you’re a regular reader of this site, you’ll know most blogs have a bit more to them than that. They have features to categorise and file articles, such as categories and tags, and they also have date-based archives so it’s easy to, say, go and read everything posted in May 2008 or any other arbitrary month of your choice. Well, I thought, that’s straightforward. All we have to do there is to reuse the paginator.coffee plugin, and go in and fiddle with the code. So, I copied the logic from paginator.coffee and produced categoriser.coffee, archiver.coffee and tagulator.coffee to handle the different types of archive page. Pure copy-and-paste code would result in a lot of duplication, so to prevent that, I also created an additional “plugin” called common.coffee. Any code that is repeated across more than one of the page-generator plugins was pulled out into a function in common.coffee, so that then it can be called from anywhere in the generator code that needs it. Moreover, this blog and the garden blog are structured as separate Wintersmith sites, so I pulled out all of the CoffeeScript code (including the supplied but now much-altered paginator.coffee) into a separate shared directory tree, equally distant from either blog. The plugins section of the configuration file now looked like this:

"plugins": [
    "../shared/wintersmith/plugins/common.coffee",
    "../shared/wintersmith/plugins/paginator.coffee",
    "../shared/wintersmith/plugins/categoriser.coffee",
    "../shared/wintersmith/plugins/tagulator.coffee",
    "../shared/wintersmith/plugins/archiver.coffee"
]

The original paginator page generation function has now turned into the below: note how the only logic here is that which slices up the list of articles into pages, because everything else has been moved out into other functions. The getArticles function weeds out any maybe-articles that don’t meet the criteria for being an article properly, such as not having a template defined.

env.registerGenerator 'paginator', (contents, callback) ->
  articles = env.helpers.getArticles contents
  numPages = Math.ceil articles.length / options.perPage
  pages = []
  for i in [0...numPages]
    pageArticles = articles.slice i * options.perPage, (i + 1) * options.perPage
    pages.push new PaginatorPage i + 1, numPages, pageArticles
  env.helpers.pageLinker pages
  rv = env.helpers.addPagesToOutput pages, 'default'
  callback null, rv

This is the simplest of all the page-generators: the others have slightly more complex requirements, such as creating a fake “Uncategorised posts” category, or labelling the archive page for January 1970 as “Undated posts”.

There we go: my Wintersmith installations are now reproducing pretty much all of the different types of archive that Wordpress was handling dynamically for me before. The next time I come back to this topic, we’ll move onto the template side of things, including some nasty performance issues I found and then sorted out along the way.

The next part of this post, in which we discuss website templating using Pug, is here

Keyword noise: blogging, Wintersmith, coding, websites, CoffeeScript, refactoring.