Agility part two: Burndown tips and tricks

Posted on by
4

Further to my post last week on agile processes, here are more ideas on how to use a burndown chart.

Tip 1: Separate Projects

I’m currently running with two burndown charts: one for my software job, and one for my writing. This makes sense because the only interaction between them is when one steals time away from the other – there is no dependency between any of the tasks in either place.

The software burndown is a text file generated every other week along with my weekly tracking log, while the writing burndown is a spreadsheet. This is my writing burndown chart for the first three weeks after Beltane (or thereabouts – May Day was mid-week and I like to start my sprints on a Monday).

I actually started this burndown a week into the sprint period, hence the blank week at the beginning. I use OpenOffice and LibreOffice for my spreadsheeting needs, but I believe recent Excel versions can read these too.

Tip 2: Keep Sprints Short

I use a two week sprint for my software burndown since that coincides with my working schedule, and a three week sprint on the writing. I’ve seen four week sprints work, but a week is too short – particularly in a team environment, because you always seem to lose a day to sprint planning and if you only five days to work with, then that is large overhead.

The writing case is interesting because of the low daily load – I really can’t expect to spend more than two hours a day on my creative projects, so I tried setting up six week sprints for that so as to be able to achieve a significant amount over a sprint (the six week length would also have matched the period of my progress posts)

However, that was too unwieldy and rather overwhelming – it’s easy to come up with six weeks of work (for example, the “outline all the scenes” task I put in on the above-linked sheet) but breaking down tasks into achievable chunks too far ahead of time is exactly the kind of brittle long term scheduling that this technique is intended to avoid.

Tip 3: Add Tasks For Follow-Up Work

On the Beltane burndown sheet, I have a task “brainstorming” for the New Dawn project which I implemented by scribbling on my Noteboard. When the brainstorms had passed, the follow-up task was to capture the contents of the Noteboard in a more durable form – that could have been a photo, but I wanted a Freemind mind map. So I created a task for that work mid-sprint.

Tip 4: Keep a Backlog

My sheet has an “above the line” and a “below the line” section. Tasks that are active are above the line – all of the formulae operate on a range down to the line boundary – while below the line is the holding area for tasks that have been pushed out of the sprint or which are not yet planned in.

In Agile terms, this is the backlog: tasks which are known to need to be worked on, but which aren’t actually in a sprint yet.

Backlog items may be very specific, or they may be amorphous blobs of work that encompass an entire project. For requirements-driven creative endeavours, this backlog can be very large and may need to be put somewhere else (such as a ticketing system). For generative creativity the backlog is likely to consist of work phases that will need to be started once the current phase is done.

The division between a well-formed backlog and an endless to do list can be fuzzy.

In the example sheet, the “outline all scenes” task used to be above the line. I don’t have any other backlog items on here yet.

Tip 5: Split Tasks That Are too Big

Sometimes tasks are bigger than we think at first, or there are consequential tasks that come out of the initial work.

The “outline all scenes” task I mentioned before was too large and ill-formed to be planned as single task. So I have moved it out of the active task area and am working instead on outlining the different acts of the story. I expect the actual amount of time taken will be equivalent, but having small tasks with specific objectives will be easier to stay motivated on.

Tip 6: Postpone Tasks Which Don’t Fit Any More

If one piece of work takes longer than expected so that there is no longer time in the sprint to complete other tasks, then push those tasks out (ie move them to the backlog).

This was initially what happened to the “outline all scenes” task, although that was malformed for other reasons of course.

Tip 7: Add Tasks For Unexpected Work

Let’s say that I am working on a story and I suddenly have a burning idea for another tale. What I ought to do is to note it down and carry on with the task in hand, but that isn’t always the right approach if the new story is time-sensitive, for example, or if it’s one of those stories that just shoulders all else out of its path shouting “Look at ME!!!”

If you spend an hour or more on this unexpected work, add a task to the burndown to capture that you spent time on that rather than what was planned – even if there is no more work to be done on it. This leads to erratic zeroes floating around, but it’s helpful to me at least to be able to look back and see that I was doing something with my time even if it was not exactly what I had intended.

In the example spreadsheet, the “outline transfer” task was one that I wasn’t really intending to do but it was necessary to put everything I needed in one place. The copying took about an hour and was done in a day’s writing time.

Do you have other task tracking or planning tricks you use to keep yourself on task?

Agility

Posted on by
Reply

I’ve been doing software for long enough that I have seen a silver bullets: methodologies that will magically slay the monster of out-of-control development times, while banishing bugs and washing whiter than white.

They don’t, of course. Software is an unpredictable thing to make because of its innate complexity: the approaches which are most durable are those which emphasise adaptability. All of these are built on iteration, and the best of breed are the agile methods*. These cover best practices for development, testing, requirements gathering, and signing up for tasks, but the bit which is most revolutionary and which can be applied to any project – including, of course, writing – is the task tracking mechanisms.

In a standard project plan (for example, a Gantt chart or any kind of dependency graph) you would go through your list of tasks and make an estimate of how long you think it will take. Then you’d break down the task into smaller pieces and fit the work into the initial estimate. If you’re smart (or lucky, depending on circumstances) you’ll adjust your overall estimate based on how long the sub-tasks take, but you would still make an overall estimate and then work towards that. The thing you make is based on the initially agreed features, and the release will slip if any of those features are not ready.

What agile methods do is to set a fixed interval for releases, and then divide up that interval into short work periods called sprints – two weeks is a pretty common sprint length. Classically, you would be ready to deliver at the end of each sprint although I’ve never worked that way. Rather, you pick tasks that will fit into the sprint. When you are making estimates, you always pick tasks that you can estimate at two or three days of effort. If a task is too big to manage in that time, then break it into smaller tasks that can.

Here’s the clever bit.

At the end of each day, you estimate how much effort is left on the tasks you have. This is called the burn down.

That doesn’t seem terrifically powerful, but it means you can adjust: if a task is more complicated than it appeared at first then you can expose that early and get help or push out tasks to the next sprint that will no longer fit. If a task took less time than initially expected (it does happen…) then you can pull in more tasks, or help someone else who is having trouble. The analogy often drawn is between firing a gun and driving a car: everything about firing a gun is in the aim, whereas when you’re driving a car you make adjustments and steer the car as you go.

This task tracking is what I like to use for other projects. I will generate a list of broad project activities, and then break each activity into more specific tasks. For a novel, this may be outlining, then a list of scenes or chapters to be written, then reviewed, then edited, and so on. In point of fact, during the drafting process the novel is its own progress metric since you see the word count climbing and the number of completed chapters rising, but for some tasks which are bit more gnarly then the burn down chart may be more immediately useful. A burn down will help tell you if you are writing a lot more or less than initially planned, anyway.

Another major benefit of the burn down is that you can learn how long it really takes you to do particular kinds of tasks. If you always estimate consistency edits on a chapter as taking two days, but it always actually takes you three then you have the information to learn that you should start estimating it at three days in the first place.

How do you track a burn down? Well, you can find tools on the web although they are mostly not free and tend to be tuned towards software development. And really, you don’t want to be learning Rally unless you have to. For my own projects I tend to either work in a spreadsheet or a text table.

Let’s consider a short story project, for example, since that might conceivably fit into a two week sprint. I’m going to name the sprint Pony, for reasons which will become obvious, and I’m going to assume a six day week, and that you can write for two hours a day. All estimates are in hours.

Pony sprint – a short story about ponies Est M T W T F S M T W T F S
brainstorm pony story ideas 2
outline story 2
write first draft 4
edit for consistency 4
edit for plot 4
edit for character 2
edit for word choice 2
polish 4
submit 1
Time left 24 22 20 18 16 14 12 10 8 6 4 2 0
Total (hrs) 27
Total (%) 112

The top line of the left column is the sprint name, and its theme (an entirely optional element, but useful to give overall direction). Most of the rest of that column consists of tasks you think you need to do for the story. Other stories might need more research tasks (what do ponies like? How much sleep do they need? Would an unfamiliar pony be eaten by the herd?) but here I mostly have writing tasks. Then at the bottom we have headings for the time totals.

The second column is where the numbers start: an estimate for how many hours each task is expected to take. The totals at the bottom of the column are the number of days left multiplied by the amount of time available at the end of each day – this is the time left in the sprint – and totals for the hours expected to be worked and the percentage of those hours compared to time left. The percentage (also called loading) is particularly interesting, because it gives you some sense of how likely you are to finish in the sprint. In this case, high is definitely bad… normal loading at the start of a sprint in a software project would be 80-90% to allow for the unexpected.

The remaining columns are the days in the sprint. I’ve labelled these with days of the week, but calendar dates are more usual.

As you progress through the sprint, at the end of the work period you note the hours remaining to be done on each task. So, if at the end of Monday the brainstorming is done and you’ve made a good start on the outlining, then you can complete one task and burn off some time from another. Then you add up all the remaining tasks and recalculate the totals at the bottom.

An obvious benefit of a spreadsheet is that all this adding up can be done for you.

So, let’s look at this burn down chart after a week.

Pony sprint - a short story about ponies Est M T W T F S M T W T F S
brainstorm pony story ideas 2 0
outline story 2 1 0
write first draft 4 4 2 0
edit for consistency 4 4 4 3 3 0
edit for plot 4 4 4 4 4 2
edit for character 2 2 2 2 2 1
edit for word choice 2 2 2 2 2 2
polish 4 4 4 4 4 4
submit 1 1 1 1 1 1
Time left 24 22 20 18 16 14 12 10 8 6 4 2 0
Total (hrs) 27 22 19 16 16 10
Total (%) 112 100 95 89 100 71

From this we can see that the brainstorming went well (zero hours left) and outlining was started early (one hour remaining). The outline was completed and first draft was started and half done on Tuesday, then complete a day early on Wednesday. The consistency edits went slower, either because there were more problems than expected which needed fixing, or because other things popped up that interfered with the work (on a software project, I would usually add tasks in to document these other things, but that’s not really appropriate here). Finally, at the end of the week, the plot editing started along with some character tweaks.

Looking at the loading along the bottom, you can see that it dropped below 100% on Tuesday and kept falling, but drifted back up as the first edit task stalled. However, that was recovered at the end of the first week and this story looks like it will be finished by the end of the two weeks.

Is this tool for everyone? Not at all, but I find it useful to keep me motivated. There’s a gamification** element here – the game is to keep the load as low as you can – which makes this quite a fun planning tool as well.

[*] there’s a whole cluster of these, but a full history is not very relevant here. The first widespread one was XP (eXtreme Programming) and the one I have encountered most frequently in my professional life is Scrum.

[**] yes, this is a hideous word, but it is a term of art which captures a useful concept of making otherwise intractably large tasks manageable by turning their completion into a game.

Outlining, part two

Posted on by
Reply

Right at the moment I have a broad outline: events are described in more or less detail down to the scene, and all of the scenes serve some kind of purpose.

What’s next?

Traditionally in my process this is where I would start NaNoWriMo and splurge words onto the page, but I am trying to be more deliberate and thoughtful here – attempting to observe how to make storytelling work by choice rather than by accident.

There are two things I will work on now.

Firstly, reviewing the characters – ensuring that the characters I have in play are being consistent, and have known motivations (both positive and negative) – by “known” here I mean known to the author rather than known at this point by the reader. Getting to know the characters, basically, but also making sure that the characters have a purpose in being there.

Some writers do character building a lot earlier of course – I’ve done it earlier on other stories – but this project, Song, was a story where characters for the most part emerged as they were needed by the plot and were coarsely sketched. Even when I have characters described before the initial drafting starts I don’t always have everything captured that I need, so some aspect of this character consistency work will probably be needed for any novel I work on.

In any case, I’ve started character description sheets for the major and supporting cast which reveals the depth of my ignorance about most of the characters.

(a digression – I have encountered writers using roleplaying game character sheets as templates for this kind of character description, and I think there’s a place for that, but most RPGs are too crunchy* for that to work well. If having game stats for your characters is going to help you visualise them then I say go for it, but I would find myself getting too caught up in the details)

My second task is to make more detailed notes on the scenes themselves: not writing the words, but laying out in rough descriptions what the scene structure is.

This is straying into word-making mode since this is where I will begin working in the manuscript itself: I will copy the scene descriptions from the scratch area where I’ve been working and start in on detailed outline in the scene documents.

From this workflow you may gather that I have not been using Scrivener for all of my outlining. My spreadsheet has columns for event, characters, and various kinds of notes. There aren’t enough top level fields in the Scrivener outline view to accommodate all of that: by default you have the scene summary card, and you could subdivide that with format conventions, but I would rather have this stuff more explicit.**

One interesting side effect of this process is that I will be abandoning the chapter structure from my original draft, working with scenes alone. We’ll see how I divvy up the text – not every book needs to be arranged in chapters – but keeping it more free-flowing now will help in inserting necessary scenes that wouldn’t have fitted into the former chapter structure.

Anyway, that’s the plan. See how it pans out.

Also, the profane Saint Chuck of Wendig has just posted 25 Things You Should Know About About Outlining – worth a look.

[*] crunchy here means having lots of stats to play with, and lots of game system details to manipulate to maximize the power of your character.

[**] I’ve since found that I can add this kind of custom information to Scrivener, but I’ve mentioned before how I tend to get too involved with the tool and not enough with the work so didn’t find it soon enough. I will probably spend a bit of time copying all of this across from the spreadsheet – pretty good way to stay in touch with the story though!

Directions

Posted on by
5

I identify very firmly as a novelist – I haven’t written much short fiction, and every time I try the plots and the characters and the story just seem to expand so that I end up with another novel.

However, writing short fiction seems like a generally good thing: it’s much easier to workshop a complete story than it is for long form pieces, and being able to hand someone a 2k or 5k piece and say “look, this is what I do” makes short fiction a useful marketing tool too.

The risk involved in writing a piece of short fiction is also lower than with a novel. A novel takes at least a year to write, especially with a full time job. Jamming out an incomplete first draft can be done in a month, but turning the book into something that other people will want to read will take the rest of the year at best. And sometimes novels don’t work. I sometimes look at the morass I have walked into with Bluehammer and wonder if I will ever be able to walk out again, or whether the years I have spent are best treated as a learning experience and if I should just call the helicopter to come and rescue me.

Short fiction is, in those terms, a more attractive option to write. It’s not for nothing that experimental work tends to be done in the short form.

I also look at the recent experiments in serialised novels – specifically John Scalzi’s success with The Human Division, or Charles Stross’s Accelerando – and wonder if those are worth exploring.

Are we moving into a phase where short stories are a way to make money again?

Can I do both? Can I do more than two things?

Chuck Wendig has spoken pretty consistently about trying lots of things to see what sticks, and Neil Gaiman’s recent keynote at the Digital Minds Conference counselled much the same thing: do many things, rather than just the one. And writing novels is just the one – diversification is the future.

But I still identify as a novelist; I still want to build my complex plots and large casts and detailed settings. I still want my worlds to live.

And how, honestly, can I commit to doing more when I haven’t even finished the one thing?

This is a very confusing time. There is no map, and the compasses don’t work any more – the landscape is changing as we walk it.

People will always want stories, though. That much is certain. I just need to find good ways to make the stories and get them into people’s hands.

The other goal

Posted on by
Reply

In talking about my writing goals, I omitted to mention my health goals, mostly because they’re not directly related to my writing.

I have been working to change my habits, mainly. I try to avoid snacking – breakfast, then lunch, then dinner, but minimise the grazing unless I am actually hungry. It’s difficult to manage at the job because there is so much food there, but I’ve been doing fairly well. Drinking more water has helped a lot.

I have been getting back to more frequent running. I really need to run four times a week to improve my fitness level, and I managed that this week (four days in a row, in fact, which I don’t do often at all). Indeed, I’ve been doing more hills and this week I broke nine minute miles for the first time in a long time.

But I think the simplest thing is that I have been walking up the stairs to get to the job in the morning rather than taking the lift. Four flights of stairs, enough to get me out of breath, but that’s been helping my whole day. I’ve only been doing that for a couple of weeks and I can feel it helping me already.

Anyway, here are the goals I set and what I’ve been doing for them.

  1. to track what I eat, with the intent being to keep daily intake below a certain threshold – as I say, I have been cutting the snacks and also trying to wait to have seconds or afters. I will get back to tracking in due course.
  2. to track what I weigh – I started a regular Monday morning weigh in about a month ago. I am not down much, but I am down.
  3. to continue exercising even when I am labouring in the word mines – can’t comment on this really, since I haven’t had an intense word-mining session in a while. See how things go once I start on the Song rewrite in earnest.

Sometimes, though, just being conscious of these things is helpful.

The Writer’s Notebook II

Posted on by
Reply

I’m a sucker for orange.

Orange is my favourite colour, and so whenever I see a book cover that features orange on it I am always going to give it a second glance.

When I was at Wordstock last year, I picked up a freebie from the Tin House booth: an essay extract from The Writer’s Notebook II, with a bold stripe of orange across the cover. While waiting for one of the panels to start, I opened it up and started to read.

After the panel, I went back to the TIn House booth and bought the book.

The Writer’s Notebook II is a collection of craft essays. They are arranged roughly by sequence of story telling, so the beginnings essay appears first and the endings one is at the end, and there are pieces on the use of poetic language, on suspense, on non-linear storytelling, and even on the use of fantastic elements in your writing.

Not all of the essays connected with me, but many of them did, leaving behind inspiration to be more precise in my writing, to be more effective – just generally more, in fact.

The only real sourness I found was the occasional literary jab at genre fiction – oddly, there was a hint of that even in the piece which extolled the virtues of fantastic elements. It’s very curious. I would say that I don’t understand, but that would be disingenuous – I have my own biases against literature when labelled as such. But it’s all writing, and good writing is just good regardless of the target market.

Still, it’s an insightful collection. I will be referring to this again

Tools of scribbling

Posted on by
Reply

I’ve talked before about my capture and tracking system for lists and project details, but that post only concerned itself with electronic process. This post is about things for hand-written notes.

Notebooks

I almost always carry a pen, and I very often carry a notebook. The one I have with me most often at the moment is a small leather-bound notebook from PaperThinks which I bought at a local gift shop when it was going out of business. The pages are 3″x5″ plain paper (that is, not lined or gridded) and the cover (of course) is orange. I use it for writing notes in situations where an electronic device is too bulky or inappropriate, or if I want to make a drawing. To be honest, I should probably use a larger format page than this for comfort, but this size is very good for portability.

One thing I do with notebooks like this is that when when I consume a note, I cross it out – usually just a single line through the page, but I will sometimes break down the erasure to particular concepts if that’s how the consumption works best. Crossing stuff out means that its management has been moved into the more searchable electronic realm and I can ignore it on the page.

Drawings

I do not make any claim to be an artist. My drafting skills are strictly utilitarian rather than capable of conveying any kind of emotional content, but there are times when I need to draw something to work through it or record a concept diagrammatically since that is just quicker than writing out (say) a list.

I’ll draw on the pages of my notebook or on 3″x5″ index cards, but I also have stacks scrap paper on my desk and blank-leaved journals on the shelves in my office at home.

What I draw are maps, relationship diagrams, plans, and sometimes even pictures of scenes.

Scribblings

Something new I saw recently was the Noteboard, a folding whiteboard which comes in a pouch with a pen. It looks pretty useful – portable and flexible. The little video on the page shows a lot of things which I would still probably prefer to use paper for, but having a reusable scrap surface could be very useful for noodling around with things.

Writing

I don’t write drafts longhand. I do sometimes write character notes in the aforementioned journals, and if I do am using one of two kinds of pen:

  • a Uni Jetstream – my youngest sister introduced me to these pens, and I love them. The ink flows freely and evenly, and I can write with one of these for a long time.
  • a fountain pen – I have a Cross and a Waterman, both of which write beautifully smoothly when they are clean. That’s the rub, of course. Since I mostly write with my Jetstreams now, the fountain pens aren’t often in use enough to stay usable.

What tools do you use for taking notes when you are electronically embarrassed?

I’m in the mood for writing

Posted on by
Reply

“I’m in the mood for dancin’, romancin’

Ooh I’m given’ it all tonight”

- The Nolans

I’ve written before that words need to be mined whether you feel like it or not, but there are times when one does feel more like writing than other times.

For me, this is partly a brain thing. My ADD and other chemical brain factors mean that there are times when I am too ground down to be very creative, or if I have had a cold I can still work but not always very imaginatively. The up slope is much more fun.

As an example, I had a cold recently and it just did me in. I couldn’t write, I couldn’t think. I was doing garden projects and stupid online stuff just to occupy the time.

But when I came back from the cold I found that the ideas were flowing. I had an idea for an interesting board game mechanic based on concentric circles. That led into the idea of dice of different sizes (shades of the Savage Worlds system, there) and whether they could be used for the strategy game in Bluehammer. My writing on tools led me to mentally revisit DataFrame again, and I started wondering about implementing that as a combined wiki and graphing tool: one pane is a wiki editor, the other is a graph rep of that wiki. That sounds quite doable, in my copious free time.

… and so on. Lots of good stuff, which at least shows that the engine is turning over.

It’s a very exciting time.

Second Chances

I should also add that I first started writing this post in the wake of our trip away to California over spring break. I returned refreshed and reinvigorated, but the ebullient mood lasted barely a day of being back at work, insufficient sleep, and various other stresses before my creative flow dried up again. It is often said that our brains are better at doing different things at different times of day, an observation which I don’t disagree with but which I have never fully applied to my own situation. The same is true, at least for me, of emotional cycles over longer periods.

It is important to keep creating at all times, but it is also important to be aware of what creative work you can do depending on the spot in the cycle at the time.

2013 Goals Post: Beltane Edition

Posted on by
1

The lesson from the last six weeks is that I should have been writing – this has not been a productive period.

1. Finish Bluehammer

Last action: fix the outline.

I haven’t done any direct work on this, but I have read a craft book which has given me more to think about and the characters and setting are certainly in mind.

  1. typo/consistency edit
  2. improve outline – ongoing.
  3. make the text match the outline
  4. hone the text
  5. make submission materials

Next action: fix the outline.

2. Execute the Song plan

Last action: polish the outline.

This is basically all I have been working on over the last six weeks*.

I do have a better story with stronger plot and characters, but it is still a mess of unconnected scenes rather than a narrative.

I did make some interesting decisions about the story, though, such as switching to a first person POV, and defining characteristics of the antagonist, but most of this has been head work rather than keyboard work.

Right at the minute, I am fleshing out my outline using a spreadsheet to fill in details about scenes so that I analyse the outline further. So, I need to finish that then crack on with the rewrite. I am going to give this another two weeks.

  1. To outline what I have
  2. To expand the outline of the first half into a complete story
  3. To work on that outline until the story is good.
  4. To plug in text I can use from Song 2011 – this is no longer a meaningful goal since I need to rewrite everything.
  5. To write the new stuff needed to complete the outline – or, as it should be cast now: To rewrite to match the outline.
  6. Make submission materials – synopsis, pitch, hook, and all of that.

Next action: populate the outline.

Next action: rewrite.

With the outline be done in two weeks or less, I will want to get on with doing the necessary rewrite. I expect that next draft to take about two months.

3. Submit one novel.

Last action: finish a novel

Still working on Song as the most likely submittable thing this year, but if anything I am further from submission than nearer to it. We will see what the summer brings.

Last action: to find some markets

No work on this.

4. Start looking for an agent.

Last action: research agents who represent science fiction.

No work on this.

5. Establish a daily writing practice.

Last action: write every day.

This blog has been my most consistent fingers-on-keyboard writing, and I am glad I have had it for that reason alone, although there have been times where I have been blogging instead of writing fiction which is less satisfactory. Much of my time has gone on other things such as garden projects and even the last throes of tax season – and the idea that I would get any writing done on spring break was fanciful.

I have been working on my stories even if I have not been adding words, but I need to be adding words.

But I’m really looking forward to getting down to the rewrite on Song.

The next six weeks are going to be critical in keeping going, because after that it is summer and there is a constant pressure to spend time on family and household activities instead of writing – which is, y’know, what makes life worth living, but it doesn’t get the book written.

Last action: participate in April Camp NaNoWriMo.

That didn’t go so well.

Overall

I think the best that can be said is that I have been breathing in. This blog is still going, but other than that I have been struggling to stay focussed on the work. This is going to change.

Next update will be a little after the summer solstice (21-Jun-2013). The solstice itself is my writing weekend, a generous birthday/father’s day gift from my family so I will update again after that.

[*] and what is another term for “six weeks” in this context? The Wiccan/pagan festivals I am using as my update markers are also known as sabbats, but is there another term for the time period between sabbats? Intersabbat? Sabbatical? Sabbatini?

Curation and the inevitable data apocalypse

Posted on by
1

I have a lot of data. Not, I am sure, a lot by the standards of an experimental scientist or someone who shoots a lot of video or takes a great many photographs, but a lot of small files which I don’t want to lose.

I have also had a number of data losses over the years – some from media failure, some from user error*. The most catastrophic was a motherboard failure on an old PC In the days when storage was expensive and removable discs were tiny. The computer worked for the most part, but there was a fault which meant that it could not calculate checksums correctly. The first things to go were the compressed directories that I had set up so as to make better use of the hard disc – that was the signal to me that I needed to do a comprehensive backup. Which I did, very carefully zipping up all the stuff I cared about most onto multi-floppy archives. An operating system reinstall did not fix the problem, and I was more than a little put out when the computer came back from the mender and I found that every single one of those archives was corrupt, the cherished data lost.

I suppose I should be glad that this happened before I had a digital camera, but I lost a lot of writing.

I have just performed an upgrade on my laptop to allow more current software to run on it, and was looking at my hard disc to clear things out a bit. What I found was nested machine images, copies of previous computer hard drives resting inside each other, copies made as I have moved from machine to machine to machine: yig contains wendigo, which contains ithaqua, which contains dagon, nyogtha and daoloth – generations of computers going back fifteen years. There’s even a disc image from the first machine I owned with a hard disc in it, kurt the Acorn Archimedes, although that was recovered in a recent bout of data archaeology.

Each previous image contains the seed of the data I have on the system that superseded it.

I’ve been using the same basic layout of my data directories for a long time, so I can trace the development of projects over the lifetime of these machines. I have early versions of programs and websites that I’ve maintained over this whole period, and larval versions of stories, and early snapshots of photo directories. I also have multiple copies of large data projects, like the effort to digitise my vinyl, which at least means I haven’t lost anything.

The question really comes down to how much of this ancient data I want to keep around. The large data projects I want to make sure are backed up in a couple of places, but the prehistoric data directories? Not so much, I think – make sure nothing is unique, but I feel less need to retain these old versions than I once did, when I started a repository project which was intended to act as a versioned history of all the work I have done.

Half the data on my hard disc on yig is duplicated, or stale, or just doesn’t need to be available at all times.

Time to clean up.

[*] aka PEBKAC – Problem Exists Between Keyboard And Chair