Documentation is always out of date
"Everything Starts Out Looking Like a Toy" #85
Join smart, curious folks to get the Data Ops 📊 newsletter each week (it’s free!)
Hi, I’m Greg 👋! I publish this newsletter on finding data products and interesting data observations with the goal of finding patterns and future product insights. (Also, it’s fun.) If you need a background on how we got here, check out What is Data Operations?
This week’s toy: an open source LEGO microscope. Edition 85 of this newsletter is here - it’s March 21, 2022 (happy Spring)!
The Big Idea
A short long-form essay about data things
⚙️ Documentation is always out of date
Think for a moment. The last time you looked at the documentation for something (most likely software), you found something different than you expected. Maybe it looked like an under-documented flow chart and was missing a few key technical steps. Perhaps it was a line by line description of what actually happened during the process. Either way, you got something different than you bargained for when you were seeking “good documentation.”
One reason for this problem is that it’s not easy to balance the feeling of knowing what the process is going to be with the exact details of what is happening during, through, and after the process. We’re trying to hold in our head the idea of what should be happening and reconcile it with the facts of the matter.
As a user, you want the product team to be describing your perfect use case. As a product owner, you want to be speaking to all of your active (and potential active users). And as a documentation owner, you want to get this set of docs done so that you can move on to the next set without driving increased support tickets.
What’s the best way to get from point A to point B?
Great documentation does a few things well. Even for a well-known idea – the classic one is explaining how to make a peanut butter and jelly sandwich – well-made documention explains the general details and goal of a process, while also letting you know:
Any required pre-conditions for that process to exist (to make a peanut butter sandwich, you need ingredients!)
Entry criteria for the process (you must have decided to make a sandwich).
Actors who will be taking actions (the sandwich maker and the sandwich-eater are not always the same person. Perhaps there are other steps involved as well)
Operations in a standard process (you need to know where the ingredients and containers are, what instruments to use to make the sandwich, and what side of the bread on which to place the peanut butter)
How to respond to unexpected changes in the process, like what to do when your peanut butter is not emulsified and needs stirring
And the exit criteria for the process (how do you know when you’re done)
Great, so why does making a peanut butter sandwich have anything to do with documentation, particularly software documentation?
Documentation is always out of date
A typical software company these days releases as many as several times a day. Documentation may lag days, weeks, or even longer before the person who writes the documentation gets a chance to learn or experience the new features, as the team building the software is not always the team documenting it.
What are some other reasons why documentation might not be up to date?
1. It’s not always easy to know what changes are breaking changes for a customer.
When you introduce a new feature, you know noone’s ever seen it before, so it needs to be documented. But how about when you change the behavior of an older one in a subtle way. You might document the change but not know exactly which customers it affects or where in their process something needs to change.
2. You’re not sure how many people read the documentation
People typically read the documentation when they need to, but before they need it? They might skim or ignore it entirely. So part of the accessibility challenge for documentation is to make it easy enough to absorb quickly while not omitting detail, and also providing ways to dive deeper when someone needs exact details.
3. It takes time to write something good, and you’re going fast
Remember how we’re releasing several times a day? Even if you go through a process of reviewing customer notes a couple of times a month, make sure the information is clear, vet it with the team, and write it well, you’re going to miss some things. As a small team working on documentation, it’s a lot to take on.
A Modest Proposal: Auto-documentation
What can we do to make this a little bit better? We’re not going to get perfect documentation unless building better documentation can be linked to Net Promoter Score, Net Dollar Retention, and so on. (And maybe it can. Companies like Stripe consistently win accolades for their documentation and user empathy, but they also have an amazing product with strong product-market fit).
Great documentation does a few things:
It makes it easy for a novice to gain understanding of the problem at hand, know what’s required to excel, and where to find more information
It makes it easy for an expert to find the information they need with a minimum of fuss.
It’s highly visually oriented while also allowing searchable text strings to find specific information
It updates itself automatically when the underlying code of the software is updated.
Yes, automatically. I’m not talking about using GPT-3 here, I’m suggesting instead that part of every major commit needs to be documentation of “what changed”, “why did it change”, and “how does it change the experience for a typical user.” That way, the metadata included in code releases will help to automatically build documentation that humans can read.
This may sound like a utopian fantasy, yet I really do think it’s possible to build better documentation into the software we build. (Here’s an example from the DBT project). Great documentation starts at the core of providing a better user experience. If a usability (and productivity) test of software starts with “what changed”, it’s a lot harder to get stuck in the feeling that documentation is always out of date.
What’s the takeaway? Building constantly updating documentation should be a goal for every software team. By creating auto-generating documentation, you build a virtuous cycle that’s beneficial for users and developers and sets the expectation of sharing information, not hiding it. Great documentation is a craft, and honing that craft is something we should strive for when building software.
Links for Reading and Sharing
These are links that caught my 👀
1/ Save this link - It’s a document from Chris Paik that will get you thinking. Specifically, it’s a list of frameworks and ways to think about strategy for startups.
When you encounter a statement like this from Kyle, Chris’s document gives you some advice on evaluating the situation so that you know quickly if you are in a horror story or in a great situation when you match a well-known pattern.
2/ Electrified transportation, not a car - I recently bought a hybrid (a Kia Niro) and I’m pretty sure that my next car is not going to be an internal combustion car. Late to the party? Maybe. I’m still not sure about how to handle intra-city travel in an EV. But local travel? Sign me up. Until the prices come down, I’m probably going to consider a late model Nissan Leaf for my next car – I’ll be using Recurrent to measure the battery first – or one of these e-bikes.
3/ How to do less, better - Alex Turek’s short piece on prioritization should be a must-read.
What to do next
Hit reply if you’ve got links to share, data stories, or want to say hello.
I’m grateful you read this far. Thank you. If you found this useful, consider sharing with a friend.
Want more essays? Read on Data Operations or other writings at gregmeyer.com.
The next big thing always starts out being dismissed as a “toy.” - Chris Dixon