Developer pays volunteers €25 an hour to follow his README aloud, and finds a long list of mistakes

Terence Eden says the best way to test a README is to watch real people try to follow it. In a blog post, he describes paying volunteers to do exactly that for ActivityBot, an ActivityPub tool, and what he learned from it.

He starts from a familiar complaint: developers are often bad at writing documentation. Sometimes there is no manual at all, just scattered notes, old tips and a forum where the same questions keep coming up. Eden says he has long argued that you should test your README on a fresh virtual machine, which is a clean, disposable computer set up inside your own. The trouble, he says, is that it is hard to ignore your own biases. You already know a command needs sudo, or that -foo really meant --foo, or that a reboot is needed at the end.

So he paid people. In his grant application to NLnet for ActivityBot, he had said he wanted to test the install experience with real users and would pay €25 for an hour of their time. He put out a call on Mastodon, found a few volunteers and met each of them on a video call.

The format was simple. Eden said he wanted feedback on the first-run experience and wanted honest criticism. Each volunteer shared their screen and talked through what they were doing, including what confused, frustrated or pleased them. He took notes by hand.

The sessions showed him a lot of problems. The link to the demo tool was wrong. Some people read the README in the terminal. Some were unsure what it means to rename a file, or how to rename a hidden one. Others asked why some text had to be quoted and some did not, and whether a demo tool should run on the web or on the user's own machine.

There were bigger gaps too. His jokes were not funny and sometimes confusing. Some technical terms needed explaining, and the order of the sections was hard to follow. He had not actually explained what the software would do. One section he found technically interesting was confusing to everyone else. And not all web servers handle permissions the same way.

After each session, Eden updated the README based on what people struggled with, then tested the new version with the next person. In total he paid out about €150.

He expects some people to ask why he did not use an LLM to simulate different users. His answer is that he wanted to talk to real people. He points to his earlier work as a technical writer for GOV.UK, where another person reviewed his writing and caught mistakes a spellchecker could not. Hearing frustration in someone's voice, he says, is what tells you something needs fixing.

Eden does not claim the ActivityBot README is perfect. He says it is now demonstrably easier to follow. His advice to other developers is to find a few people who will talk aloud while they try the instructions. If you cannot afford to pay them, he says, it is still worth asking.