How to Solve the Problem of Knowing What to Automate—and What to Leave Alone

How to make your team faster without wasting time on the wrong kind of “efficiency.“

There’s always too much work. Always. Backlogs never end, deadlines don’t care, and some of the stuff you’re dealing with is so old and clunky it should come with a warning label.

In infrastructure, DevOps, product, or frankly any team, the temptation is the same: automate everything. But that’s not actually the move. First, you’ve got to ask: what’s worth automating, and what’s just noise?

Let’s break it down.

Step 1: Write the Runbook (even though it takes time)

Before you even think about automation, start with documentation. I don’t care if you call them runbooks, guides, cheat sheets, or Grandma’s Secret Playbook—the idea is the same: anyone on the team should be able to follow the steps and get the same result.

If a new hire, an intern, or someone on another team can’t follow your process, then your automation is going to be fragile, incomplete, or just straight-up wrong. Runbooks force clarity. They expose hidden institutional knowledge. They make you less of a bottleneck. And they are, quite literally, the blueprint for automation.

Where to Keep Your Runbooks (So People Actually Use Them)

Here are five popular, accessible places teams store runbooks:

1. Confluence – Great for structured documentation with permissions, templates, and easy search. Integrates well with Jira.
2. Notion – Flexible and fast. Good for teams that want docs, checklists, and embedded media in one place.
3. GitHub / GitLab / Azure DevOps Wikis – Ideal if your team lives in version control. You can manage updates via pull requests and keep docs alongside code.
4. Google Docs – Everyone knows how to use it. Good for quick sharing, collaborative editing, and low friction access.
5. Internal Portals (e.g., SharePoint, custom tools) – If your org has a central knowledge hub, it’s often best to store runbooks there for visibility and long-term adoption.

No matter where you put them, the key is this: make sure they’re easy to find, easy to follow, and easy to update.

Step 2: Track Usage

Next, figure out how often these things are actually being used. This is where most teams fail—because people just do the work silently.

Use whatever tool you have (JIRA, Azure DevOps, Notion, even a spreadsheet) and tag tasks related to each runbook. Use labels like manual, runbook, candidate-for-automation, whatever makes sense. The goal is to create a trail of usage you can query later.

For example, on our team, we have a runbook for binding sites to IIS (I know it’s old school). Part of it’s automated, part of it’s manual. But I can look back over the last 3–6 months and see how often that runbook has been referenced or followed—and from that, we can make an actual decision based on usage data, not just gut feel.

Here is an example of a query to see how often these type of tasks are completed in the last 3 months based on a label for team devops

labels = candidate-for-automation AND (created >= -90d OR updated >= -90d) AND team = DevOps AND status = Done ORDER BY updated DESC

Step 3: Spot the Pattern

Now that you’re tracking it, ask yourself: how often is too often?

For most teams, anything done more than once or twice a week is probably worth automating. But this depends on complexity, risk, and how many people touch it. If it’s something that breaks often, blocks other work, or frustrates your team, it’s a high-value target—even if it only happens once a week.

Runbooks also help you “modularize” the work. You can pick out just the high-friction parts to automate first and leave the rest for later.

Step 4: Automate the Runbook

Here’s the trick: your runbook is the automation spec.

Once it’s solid, literally attach it to a JIRA ticket and hand it off to your devs or platform engineers. Make sure it includes:

• Clear steps
• What success looks like
• Where it runs (env, triggers)
• Known edge cases

Boom—you just went from institutional knowledge to automation without guessing.

🔧 Practical Takeaways

• Write clear, step-by-step runbooks before automating anything
• Tag and track runbook-related tasks in your task system
• Query usage over time to see what’s worth automating
• Prioritize anything happening weekly or more
• Use the runbook as the blueprint for your automation spec