How to measure the real impact of AI on your team.

Here's a pattern that plays out constantly right now: a company deploys AI tools, checks in after 90 days, and finds... nothing they can point to. People used it sometimes. Some employees liked it. Some didn't. But nobody can say with confidence whether the business is better off for having deployed it.

Measuring AI impact at work is harder than it should be — mostly because most organizations start measuring after deployment instead of before. The most reliable way to know whether AI is producing real impact is to define and record your baseline before anyone touches the tool. Here's how to actually do that.

Why most AI measurement is measuring the wrong thing

The instinct when deploying any software is to look at usage. How many seats are active. How often people log in. How many interactions happened in a given week. Software vendors encourage this because usage is what they can measure — and it's a metric that trends up, which makes the dashboard look good.

The problem is that none of these metrics connect to business outcomes. An employee who opens Claude every morning to ask it to rewrite subject lines she would have sent anyway is using AI — and producing exactly zero incremental value. The G-P survey of 6,000 executives that found over 80% of companies reporting no measurable AI productivity gains is a direct consequence of measuring the wrong thing. If you measure logins, you will optimize for logins. Logins don't pay for the tool.

The right unit is not the session. It's the task.

The only measurement framework that actually tells you something

Pick a specific, recurring task. Something that happens at least weekly, produces a clear output, and has a time component you can actually measure. Good candidates: first-draft proposals, research summaries, meeting prep documents, client status updates, data interpretation writeups.

Before anyone uses AI on this task, record the baseline. How long does it take? What does the output look like? How many revision cycles does it go through? Run this for two to four weeks if you can, to get a stable number.

Then implement AI on this specific task, with structured practice — not optional experimentation. After 30 to 60 days of consistent use, measure again. Time per task. Output quality. Revision cycles. Compare the two numbers.

That comparison tells you something real. Stanford researchers confirmed in a 2025 study that knowledge workers using AI on appropriate tasks cut first-draft time by 25-40% on average — but the range varied enormously by how the tool was used, not just whether it was used. Your before/after comparison tells you where on that range your team landed, and why.

Three metrics that actually connect to business outcomes

Throughput: How much work does the team complete in a given period? Not hours worked — deliverables produced. Proposals sent. Reports completed. Deals moved through a pipeline stage. If AI is working, throughput should go up without adding headcount or extending hours.

Time-per-task: How long does a specific task take now versus before? This is the most direct measure of individual productivity change. Track it for the tasks most amenable to AI assistance, not across all work. Not all tasks benefit equally.

Quality: How many revision cycles does an output require? What does client or manager feedback look like? This is harder to measure but important — AI that makes things faster but worse isn't creating value. The goal is faster and at least as good, ideally better.

These three metrics are enough for most teams. They require more setup than usage dashboards, but they answer the question that matters: is the business actually better off?

What this means for teams building AI capability

The measurement problem is one layer deeper than it looks. Most teams don't have good AI metrics because they never developed the underlying AI capability in the first place. You can't measure the impact of something people aren't really doing.

MakerSquare is a 2-week in-person AI builder program in Austin, TX — we work with operators, founders, and professionals who want to build real AI tools, not just use them. The curriculum is built around shipping working tools, which means every student has a concrete output to measure by the end. That output-first design is the same discipline that makes AI measurement possible.

If you can't answer the question "what specifically did AI help us produce that we couldn't produce before?" — that's not a measurement problem. It's a capability problem. Fix the capability first, and the measurement becomes obvious.