The evidence
The one claim that was answered properly
Out of twenty-four, exactly one empirical claim got evidence that measured it. Worth describing in full, because it took an afternoon.
Claims this part is about, and what each was answered with
New starters take longer to become productive remotely
Empirical→EvidenceThe one claim answered with evidence that measured it
Productivity fell when we went remote
Empirical→DataRemote workers put in longer hours than office workers
Empirical·Nothing; it was never answeredthree claims · three empirical · none of the answered ones was answered in the wrong kind · one was never answered at all
Three empirical claims with three different fates.
One claim out of twenty-four was both empirical and answered with evidence that actually measured it. It is worth setting out in full, because most of this site describes failures and the successful case is short, cheap and repeatable.
The claim
New starters take longer to become productive when they start remotely.
Made in the first all-hands, by somebody who manages a team that had taken on four people in two years.
What happened next
Somebody asked what would show it. Not aggressively; the exact words were "is there anything we could look at?"
The answer took about a minute to arrive, in the room: time from start date to first independent delivery, by starting cohort. Every team already tracks when work ships and who shipped it. Nobody had thought of it as a measurement before.
Two days later somebody produced it. It took an afternoon.
What it showed
It supported the claim. Starters who began entirely remotely reached first independent delivery later than those who began with their team present, by a margin large enough not to be noise given the numbers involved.
It also showed something nobody had claimed: the gap closed almost entirely for people who had any substantial period alongside the team in the first six weeks, whether or not the rest of their arrangement was remote.
That second finding is the one that mattered, and nobody would have got it from a general argument about offices.
Why this one worked
Four conditions, all cheap:
- The claim was specific. Not "new people suffer" but a statement about time to productivity.
- Somebody asked the question rather than agreeing or disputing.
- The measurement already existed, as a by-product of something else, which is where most usable measurements come from.
- The person who asked was prepared for it to go against them. They were on the other side of the broader argument, and said so when it came back.
The fourth is the rare one. An empirical question only gets asked honestly by somebody willing to be wrong about it.
What followed
The finding changed one thing and did not change the rest.
It changed the arrangement for new starters: six weeks alongside the team, by name and by date, which is now what happens. That is a real outcome from a real measurement and it is the only such outcome in eleven weeks.
It did not settle the broader argument, and nobody pretended it had. The claim it answered was one claim out of twenty-four, and the other twenty-three were exactly where they had been.
The transferable part
Of everything on this site, this is the cheapest thing to copy.
When an empirical claim is made in one of these discussions, ask what would show it, and then look. Most of the time the answer will be that nothing records it, which is useful. Occasionally the measurement turns out to be sitting in a system somebody already runs, and an afternoon settles something that would otherwise have been argued about for a year.
What the finding did not license
Worth marking, because this is where a good measurement usually goes wrong afterwards.
The finding was about new starters in their first weeks. It was immediately quoted, twice, in support of the general three-day requirement for everybody.
It does not support that. It supports an arrangement for new people, which is what it measured, and extending it to a population it did not examine is the same error as every mismatched citation elsewhere on this record — committed, this time, with the organisation's own data.
The person who produced the measurement said so, both times. It was quoted that way anyway.
The lesson about good evidence
A measurement that bears on a claim is valuable and narrow. The narrowness is the thing that makes it valuable, and it is the first thing lost when the finding enters an argument.
So the useful habit when producing one is to state the scope in the same breath, every time, including when it is tiresome:
This is about people in their first six weeks. It says nothing about anybody else, and I would rather it was not used as if it did.
The cost of the measurement
Worth recording precisely, because the perceived cost of measuring things is usually the reason nothing gets measured.
An afternoon of one person's time. No new system, no instrumentation, no survey. The data existed because delivery dates are recorded for unrelated reasons, and the analysis was a sort and a subtraction.
Set that against eleven weeks of discussion, six sessions and the senior time in them. The measurement that settled one claim cost roughly a thousandth of what the argument about all of them cost.
Where the next one would come from
Every organisation has a few of these: data collected for one purpose that answers a question somebody is arguing about.
Delivery dates answer questions about ramp-up. Logged hours answer questions about working patterns. Meeting calendars answer questions about who actually overlaps with whom. None of it was collected to settle an argument about offices, and all of it bears on one.
The habit worth having is to ask, of any empirical claim, whether something already recorded would speak to it. On this record the answer was yes twice out of nine.