The evidence
Evidence that measures something else
The commonest failure in this argument, and the one that looks most like rigour. Three cases from the record, and the two questions that catch it.
Claims this part is about, and what each was answered with
New starters take longer to become productive remotely
Empirical→EvidenceThe one claim answered with evidence that measured it
Productivity fell when we went remote
Empirical→DataRemote workers put in longer hours than office workers
Empirical·Nothing; it was never answeredthree claims · three empirical · none of the answered ones was answered in the wrong kind · one was never answered at all
Three empirical claims with three different fates.
A study is cited. It is a real study, well conducted, with a large sample. It is produced in support of a claim, and it does not bear on the claim at all.
This happens constantly, it is almost never deliberate, and it is the hardest failure to call out because everything about it looks like doing the work properly.
Three cases from the record
The company-level paper at a team-level claim. The claim was that teams who sit together ship faster. The paper compared firms with different working policies on firm-level outcomes. Both are about co-location and productivity; the unit of analysis is different, and the claim is about the unit the paper does not examine.
Revenue per head across a changing product. The claim was that productivity fell after going remote. The figure offered was revenue per head, before and after, over a period in which the product changed, the pricing changed and two large customers left.
Sick pay rules in another country. The claim was that absence is lower when people can work from home. The study cited was from a country where statutory sick pay works differently enough that the behaviour it measures is not the behaviour claimed.
In all three the person citing had read the study and could summarise it accurately. None of them had asked whether it addressed the claim, because that is a separate question and it does not feel like one.
The two questions that catch it
Ask them of any study, including your own.
What is the unit? Companies, teams, individuals, days? A finding about one unit frequently says nothing about another. Most citation failures in this argument are unit mismatches.
What else changed? Any before-and-after comparison across 2020 to 2023 is a comparison across a period in which almost everything changed. A difference between those two periods is not evidence about working location unless something has isolated it.
Those two catch most of it. A third is worth adding for this particular subject:
Who chose? Where people chose their own working pattern, any difference between the groups may be about who chose what. The people who opted to come in were different people before they came in.
Why it is so hard to raise
Because the obvious phrasing sounds like an accusation of incompetence, and the person has just demonstrated effort.
The phrasing that works puts the question on the claim rather than on the reader:
That is useful. Can I check what it would mean for our case — is it comparing teams or companies? Ours is a claim about teams and I want to be sure we are reading it onto the right thing.
"Our case", "we", "I want to be sure". The study is not attacked, the reading is examined jointly, and the person who produced it is given somewhere to go that is not a retreat.
The most useful habit
Before citing anything, write one sentence: this study shows X about Y, and my claim is about Y. If the second Y is a different thing from the first, the citation does not work, however good the study is.
Thirty seconds, and on this record it would have removed three of the four most-repeated citations in eleven weeks.
The hardest case: a study that nearly fits
The three on this record are clear mismatches. There is a harder version, and it causes more trouble because reasonable people disagree about it.
A study measures something adjacent to the claim: reported belonging rather than retention, self-assessed productivity rather than output, manager ratings rather than delivery. It is not irrelevant. It is also not the thing.
There is no rule for these. What helps is saying out loud what the gap is, rather than treating the study as either decisive or worthless:
This measures how connected people say they feel, and the claim is about whether they leave. Those are related and I do not know how tightly. I am happy to treat it as some evidence, not as settling it.
That sentence is more work than either dismissing or accepting, and it is the honest position in a surprising number of cases.
The rule for your own citations
Before you cite, name the gap yourself.
A person who says "this measures X and my claim is about Y, which I think are close" is much harder to argue with than one who presents the study as a straight answer, because the obvious objection has already been made and conceded.