The evidence standard
Every source this publication relies on carries a grade, and the grade is the first thing to read. It answers one question: who measured this, or who said it, and what did they have to gain by the answer?
| Grade | What it is | What that looks like |
|---|---|---|
| 1 Primary | A body with the power to compel an answer, and nothing to sell you | A regulator’s decision, a court ruling, a statistics office’s release, the text of a law |
| 2 Published method | A study or survey that publishes its method and states its sample | A peer-reviewed paper, a preprint marked as not yet reviewed, a workforce survey with its sample |
| 3 Filed record | A record of money or facts that actually moved, filed where somebody has a reason to check it | A company’s filing with a regulator, audited accounts, a layoff notice, a filed wage |
| 4 Aggregator | Figures people typed in themselves, collected and republished | A salary comparison site, a crowd-sourced layoffs tracker, job-board posting counts |
| 5 Own claim | Somebody describing the thing they sell, or describing themselves | A company’s announcement or benchmark of its own product, a training provider’s own outcome figures |
What a grade lets you do with a source
A grade is not a score out of five. It is a statement about what the source can be used for, and the rule differs at each step.
Grades 1 to 3 can be stated as fact. Somebody with the standing to know measured it, ruled it or filed it, and you can act on it.
Grade 4 carries a direction and never an amount. If it says postings fell, believe that. Do not believe the digits after the first one.
Grade 5 is reported only as a claim. It is somebody’s own account of themselves or of what they sell. It is news when the claim itself is the story, and then it is written as “the company says”, never as fact.
The grade travels with the source, beside the story, rather than sitting in a note on methodology at the back. That is the whole of the discipline: a reader should never have to go looking for how much weight a source will take.
The second question, which the grade does not answer
A source can be perfect on provenance and still not answer the question.
A government publishes exactly what one small programme pays in one expensive city. That is grade 1, audited, impossible to argue with, and a bad answer to “what does this kind of work pay?” A national jobs figure for every industry is grade 1 too, and it says little about software engineers.
So there are two questions, in this order. Who produced it, and what do they sell? That is the grade. Is it about what the story says it is about? That is the fit. A source has to pass both, and one that passes the first and fails the second is marked rather than quietly used.
What happens when nothing meets the standard
Nothing is estimated.
In the news, the gap is said out loud. Every story states what its evidence does not tell you. A story whose original document cannot be read, or rests only on another outlet’s account, is not published.
In the research, the gap is published. Where the best figure found falls below the required grade, it is printed with a mark saying so, because hiding it would be worse. Where nothing was found at any grade, nothing is printed: a number with nothing behind it is a rumour with the decimal point still attached. Every question that could not be answered goes into a register that ships with the research, free, before anybody has paid for anything.
The news and the figure
Most news is a change rather than a number: a new rule, a ruling, a filing, a launch. A story like that is traced to the document the change is in, and the grade shown on its card is that document’s. When a story does print a figure, the figure carries its own source in the same paragraph, and is held to everything above.
It applies to the work published here
The standard is not a claim about other people’s evidence. It is applied to this publication’s own, and it is applied in public where the result is unflattering.
Of the 46 careers examined in the first research project, 16 have a pay figure meeting the standard and none has one for how long entry takes, so none clears both. Until September 2026 this page said eleven and five, because the book had counted its own inferences as measurements. The correction is on the findings page. That count is published with its working, recomputed from the shipped data on every build, and the build refuses to publish a figure it cannot recompute.
There are 154 questions in that project’s gap register. 22 of those gaps are marked closeable, meaning a published source exists and nobody here has compiled it yet.
Where it came from
The standard was first written down for the research’s first book, about computing careers, and the long form runs to a chapter there, with worked examples of each grade failing in public. It is free to read with nothing to sign. The book names grades 2, 3 and 5 “Structured survey”, “Verified transactional” and “Vendor”, because it was grading career data; the names above are the same grades, worded for any subject.