Opinions diverge. Outcomes do not.

Thirteen specialist providers were given the same one hundred properties and asked what the damage would be. Their estimates correlated as weakly as 0.2. Not one of them publishes an error rate, because none of them has anything to measure an error against. This is not a scandal about vendors. It is what every market looks like before it has a shared record of outcomes, and it is exactly where carbon sits today.

The finding, and why it should not be dismissed

The benchmarking work was run for the Climate Financial Risk Forum and published through the Global Association of Risk Professionals. It compared physical climate risk vendors on identical assets. The headline is the dispersion: on the same properties, facing the same hazards, the damage estimates ranged from broadly agreeing to barely related.

The obvious response is that some vendors are simply better. That may be true and it is unfalsifiable, which is the problem. Without a record of what actually happened to those properties, there is no way to say which estimate was closer. The dispersion is not evidence that the vendors are bad. It is evidence that the market has no mechanism for finding out.

The academic literature makes the same point from another direction: a climate risk premium identified using one vendor's data disappears when the analysis is repeated with another's. If the answer changes with the supplier, the answer was never about the asset.

Carbon has the identical structure, one step earlier

Carbon credit ratings are ex ante opinions. They estimate the probability that a credit represents what it claims, before anyone knows. There are several providers, they use different methodologies, and they disagree. That disagreement is normal, healthy and completely unresolvable, because no rating product in this market has ever been publicly backtested against realised outcomes.

The reason is not negligence. It is that the outcomes have never been collected. There is no cross scheme record of reversals, invalidations, eligibility losses, delivery failures and insolvencies, dated and typed and attributable to specific projects and vintages. So the question "was this rating right" has no data behind it, and everyone in the market is arguing about opinions instead of comparing them against events.

What an outcome record does that an opinion cannot

The distinction is worth being pedantic about, because the whole argument rests on it.

An ex ante opinion is a view about the future formed from present evidence. It can be well reasoned, expensively produced, and still unverifiable. Its quality is asserted rather than demonstrated.

An outcome record is an account of things that occurred, each with the document that decided it. A regulator retired reserve credits on a date. A registry recorded units relinquished. A court recorded an insolvency. A standard removed a methodology from an eligibility list. None of these are judgements. They are events, and the only skill involved is finding them, classifying them consistently and stating what they are divided by.

The record does not replace the opinions. It makes them testable, which is the first time anyone can say which opinion was worth paying for.

Two things the record has to get right

Denominators travel. The same eligibility loss has been quoted as 236 million tonnes and 32 per cent, as around 240 million and roughly a third, and as 268 million and 30.6 per cent. None is wrong. They are unretired credits against total market, credits in circulation, and total available supply. Anyone can select the pair that suits the argument, and in this market everyone does. A ratio published without its denominator is a rhetorical device, not a measurement.

Abstention is a finding. Where attribution is not published, the honest output is not an estimate. It is a labelled abstention with a reason. A record that always has a number is not confident, it is unfalsifiable, and it will be treated as such by exactly the audience that matters.

Why this matters to the person carrying the risk

An underwriter pricing reversal risk uses a rating as one input among several. Fine. But when that book is presented to a reinsurer, a capital provider, a rating committee or an auditor, the question changes from "what do you think" to "what does the evidence say, and how often have views like yours been wrong". Today there is no answer, and the absence is structural rather than anyone's fault.

The same holds for a valuation committee marking a Level 3 position, and for a credit officer lending against carbon collateral. The input they need is not a better opinion. It is the record that lets any opinion, including their own, be checked.

Wherever there is no shared outcome record, models disagree and nobody can say who is right. That is not a permanent condition. It is simply what a market looks like before someone does the reconciliation.