Building explainable geospatial AI

An AI output that a person cannot interrogate is not evidence. That is not a philosophical position; it is a practical one. Anyone whose job involves acting on a finding needs to know what the finding rests on, and a number without provenance gives them nothing to weigh.

Geospatial machine learning also has specific technical failure modes that general practice does not prepare you for.

The mistake that makes models look better than they are

Spatial data is autocorrelated. Places near each other resemble each other, in bathymetry, sea state, traffic density and almost everything else.

Split such a dataset randomly into training and test sets and something quietly breaks. Points a few hundred metres apart end up on both sides of the split. The model is then evaluated on locations that are nearly identical to ones it memorised, and reports excellent accuracy that will not survive contact with a genuinely new area.

This is not a small effect. It is one of the most common reasons a geospatial model performs far worse in deployment than in validation, and it usually goes undiagnosed because the validation numbers looked fine.

The fix is spatial cross-validation: partition into geographically separated blocks so that test regions are genuinely unseen. Accuracy usually drops. That drop is not a regression, it is the removal of an illusion, and the lower figure is the one worth quoting.

The same argument applies in time. A model validated on randomly sampled dates has seen the days either side of every test day. Where a system is expected to generalise to future conditions, the split has to respect time as well as space.

Imbalance, and the seduction of accuracy

Anomaly detection is by definition a rare-event problem. If unusual activity is a fraction of a percent of observations, a model that reports “nothing unusual” for every input scores over 99 percent accuracy and is completely useless.

Accuracy is the wrong measure here. What matters is precision and recall at the operating point actually being used, and the shape of the trade-off between them. An operator’s real question is rarely “is the model accurate”. It is “if I investigate the twenty highest-ranked items this week, how many will be worth my time” — and that is a question about ranking quality, not classification accuracy.

Confidence has to mean something

Most classifiers emit a number between zero and one. It is tempting to read it as a probability. Usually it is not one.

A well-calibrated model is one where predictions made with 80 percent confidence turn out correct about 80 percent of the time. Modern models are frequently overconfident, and calibration has to be measured and corrected rather than assumed.

This matters more here than in most applications. If a system exists to prioritise scarce human attention, the confidence value is the thing doing the prioritising. An uncalibrated confidence is a broken ranking wearing a number.

What an interrogable output contains

If an insight is going to inform a decision, it should arrive with:

Evidence. The specific observations that produced it, retrievable, not summarised away.

Confidence, and its meaning. A stated level, plus what that level was measured against.

Sources. Which sensor, which acquisition, which auxiliary feeds.

Context. Environmental and historical conditions at the time, because they determine how much weight the observation can bear.

Timestamps, kept separate. When the observation was acquired, and when it was processed. Collapsing them into one field hides latency at the moment latency matters most.

Processing lineage. Model version and processing chain, so a result can be reproduced and so a later correction can identify everything it affects.

None of this requires the model itself to be simple. It requires the system around the model to preserve the chain from observation to statement, rather than discarding it and presenting a conclusion.

The human is not a fallback

There is a tendency to treat human review as what happens when automation is not yet good enough, and to frame improvement as removing the person.

For this class of problem that framing is wrong. The value of automation is that it can examine far more observations than a person can, and rank them. The value of the person is judgement about context, consequence and accountability that the model does not have and is not being asked to have.

Designing for that division from the start produces a different system than designing for eventual autonomy. It also produces one that people are willing to use.