Geo-Guess Evals

How well can today’s models place a photo on the map? For this experiment, I ran 10 webcam frames through 3 prompts and 3 models over the course of 90 total runs. Each model guessed a latitude and longitude, and then this guess was scored by its distance to the true point.

Testing Methodology

Prompt & Cases

The whole scenario is wired up in promptfoo, which handles the matrix of prompts and providers and lets me diff results across runs. The score for any guess is the great-circle distance, in kilometers, between the guess and the truth.

The three prompts are additive: each one is the previous prompt plus one new instruction. Because every variant shares the same base, the variants are a single ablation, meaning any change in the score is attributable to the one module that was added.

1. Base (image only). “You are a geolocation expert. You are shown ONE static frame from a YouTube live stream. Using any cues you can (landmarks, architecture, landscape, vegetation, coastline, signage, any visible text, sun position), estimate where on Earth this stream is.”
2. Base + landmarks. Adds: “Before deciding, list any recognizable landmarks or distinctive natural/built features you identify (named towers, bridges, mountains, coastline shape, biome, plus any readable signage or text), then use them to inform your estimate.”
3. Base + landmarks + time of day. Adds: “Also work out the LOCAL TIME OF DAY at the scene—from sun height and shadows, sky color, lighting, and any on-screen clock—then reason about which longitude (timezone) bands are consistent with that time, and use it to narrow the location.”

Each model returns a JSON object with its lat/lon, a short place label, and a self-reported confidence from 0 to 1. Only lat/lon are scored; the confidence is logged but left out of scoring, which leaves room to check later whether a model’s stated confidence actually tracks its accuracy.

Model Selection

The three models were chosen as roughly contemporary, capable competitors.

I should have set the models to all use reasoning or not use reasoning, but I missed this at run-time. Claude was run with thinking off (0 reasoning tokens) while the other two used somewhere between 3k and 9k. Also, I missed setting temperature, so Gemini ran at temperature 0 (deterministic), but I can’t be assured of the temperature used by Claude and Gemini.

Results

A Bimodal-ish Distribution

Of the 90 guesses, 53 land under a kilometer from the base-truth, but then the middle of the distribution is nearly empty, and a second, smaller pile forms out at 25 to 50 km.

I take it to be the case that the models don’t make a general guess in the area of their best estimate, instead maybe preferring to lock onto a named place. This could explain being wrong by so much without having the middle covered, but there are probably other good explanations as well. I suppose what jumps out is that this seems more like recognition based on known places rather than more of the spatial estimation I’d been hoping for. The spatial estimation is probably what needs more attention, since there are only so many landmarks. Or maybe rethinking what it means to be a landmark to the more general sense and pushing the model to accept a more general interpretation. Or maybe prompting it with candidates from some tooling it can access?

Impacts of Each Prompt

The whole point of the additive prompts was to see whether each extra instruction earned its place. By the typical guess, the answer is that they barely move the needle.

The median for every variant sits right at the bottom: 1 km for the image-only base, 0.5 km once I ask it to list landmarks first, and back to 1 km after adding the time-of-day reasoning. So on a normal frame all three prompts land essentially on top of the answer, with the landmarks variant edging slightly ahead.

That tidy picture falls apart the moment you look at the mean.

The medians are the same gray bars as before, but the coral mean bars tell a different story: 8.3 km for the base, 24.8 km for landmarks, and 14.2 km for landmarks-plus-time. The landmarks prompt has both the best median and the worst mean. That gap between the gray and coral bar is the whole point—it’s the tail. The mean isn’t high because the prompt got generally worse; it’s high because a few guesses detonated (the 455 km Key West miss is the clearest one). So the instruction that sharpened the typical guess is the same one that opened the door to catastrophe, which is exactly what the next section digs into.

Claude’s Tail

ChatGPT and Gemini did not miss past ~40 km, whereas every error beyond 50 km belongs to Claude; these were 84km, 212km, and a 455km. Claude’s median was 5km, but its mean was 35.7km, so we can infer (and indeed we otherwise know from counting) that most guesses are good with just a few that were really far off. This kind of difference seems suspect except the asymmetry I introduced earlier with thinking puts this in question.

Reasoning Can Backfire

In the case of the “Key West” row, Claude landed within 1km using the base prompt, but then under the “list landmarks first” prompt, it jumped to 455km off. Although thinking isn’t on, so we can’t know for sure if Claude essentially talked itself out of this, that seems like a good candidate explanation. This makes me wonder about what role scaffolding can play in actually misleading rather than clarifying.

Clustered Failure Across Models

The Tehachapi loop saw all three models land around 37km off. It is a rural spot, and it seems on inspection that the models snapped to the nearest named town.

Efficacy by Cost: Gemini Coming Out Ahead

gemini-3-flash was both the cheapest and the most accurate—a mean of 5.1 km for $0.10 total. Claude cost the most at $0.18 and scored the worst.

Was The Confidence Earned?

Does a model’s self-reported confidence actually track whether it was right? Plotting confidence against error: basically not.

The points pile against the right-hand wall shows almost every guess came stamped 0.85 or higher—so there is barely any range for confidence to predict with. And at any given confidence the error spans the whole axis: in the 0.95 column there are guesses under half a kilometer sitting alongside guesses 30 to 40 km out. The two worst misses of the whole run, Claude’s 455 km and 212 km on Key West, were both stated at 0.85—a higher confidence than the 0.62 GPT attached to a guess that landed 1 km away.

• Most often right when confident → Gemini, no contest.
  • 83% of its high-confidence calls landed within 5 km, 90% within 25 km.
  • Well ahead of GPT (71% / 86%) and Claude (61% / 74%).
  • But the lead is hollow — it rubber-stamped nearly everything at 0.95, so its confidence tells you nothing about when to doubt it. A number that never moves can’t warn you.
• Confidence that actually moved with reality → Claude.
  • Confidence-to-error correlation of −0.37 — weak, but the only one pointing the right way.
  • Gemini: a flat −0.10.
  • GPT: +0.15 — meaning it was, if anything, more sure of its worse guesses. The one thing you never want confidence to do. So… the most accurate model and the only model whose confidence is worth trusting are not the same model.

Where I’d Go Next

Most of what I learned was about the experiment rather than the models. A few things I would fix before trusting any ranking:

• Match the thinking setting. Claude ran with thinking off and the others on. Turn it on for all, or off for all, and re-run before concluding anything about which model is better.
• Give the model tools. Let it atomize the problem—read on-screen text, look up a landmark, check the sun’s position—rather than guessing from a single glance.
• Harden the test set. Ten rows is too few; one outlier swings the mean. Aim for 30 or more, and deliberately add hard cases: no signage, look-alike cities, open ocean.
• Run each cell many times. I sampled each cell once, and these models are random. The 455 km miss might not repeat. Run each three to five times and report the spread.
• Fix the metric. Set a real fail cutoff (a miss over 50 km fails), report both median and mean in kilometers, and stop rounding to whole km.
• Add an image-only baseline. Famous landmarks let a model recall a saved coordinate—memory, not vision. A plain image-only prompt should be the baseline, and any fancier prompt must beat it to earn its place.

Conclusion

Most of these frames were fairly close the experiment’s real lesson is that the harness, not the model, is what most needs fixing: an unfair thinking setting, a single sample per cell, a metric that never fails, and famous landmarks that reward memory over perception.

So before I rank any model, I have to answer a more basic question: when the guess is right, did the model see the place, or merely remember it? Until the eval can tell those apart, the leaderboard is measuring the wrong thing.

Code and raw results on Github.