Gemini 3 at launch: benchmarks, costs and limitations

Alright, let's talk about Gemini 3. Google dropped this bomb on November 18, 2025, introducing Gemini 3 Pro in preview and calling it its most intelligent model yet. And you know what? There was plenty here worth paying attention to.

These were my impressions after 48 hours reviewing the launch material, Artificial Analysis's evaluations and early testers' accounts. Those are different kinds of evidence: Google's reported results, an independent evaluator's measurements and individual experiences. Here is what they can tell us, and why I still wouldn't hand over the keys to a business without supervision.

The Data That Made Me Do a Double-Take

First, the headline number: 1,501 Elo. That was the LMArena score Google reported for Gemini 3 Pro at launch. It was an impressive signal of human preference in those comparisons; it wasn't a certificate of superiority on every job you might give the model. Google's launch announcement

Artificial Analysis's November 18 review placed Gemini 3 Pro Preview first on its Intelligence Index, three points ahead of GPT-5.1. That's a strong launch result on that evaluator's mix of tests. Keep the date attached: an index ranking is a snapshot, not a lifetime achievement award. Artificial Analysis's launch review

But here's where it gets wild. The reported mathematical reasoning gains were striking:

• MathArena Apex: 23.4% for Gemini 3 Pro versus 0.5% for Gemini 2.5 Pro — a rise of 22.9 percentage points, or about 47 times the displayed baseline.
• ARC-AGI-2: Google reported 45.1% for the separate Gemini 3 Deep Think mode with code execution. That mode was still with safety testers at launch.
• AIME 2025: 100% with code execution, versus 95% without tools for Gemini 3 Pro.

Those settings matter. A result using code execution or a separate reasoning mode should come with its label attached. Google's launch announcement · November 2025 evaluation table

A Million Tokens Still Needs Checking

A million-token context window sounds glorious. It describes how much input the model accepts; it doesn't promise perfect recall of every detail.

Google's November 2025 table reported 26.3% for Gemini 3 Pro on MRCR v2 at 1M tokens, against 16.4% for Gemini 2.5 Pro. The 128K figure was a cumulative average, while the 1M figure was a point measurement. Neither establishes a universal 128K reliability cutoff.

My practical takeaway: test retrieval and reasoning on your own documents, and check important details wherever they appear. The input fitting is only the beginning. Google's evaluation methodology

Your Wallet's About to Feel This

Let's talk money because Google certainly wants yours. The launch rates quoted here were per million tokens:

• For prompts up to 200K tokens: $2 input / $12 output.
• For prompts over 200K tokens: $4 input / $18 output.

Artificial Analysis reported a 12% increase in the cost of running its Intelligence Index compared with Gemini 2.5 Pro. That combines prices and token usage; it isn't a general 12% price rise. Artificial Analysis's launch review

Using those quoted rates, 350K input tokens plus 15K billed output tokens cost $1.67 per request. At 100 requests daily for 30 days, that is $5,010 monthly, without caching or discounts. The original $50–70 monthly calculation was incorrect.

The useful comparison is the cost of completing your workload at an acceptable quality level. A leaderboard won't do that budget for you.

The "Autonomous Agent" That Needs a Babysitter

This is where things get hilarious (or frustrating, depending on your perspective). Google's pushing their new Antigravity platform hard - an AI-controlled development environment that supposedly builds software autonomously. The demos look like magic. The reality? Not so much.

In his launch review, Matt Shumer reported that Antigravity could overlook build errors and problems visible in browser checks. He recommended checking its work and rerunning tests. That is one tester's experience, but a useful reminder that a confident completion message isn't a test result. Matt Shumer's review

My favorite reality check came from Simon Willison's audio test. For a 3-hour, 33-minute council recording, the output placed the adjournment at 1:04:00 instead of 3:31:05 and summarized some speech instead of transcribing it verbatim. So much for calling that a perfect transcript. This was a documented example, not a universal test of whether the model can tell time. Willison's original test

Where the Launch Results Looked Strong — and Their Limits

The November 2025 comparison table gave Gemini 3 Pro several strong results. These were results reported by Google and other providers, with evaluation-specific settings:

• Screen understanding: 72.7% on ScreenSpot-Pro, compared with Claude Sonnet 4.5's 36.2% and GPT-5.1's 3.5%. Google's top result used a screenshot tool and extra_high resolution, then described as forthcoming; at high resolution, it reported 60.5%.
• Factual accuracy: 72.1% on SimpleQA Verified, versus GPT-5.1's 34.9%.
• Video analysis: 87.6% on Video-MMMU.
• Coding: 76.2% on SWE-bench Verified, versus Claude Sonnet 4.5's 77.2%. That is a 1.0-percentage-point gap between reports using different scaffolding and infrastructure, not proof of a production reliability gap.

Google's November evaluation table and methodology

Separately, Artificial Analysis reported an 88% hallucination rate on AA-Omniscience. That metric counts incorrect answers among non-correct responses; it doesn't measure confidence or say that 88% of all answers are false. Launch review · Metric definition

Availability also depended on the product. Gemini 3 Pro was rolling out globally, while Gemini Agent initially required Google AI Ultra and was available on the web in the U.S. Those are different claims from saying Gemini was blocked across Europe. Original app announcement

The Three Things That Are Actually Revolutionary

After filtering through all the marketing BS and looking at real data, three capabilities represent genuine breakthroughs:

1. The math jump isn't evolution, it's mutation - Going from 0.5% to 23.4% on contest problems is like your calculator suddenly understanding philosophy

1. Screen comprehension looked promising — the launch results made visual automation worth testing, with the screenshot tools and resolution settings kept in view.

1. Better coding tools — in a launch testimonial published by Google Cloud, JetBrains said Gemini 3 Pro solved over 50% more benchmark tasks than Gemini 2.5 Pro. That was a partner's model evaluation, not an independent study of Antigravity. Still, it was an encouraging reason to try the tools. Google Cloud's launch article

My Honest Take After 48 Hours

Google built something impressive. The launch results gave me reasons to try it on difficult mathematical problems, video analysis and interfaces. They didn't give me a reason to stop checking its work.

The context window can hold a lot of material, but capacity isn't reliable recall. Coding agents still need review. And being good at answering questions is different from knowing when to admit uncertainty. Talented intern, meet responsible supervisor.

Should You Jump Ship?

At launch, my advice was to test Gemini 3 Pro on work where the reported gains might matter:

• Mathematical and scientific questions with answers you can check.
• Image and video analysis, including checking timestamps and details.
• Long documents with a set of important facts to retrieve and verify.
• Coding tasks with working tests and a realistic review process.

Compare it with your existing tool on the same tasks. Measure quality, corrections, time and total cost. Check the availability of the particular feature you need. A small benchmark lead is not a reason to migrate production blindly.

At the time of writing in November 2025, Gemini 3 had just launched. We were still in the honeymoon phase. My verdict: an interesting, powerful tool worth evaluating for specific jobs, with supervision still part of the deal.

Welcome to the future, where impressive reasoning can coexist with an audio timestamp that sends you to the wrong part of the meeting.