This is an opinion column. Within 24 hours, two labs took opposite routes with models that are good at finding and fixing software flaws. Google DeepMind announced Gemini 4 Carbon and gave it only to a small group of cyber defenders. Mistral released the weights of Large 4 for anyone to download. Our view is that neither route is wrong, but neither is enough. The part both leave out, independent testing of the cyber claims, is the part defenders most need.
Two answers to the same question
The question is how to release a model that can reproduce and patch vulnerabilities, and could in other hands help exploit them. Google’s answer, set out in its announcement, is staged access: a subset of defenders first, then more trusted partners through its Fairwind Program, while it takes part in the US government’s voluntary pre-release testing. Mistral’s answer, in its own announcement, is the opposite: because refusals from closed providers can block legitimate vulnerability research and an API can vanish mid-incident, put the weights in defenders’ hands and let them run the model under their own policies.
Each argument has merit. A gated rollout lets a company watch how a model behaves before the whole internet touches it. An open-weight release lets a hospital, a government agency or a small security firm run a capable model without sending sensitive logs to someone else’s servers. The two are not mutually exclusive, and both labs say they care about defense.
What the gated route gets wrong
The trouble with staging is that the evidence for the model’s strength stays inside the company. Google reports 88.2% on SEC-bench Pro, 54.8% on ExploitGym and 71.0% on CWE-bench, and says its Chrome team saw patches of much higher quality than a rival’s model produced. Some of those comparisons are internal. Until outside groups can run the same tests, defenders choosing where to spend scarce time are taking a vendor’s word, and those with no access cannot judge at all.
There is also a quiet competitive effect. Early access goes to partners the company selects. That may be perfectly sensible for safety, but it concentrates insight in the hands of a few firms and lets the vendor shape the story of what the model can do.
What the open route gets wrong
Open weights solve the access problem and create a different one. Mistral says Large 4 declines harmful cyber prompts more often than any other open-weight model, according to the figures cited in the Hugging Face walkthrough. That is a useful result about the weights as released. But anyone can fine-tune or modify downloaded weights, so a refusal rate describes a starting point, not a guarantee. The company’s own pitch assumes users will apply their own policies, which also means misuse controls are no longer the lab’s.
The cyber numbers also come from the vendor and its launch partners: 82% on a reproduce-and-patch task and 93% on Cybench. The one independent placement, fifth among open-weight models on the Artificial Analysis Intelligence Index, is a general measure rather than a cyber one. TechCrunch reports Mistral’s comparison with GLM-5.3, which is again the company’s own claim.
What we would want instead
A shared standard would help more than either release strategy. We would like to see three things:
- Independent cyber evaluations published at launch, run by third parties on the final model, with the same tasks and scoring for every lab so that numbers are comparable.
- Clear reporting on guardrail changes. Google’s headline benchmarks were measured without guardrails, while Mistral reports refusal behavior with them. Buyers need both, labeled.
- Access rules that are stated, not implied. If a model is limited to approved defenders, say who qualifies and how to apply.
None of this requires slowing anyone down. It requires treating cyber capability like any other claim that deserves a second pair of eyes. For readers new to the topic, our overview of AI in cybersecurity explains the attacker and defender sides, and our news reports cover the Carbon rollout and the Mistral release in detail.
Where this could go wrong
We could be wrong on timing. Independent evaluators may already be testing both models and publish soon, in which case this column will age quickly, which would be a good outcome. We could also be underrating the value of simply getting a strong model into defenders’ hands now, imperfect evidence and all. Many security teams need working tools faster than any evaluation ecosystem can mature.
What to watch next
- Whether outside groups publish results for Carbon and Large 4 on the same cyber tasks.
- How many defenders actually get Carbon access, and on what terms.
- Any reports of misuse or fine-tuned variants of Large 4.
- Whether other labs adopt either release pattern for their next security-capable model.
Key takeaways
- Google gates Gemini 4 Carbon; Mistral opens Large 4: two opposite strategies for cyber-capable models.
- Both rely mainly on vendor-reported cyber benchmarks.
- Gating hides evidence; open weights hand over control and risk.
- Comparable, independent evaluations published at launch would serve defenders better than either approach alone.
Frequently asked questions
Why is Gemini 4 Carbon not publicly available?
Google says it is taking a deliberate approach, giving access first to cyber defenders and trusted partners while it takes part in the US government’s voluntary pre-release testing.
Does open-weight mean unsafe?
Not necessarily, but anyone can modify downloaded weights, so safeguards built into the model are only a starting point. Safe use depends on the people running it.
What would independent testing look like?
Third parties would run the same cyber tasks, such as vulnerability finding, exploitation and patching, on the final model and publish scores with and without guardrails.
