An estimated 250 billion lines of COBOL are still running in production today, by a widely cited 2021 TechChannel survey figure that both IBM and most of the trade press continue to reference. Every vendor with a stake in enterprise software — IBM, Google, AWS, and a wave of startups — now has an “AI agent” pitch for modernizing that code, whether that means explaining it, rewriting it in Java, or migrating it off the mainframe entirely. The pitches are consistent; the actual, independently measured performance of these agents on real legacy code is a lot more mixed than the marketing suggests, and 2026 is the first year there’s hard benchmark data to check the claims against.
At a Glance
| Approach | What It Does | Representative Tools | Maturity |
|---|---|---|---|
| Code understanding / documentation | Uses an LLM to read legacy code and generate plain-language explanations, business-rule extraction, and dependency maps | IBM watsonx Code Assistant for Z, Google’s Mainframe Assessment Tool (MAT), Swimm | Most mature — lower risk, no code changes, used as a first step before any rewrite |
| Deterministic (like-for-like) modernization | Restructures code/data to a new platform while requiring provably identical outputs for identical inputs | Google’s Dual Run (parallel-run verification), Mainframe Connector (data migration) | Moderate — well-suited to regulated, high-stakes systems where behavior can’t change |
| Agentic rewrite / translation | Multi-agent systems that translate COBOL to Java (or another target language) and review their own output | Persistent Systems’ coder/reviewer multi-agent system, Google Antigravity (agentic harness) | Least mature — independent benchmarks show current agents solving well under half of real-world legacy tasks |
Short Verdict
AI agents are genuinely useful for the first, lowest-risk part of legacy modernization — reading 50-year-old COBOL and producing documentation, dependency maps, and business-rule extraction that would otherwise require retiring subject-matter experts to explain line by line. They are considerably less reliable once you ask them to autonomously rewrite or migrate that code: the first independent benchmark built specifically for this task, Factory’s LegacyBench (announced April 1, 2026), found the best-performing model-agent combination solved only 42.5% of its legacy coding tasks overall, and just 34.8% of its COBOL-specific tasks — compared with the same models scoring above 70% on modern-code benchmarks like SWE-bench Verified. Treat vendor claims about full autonomous migration with real skepticism, and treat AI-assisted code understanding as the safer, more defensible place to start.
Why COBOL Modernization Is Hard — With or Without AI
Google’s own engineering materials on mainframe modernization are unusually candid about why this problem resists simple automation: it “isn’t a pure code-to-code conversion problem.” The hard parts are tightly coupled data and logic (COBOL programs often embed business rules directly in record layouts and file structures that have no modern equivalent), transaction scenarios that span legacy utilities and protocols with no modern documentation, undefined application boundaries after decades of ad hoc patches — commonly described as “spaghetti code” — and intricate sequential batch workflows that assume mainframe-specific execution guarantees. None of that is a language-translation problem an LLM can solve by pattern-matching COBOL syntax to Java syntax; it requires understanding what the code is actually for, which is exactly where AI coding tools have historically been weakest.
What the Major Players Are Actually Shipping
IBM positions watsonx Code Assistant for Z as an AI-agent layer specifically for explaining mainframe code in natural language and automatically refactoring COBOL applications into modular business services — notably, IBM frames this as something that increases the need for mainframes rather than eliminating them, since the goal is adding AI-assisted capability to systems that stay on Z hardware rather than migrating off it entirely.
Google Cloud offers a more staged toolchain: its Mainframe Assessment Tool (MAT) reverse-engineers legacy code for dependency visualization, automated business-rule extraction, and documentation, connecting into agentic workflows via MCP. From there, Google supports two distinct modernization patterns — a “Rewrite/Reimagine” path using Gemini models and its Antigravity agentic harness (with optional human-in-the-loop governance) for applications where new business logic justifies a rebuild, and a “Deterministic” path for applications that must produce bit-for-bit identical behavior after modernization. The deterministic path’s key safeguard is Dual Run: it executes live production workloads on both the old mainframe and the new Google Cloud environment simultaneously, comparing outputs continuously until logic and data equivalence are proven — which is a meaningfully more conservative approach than trusting an agent’s rewrite on the strength of a code review alone.
Third-party and specialist vendors round out the toolchain: Swimm pairs static analysis with LLMs specifically for business-rule extraction and dependency mapping (the documentation step, not code rewriting), and Persistent Systems has built a multi-agent system that explicitly separates roles — a “coder” agent translates COBOL to Java, and a separate “reviewer” agent checks that output for errors and optimization opportunities before a human sees it, an architecture that mirrors the broader industry pattern of using one AI agent to generate code and a second, independent one to critique it rather than trusting a single model’s self-review.
What Independent Benchmarks Actually Show
Vendor case studies are, understandably, selective. The more useful signal comes from Factory’s LegacyBench, an independent benchmark (published by an AI coding-agent company, so its own incentives are worth keeping in mind) built specifically because existing coding benchmarks like SWE-bench and Terminal-Bench test only modern languages and don’t reflect legacy work at all. LegacyBench covers six legacy language families — COBOL (46% of tasks), Java 7 (32%), BASIC, C89, Fortran, and Assembly — across three task types: bug fixing, implementing new functionality, and migrating code to other languages.
| Model / Agent | Overall Pass Rate | COBOL-Specific Result | Notable Strength / Weakness |
|---|---|---|---|
| GPT-5.3-Codex | 42.5% (highest overall) | 34.8% | Weak specifically on C |
| Gemini-3.1-Pro | 38.7% | No language below 29.7% | Most balanced profile across all six languages; 81.0% on BASIC |
| Opus 4.6 | Tied highest on Java 7 (54.2%) | 18.1% (its lowest result) | 34.3% on C |
| GLM-5 | 40.6% on Java 7 | Not separately reported as highest/lowest | Tied-best Fortran migration score (67%); 0% on Assembly bug fixing |
The headline comparison in Factory’s own announcement is the one worth sitting with: these same models score above 70% on modern-code benchmarks like Terminal-Bench 2 and SWE-bench Verified, but the best agent on LegacyBench still solves fewer than half of its legacy tasks. That gap is the honest current state of the technology — AI agents aren’t incapable of legacy code work, but they are measurably, substantially worse at it than at the modern software engineering tasks most benchmarks (and most marketing) are built around.
A Minimal Example of the Documentation-First Approach
The lower-risk, higher-maturity use case — having an AI agent explain a COBOL paragraph rather than rewrite it — looks roughly like this in practice: a short COBOL paragraph computing an interest accrual, and the kind of structured, plain-language extraction a documentation-focused tool is designed to produce from it.
COMPUTE WS-INTEREST = WS-PRINCIPAL * WS-RATE / 1200
IF WS-INTEREST > WS-MAX-INTEREST
MOVE WS-MAX-INTEREST TO WS-INTEREST
MOVE "Y" TO WS-CAPPED-FLAG
END-IFA documentation-generation tool’s output for this snippet should read something like: “Computes monthly interest as principal times annual rate divided by 1,200 (converting an annual percentage rate to a monthly decimal amount). If the result exceeds a configured maximum, the interest is capped at that maximum and a flag is set.” That’s a comprehension task — turning code into a verifiable business-rule statement a human reviewer can check against the actual requirement — which is precisely the kind of task LegacyCodeBench and similar comprehension-focused benchmarks report much higher, more reliable scores on than autonomous rewriting tasks.
Trade-Offs
- Pros: AI-assisted documentation and business-rule extraction can compress months of manual code archaeology (especially valuable as the pool of COBOL-literate engineers shrinks through retirement) into days; deterministic/parallel-run approaches like Dual Run let you validate an AI-assisted rewrite against the real production system before cutting over, rather than trusting code review alone.
- Cons: independent benchmarks show current agents solving well under half of realistic legacy coding tasks autonomously, with COBOL specifically among the weaker categories for at least one leading model; vendor case studies citing speed or cost improvements are typically unquantified or self-reported, and none of the major vendors’ own modernization blog posts in our research included independently verifiable statistics for time or cost savings.
Who This Is a Good Fit For
Organizations with large COBOL estates and a shrinking pool of engineers who actually understand them are the clearest beneficiaries of the documentation and business-rule-extraction tools today — that’s the use case with the most mature tooling and the least downside risk, since nothing in production changes. Organizations in regulated industries (banking, insurance, government) that need to modernize without changing behavior should look specifically at deterministic/parallel-run approaches rather than agentic rewrites, given how much verification those regulated environments require regardless of which tool wrote the new code.
Who Should Look Elsewhere
Teams hoping to fully automate a COBOL-to-Java migration with minimal human review should treat that as a 2026-and-beyond aspiration, not a current capability — LegacyBench’s sub-50% pass rates for the best available agents are a direct, independently measured counterpoint to “fully autonomous migration” pitches. Teams without the budget or appetite for a multi-month pilot (most vendors here, including Google, explicitly structure their offering around a pilot engagement — an automated scan of one application followed by a modernization workshop) should expect this to be a longer engagement than a straightforward SaaS purchase.
Bottom Line
AI agents have made real progress on the comprehension side of legacy modernization — reading COBOL and producing documentation, dependency maps, and extracted business rules is a task where current tools are genuinely useful. Autonomous rewriting and migration is a different story: the first purpose-built independent benchmark for this exact problem shows even the best current models solving fewer than half of realistic legacy-code tasks, and specifically struggling with COBOL. The responsible way to use AI in a legacy modernization program in 2026 is to lean hard on it for understanding and documentation, use deterministic parallel-run verification wherever behavior must stay identical, and keep a human reviewer firmly in the loop for anything an agent rewrites — not to expect it to migrate your mainframe unsupervised.
FAQ
How much COBOL is actually still in use? A widely cited 2021 TechChannel survey estimate puts it at around 250 billion lines in production, a figure IBM and much of the trade press continue to reference as the current best public estimate.
Can AI agents fully automate COBOL-to-Java migration today? Not reliably, based on independent benchmark results. Factory’s LegacyBench found the best model-agent combination tested solved 42.5% of legacy coding tasks overall and 34.8% of COBOL-specific tasks, versus over 70% on modern-code benchmarks like SWE-bench Verified for the same models.
What’s the safest way to start a COBOL modernization effort with AI? Begin with AI-assisted code comprehension and documentation — extracting business rules and dependency maps from existing code — rather than autonomous rewriting. This is the most mature, lowest-risk application of the technology and doesn’t touch production behavior.
What is “Dual Run” and why does it matter? It’s Google Cloud’s approach to deterministic modernization: running the same production workload on both the old mainframe and the new target environment simultaneously and comparing outputs until they’re proven equivalent, rather than relying on code review alone to catch behavioral differences introduced during a rewrite.
Does using AI mean a company can get rid of its mainframe? Not necessarily, and IBM explicitly argues the opposite for its own customers — that AI-assisted modernization increases the value of keeping workloads on Power/Z systems by adding new AI-driven capability to them, rather than being primarily a tool for migrating off the platform.
Sources
- IBM — COBOL Modernization
- Google Cloud Blog — Mainframe Migration and Modernization With AI
- Factory — Introducing Legacy-Bench
- Factory — Legacy-Bench Documentation
Related reading on this site: What OWASP’s Guidance Says About Securing AI-Generated Code in 2026.
Discover more from Software Online Review
Subscribe to get the latest posts sent to your email.
