Analysis
OpenAI's new models migrate COBOL with 86.6% accuracy on its own test. What that changes for legacy estates
1AYM is an OpenAI Select Partner.
Published . Vendor facts checked on against each vendor’s own pages, listed at the end. Plans, prices and availability change. Neither OpenAI nor Anthropic has reviewed this analysis.
The short version
One publication with two evaluations. The first section covers what OpenAI measured and reported; the rest covers what it means for organisations that run COBOL, what is still unknown and what to set up before a pilot.
- COBOL migration and MultiLangBench
- OpenAI reports GPT-6.1 Sol migrates COBOL programs with 86.6% accuracy at low reasoning effort, up from 5.4% for GPT-5.6 Sol at 61% of the cost, and 97.9% at higher effort. GPT-6 Astra passes 87.4% of tests on a 20-language build task.
Two evaluations: building in 20 languages, and migrating COBOL without changing what it does
What was announced
On 19 October 2026 OpenAI published results for coding in low-resource programming languages, which it defines as languages such as COBOL, Fortran and Visual Basic that have far less public code and documentation than Python or JavaScript. OpenAI says the work follows feedback from developers and enterprise customers that coding models struggle with these languages, and it introduced two evaluations to measure it.
The COBOL migration evaluation asks a model to migrate COBOL programs to C++, Java, Kotlin, Python or Rust while keeping their behaviour. Hidden tests require byte-for-byte parity on observable outputs, and a migration counts only if every test passes. At low reasoning effort GPT-6.1 Sol scores 86.6%, against 5.4% for GPT-5.6 Sol at the same setting, at 61% of the API cost. At higher reasoning effort it reaches 97.9%, which OpenAI describes as close to saturating the evaluation.
MultiLangBench asks a model to build a SQLite-compatible database engine from scratch in one of 20 languages, checked against the reference SQLite. At maximum reasoning effort GPT-6 Astra passes 87.4% of tests on average, up from 26.8% for GPT-5.6 Sol, and above 83% in 19 of the 20 languages. COBOL remains the hardest at 72.1%, up from 10.9%; Fortran moves from 21.4% to 83.5% and Visual Basic from 27.5% to 89.4%. GPT-6 Luna, the lower-cost option, averages 40.3% at an estimated $0.77 per sample against $3.48 for GPT-5.6 Sol, which OpenAI puts at about 78% lower.
OpenAI has released the MultiLangBench setup and test suite on GitHub under the Apache 2.0 licence. Its own post adds a caveat in plain terms: applying the capability to production software still requires expert review and testing against the system's business rules and dependencies.
What it means for enterprises
The cost of finding out has likely dropped. Banks and insurers that run COBOL have often treated migration as a multi-year programme because the business rules live only in the code and the people who wrote it have left. A model that can translate a program and pass a strict parity test on most attempts makes a bounded pilot on one batch job or one calculation module a realistic first step, rather than a decision about the whole estate.
The hard part moves to the tests. OpenAI's evaluation counts a migration as successful only when hidden tests show identical outputs. Most legacy estates do not have that test suite; it has to be built by recording what the current system does with real inputs. OpenAI's own example shows why: COBOL fixed-point arithmetic drops excess decimal digits unless rounding is asked for, so a rewrite that rounds can run cleanly and still produce different financial results.
These are a vendor's numbers on a vendor's test. They are a reason to run your own evaluation on your own programs, not a forecast of what a migration will cost or how long it will take. For a regulated firm, a migrated system is also a changed system, which likely brings model risk and change management teams into the pilot from the start.
Before you switch it on
- Parity tests first
- Before any translation, capture the current program's outputs for a representative set of real inputs. Those recordings become the pass mark.
- Arithmetic rules
- List the rounding, truncation and missing-record behaviour the business depends on, and test each one explicitly in the migrated code.
- Effort and cost
- OpenAI reports both low and high reasoning effort results. Measure accuracy and spend at each setting on your own programs before budgeting.
- Where the code goes
- Source code often encodes pricing and customer logic. Confirm the data handling and region terms of the API account before sending it.
My take
This is a game changer and right up our field, as I have associates that have worked in deep banking and highly regulated industries, so this capability will be awesome for finance.
Where the market is heading: legacy code as a target for coding agents
Coding model results have mostly been reported on popular languages. OpenAI publishing COBOL and Fortran scores, alongside an open test suite, signals that vendors now see legacy maintenance and migration as a market for coding models in their own right, and that cost per task is part of the pitch as much as accuracy.
If the numbers hold on real estates, the bottleneck likely moves from writing the new code to proving it behaves the same: test capture, reconciliation runs and sign-off by the people who own the business rules.
What is not known yet
The post, read a day after publication, leaves several questions open.
- Size of the COBOL test
- OpenAI does not say how many programs the COBOL migration evaluation contains or how large they are.
- Whether it is released
- The MultiLangBench repository does not include the COBOL migration evaluation, and the post does not say it will be released.
- Independent results
- No third-party replication of either evaluation was published at the time of checking.
- Real estates
- The post does not say how the models handle the code that usually sits around COBOL in production, such as shared copybooks and batch scheduling.
For engineers: how the evaluations work and how to run your own
What MultiLangBench actually tests
The model implements SQLite 3.40.1 in the requested language as an executable that takes a database path and reads a query on stdin, with no internet access and SQLite removed from the environment. The repository's tests were converted from SQLite's own TCL tests into an initial database, a query and a validation query; the output database is checked with the real SQLite. Crashes and runs over 60 seconds fail, and code in the wrong language scores zero.
Results are reported as a pair of API cost and test pass rate. Routine runs use a subset of 272 tests; the repository allows the full converted suite for higher precision if that is stated.
Sources: [2]
Borrowing the method for a COBOL pilot
OpenAI's migration test uses hidden tests with byte-for-byte parity on observable outputs. The same shape works in-house: record inputs and outputs from the running system, keep them out of the model's context, and treat any difference as a failure until a business owner signs it off.
Run the same programs at low and high reasoning effort and record cost and pass rate for each, so the choice of model and setting is made on your data rather than on the published averages.
Sources: [1]
Where 1AYM fits
A model that writes the migrated code is half of the job; the other half is making sure that code takes the same path to production as every other change. Our AI Engineering Transformation engagement adds the architecture, guardrails and CI around the coding tools teams already use, so AI-assisted changes pass the same review and gates as the rest.
If the open question is which legacy system is worth a pilot at all, we would start with our AI Opportunity & Feasibility Sprint: fixed scope, typically two to four weeks.
Sources
Frequently asked questions
What did OpenAI announce about COBOL?
On 19 October 2026 OpenAI published a COBOL migration evaluation in which GPT-6.1 Sol migrated COBOL programs to modern languages with 86.6% accuracy at low reasoning effort, against 5.4% for GPT-5.6 Sol, and 97.9% at higher effort.
Are these results independent?
No. Both evaluations were designed and run by OpenAI. The MultiLangBench test suite is public on GitHub, so others can run it; the COBOL migration evaluation is not in that repository.
Which languages can the models migrate COBOL into?
The evaluation migrates COBOL programs to C++, Java, Kotlin, Python or Rust.
Does this mean a COBOL system can be migrated automatically?
OpenAI says no: applying it to production software still requires expert review and testing against the system's business rules and dependencies. Strict tests of current behaviour are the first thing to build.
What is MultiLangBench?
An OpenAI evaluation, released under Apache 2.0, that asks a model to build a SQLite-compatible database engine in one of 20 languages and checks the result against the real SQLite.
Further
- Rolling out Codex and Claude Code across engineering · How to roll coding agents out across an engineering organisation with shared skills, guardrails and review.
- SR 26-2 and generative AI in banks · What US banks still owe their LLM systems now SR 26-2 leaves generative AI out of scope, with a control set.
- Codex pricing · Which ChatGPT plans include OpenAI's coding agent, how its credits work and how to budget a pilot.
- OpenAI DevDay 2026 · The October launches from OpenAI, including GPT-6.1 Sol and the plans it is available on.
- Meta's Muse Wonder: price and data terms · Another vendor's API price per million tokens and data modes, for comparing the cost of a pilot across providers.
Weighing an AI pilot on a legacy system?
Half an hour is enough to pick a bounded first program and agree how its current behaviour will be tested.
Last reviewed · 1AYM