Last winter, an SAP consultant named Lao Zhang retired after 20 years at a manufacturing company in eastern China. He took with him more than just a badge—he carried the "living map" of hundreds of ABAP programs across the ECC system: which code block once fixed a bizarre tax code bug at 3 AM, which RFC interface connected to which authentication layer of the Golden Tax system, and which Z-table fields named SGTXT actually stored invoice remarks.
This knowledge was never written in any document. Now, it is gone.
Lao Zhang's story is not unique. SAP is everywhere—over 77% of the world's transaction revenue flows through SAP systems, and 87% of the Fortune 500 use them. SAP has announced that ECC (Business Suite 7) will end mainstream maintenance at the end of 2027, with extended maintenance available through the end of 2030 (SAP official). Companies still on ECC will sooner or later have to face migration to S/4HANA.
Why SAP Migration Is Different from Ordinary Code Migration
If you've done a framework upgrade for a web application, you might think "code migration is just code migration." But SAP migration is an entirely different species. You're not just updating familiar code to better patterns—you're reverse-engineering a complex business system that intertwines finance, supply chain, human resources, and tax compliance.
The code is the living documentation, and the people who could read it are gone.
The traditional solution was to assemble massive consultant teams, spending months or even years mapping out workflows. But the cost deterred most companies—so the systems just sat there, year after year, with the refrain "let's wait a bit longer."
AI is changing all of this. But to what extent, exactly? Imagination alone won't answer that—we decided to use data.
ABAP-Bench: Giving AI an SAP Consultant Qualification Exam
Existing AI code evaluation benchmarks are no help here. SWE-bench tests GitHub bug fixes, BigCodeBench tests general programming, and HumanEval only has algorithm problems. The SAP/ABAP domain is a complete blank on the AI evaluation map.
So we built one ourselves. ABAP-Bench—60 expert-authored SAP migration tasks, covering 9 key dimensions. Each task is worth 20 points, independently judged by a three-layer scoring engine. We gave AI an SAP consultant qualification exam, then graded it meticulously.
Building ABAP-Bench
Task Design
Every task originates from real SAP migration project scenarios, written by domain experts. We accept no vague answers: if the AI responds "consider using a CDS view instead of the BKPF table"—zero points. You must specify I_JournalEntry, with field BUKRS mapped to CompanyCode.
The test code uses Chinese variable names (e.g., 发票号码, 含税金额), which is extremely rare in public code repositories—ABAP-Bench's "anti-cheating watermark."
Three-Layer Scoring Engine
Click each layer to learn the scoring mechanism:
I_JournalEntry earns points; incorrectly claiming "BKPF is the primary table in S/4HANA" deducts 3 points—saying something wrong is worse than saying nothing at all.Preventing Evaluation "Expiry"
Every quarter, 10% of old tasks are retired and replaced with new ones; tasks are anchored to specific versions of SAP Notes and regulatory documents. Like annual exams that change questions every year, ABAP-Bench's validity doesn't decay over time.
Engineering Quality
First Evaluation: GLM-5
We chose Zhipu GLM-5 as the first challenger—Zhipu AI's flagship reasoning model, equipped with internal chain-of-thought reasoning capability. The evaluation covers the full lifecycle of SAP migration, with temperature fixed at 0.1 to ensure reproducibility. Then, we quietly waited for 145 minutes.
Take a guess: what score can GLM-5 achieve?
60 SAP expert-level tasks, max score 100
Evaluation Results: 61 / 100
At first glance, it might seem "just okay." But dig into the specific tasks, and a more vivid picture emerges—this unevenness itself is the most valuable finding.
Can you find this ABAP bug? — After the first LOOP AT gt_bkpf INTO DATA(ls_bkpf) loop ends, what will be the value of ls_bkpf?
DATA() inline-declared variable persists after the loop ends, retaining the value from the last iteration. GLM-5 correctly identified this bug and pointed out that it would cause "the credit memo/normal status judgment for all invoices to be completely wrong." This was one of the most surprising findings in the evaluation.
Surprising Moments
It wrote directly compilable S/4HANA code
T03 · 16/20Rewriting the ECC-era nested SELECTs (BKPF + BSEG, a classic N+1 disaster) into an S/4HANA-compatible version.
* ECC: N+1 query anti-pattern SELECT * FROM bkpf INTO TABLE @DATA(lt_bkpf) WHERE bukrs = @p_bukrs AND gjahr = @p_gjahr. LOOP AT lt_bkpf INTO DATA(ls_bkpf). SELECT * FROM bseg INTO TABLE @DATA(lt_bseg) WHERE bukrs = @ls_bkpf-bukrs AND belnr = @ls_bkpf-belnr AND gjahr = @ls_bkpf-gjahr. ENDLOOP.
* GLM-5: CDS View JOIN, single query SELECT j~CompanyCode, j~AccountingDocument, j~FiscalYear, i~AccountingDocumentItem, i~GLAccount, i~AmountInCompanyCodeCurrency FROM I_JournalEntry AS j INNER JOIN I_JournalEntryItem AS i ON j~CompanyCode = i~CompanyCode AND j~AccountingDocument = i~AccountingDocument AND j~FiscalYear = i~FiscalYear WHERE j~CompanyCode = @p_bukrs AND j~FiscalYear = @p_gjahr INTO TABLE @DATA(lt_result).
Correctly chose CDS view JOIN, precisely mapped field names (BUKRS → CompanyCode), and used modern inline declaration syntax. This code can compile and run directly.
Found hidden bugs like a meticulous code reviewer
T02 · 12/20Five bugs were planted in a VAT invoice processing program. GLM-5 precisely hit the most critical one: the residual value problem of variable ls_bkpf after the loop ends, and accurately described the business impact—"the credit memo/normal status judgment for all invoices is completely wrong, leading to severely distorted tax filing data."
This is not a textbook answer, but one that truly understands ABAP runtime behavior and the VAT business context.
China-specific regulations are its home turf
T16 · 17/20The highest score in the entire evaluation. Correctly identified the five-insurance-one-fund calculation rules, specific fields for the cumulative withholding method for individual income tax, and the standard usage of the HR_READ_INFOTYPE function module. Full rubric score, full structural quality score.
VAT red-letter invoice rules, e-invoice implications, PIPL data compliance—these answers are scattered across Chinese policy documents and SAP configuration depths, requiring even senior consultants to research for half a day.
Risk analysis on par with a junior consultant
T43 · 16/20Evaluating the migration risk for an ECC system with 800 custom tables. GLM-5 identified the impact of Pool/Cluster tables, Z-table issues referencing deleted fields, and accurately mentioned the specific rule set names of SAP Readiness Check 2.0 and ABAP Test Cockpit.
This kind of analysis previously required senior consultants days to complete. AI produced a qualified first draft in under 4 minutes.
In which dimension did GLM-5 score the lowest?
LOOP...ENDLOOP immediately followed by IF sy-subrc = 0 will instinctively frown. This intuition remains a uniquely human advantage for now.
Notable Limitations
Steady on familiar paths, stumbling on unfamiliar ones
16 vs 5T03 (classic CDS rewrite) 16/20, T37 (RAP EML rewrite) 5/20. Classic scenarios have abundant documentation; cutting-edge paradigms have scarce resources. In T37, GLM-5's direction was correct (I_Product, MODIFY ENTITIES), but the EML syntax had subtle errors.
Like a newly hired consultant: knows which road to take, but hasn't yet learned every stone along the way.
The most hidden bugs still belong to humans
5 bugs, hit 3The 5 bugs in T02 were ranked by subtlety. GLM-5 hit the first 3 but missed the deepest 2—internal table/structure type confusion, and SY-SUBRC being unexpectedly overwritten in nested LOOPs.
AI can learn knowledge, but the intuition born from "having been burned" remains a uniquely human advantage for now.
Thinking longer doesn't mean answering better
155s → 17pts 447s → 6ptsThe reasoning model averaged 145 seconds per task. The highest-scoring T16 (17/20) took 155 seconds, while the lowest-scoring T36 (6/20) took 447 seconds. Thinking longer cannot compensate for knowledge gaps.
Human + AI: The Optimal Solution for SAP Migration
ask where AI can help consultants save the most time.
AI Can Handle Today
- Pre-migration risk assessment and roadmap drafts
- China-specific compliance mapping
- Architecture solution discussions (FI-CO / ACDOCA / CDS)
- Code rewrites for classic scenarios
Still Requires Human Oversight
- Production-grade code for cutting-edge paradigms
- Final confirmation of deep defects
- Implementation of performance optimizations
This is not a story of AI replacing people, but of AI changing how people work.
In the past, senior consultants spent two weeks writing a migration assessment report. Now, AI generates a first draft in hours, and consultants shift from "writing from scratch" to "reviewing and optimizing"—efficiency increases several-fold, and quality doesn't drop but actually improves.
Conclusion: This Future Is Closer Than You Think
SAP migration has stalled for years; at its core, this is an economics problem: the cost of understanding legacy systems is simply too high.
AI is breaking down this cost barrier. 61/100 is not "barely passing"—it is a signal: AI in the SAP domain has crossed the "toy" phase and entered the starting point of "usable." It can write compilable S/4HANA code, find variable-scope bugs, and produce migration risk reports with specific tools and steps.
The shortcomings it reveals—precision in cutting-edge syntax, capturing the deepest defects—point precisely to the next direction. These are not fundamental barriers, but capability boundaries that can be continuously improved as training data grows richer.
The S/4HANA migration timeline won't wait, and consultants like Lao Zhang aren't coming back.
Fortunately, a new helper has arrived—it's not perfect, but it's online 24 hours a day, never forgets a single field mapping, and gets a little better every day.