Transpiling the Untranspileable: Patterns for Building Clean Transpilers

Building a production-grade transpiler requires strict separation between syntax parsing, semantic binding, and target AST lowering. Using a UniBasic-to-Java case study, this talk presents practical design patterns for engineering clean, modular code transpilers.

What problem are you addressing?

Building transpilers, DSL engines, or code generators often becomes convoluted when parsing rules are blurred with semantic resolution, when transformation passes fail to lower source concepts to native ASTs, or when emitters attempt to inline complex built-in functions directly into target expressions.

When modernizing 40-year-old legacy systems (UniBasic) into modern Java, early architectures lose modularity under complex language quirks: ambiguous operator syntax, global memory blocks (COMMON), procedural GOSUB routines, pass-by-reference parameters, and built-in functions (TRIM, OCONV, INDEX).

This session addresses how to structure a production-grade transpiler using formal design patterns so that parsing remains syntax-only, transformations lower cleanly to native ASTs backed by a thin runtime library, and emitters remain 100% dumb and deterministic.

Who is the intended audience?

Developers / Software Architects / Language & Tooling Engineers

Level

Intermediate

List one or two practical takeaways

1. A Decoupled 7-Stage Pipeline Blueprint: A proven architecture for structuring code translators into single-responsibility passes—keeping parsing 100% syntax-only, semantic binding isolated, and AST lowering distinct from code printing.
2. The Thin Runtime Offloading Strategy: How delegating complex built-in functions (TRIM, INDEX) and dynamic value behaviors (UnibasicValue) to a lightweight runtime library reduces transpiler AST complexity by orders of magnitude and produces clean, readable Java.
3. Template-Driven Emission over String Appends: How replacing brittle StringBuilder code generators with structured layout objects (Template, FixedLayoutTemplate) automatically enforces formatting, indentation, and statement termination rules.

What will you share?

We will walk through our production pipeline architecture, contrasting our initial monolithic prototype with our refactored multi-stage design:

Source Code
    │
    ▼
[Preprocessing & Inlining]  ──► Resolve $INCLUDE graphs & fragment boundaries
    │
    ▼
[Lexical Analysis]          ──► Generate flat token stream
    │
    ▼
[Parsing]                   ──► Build syntax-only UniBasic AST 
    │
    ▼
[Semantic Analysis]         ──► Construct COMMON/DIM registries & bind types
    │
    ▼
[AST Transformation]        ──► Lower UniBasic AST to Java AST (Domain-directed naming)
    │
    ▼
[Fact Collection]           ──► Collect imports and class fields
    │
    ▼
[Emission]                  ──► Template-driven emission
(Zero logic/lookups)

• Architecture Decisions: Designing a 7-stage compiler pipeline where each pass has a single responsibility.
• Code & Implementation Details: Pratt parselets with same-line guards, AST lowering rules, parameter write-back generation, and thin runtime helpers.
• Trade-offs & Alternatives: Choosing mechanical type binding over complex global inference, and delegating legacy built-ins to a runtime library instead of inline code generation.
• Before-and-After Experience: Real code examples showing how refactoring from stateful emitters to pure printers improved testability and modularity.

What is your experience with this problem?

• Internal engineering project
• Hard-earned engineering lesson
• Retrospective/lessons from a previous system

What approaches failed, disappointed, or created unexpected problems?

In our initial implementation attempt, the architecture became overly convoluted and non-modular due to:

1. Blurring Parsing and Semantics: Performing symbol lookups, type checking, and COMMON block resolution directly inside parser rules, creating an entangled parser call graph.
2. Incomplete AST Lowering: Transforming source trees without technically lowering them to native target AST constructs, retaining high-level UniBasic nodes (e.g. MatAssignment) in the target representation instead of decomposing them into Java primitives (e.g. Arrays.fill(...)).
3. Stateful Emitters using String Concatenation: Writing code emitters that performed symbol table lookups and string appends, making code generation brittle and hard to extend.

What will you do differently today? (we are doing these)

We enforce clear, modular transpiler design patterns:

1. Pre-Lexing Graph Closure: Resolving $INCLUDE directives and fragment inlining before tokenization.
2. Syntax-Only Parsing: Enforcing a pure Directed Acyclic Graph (DAG) for parser rules with zero semantic state.
3. Pratt Parselets with Same-Line Guards: Handling ambiguous legacy operators and line-dependent formatting specifiers cleanly.
4. Dedicated Semantic Analysis: Building symbol tables, COMMON/DIM block registries, and type bindings in a separate pass.
5. True AST-to-AST Lowering: Decomposing UniBasic language constructs directly into native Java AST nodes (ClassNode, MethodNode, FieldNode, Arrays.fill invocations, domain-directed method names).
6. The Thin Runtime Library Pattern: Translating built-in functions (TRIM, INDEX, OCONV) into invocations of a thin, specialized runtime helper (UnibasicRuntime.trim(var)), encapsulating 1-based indexing, multi-value delimiters (@AM, @VM), and null-safety outside the transpiler AST
7. Parameter Promotion & Write-Backs: Mapping procedural pass-by-reference to Java method parameters + instance field write-backs (this.x = instance.x).
8. The Fact Collector Pattern: Gathering import statements, instance variables, and dependencies via a read-only AST walk prior to emission.
9. Template-Driven Emitters: Mapping target AST nodes into structured layout templates (FixedLayoutTemplate, StringTemplate) that handle layout, indentation, and statement termination.

What trade-offs did you consider?

1. Mechanical Type Binding vs. Type Inference: We opted for mechanical type binding anchored by explicit assignments and language anchors, avoiding overly complex global type inference that could introduce silent compilation bugs.
2. Thin Runtime Library vs. Inline Code Generation: We chose to route built-in functions and dynamic array behavior (TRIM, OCONV) to a thin runtime library rather than emitting complex inline Java expressions, optimizing for target code readability, testability, and simple AST transformation logic.
3. Pure Emitters vs. Flexible Emitter Logic: We knowingly gave up emission-time flexibility in exchange for strict upstream encoding: if an emitter needs information, it must be explicitly placed into the Target AST by upstream passes first.

How can this help other practitioners?

A Design Approach & Blueprint:

Provides a reusable, step-by-step pipeline architecture for anyone building transpilers, DSL engines, code generators, or modernizing legacy codebases.

Concrete Patterns to Adopt:

• Pratt Parsing with Line-Aware Guards: Parse ambiguous or operator-dense grammars cleanly using token-registered parselets instead of exploding recursive descent rules.
• Contextual Disambiguation: Resolving syntactic ambiguities on the fly, such as splitting greedily-matched tokens (e.g. decomposing >= into a closing delimiter > and an assignment operator =) based on parser context, without breaking operator precedence loops.
• True AST-to-AST Lowering: Decompose source language abstractions into native target AST nodes before code emission.
• The Fact Collector Pass: Aggregate imports, instance fields, and class dependencies via a read-only AST walk to keep code emitters completely stateless.
• Procedural-to-OO Bridge Patterns: Translate pass-by-reference parameter mutation (CALL SUB(A,B)), global session state (COMMON blocks), and GOSUB routines into Java domain classes.

Costly Mistakes to Avoid:

• Mixing semantics into parser rules: Blurring symbol lookups into parsing creates an entangled, un-testable parser call graph.
• String-concatenation emitters: Raw string appends scatter layout logic, semicolon handling, and type checks across printers.
• Inlining complex built-ins inside AST nodes: Inlining complex string/array logic directly into target AST nodes bloats transpiler code generation instead of delegating to a thin runtime.

#architecture #compilers #transpiler #designpatterns #softwarecraftsmanship #codegeneration #dsl #refactoring #casestudy