l2/SPEC.md

L2 Language Specification (Draft)

1. Design Goals

• Meta-language first: L2 is a minimal core designed to be reshaped into other languages at runtime, matching Forth's malleability with modern tooling.
• Native code generation: Source compiles directly to NASM-compatible x86-64 assembly, enabling both AOT binaries and JIT-style pipelines.
• Runtime self-modification: Parsers, macro expanders, and the execution pipeline are ordinary user-defined words that can be swapped or rewritten on demand.
• Total control: Provide unchecked memory access, inline assembly, and ABI-level hooks for syscalls/FFI, leaving safety policies to user space.
• Self-hosting path: The bootstrap reference implementation lives in Python, but the language must be able to reimplement its toolchain using its own facilities plus inline asm.

2. Program Model

• Execution units (words): Everything is a word. Words can be defined in high-level L2, inline asm, or as parser/runtime hooks.
• Compilation pipeline:
  1. Source stream tokenized via active reader (user-overridable).
  2. Tokens dispatched to interpreter or compiler hooks (also user-overridable).
  3. Resulting IR is a threaded list of word references.
  4. Code generator emits NASM .text with helper macros.
  5. nasm + ld (or custom linker) build an ELF64 executable.
• Interpreted mode: For REPLs or rapid experimentation, the compiler can emit temporary asm, assemble to an object in memory, and dlopen or execve it.
• Bootstrapping: main.py orchestrates tokenizer, dictionary, IR, and final asm emission.

3. Parsing & Macro System

• Reader hooks:
  • read-token: splits the byte stream; default is whitespace delimited with numeric/string literal recognizers.
  • on-token: user code decides whether to interpret, compile, or treat the token as syntax.
  • lookup: resolves token → word entry; can be replaced to build new namespaces or module systems.
• Compile vs interpret: Each word advertises stack effect + immediacy. Immediate words execute during compilation (macro behavior). Others emit code or inline asm.
• Syntax morphing: Provide primitives set-reader, with-reader, and word-lists so layers (e.g., Lisp-like forms) can be composed.

4. Core Types & Data Model

• Cells: 64-bit signed integers; all stack operations use cells.
• Double cells: 128-bit values formed by two cells; used for addresses or 128-bit arithmetic.
• Typed views: Optional helper words interpret memory as bytes, half-words, floats, or structs but core semantics stay cell-based.
• User-defined types: struct, union, and enum builders produce layout descriptors plus accessor words that expand to raw loads/stores.

5. Stacks & Calling Convention

• Data stack: Unlimited (up to memory). Manipulated via standard words (dup, swap, rot, over). Compiled code keeps top-of-stack in registers when possible for performance.
• Return stack: Used for control flow. Directly accessible for meta-programming; users must avoid corrupting call frames unless intentional.
• Control stack: Optional third stack for advanced flow transformations (e.g., continuations) implemented in the standard library.
• Call ABI: Compiled words follow System V: arguments mapped from data stack into registers before call, results pushed back afterward.

6. Memory & Allocation

• Linear memory primitives: @ (fetch), ! (store), +!, -!, memcpy, memset translate to plain loads/stores without checks.
• Address spaces: Single flat 64-bit space; no segmentation. Users may map devices via mmap or syscalls.
• Allocators:
  • Default bump allocator in the runtime prelude.
  • install-allocator allows swapping malloc/free pairs at runtime.
  • Allocators are just words; nothing prevents multiple domains.

7. Control Flow

• Branching: if ... else ... then, begin ... until, case ... endcase compile to standard conditional jumps. Users can redefine the parsing words to create new control forms.
• Tail calls: tail word emits jmp instead of call, enabling explicit TCO.
• Exceptions: Not baked in; provide optional libraries that implement condition stacks via return-stack manipulation.

8. Inline Assembly & Low-Level Hooks

• Asm blocks: asm { ... } injects raw NASM inside a word. The compiler preserves stack/register invariants by letting asm declare its stack effect signature.
• Asm-defined words: :asm name ( in -- out ) { ... } ; generates a label and copies the block verbatim, wrapping prologue/epilogue helpers.
• Macro assembler helpers: Provide macros for stack slots (.tos, .nos), temporary registers, and calling runtime helpers.

9. Foreign Function Interface

• Symbol import: c-import "libc.so" clock_gettime loads a symbol and records its address as a constant word. Multiple libraries can be opened and cached.
• Call sites: c-call ( in -- out ) symbol pops arguments, loads System V argument registers, issues call symbol, then pushes return values. Variadic calls require the user to manage al for arg count.
• Struct marshalling: Helper words with-struct and field macros emit raw loads/stores so C structs can be passed by pointer without extra runtime support.
• Error handling: The runtime never inspects errno; users can read/write the TLS slot through provided helper words.

10. Syscalls & OS Integration

• Primitive syscall: syscall ( args... nr -- ret ) expects the syscall number at the top of stack, maps previous values to rdi, rsi, rdx, r10, r8, r9, runs syscall, and returns rax.
• Wrappers: The standard library layers ergonomic words (open, mmap, clone, etc.) over the primitive but exposes hooks to override or extend them.
• Process bootstrap: Entry stub captures argc, argv, envp, stores them in global cells (argc, argv-base), and pushes them on the data stack before invoking the user main word.

11. Module & Namespace System

• Wordlists: Dictionaries can be stacked; within wordlist ... end temporarily searches a specific namespace.
• Sealing: Wordlists may be frozen to prevent redefinition, but the default remains open-world recompilation.
• Import forms: use module-name copies references into the active wordlist; advanced loaders can be authored entirely in L2.

12. Build & Tooling Pipeline

• Compiler driver: main.py exposes modes: build <src> -o a.out, repl, emit-asm, emit-obj.
• External tools: Default path is nasm -f elf64 then ld; flags pass-through so users can link against custom CRT or libc replacements.
• Incremental/JIT: Driver may pipe asm into nasm via stdin and dlopen the resulting shared object for REPL-like workflows.
• Configuration: A manifest (TOML or .sl) records include paths, default allocators, and target triples for future cross-compilation.

13. Self-Hosting Strategy

• Phase 1: Python host provides tokenizer, parser hooks, dictionary, and code emitter.
• Phase 2: Re-implement tokenizer + dictionary in L2 using inline asm for hot paths; Python shrinks to a thin driver.
• Phase 3: Full self-host—compiler, assembler helpers, and driver written in L2, requiring only nasm/ld.

14. Standard Library Sketch

• Core words: Arithmetic, logic, stack ops, comparison, memory access, control flow combinators.
• Meta words: Reader management, dictionary inspection, definition forms (:, :noninline, :asm, immediate).
• Allocators: Default bump allocator, arena allocator, and hook to install custom malloc/free pairs.
• FFI/syscalls: Thin wrappers plus convenience words for POSIX-level APIs.
• Diagnostics: Minimal type, emit, cr, dump, and tracing hooks for debugging emitted asm.

15. Command-Line & Environment

• Entry contract: main receives argc argv -- exit-code on the data stack. Programs push the desired exit code before invoking exit or returning to runtime epilogue.
• Environment access: envp pointer stored in .data; helper words convert entries to counted strings or key/value maps.
• Args parsing: Library combinators transform argv into richer domain structures, though raw pointer arithmetic remains available.

16. Extensibility & Safety Considerations

• Hot reload: Redefining a word overwrites its dictionary entry and emits fresh asm. Users must relink or patch call sites if binaries are already running.
• Sandboxing: None by default. Documented patterns show how to wrap memory/syscall words to build capability subsets without touching the core.
• Testing hooks: Interpreter-mode trace prints emitted asm per word to aid verification.
• Portability: Spec targets x86-64 System V for now but the abstraction layers (stack macros, calling helpers) permit future backends.