Geode

Geode is a high-performance platform for training neuro-symbolic programs with Evolution Strategies (ES) — gradient-free optimization that sidesteps backpropagation and parallelizes naturally across a large population. Because ES needs no gradients, the thing you train can be any program you can compute, not just a differentiable network:

• neural networks — layers and nonlinearities
• simulations and other non-differentiable objectives
• symbolic / algorithmic logic — conditionals, loops, arithmetic, trig
• custom loss, fitness, or reward functions
• hybrids that interleave all of the above

Programs are compiled to standalone native C (CPU or Apple Metal), so the ES inner loop runs at near-inference throughput instead of being bottlenecked by a Python training loop.

The goal is production-grade ES at scale. Geode implements EGGROLL's low-rank perturbations — the technique that makes ES tractable on large models — and applies modern, production-ready optimizers (MUON, Adam, SGDM, RMSProp) to the ES gradient estimate, with exact checkpoint/resume. A program is written in a compact stack-based DSL, but the DSL is just the front end; the substance is the compiled ES engine.

It is currently exercised on small networks (see the quick start below), with the roadmap aimed at productionizing EGGROLL-scale training and validating production optimizers in the ES setting.

How it works

Geode doesn't interpret your program step by step. It compiles it to native code — the way C source is built into an executable — so training runs with no interpreter in the loop. The flow is three stages:

1. Write a Geode program (.gs) describing the network, the objective, and the training settings.
2. Compile (compile.sh): Geode's compiler reads your program and emits specialized C source — your forward pass, your objective, and the entire ES training loop, written out as plain C (plus Metal kernels for the GPU target). It then builds that with clang into one self-contained native binary at build/out/program.
3. Run that binary: it executes the whole training loop natively — drawing the population of perturbations, evaluating your objective across them, scoring, and applying the optimizer — printing metrics each round and writing checkpoints. No Python is involved at this stage.

Why generate C? The expensive part of ES is evaluating the objective for a large population, every round. Compiling your specific program to native code lets that inner loop run at full CPU/GPU speed with no per-step overhead — and the result is a portable, standalone binary (Python is needed only to build it, not to run it).

Requirements

• Python 3.10+ — runs Geode's compiler
• A C compiler (clang; on macOS install Apple's Command Line Tools with xcode-select --install)
• For GPU training, a Mac with Apple Silicon / Metal — Apple GPUs (Metal) for now; CUDA support is coming soon. The CPU backend needs nothing more and runs anywhere.

Install

Geode runs straight from its source folder — nothing to build or install first:

git clone <repo-url> py-geode
cd py-geode

Check it works:

python3 -m pytest

Quick start

A minimal smoke test: a small autoencoder trained on the bundled dataset (datasets/sine-1000.csv — 1000 examples of 100 numbers each). The same program shape scales up by swapping in a larger network and perturbation, or generalises by replacing the objective with a simulation or symbolic program. Save this as sine-ae.gs in the project root:

TARGET-CPU

: NEURON ( <layer> n -- <layer> ) LINEAR INIT-HE TRAINABLE RELU ;   \ a layer of ReLU units
: LINO   ( <layer> n -- <layer> ) LINEAR INIT-XAVIER TRAINABLE ;    \ a plain linear layer

: FORWARD   ( r:* -- r:100 ) 8 NEURON 100 LINO ;               \ the network: 100 -> 8 -> 100
: OBJECTIVE ( r:b:100 -- r ) DUP FORWARD LOSS-EUCLIDEAN SUM ;  \ reconstruct input; error to minimise

: SETTINGS ( -- )
  OBJECTIVE-LOSS
  200 EPOCHS 4 WORKERS 0.01 ALPHA 42 SEED       \ 200 rounds, fixed seed for repeatability
  MUON-UPDATES 5 MUON-WARMUP! 0.9 MUON-BETA! 0.1 MUON-KAPPA!
  256 POPULATION ANTITHETIC SCORE-RANK-SHAPED   \ 256 perturbations per round
  0.01 ES-GAUSSIAN
  .METRICS ;

: MAIN ( -- )
  SETTINGS
  "datasets/sine-1000.csv" 100 CSV-INPUT        \ dataset path, resolved at runtime relative to your cwd
  ALL ['] OBJECTIVE TRAIN ;

(Geode's syntax is stack-based, in the Forth tradition: each definition's ( before -- after ) signature declares what it consumes and produces.)

Compile it to a native program, then run the binary from the project root (the datasets/... path is resolved at runtime relative to your current directory, so running from elsewhere won't find the file):

./compile.sh --fast sine-ae.gs build
./build/out/program            # run from the project root so datasets/ resolves

The error falls each round as the network learns:

epoch=0    ... best=57476 ...
epoch=99   ... best=44180 ...
epoch=199  ... best=36307 ...

For GPU training, use Apple GPUs via Metal today — change the first line to TARGET-APPLE-METAL (CUDA support is coming soon). To inspect the compiled assembly instead of running, use ./assembly.sh sine-ae.gs.

What's inside

• Perturbations: dense Gaussian (ES-GAUSSIAN) and EGGROLL low-rank (EGGROLL-R1, OMELETTE-R1), with antithetic sampling and rank-shaped scoring.
• Optimizers: MUON (Newton–Schulz-orthogonalized momentum), ADAM, SGDM, RMSPROP, applied to the ES gradient estimate.
• Checkpointing: SAVE-MODEL / SAVE-MODEL-ALL / RESUME-MODEL for exact, resumable runs.
• Targets: native C for CPU and Apple Metal (CUDA coming soon).
• Weight tying: distinct weights per call site by default; opt into sharing with SHARED / (SHARED) / ~ (see below).

Weight tying

By default every call site gets its own weights, so the parameter count matches what the source reads as having — 12 NEURON 12 NEURON is two independent layers. When you want applications to share one set of weights — recurrent steps, deep-equilibrium fixed points, Siamese encoders, repeated transformer blocks — opt in explicitly:

• SHARED marks the most-recently defined word as tied: every call to it shares one set of weights. (SHARED) does the same to a named word by its token (['] BLOCK (SHARED)), so you can mark a word defined elsewhere.
• ~ ( xt -- m ) makes a fresh tied instance from an execution token and binds it to a local; referencing that local applies the word with shared weights. Each ~ is a separate instance, so two instances of the same word stay independent.

: BLOCK ( r:b:100 -- r:b:100 ) 100 LINEAR INIT-HE TRAINABLE RELU ;

\ One tied instance, applied twice -> a single shared 100x100 weight set
\ (a deep-equilibrium step: reconstruct the input from BLOCK(BLOCK(input))):
: OBJECTIVE ( r:b:100 -- r )
  ['] BLOCK ~ { block }
  DUP block block LOSS-EUCLIDEAN SUM ;

A tied unit shares one set of weights, so all of its applications must have the same input shape and constants — applying it at a different shape or literal is a compile-time error, not a silent split. Tied networks save and load at the reduced (unique) parameter count, and a checkpoint from a tied network is rejected by an untied build (and vice versa). Runnable examples are in examples/.

Project layout

• geode/ — the compiler, C/Metal backend, and runtime
• examples/ — runnable example programs (including weight tying)
• tests/ — the test suite (run with python3 -m pytest)
• datasets/ — bundled example data