🌼 DaisyChain-Train — Old Hardware Training Pipeline

Part of DaisyChain on 🤗 Hugging Face → https://huggingface.co/DaisyChainAI Model page (weights + card): https://huggingface.co/DaisyChainAI/DaisyChain-Train

In plain terms: DaisyChain-Train lets you use old / spare machines to train neural networks. The training runs through emulated GPU logic — verified INT8 units (GUDA-style) that stand in for a GPU's math — so machines without a modern GPU can still do the work. Chain several together and they train one shared model as a cluster. Before you rely on it, see what it can't do → Limitations.

Use the hardware you already have to train. Each machine runs the emulated GPU logic (verified INT8 units — multiply / requantize / ReLU) to compute the model, and DaisyChain pools the machines data-parallel: device selection, capacity-weighted sharding, gradient sync, a P2P setup, and a live dashboard. Two ways to run — Docker or Python.

⚠️ Read this first

DaisyChain-Train is for small models on spare hardware. It pools compute, not memory (the model must fit on one machine), scaling is sublinear, and it is not a substitute for a real GPU on large models. Full envelope in docs/LIMITS.md — please read it before relying on it.

Feature list

Python cluster trainer (`daisychain/`)

Data-parallel training across mixed machines — each node trains its own shard; gradients combine into the exact full-batch gradient, replicas stay bit-identical.
Capacity-weighted sharding — faster machines automatically take a bigger share of the batch.
Emulated GPU compute (verified INT8 units) — VerifiedLinear layers run every forward multiply / requantize / ReLU through the bundled trained units; cluster-wide unit-invocation counts printed by rank 0.
Bring your own model — any Task (build_model / sample / loss) via DAISY_TASK; template in examples/my_task_template.py.
Plain-float alternative task — same cluster and pooling with ordinary float math.
Live dashboard (daisychain-dashboard) — readiness banner, P2P connectivity scan, pooled cores/RAM, per-node capacity plan, live loss.
SpikeWhale control panel (spikewhale_panel, localhost:8899) — sliders for model size / training settings, any HF dataset you can access (default streamed FineWeb-Edu), start/stop/re-adjust, live loss.
Docker demo cluster — 3 nodes + dashboard in one command.
Windows helper (scripts\setup.bat) and Tailscale mesh guide.

DaisyChain-Web (`web/`) — browser P2P training

Zero-install nodes — opening the page IS joining; devices on one network auto-group (Snapdrop-style, by public IP).
Private cross-network rooms — ?room=CODE with host approval for every join.
Full WebRTC mesh — gradients travel peer-to-peer; the server only signals and serves static files, it never sees weights or gradients.
Leader-follower runs — whoever presses Start sets width / sequence / batch-per-device / steps / learning rate for the whole group; config broadcast on the wire.
Mid-run join — late devices are synced in (weights + step) and contribute from the next step.
Bit-identical replicas — same seeded init, strict roster-order gradient averaging, deterministic Adam with identical state on every peer; verified live by per-step weight hashes.
Sync guard — any weight-hash mismatch stops the run instead of training past a fork; the step roster forbids silent partial averages.
Gradient repair — a follower missing a roster gradient re-requests it from the leader (8 steps retained), bit-exact, and the run continues.
Cross-device kernel probe — every step, every device re-hashes a fixed seeded int8 GEMM through its live kernel; catches broken arithmetic that weight hashes cannot see.
Hardcoded FineWeb-Edu streaming — the server reads random slices of the 10BT parquet shards straight off the HF CDN via HTTP range requests (pure-JS hyparquet); built-in corpus fallback offline.
Checkpoints — download .pt, upload → broadcast to the whole group; validated (magic, dims, tokenizer vocab) before accepting.
Inference kit — one self-contained HTML file with the trained weights baked in; generations offline, anywhere.
In-page generation — prompt box on the trained model.
Old-hardware tier — no WebGPU? The identical units run on CPU (same bits, so CPU and GPU devices co-train in one group). There is no plain-float path.
Large-message fragmentation — multi-MB gradients/checkpoints chunked at 48 KB over the data channels.

Verified compute & kernels (web)

Verified INT8 units everywhere — block-scaled int8 GEMM: exact LUT products, exact int32 accumulation, bit-exact f32 epilogue with a pinned rounding schedule; scales derived in JS f64 (division never runs on GPU).
Backends, best-first — DP4A hardware int8 dot → LUT compute shader → CPU mirror; every kernel exact-gated at init (bit-level compares) and demoted to the mirror on any mismatch.
Continuous random-cell audit at live training shapes.
Fused attention kernels — gather/scatter head-strided q·kᵀ and a·v straight from BT×C layout (CUTLASS ex. 36/52 style).
QKV dual-GEMM fusion — shared left operand quantized once, one batch-3 dispatch (ex. 45); bit-identical.
B2B MLP chain — both MLP GEMMs back-to-back on GPU with fused per-row absmax reduction and on-device quantize (ex. 13 + 23); WGSL-exact respec with a fround-stepped JS mirror; fma-contraction-immune by construction.
Dispatch-optimized backward — overlapped independent GEMMs, batch-3 sibling fusions (ex. 05/24); bit-identical gradients; optional int8 STE backward path (dormant, 1.21× vs float).

Verification stack (web)

Exact init gates on every kernel, every device, every boot — including gates that "gate the gate" with discriminating boundary inputs.
IEEE-754 binary32 oracle in exact BigInt arithmetic — proves the JS epilogue mirror is spec-correct (rejects the old mirror on 34% of inputs).
Metamorphic property suite — reference-free relations + definitional absolutes; 4/4 on an externally-authored bug corpus, matching the differential gate.
RDNA2 ISA audit hardenings — bit-level (−0-aware) gate comparisons; proof that FMA contraction cannot change the quantize.
Eleven-suite test chain (cd web && npm test) — convergence, replicas, oracle, gates, properties, external corpus, self-corpus (the instruments scored against my own bugs), B2B, optimizer, transformer LM, int8 backward; results in web/TEST_RESULTS.md.
Dirty-buffer gate — the pool is poisoned before a re-sweep so state bugs (a kernel assuming zeroed memory) are caught deterministically rather than by ordering luck.

Documentation

Python: QUICKSTART, LIMITS, CUSTOM_TASK, TAILSCALE.
Web: Getting started, Architecture, Verification, Troubleshooting.

Quick start

Docker (most reliable — one command)

docker compose -f docker/docker-compose.yml up --build
# open http://localhost:8080

Brings up a 3-node demo cluster + dashboard on one machine.

Python (real machines)

On every machine (pip install -e .):

export MASTER_ADDR=100.101.102.10   # coordinator IP (Tailscale 100.x recommended)
export MASTER_PORT=29560
export WORLD_SIZE=3
export RANK=0                        # 1, 2, ... on the others
export GLOO_SOCKET_IFNAME=tailscale0 # your mesh / LAN NIC
daisychain-train

Windows helper

scripts\setup.bat

An interactive menu: Docker, Python node, or just install deps.

Full walkthrough: docs/QUICKSTART.md.

🐋 SpikeWhale control panel (sliders → real training)

python -m daisychain.spikewhale_panel
# open http://localhost:8899

A web control panel: pick model size / training settings with sliders, choose any HuggingFace dataset you have access to (default: streamed FineWeb-Edu), hit Start, and watch the live loss. Stop and re-adjust any time with ← Back to settings. Launches the real DaisyChain training underneath.

🌐 DaisyChain-Web (train by opening a browser tab)

cd web && npm install && node server.js
# open http://localhost:8787 on every device

Zero-install browser training: devices on the same network auto-group (Snapdrop-style) and train a shared model peer-to-peer over WebRTC, computing through the same verified INT8 units (WebGPU, with the identical units on CPU for machines without it — there is no plain-float path). Private cross-network rooms via ?room=CODE with host approval — the room creator accepts each device before it can join. Includes gradient averaging with a deterministic Adam optimizer (identical state on every peer, nothing extra over the wire), checkpoint download (.pt) and upload → broadcast so one device can restore the whole group after a failure.

Live demo: https://huggingface.co/spaces/Quazim0t0/DaisyChain-Web

Recent updates (July 2026) — DaisyChain-Web

Verification stack — the browser trainer's correctness is now checked by things that run, not argued (full results):

IEEE-754 oracle (web/test_ieee.js): a binary32 oracle built from the standard in exact BigInt arithmetic proves the JS epilogue mirror is spec-correct — and rejects the old round-once mirror on 34% of inputs.
Metamorphic properties + oracle mutation scoring (test_metamorphic.js, test_corpus.js): properties needing no reference implementation, scored against an externally-authored bug taxonomy — 4/4, matching the exact differential gate's 4/4. Relations own the loop bugs; two definitional absolutes (ReLU output range, a unit-scale integer anchor) own the value bugs no relation can see.
Exact kernel gates on every live kernel, a continuous audit at live shapes, and a cross-device kernel probe (same seeded int8 GEMM, same hash on every honest device, any backend). The audit's sampling was rebuilt against a named bug class: its old constants (6 cells, 2% of GEMMs) bounded an overhead that had never been measured (auditing every GEMM costs <0.01% of a step), and uniform random cells cannot see a last-row/column bug at a 16512-wide output. Sampling is now stratified — the first cells are the structural danger points, chosen deliberately. On a last-column bug, same cell budget: uniform caught 5/300 audits, stratified 300/300, with zero false positives.
RDNA2 ISA audit: reading a real GPU's shader ISA against our determinism assumptions confirmed three of them on silicon (exact packed int8 dot; correctly-rounded f32 add/mul; 1-ULP reciprocal — division stays off the GPU) and produced two hardenings. (1) Real ISAs have non-IEEE variants that flush −0 to +0; JS !== can't see that (-0 !== 0 is false), so all gates and audits now compare bit patterns — exactly what the replica hash sees. (2) FMA contraction of the quantize's x·inv + 0.5 (one rounding instead of two) turned out to be floor-invisible by construction — proven in test_b2b.js with 175k+ last-ulp anomalies at binade edges, zero surviving floor(). Rounding mode and denorm flushing are runtime driver state on real hardware, which is why every device re-runs the exact gates at every init.

Training data — FineWeb-Edu (10BT sample) is the hardcoded dataset. The Space reads random slices of the parquet shards straight off the HF CDN with range requests (pure-JS hyparquet, SNAPPY) and serves plain text at /data — no dependency on the datasets-server rows API and its 503s.

Resilience — the sync guard now repairs instead of halting: a roster gradient that reached the leader but not some follower (asymmetric WebRTC mesh) is re-requested from the leader, bit-exact, and the run continues. The guard still stops anything that would fork the weights.

CUTLASS-style kernel work, each step proven bit-identical or exact-gated:

Dispatch-optimized backward (ex. 05/24): independent GEMMs overlapped, sibling trios fused into batch-3 dispatches — bit-identical gradients, dormant int8-backward path down from 1.63× to 1.21× vs float.
QKV dual-GEMM fusion (ex. 45): q/k/v share one left operand — quantized once, one batched dispatch, zero changed bits.
B2B MLP chain (ex. 13 + 23): both MLP GEMMs back-to-back on the GPU with a fused per-row absmax reduction; the intermediate is quantized on-device via a WGSL-exact respec (floor(f32(x·invScale)+0.5) — no GPU division) whose fround-stepped JS mirror keeps mixed GPU/CPU fleets bit-identical.

Profile-driven speed work — every change below is bit-identical (gradient and loss hashes unchanged), so none of it trades correctness for wall clock:

Buffer pooling: GPU buffers are recycled by size bucket instead of being created and destroyed per dispatch (~19 per MLP call, per layer, per step). 6–10% faster, every hash unchanged.
Shared-operand embedding GEMMs: profiling put two f32 backward GEMMs at 55% of the entire step, and both consumed the same dlogits operand — ~17 MB at the 16512-token vocab, uploaded twice. One upload, one encoder, one submit: that pair went 205 → 90 ms and the step 12% faster. The fusion is gated bit-for-bit against the two calls it replaces.
A negative result, kept on purpose: the remaining hot kernel looked cache-hostile (adjacent lanes wrote 66 KB apart), but making the writes contiguous changed nothing. Two probes explain why — holding the output at 17 MB while cutting compute 32× barely moved the time. The logits GEMM is transfer-bound, not compute-bound, and the readback cannot be removed because softmax must stay in JS (WGSL's exp is not correctly rounded, and a per-vendor exp would fork replicas). The real lever there is the vocabulary, not the kernel.
An init backend race was tried and removed: it tied on the shipped path, cost ~430 ms of init, and made the backend vary between page loads, which silently invalidated three A/B comparisons before it was caught. A knob that changes what you are measuring is worse than a fixed choice.

Dirty-buffer gate — and the assumption it falsified. Pooling introduced a bug class the gates predate: a pooled buffer is not zero-initialized, so a kernel that assumes zeros is right on step one and wrong on step two. That is a state bug, where no single call is wrong and the sequence is, which is the family no oracle can reach. The assumption was that the gates were blind to it. Mutation-testing the gate proved otherwise: deleting the zeroing made the plain gate fail at its second shape, because the sweep's own shapes recycle each other's buffers. The suite had incidental coverage nobody designed, which is coverage nobody can rely on — shorten the shape list and it evaporates with the gate still green. It is now deliberate: the pool is poisoned with 1e4-magnitude residue before a re-sweep, so detection no longer depends on ordering luck. ~90 ms one-time.

Scoring the oracles against my OWN bugs (web/test_selfcorpus.js) — the external corpus measures kernel bugs someone else wrote down, so this suite asks the harder question: what do the instruments score against the four real bugs of the month? Properties 0/2 on the data-plane pair (the c·out theorem again), differential 2/2 — but half the bugs were not in the kernels at all. A dead gate is a bug in a checker, caught only by mutating the gate; a stalled roster gradient is a bug in the protocol, where every computed value on every peer was correct, so no data oracle could fire. Those needed different instruments, not better oracles.

All eleven test suites (cd web && npm test) pass; results with methodology in web/TEST_RESULTS.md.

How it works

Each machine runs the same command; they form a cluster and train one shared model. Two things happen:

The compute runs through the emulated GPU logic. By default the model is built from VerifiedLinear layers, so every forward multiply / requantize / ReLU is done by the bundled verified INT8 units (daisychain/verified/) — the emulated GPU math. Rank 0 prints cluster-wide unit-invocation counts so you can see the emulated logic doing the work.
The machines are pooled data-parallel. Each node trains on its own shard; gradients are capacity-weighted and combined into the exact full-batch gradient, so replicas stay bit-identical. Faster machines automatically take a bigger share.

  old machine A ─┐
  old machine B ─┼─►  each runs the emulated GPU logic on its shard  ─►  one model
  old machine C ─┘        (gradients combined across the cluster)

Bring your own model

DaisyChain-Train trains any Task (build_model / sample / loss). Copy examples/my_task_template.py, set DAISY_TASK=your_module:YourTask. Use VerifiedLinear (see daisychain/verified_task.py) to run your model's compute through the emulated units. See docs/CUSTOM_TASK.md.

Plain-float alternative

To skip the emulated units and train with normal float math on each machine, set DAISY_TASK=daisychain.example_task:ExampleTask. Same cluster, same pooling — the model math just runs as ordinary float instead of through the verified units.

The dashboard

daisychain-dashboard (or the Docker service) serves a Tailwind page at :8080 — readiness banner, P2P connectivity scan, pooled cores/RAM + capacity plan (per-node device, weight, batch), and live training loss.

Networking

Use Tailscale for a P2P mesh so machines on different networks get stable IPs on one interface — docs/TAILSCALE.md.

Layout

daisychain/cluster.py        capacity-weighted CPU/GPU data-parallel trainer
daisychain/train.py          entry point (daisychain-train)
daisychain/verified/         bundled trained N/N units + VerifiedLinear (train through them)
daisychain/verified_task.py  default task: forward runs on the verified units
daisychain/example_task.py   plain-float alternative task
daisychain/task.py           the Task interface + loader
daisychain/dashboard/        agent + P2P scanner + Tailwind server
docker/                      Dockerfile, dashboard image, compose (demo cluster)
scripts/setup.bat / setup.sh interactive setup helpers
config/                      nodes + cluster env examples
examples/my_task_template.py starting point for your own model
docs/                        QUICKSTART, LIMITS, CUSTOM_TASK, TAILSCALE
daisychain/spikewhale_task.py   trains the real SpikeWhale on streamed HF datasets
daisychain/spikewhale_panel.py  slider control panel (localhost:8899)
web/                         DaisyChain-Web: P2P browser training (WebRTC + WebGPU)
export_luts_web.py           regenerates web/public LUTs from the trained units

Install

pip install torch numpy psutil
pip install -e .          # exposes: daisychain-train, daisychain-agent, daisychain-dashboard

Requires Python ≥ 3.9, PyTorch ≥ 2.0. Multi-node is reliable on Linux/macOS; on Windows use Docker/WSL (see Limitations).

Citation

@misc{byrne2026daisychain,
  title        = {DaisyChain-Train: An Old Hardware Training Pipeline},
  author       = {Byrne, Dean (Quazim0t0)},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/DaisyChainAI/DaisyChain-Train}},
  note         = {Chain spare/old machines into a data-parallel training cluster}
}

Dean Byrne (Quazim0t0) · 2026

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

DaisyChainAI
/

DaisyChain-Train

🌼 DaisyChain-Train — Old Hardware Training Pipeline

⚠️ Read this first

Feature list

Python cluster trainer (`daisychain/`)

DaisyChain-Web (`web/`) — browser P2P training

Verified compute & kernels (web)

Verification stack (web)

Documentation

Quick start

Docker (most reliable — one command)

Python (real machines)

Windows helper

🐋 SpikeWhale control panel (sliders → real training)

🌐 DaisyChain-Web (train by opening a browser tab)

Recent updates (July 2026) — DaisyChain-Web

How it works

Bring your own model

Plain-float alternative

The dashboard

Networking

Layout

Install

Links

Citation

🌼 DaisyChain-Train — Old Hardware Training Pipeline

⚠️ Read this first

Feature list

Python cluster trainer (daisychain/)

DaisyChain-Web (web/) — browser P2P training

Verified compute & kernels (web)

Verification stack (web)

Documentation

Quick start

Docker (most reliable — one command)

Python (real machines)

Windows helper

🐋 SpikeWhale control panel (sliders → real training)

🌐 DaisyChain-Web (train by opening a browser tab)

Recent updates (July 2026) — DaisyChain-Web

How it works

Bring your own model

Plain-float alternative

The dashboard

Networking

Layout

Install

Links

Citation

Python cluster trainer (`daisychain/`)

DaisyChain-Web (`web/`) — browser P2P training