#Architecture overview (ByteOr OSS)

ByteOr is an OSS pipeline/runtime substrate. The workspace is intentionally split so the deterministic spec model, hot-loop kernel, SHM transport, and executor layers can evolve with auditable boundaries.

#Layering

• byteor-abi
  • C-facing repr(C) layouts and the generated spec/byteor_v1.h header.
  • This is the ABI source of truth for exported ByteOr layouts.
• byteor-pipeline-spec
  • The spec contract crate.
  • Owns the v1 data model, validation, kv encode/decode, canonicalization, and human-readable describe helpers.
• byteor-pipeline-plan
  • Model-neutral authoring surface.
  • Owns plan validation, execution target selection, and author-intent structures shared by OSS, WASM, and Cloud.
• byteor-pipeline-lower
  • Explicit lowering from PipelinePlanV2 into PipelineSpecV1::{SingleRing,LaneGraph}.
  • Reports plan diagnostics separately from lowering diagnostics.
• byteor-pipeline-kernel
  • Hot-path lane traits, barriers, sequencing, and wait primitives.
  • Contains no product policy.
• byteor-transport-shm
  • File-backed shared-memory transport/open-create-validate layer.
• byteor-pipeline-backings-mem
  • In-memory backings for tests, deterministic examples, and non-SHM paths.
• byteor-pipeline-backings-shm
  • SHM-backed lane implementations and attach/create helpers.
• byteor-pipeline-exec
  • In-process executors and runtime bring-up helpers.
  • Validates specs before running and stays free of product policy.
• byteor-pipeline
  • Facade crate that re-exports the main OSS authoring/runtime surface.
• byteor-pipeline-wasm
  • Browser-facing bindings around spec authoring/validation and format conversion.
• xtask
  • Workspace automation for CI gates, header generation, perf, fuzz, soak, and policy checks.

#Two v1 spec shapes

ByteOr v1 intentionally has two different authoring/runtime shapes:

• PipelineSpecV1::SingleRing
  • Explicit stage DAG over one sequenced ring.
  • Used for bring-up, bounded runs, sharded per-key execution, and microbench-style paths.
• PipelineSpecV1::LaneGraph
  • Whole-plant graph over named lanes, endpoints, and roles.
  • Used when the topology needs fanout, merge, bridges, explicit endpoint wiring, or role-based MP execution.

The supported OSS authoring path is plan then explicit lowering. Callers that already know the exact canonical shape may still author specs directly with byteor-pipeline-spec builders.

#Data and control flow

At a high level, ByteOr follows this sequence:

1. Author a PipelinePlanV2 or construct a PipelineSpecV1 directly when appropriate
2. Validate the plan with byteor-pipeline-plan::validate_plan_v2 when using the planning path
3. Lower explicitly with byteor-pipeline-lower
4. Validate the resulting spec with byteor-pipeline-spec::validate_v1
5. Attach or create the required backings
6. Run an executor from byteor-pipeline-exec
7. Observe runtime health through snapshots/counters exposed by the backing/executor layer

This separation matters because the OSS workspace treats the spec model as the stable contract and treats convenience helpers as secondary.

#IndexBus dependency

Today the ByteOr OSS workspace still expects an indexbus checkout at ../indexbus.

• Several crates and xtask use sibling path dependencies.
• The generated ByteOr header includes indexbus_v1.h.
• CI checks out both workspaces side-by-side to satisfy those dependencies.

This dependency is part of the current initial-commit reality and should be treated as a workspace assumption until the path dependencies are removed.