Wire and Addressing

Chapter 10 closed with the engine emitting EngineStep::SendEnvelope out of its poll cycle and accepting IngressEvent::EnvelopeFrom back in. This chapter walks the bytes that live between those two seams. Every cross-Node edge in a compiled graph rides as one WireEnvelope. Every peer destination is a multiaddr. Every fill in an envelope carries its own per-slot suffix the receiver parses to dispatch without any subscription table.

The wire surface is narrow. One canonical proto envelope. One codec. Four routable multiaddr segment kinds. One DSL method on Graph for the producer side. One AddressBook mutation API on Node for the host. The compiler’s partition_by_wire_ops pass cuts each graph at the wire boundary, and a sister synthesize_wire_recvs pass materializes the receiver-side wire.Recv NodeProto so user code never writes one.

Addresses are multiaddrs

The Address type holds an ordered sequence of typed Protocol segments. Only four variants exist, the four the framework actually routes on:

// from bytesandbrains/bb-runtime/src/framework/address_book.rs
pub enum Protocol {
    /// Peer identity (BB `PeerId`).
    P2p(PeerId),
    /// Data-plane slot fill target - the receiver writes the payload
    /// to the slot at this `NodeSiteId` and pushes downstream
    /// consumers.
    Site(NodeSiteId),
    /// Control-plane component identity. Combined with an `Op`
    /// segment, identifies the receiver's `dispatch_atomic` target.
    Component(ComponentRef),
    /// Op name for control-plane dispatch (e.g. `"FindNode"`). The
    /// receiver calls `component[cref].dispatch_atomic(op_name,
    /// payload, ctx)`.
    Op(String),
}

/p2p/ uses libp2p’s standard varint code 421 so a peer-id minted by libp2p round-trips through this codec byte-for-byte. The other three segments use framework-internal codes in the unassigned 0xE0 range. Transport-layer multiaddr segments like /ip4/, /tcp/, and /quic/ are deliberately not recognized. Address::from_bytes rejects unknown codes with AddressError::UnknownCode so a libp2p-style multiaddr that an adapter dropped into the wrong place cannot reach engine routing.

Construct an Address through the typed builder chain. The methods are by-value so each call returns a fresh Address with one more segment.

// from bytesandbrains/bb-runtime/src/framework/address_book.rs
pub fn empty() -> Self {
    Self::default()
}

pub fn p2p(self, peer: PeerId) -> Self { ... }
pub fn site(self, site: NodeSiteId) -> Self { ... }
pub fn component(self, c: ComponentRef) -> Self { ... }
pub fn op(self, name: impl Into<String>) -> Self { ... }

The string form mirrors libp2p’s /protocol/value/protocol/value/... layout. A data-plane suffix looks like /site/17. A control-plane suffix looks like /component/7/op/FindNode. The /p2p/ segment uses the same base58btc multihash encoding libp2p does. Address::parse_str parses the string form, Display renders it back.

// from bytesandbrains/bb-runtime/src/framework/address_book.rs
pub fn to_bytes(&self) -> Vec<u8> { ... }
pub fn from_bytes(mut bytes: &[u8]) -> Result<Self, AddressError> { ... }

The Multiaddress alias exported from bb-runtime::framework is the same type. Custom ops that declare a port carrying an Address use Multiaddress so user-visible types read naturally without introducing a second name for the same thing.

The WireEnvelope proto

Every byte that crosses a transport adapter rides as one WireEnvelope. The proto definition lives in proto/bb_core.proto and prost generates the Rust struct that bb-ir re-exports.

// from bytesandbrains/proto/bb_core.proto
message WireEnvelope {
  // Ordered destination address list. The framework's wire syscall
  // populates this from the `AddressBook` at dispatch time.
  repeated bytes dest_peer_addresses = 1;

  // One or more slot fills delivered atomically.
  repeated SlotFill fills = 2;

  // Request/response correlation. Control planes MAY use it.
  WireCorrelation correlation = 3;

  // Dapper-style deadline propagation.
  uint64 remaining_deadline_ns = 4;

  // Reverse-path RTT piggyback.
  repeated EdgeRttReport edge_rtt_reports = 5;

  // Multihash bytes of the originating peer.
  bytes src_peer_bytes = 6;

  // WireEnvelope schema version.
  uint32 schema_version = 7;

  // Sender-claimed local-address bag. Snapshot of the sender's
  // AddressBook entry for its own PeerId at envelope-mint time. The
  // receiver merges into its own AddressBook entry for `src_peer` so
  // future replies can dial back on any reachable interface. Empty
  // stamps zero entries and the receiver leaves its existing entry
  // untouched.
  repeated bytes src_peer_addresses = 8;
}

The envelope carries an ordered list of peer addresses, not a single peer id. The wire syscall resolves the destination through the framework’s AddressBook at dispatch time, snapshots every address the book has for that peer, and stamps the snapshot into dest_peer_addresses. The host transport adapter picks one entry by its own capabilities. An adapter that speaks QUIC over IPv6 picks a different entry than one that speaks libp2p over relay. The framework does not care which.

Each fill carries its own per-slot multiaddr suffix. Receivers never consult a routing table. They parse the suffix and dispatch.

// from bytesandbrains/proto/bb_core.proto
message SlotFill {
  // Per-slot multiaddr suffix encoded via the canonical Address
  // binary form. Two shapes per ADDRESSING.md:
  //   `/site/{site}`                — data-plane slot fill
  //   `/component/{cref}/op/{name}` — control-plane component dispatch
  bytes dest_suffix = 1;

  // Wire-encoded bytes; empty when `trigger_only=true`.
  bytes payload = 2;

  // True when the consumer only reads the firing signal, not the value.
  bool trigger_only = 3;

  // Per-fill type-hash discriminator.
  uint64 type_hash = 4;
}

The correlation header pairs requests with responses. The framework allocates a fresh wire_req_id per outbound wire.Send so the inbound response side can look up the in-flight cohort. Control-plane components may use the same field for their own request/response pairing without inventing a separate token system.

// from bytesandbrains/proto/bb_core.proto
enum CorrelationKind {
  NONE = 0;
  REQUEST = 1;
  RESPONSE = 2;
}

message WireCorrelation {
  CorrelationKind kind = 1;
  uint64 wire_req_id = 2;
}

Encoding through EnvelopeCodec

EnvelopeCodec is the single codec entry on the wire. It is a stateless thin wrapper around prost. The buffer total-length cap fires before prost allocates anything, so an adversarial sender cannot pre-balloon memory by advertising a huge frame; the per-fill, per-suffix, and src-peer-address caps fire after prost parses the proto, before the framework propagates the values downstream.

// from bytesandbrains/bb-runtime/src/envelope.rs
pub struct EnvelopeCodec;

impl EnvelopeCodec {
    pub fn encode(env: &WireEnvelope) -> Vec<u8> {
        env.encode_to_vec()
    }

    pub fn decode_capped(
        bytes: &[u8],
        caps: &EnvelopeCaps,
    ) -> Result<WireEnvelope, EnvelopeDecodeError> {
        // 1. buffer total-length cap, pre-prost
        // 2. prost decode
        // 3. schema-version validation
        // 4. fills-count cap
        // 5. per-fill payload + dest_suffix caps
        // 6. src_peer_addresses count + per-entry size caps
        ...
    }
}

Callers stamp WireEnvelope.schema_version on push into the outbound queue rather than inside encode, so encoding does not clone the (potentially large) payload bytes just to set one u32.

The EnvelopeCaps struct holds six limits. The defaults track the “16 MiB / 256 / 4 MiB / 4 KiB / 8 / 256 B” recommendation from the architecture spec. A tighter EnvelopeCaps::edge() preset is available for embedded deployments.

// from bytesandbrains/bb-runtime/src/envelope.rs
pub struct EnvelopeCaps {
    pub max_total_bytes: usize,             // default 16 MiB
    pub max_slot_fills: usize,              // default 256
    pub max_per_fill_bytes: usize,          // default 4 MiB
    pub max_dest_suffix_bytes: usize,       // default 4 KiB
    pub max_src_peer_addresses: usize,      // default 8
    pub max_src_peer_address_bytes: usize,  // default 256
}

EnvelopeDecodeError discriminates malformed-prost, version mismatch, oversize envelope, oversize fill, too-many-fills, too-many-src-peer-addresses, and oversize-src-peer-address as typed variants so transport adapters can log + react per failure mode. The caps live on NodeConfig.envelope_caps. The host overrides the defaults via NodeConfig::with_envelope_caps(caps) on the config builder supplied to bb::install. The edge preset is EnvelopeCaps::edge(); arbitrary caps come from an EnvelopeCaps { ... } struct literal followed by the same with_envelope_caps builder method. The engine’s inbound path reads the resulting caps on every envelope.

To swap in the edge preset, build the NodeConfig once and feed it into the install bag:

use bytesandbrains::envelope::EnvelopeCaps;
use bytesandbrains::node::NodeConfig;

let config = NodeConfig::new(peer_id).with_envelope_caps(EnvelopeCaps::edge());

The Config bag forwarded into bb::install carries this same NodeConfig when a non-default node-wide setting is in play.

The producer side: g.net_out

The DSL exposes one method for the cross-Node boundary. The Module’s body calls g.net_out(name, peers, value). The compiler partitions the graph at this call and emits the receiver-side wire.Recv NodeProto on every consumer-side partition that reads the named port.

// from bytesandbrains/bb-dsl/src/graph.rs
/// Emit a `wire.Send` NodeProto and register `name` as a
/// network-typed output port. `value` is shipped to every peer
/// in `peers` (which must be a `Vec<PeerId>` at dispatch time).
/// The compiler's `partition_by_wire_ops` cuts the graph at this
/// boundary; `synthesize_wire_recvs` materializes the matching
/// `wire.Recv` on every consumer-side partition that reads the
/// named port.
pub fn net_out(&mut self, name: &str, peers: Output, value: Output) {
    let value_type = value.type_node;
    let port_name = name.to_string();
    let handle_name = self.next_site_name();
    ...
    self.push_node(NodeProto {
        op_type: bb_ir::syscall_ids::OP_WIRE_SEND.into(),
        domain: bb_ir::syscall_ids::WIRE_DOMAIN.into(),
        input: vec![value.name.clone(), peers.name],
        output: vec![port_name.clone(), handle_name.clone()],
        ..Default::default()
    });
    ...
}

The Module’s body calls net_out for every value that needs to leave this peer. The peers argument is a Vec<PeerId> produced earlier in the graph: from a PeerSelector sample, from a constant the bootstrap recorded, or from a Module input the host invokes the Module with. The value is whatever this Module computed.

The federated learning example records both sides of a round in two Modules. The server samples K clients and ships the current params:

// from bytesandbrains/examples/federated_learning.rs
struct ServerLogic;
impl Module for ServerLogic {
    fn name(&self) -> &str {
        "ServerLogic"
    }
    fn body(&self, g: &mut Graph) {
        // K random peers from the GlobalRegistryServer's announced set.
        let peers = PeerSelectorSlot::default().sample(g, 2);
        // Current global params shipped across the network.
        let current_params = ModelSlot.params(g);
        g.net_out("server_params", peers, current_params);
    }
}

The client receives the server’s params (the compiler’s synth-recv pass materializes the matching Recv on this partition), runs a local step, and ships the updated params back:

// from bytesandbrains/examples/federated_learning.rs
struct ClientLogic;
impl Module for ClientLogic {
    fn name(&self) -> &str {
        "ClientLogic"
    }
    fn body(&self, g: &mut Graph) {
        let server_params = g.input("server_params");
        let _ = ModelSlot.load_parameters(g, server_params);

        let (batch, _labels) = DataLoaderSlot.next_batch(g);
        let _prediction = ModelSlot.forward(g, batch);

        let updated_params = ModelSlot.params(g);
        let server_peer = g.input("server_peer");
        g.net_out("updated_params", server_peer, updated_params);
    }
}

The Module body never touches transport, never encodes bytes, never constructs an envelope. The compiler’s wire passes do all of that between Compiler::compile and bb::install.

Bundling multiple values into one port

net_out ships one typed value per envelope. When a producer needs to atomically attach metadata to a payload (a FedAvg contribution paired with the contributing peer’s sample count, a hierarchical aggregator’s gradient plus the child layer’s reduction metadata), the pattern is: bundle the values into one composite Output via g.bundle, ship the composite through a single net_out, and unbundle on the receiver via g.unbundle. Single-port DAG semantics hold because the bundle/unbundle pair traverses one Output between peers, so the compiler’s synthesize_wire_recvs keeps its single-port cross-partition resolution.

// from bytesandbrains/bb-dsl/src/graph.rs
pub fn bundle(&mut self, parts: &[Output]) -> Output;
pub fn unbundle(
    &mut self,
    composite: Output,
    part_types: &[&'static TypeNode],
) -> Vec<Output>;

The recorded ops belong to ai.bytesandbrains.composite (Bundle + Unbundle). Bundle’s NodeProto has variable-arity input (one entry per child) and a single composite output port; the matching Unbundle reads that composite, validates the declared child count against the runtime envelope, and re-emits each child as the concrete SlotValue carrier the sender bundled. The ValueInfoProto.denotation on each child output carries the declared TypeNode. Downstream consumers downcast directly (as_any().downcast_ref::<T>()) against that denotation rather than running bincode against it; the wire.Recv contract uses the same decoder registry so the two surfaces stay symmetric.

How partitioning lands per-target

A composed Module containing a net_out call compiles into multiple FunctionProto targets in one ModelProto. multi_target_network makes the structure visible. One FedNetwork records a graph with three leaf Modules and a single net_out between the trainer and the sink. The compiler emits one function per target peer class and inserts a wire.Send into the producer function and a wire.Recv into every consumer function.

// from bytesandbrains/examples/multi_target_network.rs
impl Module for FedNetwork {
    fn name(&self) -> &str {
        "FedNetwork"
    }
    fn body(&self, g: &mut Graph) {
        let sink_peers = g.input("sink_peers");

        // LoaderLeaf (DataLoader.next_batch) → TrainerLeaf
        // (Model.forward) — both local on the source node.
        let loader_outs = self.loader.call().build(g);
        let batch = loader_outs.output("batch");
        let trainer_outs = self.trainer.call().input("batch", batch).build(g);
        let prediction = trainer_outs.output("prediction");

        // Network output: ship the prediction to every sink peer.
        g.net_out("prediction_port", sink_peers, prediction);

        // SinkLeaf consumes the inbound payload + a metadata channel.
        let received = g.lookup_output("prediction_port").expect("net_out port");
        let _ = self
            .sink
            .call()
            .input("contribution", received.clone())
            .input("metadata", received)
            .build(g);
    }
}

After compilation, the compiled ModelProto carries one function per partition. Each function names the target peer class it installs on. The host walks compiled.functions, picks a target name per peer, and installs each one as its own Node:

// from bytesandbrains/examples/multi_target_network.rs
for f in &compiled.functions {
    let sends = f
        .node
        .iter()
        .filter(|n| n.domain == "ai.bytesandbrains.wire" && n.op_type == "Send")
        .count();
    let recvs = f
        .node
        .iter()
        .filter(|n| n.domain == "ai.bytesandbrains.wire" && n.op_type == "Recv")
        .count();
    println!(
        "  target `{}`: {sends} wire.Send, {recvs} wire.Recv",
        f.name
    );
}

The source partition gets the wire.Send op the user wrote. The sink partition gets a synthesized wire.Recv whose output flows into the sink leaf’s contribution input. Neither partition references the other’s compute. The cut is total.

The host driving the loop

The Node struct exposes one inbound and one outbound seam. Outbound is the Vec<EngineStep> the engine returns from poll. Inbound is the IngressQueue the transport pushes envelopes into.

// from bytesandbrains/bb-runtime/src/node/mod.rs
pub fn poll(
    &mut self,
    cx: &mut std::task::Context<'_>,
) -> std::task::Poll<Vec<crate::engine::EngineStep>> { ... }

pub fn deliver_inbound(
    &mut self,
    src_peer: crate::ids::PeerId,
    bytes: &[u8],
) -> Result<(), crate::errors::delivery::DeliveryError> {
    let envelope =
        crate::envelope::EnvelopeCodec::decode_capped(bytes, &self.config.envelope_caps)
            .map_err(|e| {
                crate::errors::delivery::DeliveryError::InvalidEnvelope(e.to_string())
            })?;
    // `Node::deliver_inbound` does not observe a transport-side
    // reflexive address; adapters that can supply one push
    // `IngressEvent::EnvelopeFrom` directly with
    // `src_observed_address: Some(addr)`.
    self.engine
        .ingress
        .push(crate::ingress::IngressEvent::EnvelopeFrom {
            src_peer,
            envelope,
            src_observed_address: None,
        })
        .map_err(|_| crate::errors::delivery::DeliveryError::IngressClosed)
}

deliver_inbound routes through EnvelopeCodec::decode_capped so malformed, schema-mismatched, or oversize buffers fail with DeliveryError::InvalidEnvelope before any prost allocation. The transport adapter supplies src_peer so the engine can consult PeerGovernor::check_inbound before any slot is written. The decoded envelope lands on the ingress queue as IngressEvent::EnvelopeFrom.

// from bytesandbrains/bb-runtime/src/ingress.rs
pub enum IngressEvent {
    EnvelopeFrom {
        src_peer: crate::ids::PeerId,
        envelope: crate::envelope::WireEnvelope,
        // Transport-observed reflexive endpoint (NAT-translated
        // source). `None` when the adapter cannot surface it.
        src_observed_address: Option<crate::framework::Address>,
    },
    AppEvent { ... },
    TimerMatured { ... },
    Invoke { ... },
    Completion { ... },
    CompletionFailed { ... },
    SendFailed { ... },
}

The in-process router and test buses construct EnvelopeFrom via IngressEvent::from_in_process(src_peer, envelope) (bb-runtime/src/ingress.rs), which populates src_observed_address with the sender’s /p2p/<PeerId> address. Production transports populate the field from whatever reflexive address their adapter surfaced (libp2p relay endpoint, post-NAT socket address, etc.).

Two inbound paths for transports

Node::deliver_inbound(src_peer, &bytes) is the raw-bytes path for transports that hand off the on-wire envelope undecoded. The framework runs EnvelopeCodec::decode_capped against NodeConfig::envelope_caps, so a malformed, schema-mismatched, or oversize buffer fails synchronously with DeliveryError::InvalidEnvelope before any prost allocation. The adapter only owes the engine a PeerId and a byte slice; everything else is the framework’s problem.

Adapters that have already decoded the envelope (in-process buses, control-plane fabrics, transports that know their own reflexive endpoint) push IngressEvent::EnvelopeFrom { src_peer, envelope, src_observed_address: Some(addr) } directly onto the ingress queue. The in-process common case has a one-line helper: IngressEvent::from_in_process(src_peer, envelope) stamps the sender’s /p2p/<PeerId> as the observed address so the merge path behaves the same way it would for a real transport. Both paths land on the same downstream PeerGovernor check and the same fills loop; the difference is who owns decode and who owns the reflexive address stamp.

The outbound seam is EngineStep::SendEnvelope. The engine drains the outbound queue at the end of every poll cycle and emits one SendEnvelope per envelope ready to ship.

// from bytesandbrains/bb-runtime/src/engine/step.rs
/// An outbound envelope is ready to ship.
SendEnvelope(WireEnvelope),

A transport adapter’s outbound loop reads each SendEnvelope, picks one of the addresses from envelope.dest_peer_addresses by its own capabilities, encodes the envelope with EnvelopeCodec::encode, and ships the bytes. The example bus that drives multi_target_network does the same shape in-process, pushing the encoded bytes through a HashMap of ingress handles instead of a socket.

Receiver dispatch by suffix shape

Engine::route_envelope walks each inbound envelope’s fills and parses each dest_suffix as an Address. The trailing segment shape is the routing decision. No subscription table is consulted. No opset is keyed. The address itself is the routing.

// from bytesandbrains/bb-runtime/src/engine/poll.rs
fn deliver_fill(
    &mut self,
    fill: crate::envelope::SlotFill,
    fill_index: u32,
    wire_req_id: u64,
    src_peer: Option<crate::ids::PeerId>,
    arrival_ns: u64,
    inbound_remaining_deadline_ns: Option<u64>,
) -> Vec<EngineStep> {
    let addr = match crate::framework::Address::from_bytes(&fill.dest_suffix) {
        Ok(a) => a,
        Err(e) => {
            return vec![self.emit_wire_decode_failure(
                0,
                fill.payload.len(),
                format!("dest_suffix parse: {e}"),
            )];
        }
    };

    // Data-plane suffix: /site/<NodeSiteId>.
    if let Some(site_id) = addr.site_id() {
        return self.deliver_data_plane_fill(
            site_id, fill, fill_index, src_peer, wire_req_id,
            arrival_ns, inbound_remaining_deadline_ns,
        );
    }

    // Control-plane suffix: /component/<cref>/op/<name>.
    if let (Some(cref), Some(op_name)) = (addr.component_ref(), addr.op_name()) {
        let op_name = op_name.to_string();
        return self.deliver_control_plane_fill(cref, op_name, fill, wire_req_id);
    }

    vec![self.emit_wire_decode_failure(
        0,
        fill.payload.len(),
        "address shape neither data-plane nor control-plane".to_string(),
    )]
}

Data-plane delivery materializes the payload bytes into the typed SlotValue carrier the sender stamped on the fill, writes the result into the addressed slot at a fresh ExecId, and pushes that slot’s downstream consumers onto the frontier. NodeSiteId is globally unique within a Node, minted by Engine::allocate_node_site_id from a per-Node u64 counter at graph-installation time, so the same site id never names two slots. The graph’s consumers map gives OpRefs for every op that reads the slot, and the engine pushes those onto the frontier just like any other op that gained an input.

Control-plane delivery borrows the addressed component out of Engine.components and calls component.dispatch_atomic(op_name, [(payload, correlation)], ctx). The framework synthesizes opaque payload and correlation inputs from the fill so the component’s existing dispatch path handles inbound envelopes the same way it handles user-graph DSL ops.

Anything that does not parse as either shape drops silently, with a WireDecodeFailure published on the bus + a WireDecodeFailed EngineStep so the host sees the drop.

Self-addressed envelopes on multi-target Nodes

When a peer hosts more than one target (e.g. &["Client", "Server"]), the wire syscall continues to address destinations by PeerId and the receiver dispatches by dest_suffix exactly as in the single-target case. The receiver’s site-name to NodeSiteId table is global across every installed target’s graph: Engine::install_graph populates one site_names map for every installed graph, so a wire.Send from Client’s partition addressed at self.peer_id() lands at the correct wire.Recv site even when that site lives inside Server’s partition graph. The site id is unique per Node, not per target, so the partition the suffix resolves into is determined entirely by which graph owns the NodeSiteId. No target-awareness is required in the envelope or the dispatch table. The same holds for control-plane /component/<cref>/op/<name> fills: the deduped ComponentRef is shared across every target sharing the slot, so a control-plane fill from Client’s partition reaches the same component instance the Server partition would dispatch to.

Typed wire.Recv delivery

Engine::deliver_data_plane_fill resolves the typed SlotValue carrier BEFORE allocating an ExecId or mutating the slot table. Trigger fills bypass the registry and install TriggerValue directly. Non-trigger fills go through decode_typed_fill (bb-runtime/src/engine/poll.rs:996-1083), which:

1. Validates the destination slot’s compile-time wire-type assertion via GraphSlot::recv_wire_type_hash.
2. Resolves the destination binding via GraphSlot::recv_site_to_slot_id plus Engine::slot_id_to_role_ref to discover whether the slot binds a Backend role.
3. Pre-charges fill.payload.len() against Engine::ingress_byte_budget. Overflow surfaces WireReceiveError::BudgetExceeded and drops the fill.
4. Branches:
   • Backend-mediated path when the slot binds a Backend role. materialize_via_backend mem::takes fill.payload (already framework-owned from envelope decode) and hands the Vec<u8> to Backend::materialize_from_wire by value. The bytes never re-cross the framework boundary. The backend chooses adoption strategy: zero-copy via ArrayD::from_shape_vec when alignment permits, pool-pulled buffer with copy in, or fresh-allocate. The engine wraps the result in a BackendTensorCarrier and stamps the engine-side accounting fields.
   • Framework-carrier path otherwise. The global wire_decoder_registry() decoder runs on &fill.payload and returns a typed Box<dyn SlotValue>.
5. Writes the materialized SlotValue to the slot. Downstream consumers fire. Slot overwrite or eviction calls SlotValue::charged_bytes() and releases the count via Engine::release (bb-runtime/src/engine/core.rs:554-).

There is no BytesValue fallback path. A sender that wants to ship raw bytes stamps a BytesValue on the fill; the decoder registry round-trips it to BytesValue on the receiver. Anything else lands as the typed carrier the sender stamped. Downstream consumers downcast via as_any().downcast_ref::<T>(); the previous “decode bytes via bincode against the graph’s TypeNode” hop is gone.

Failure modes

Six per-fill failures surface as InfraEvent::WireReceiveError on the bus and matching EngineStep::WireReceiveFailed so the host poll() caller observes the per-fill failure without bus subscription:

• TypeMismatch. The destination slot carries a compile-time expected type_hash via GraphSlot::recv_wire_type_hash and the inbound fill’s type_hash does not match. Checked before the decoder lookup so a mis-typed payload never reaches the decoder. Variant carries expected_hash: u64.
• UnknownTypeHash. No decoder is registered for the stamped hash. The sender shipped a value whose concrete type is unknown to this Node’s inventory (version skew, or a fuzzed envelope). The slot stays empty; the bus event surfaces the drop.
• DecodeFailed. The registered decoder ran and returned Err. Variant carries error_summary: String (typically a bincode error stringified) so subscribers can attribute the drop without re-parsing the bytes.
• AllocationFailed. A try_reserve_exact reservation somewhere on the per-fill ingress path returned TryReserveError, or a caller-side cap rejected before allocating (bb-runtime/src/bus.rs:233-241). Variant carries byte_count: usize plus reason: AllocFailReason::{HeapExhausted, PerItemCapExceeded { cap }}.
• BudgetExceeded. The pre-decode try_charge against Engine::ingress_byte_budget overflowed. The fill drops before any decoder runs. Variant carries byte_count: usize and budget_remaining: usize.
• BackendMaterializeFailed. The bound backend’s materialize_from_wire returned Err, or the engine could not borrow the backend component. Variant carries backend_ref: ComponentRef and backend_error_summary: String. The byte charge releases before the event publishes.

Each event carries src_peer: Option<PeerId>, fill_index: u32 (0-based position within the envelope), actual_hash: u64, and payload_size: usize. The sub-kind discriminator lets subscribers route on the top-level topic ("WireReceiveError") and match on the cause from the variant fields. WireReceiveError is distinct from WireDecodeFailure (which fires on envelope-level header / dest_suffix parsing); the receive event covers the typed-decode step that runs after the envelope has parsed.

Partial delivery

Engine::deliver_envelope iterates fills sequentially. A per-fill failure emits its WireReceiveError event and continues to the next fill. Other fills in the same envelope still deliver. The fill_index field on every failure event identifies which fill failed inside a multi-fill envelope so subscribers can attribute the drop precisely.

Destination metadata

GraphSlot::recv_wire_type_hash: HashMap<NodeSiteId, u64> holds the expected type_hash per wire.Recv payload site. Populated at install time alongside recv_sender_sites. Slots without an entry are treated as dynamic / Any: the registry lookup proceeds without the mismatch check. The compiler does not yet stamp ValueInfoProto.type_node on Recv payload outputs with a hash that matches the producer-side SlotValue::type_hash() derivation, so the map stays empty at install time in production. The TypeMismatch check is dormant until the compiler follow-up lands; tests populate the map manually to exercise the path.

The AddressBook

The framework owns one AddressBook per Node. Every overlay protocol, transport adapter, and application shares it. No Component duplicates address bytes in its own state.

Entries are ref-counted. The same peer can be claimed by multiple overlay protocols without duplicating address records. add_peer increments the count, drop_peer decrements it, and the entry is removed when the count hits zero.

// from bytesandbrains/bb-runtime/src/framework/address_book.rs
pub fn add_peer(
    &mut self,
    peer: PeerId,
    addresses: Vec<Address>,
) -> Result<(), AddressBookError> {
    if addresses.is_empty() {
        return Err(AddressBookError::EmptyAddressList);
    }
    match self.entries.get_mut(&peer) {
        Some(entry) => {
            entry.ref_count = entry.ref_count.saturating_add(1);
            for addr in addresses {
                if !entry.addresses.contains(&addr) {
                    entry.addresses.push(addr);
                }
            }
        }
        None => {
            if self.entries.len() >= self.cap {
                return Err(AddressBookError::Full { cap: self.cap });
            }
            ...
        }
    }
    Ok(())
}

The mutation surface is four methods:

// from bytesandbrains/bb-runtime/src/framework/address_book.rs
pub fn add_peer(&mut self, peer: PeerId, addresses: Vec<Address>)
    -> Result<(), AddressBookError>;
pub fn drop_peer(&mut self, peer: PeerId)
    -> Result<(), AddressBookError>;
pub fn register_address(&mut self, peer: PeerId, address: Address)
    -> Result<(), AddressBookError>;
pub fn forget_address(&mut self, peer: PeerId, address: &Address)
    -> Result<(), AddressBookError>;

add_peer rejects an empty Vec<Address> and rejects a new peer when the book is at its cap. drop_peer decrements the ref count. register_address appends one address, idempotent on duplicates. forget_address prunes one address; the entry stays even if pruning leaves the address list empty.

The wire syscall consults AddressBook::lookup on every outbound wire.Send. The lookup returns the entire ordered slice (the ordering is insertion order, peer-stated preference) so the host transport sees every option at once.

// from bytesandbrains/bb-runtime/src/framework/address_book.rs
pub fn lookup(&self, peer: PeerId) -> Option<&[Address]> {
    let entry = self.entries.get(&peer)?;
    if entry.addresses.is_empty() {
        None
    } else {
        Some(entry.addresses.as_slice())
    }
}

pub fn lookup_first(&self, peer: PeerId) -> Option<&Address> {
    self.lookup(peer).and_then(|addrs| addrs.first())
}

lookup returns None for both an unknown peer and a known peer whose address list is empty. Both are treated as “can’t route” by the wire syscall.

The Node struct exposes the same surface through pass-through methods so the host can manage peers without reaching into the engine. The federated learning example uses both during bootstrap and at install time:

// from bytesandbrains/examples/multi_target_network.rs
node.add_peer(
    PeerId::from(PEER_SOURCE),
    vec![Address::empty().p2p(PeerId::from(PEER_SOURCE))],
)?;
node.add_peer(
    PeerId::from(PEER_SINK),
    vec![Address::empty().p2p(PeerId::from(PEER_SINK))],
)?;

A graph can also mutate the book from inside a recorded program. The ai.bytesandbrains.address_book opset exposes three custom ops that let discovery protocols compile into a graph that announces and resolves peers without out-of-band host calls. Insert(peer, address) adds a single multiaddr (used by the engine’s transport-observed merge). InsertMany(peer, addresses) records a full address bag in one NodeProto. Lookup(peer) returns the full ordered slice on the new TYPE_ADDRESS_VEC carrier; callers needing a single entry pick one at the call site. The federated learning example’s bootstrap records a Constant for the server’s peer id, a Constant for the server’s address vec, and an InsertMany op that merges them into the book before the first poll.

// from bytesandbrains/bb-dsl/src/syscalls.rs
pub fn address_book_insert_many(
    g: &mut Graph,
    peer: Output,
    addresses: Output,
) -> Output;

pub fn address_book_lookup(g: &mut Graph, peer: Output) -> Output;

Local-address binding and src_peer_addresses

A Node binds to Vec<Address> at install, not a single address. bb::install(peer_id, addresses, model, targets, config) (bytesandbrains/src/install.rs) registers every address against peer_id in the engine’s AddressBook. Empty vec skips the self-registration step and surfaces downstream “no addresses” errors at the protocol level rather than silently synthesizing a /p2p/<PeerId> placeholder.

The addresses bag is shared across every entry in the targets slice. The install path keys the AddressBook on the Node’s own self_peer, so every target the install path registers sees the same local_addresses() view. A peer hosting both halves of a federated round shares one identity across both partitions; there is no per-target PeerId and no per-target address bag.

// from bytesandbrains/bb-runtime/src/node/mod.rs
pub fn local_addresses(&self) -> &[Address];
pub fn add_local_address(&mut self, addr: Address)
    -> Result<(), AddressBookError>;
pub fn forget_local_address(&mut self, addr: &Address)
    -> Result<(), AddressBookError>;

Each method is a thin wrapper around the AddressBook entry keyed by the Node’s own PeerId. Every dispatch context exposes ctx.local_addresses() (bytesandbrains/bb-runtime/src/runtime.rs) so wire ops and identity-bearing protocols read through one accessor; mutations via node.add_local_address propagate without rebinding.

The wire syscall snapshots ctx.local_addresses() once per Send and stamps the bag onto every envelope minted in the fan-out (bytesandbrains/bb-ops/src/network/wire/mod.rs):

let src_peer_addresses: Vec<Vec<u8>> =
    ctx.local_addresses().iter().map(|a| a.to_bytes()).collect();
// ... build envelope ...
env.src_peer_addresses = src_peer_addresses.clone();

Phase 1 of Engine::poll merges both advertised address bags into the receiver’s AddressBook (bytesandbrains/bb-runtime/src/engine/poll.rs). Sender-claimed envelope.src_peer_addresses merges first; the transport-observed src_observed_address from IngressEvent::EnvelopeFrom merges next so the observed address wins for the NAT-translated case the sender’s snapshot cannot know. Both paths use skip-on-unchanged guards: slice equality elides the rewrite when the claimed bag matches; containment-check elides when the observed address is already present. Steady-state traffic costs at most one AddressBook write per real change.

The receiver’s EnvelopeCaps bounds the pre-allocation an adversarial sender can trigger (bytesandbrains/bb-runtime/src/envelope.rs):

pub struct EnvelopeCaps {
    pub max_total_bytes: usize,                // default 16 MiB
    pub max_slot_fills: usize,                 // default 256
    pub max_per_fill_bytes: usize,             // default 4 MiB
    pub max_dest_suffix_bytes: usize,          // default 4 KiB
    pub max_src_peer_addresses: usize,         // default 8
    pub max_src_peer_address_bytes: usize,     // default 256
}

EnvelopeCodec::decode_capped checks the src_peer_addresses caps after prost parses the envelope but before the framework consumes the entries (bytesandbrains/bb-runtime/src/envelope.rs); the buffer total-length cap is the one check that fires pre-prost.

Peer resolution failure

The wire syscall cannot resolve a destination in three cases. The peer input did not carry a parseable PeerId. The PeerId is unknown to the AddressBook. The PeerId’s entry exists but its address list is empty. In each case the syscall produces no envelope, records the failure on the framework’s pending_peer_resolve_failures, publishes InfraEvent::PeerResolveFailure on the bus, and surfaces the failure on the next poll as EngineStep::PeerResolveFailed.

// from bytesandbrains/bb-runtime/src/engine/step.rs
/// `wire::Send` could not resolve its destination peer's
/// addresses against the framework's
/// [`crate::framework::AddressBook`]. Either the peer is
/// unknown, its address list is empty, or the Send op's `peer`
/// input didn't carry a valid `PeerId`. The Send op produces
/// no envelope; the host application reacts via this event.
PeerResolveFailed {
    /// The peer whose addresses could not be resolved. `None`
    /// when the Send op had no parseable `peer` input.
    peer: Option<PeerId>,
    /// The Send op that failed to resolve.
    op_ref: OpRef,
    /// Execution this Send belonged to.
    exec_id: ExecId,
},

This is a first-class lifecycle event in the same family as PeerBlocked, PeerDown, and PeerUp. It is not an OpFailed. Telemetry can surface resolution failures alongside other peer-lifecycle signals without conflating them with op-level errors.

Encoding and decoding the envelope

The codec round-trip is short enough to exercise from a unit test. The framework’s own envelope tests illustrate the full shape: build an envelope with a destination address list and a fill carrying a data-plane suffix, encode it, decode it, and verify the suffix parses back into the expected Site segment.

// from bytesandbrains/bb-runtime/src/envelope_tests.rs
fn sample_envelope() -> WireEnvelope {
    let dest_suffix = Address::empty().site(NodeSiteId::from(7u64)).to_bytes();
    let dest_addr_bytes = Address::empty().p2p(PeerId::from(42u64)).to_bytes();
    WireEnvelope {
        dest_peer_addresses: vec![dest_addr_bytes],
        fills: vec![SlotFill {
            dest_suffix,
            payload: b"hello".to_vec(),
            trigger_only: false,
            ..Default::default()
        }],
        correlation: Some(WireCorrelation {
            kind: CorrelationKind::None as i32,
            wire_req_id: 0,
        }),
        remaining_deadline_ns: 0,
        edge_rtt_reports: Vec::new(),
        ..Default::default()
    }
}

#[test]
fn envelope_codec_roundtrips_bytes() {
    let env = sample_envelope();
    let bytes = EnvelopeCodec::encode(&env);
    let back = EnvelopeCodec::decode_capped(&bytes, &EnvelopeCaps::default()).expect("decode");

    assert_eq!(back.dest_peer_addresses.len(), 1);
    let dest = Address::from_bytes(&back.dest_peer_addresses[0]).expect("address");
    assert_eq!(dest.peer_id(), Some(PeerId::from(42u64)));

    assert_eq!(back.fills.len(), 1);
    let suffix_addr = Address::from_bytes(&back.fills[0].dest_suffix).expect("suffix");
    assert_eq!(suffix_addr.site_id(), Some(NodeSiteId::from(7u64)));
    assert_eq!(back.fills[0].payload, b"hello");
    assert!(!back.fills[0].trigger_only);
}

The control-plane shape is a single segment swap. A fill targeting a component dispatch carries a suffix like /component/7/op/FindNode. The receiver’s deliver_control_plane_fill borrows component ref 7 out of the engine and calls dispatch_atomic("FindNode", ...).

// from bytesandbrains/bb-runtime/src/envelope_tests.rs
#[test]
fn envelope_codec_decodes_control_plane_suffix() {
    let suffix = Address::empty()
        .component(ComponentRef::from(7u32))
        .op("FindNode")
        .to_bytes();
    let dest_addr_bytes = Address::empty().p2p(PeerId::from(99u64)).to_bytes();
    let env = WireEnvelope {
        dest_peer_addresses: vec![dest_addr_bytes],
        fills: vec![SlotFill {
            dest_suffix: suffix,
            payload: b"query".to_vec(),
            trigger_only: false,
            ..Default::default()
        }],
        correlation: None,
        remaining_deadline_ns: 0,
        edge_rtt_reports: Vec::new(),
        ..Default::default()
    };
    let bytes = EnvelopeCodec::encode(&env);
    let back = EnvelopeCodec::decode_capped(&bytes, &EnvelopeCaps::default()).expect("decode");
    let addr = Address::from_bytes(&back.fills[0].dest_suffix).expect("suffix");
    assert_eq!(addr.component_ref(), Some(ComponentRef::from(7u32)));
    assert_eq!(addr.op_name(), Some("FindNode"));
}

Two fills with different suffixes can ride in the same envelope. A multi-fill envelope is the batched shape the compiler’s analyze_wire_edges pass produces when several edges from the same producer Send target the same receiver in the same cycle. One TCP send, one parse, multiple slot writes.

Wire-eligibility

The framework’s SlotValue blanket carries the wire-eligibility contract. Any type that satisfies Any + Send + Sync + Clone + Serialize + DeserializeOwned is a SlotValue and ships over the wire. The encoder is bincode::serialize(self); the producer also stamps SlotFill.type_hash from SlotValue::type_hash() so the receiver can route the bytes back to the original carrier without coordinating with the sender. Carrier authors register the lattice binding and the wire decoder with one register_type_node!(MyValue, &TYPE_X) line.

On the receive side, the engine reads SlotFill.type_hash off the wire and looks the carrier up in wire_decoder_registry keyed on that 64-bit FNV-1a digest. The same hash that the sender stamped from SlotValue::type_hash() is the routing key that recovers the original carrier without a coordination round-trip. The receive site then asserts the recovered carrier against the slot’s compile-time GraphSlot::recv_wire_type_hash (the destination metadata table covered earlier in the typed wire.Recv section) and surfaces a WireReceiveError if the two disagree.

The decoder registry that the wire.Recv path consults is the same one the CompositeValue Bundle/Unbundle codec uses, so a single registration participates in both surfaces.

Adding a new wire-traversable user type is two lines: derive serde, register the type node.

#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct ModelDelta {
    pub round: u32,
    pub weights: Vec<f32>,
}

The Module body emits g.net_out("delta", peers, model_delta). The compiler stamps the destination suffix on the Send NodeProto’s metadata. The runtime resolves the peer through the AddressBook, encodes the value via bincode, packs one SlotFill, builds one WireEnvelope, and emits one SendEnvelope step.

Type-node registration

The blanket SlotValue impl gives every serde-derived carrier a type_hash (FNV-1a 64 of std::any::type_name::<T>()) and a default runtime_type() that resolves through a process-wide TypeId → &'static TypeNode map. That map is the framework’s type-indexed decoder table: the registry the engine and the compiler’s TypeSolver consult to recover a value’s lattice node from its concrete Rust type. It is built once at startup from inventory-submitted RuntimeTypeBinding entries.

// from bytesandbrains/bb-ir/src/slot_value.rs
pub struct RuntimeTypeBinding {
    pub type_id_fn: fn() -> TypeId,
    pub type_node: &'static TypeNode,
}

inventory::collect!(RuntimeTypeBinding);

pub fn runtime_type_registry() -> &'static HashMap<TypeId, &'static TypeNode> {
    static REG: OnceLock<HashMap<TypeId, &'static TypeNode>> = OnceLock::new();
    REG.get_or_init(|| {
        let mut m: HashMap<TypeId, &'static TypeNode> = HashMap::new();
        for binding in inventory::iter::<RuntimeTypeBinding> {
            m.insert((binding.type_id_fn)(), binding.type_node);
        }
        m
    })
}

The registration surface is one declarative macro. Each call expands to a single inventory::submit! of a RuntimeTypeBinding paired with the lattice node the type resolves to.

// from bytesandbrains/bb-ir/src/slot_value.rs
#[macro_export]
macro_rules! register_type_node {
    ($t:ty, $node:expr) => {
        $crate::inventory::submit! {
            $crate::slot_value::RuntimeTypeBinding {
                type_id_fn: || ::std::any::TypeId::of::<$t>(),
                type_node: $node,
            }
        }
    };
}

Authors of wire-eligible carriers add one register_type_node! line per type. The framework’s own typed carriers register the same way; the full set lives next to the carrier definitions:

// from bytesandbrains/bb-runtime/src/syscall/values.rs
register_type_node!(TriggerValue, &TYPE_TRIGGER);
register_type_node!(PeerIdValue, &TYPE_PEER_ID);
register_type_node!(PeerIdVecValue, &TYPE_PEER_ID_VEC);
register_type_node!(WireReqIdValue, &TYPE_WIRE_REQ_ID);
register_type_node!(BytesValue, &TYPE_BYTES);
register_type_node!(AddressValue, &TYPE_MULTIADDRESS);
register_type_node!(AddressVecValue, &TYPE_ADDRESS_VEC);
register_type_node!(CompositeValue, &TYPE_COMPOSITE);

For a user-shipped carrier, the canonical use sites import the macro from the bb-ir companion crate. Add it alongside bytesandbrains in Cargo.toml: bb-ir = "0.3". The bytesandbrains facade does not re-export the macro by name (Rust re-exports of #[macro_export] macros are a separate path), so authors of user-defined wire-eligible carriers depend on the bb-ir crate directly.

Note: register_type_node! lives in the bb-ir companion crate, not in the bytesandbrains facade. Add it explicitly with cargo add bb-ir and import it as use bb_ir::register_type_node;. The facade re-exports the registered carrier but does not re-export the registration macro.

use bb_ir::register_type_node;
use bb_ir::types::TYPE_TENSOR_F32;

register_type_node!(ModelDelta, &TYPE_TENSOR_F32);

The same registration pattern that ships TriggerValue and PeerIdValue is what bb-ops/src/backends/cpu/tensor.rs uses to bind CpuTensor to &TYPE_TENSOR_F32, so the engine recovers a tensor’s lattice node from its TypeId on snapshot replay and at the type-solver’s app-event entry points.

Registration is optional. SlotValue::runtime_type falls back to TYPE_ANY when the concrete type has no binding in the registry. The framework keeps a CommandIdValue carrier intentionally unregistered to exercise that fallback path; the unregistered_value_runtime_type_defaults_to_any test asserts the behaviour in bytesandbrains/bb-runtime/src/syscall/values_tests.rs. Unregistered types still ship over the wire (their type_hash is still computed from type_name) but the compiler’s TypeSolver cannot narrow their lattice node beyond Any at app-event entry points.

Where this lives

The canonical sources in the bytesandbrains repo:

• bytesandbrains/proto/bb_core.proto: the WireEnvelope, SlotFill, WireCorrelation, CorrelationKind proto definitions.
• bytesandbrains/bb-runtime/src/envelope.rs: EnvelopeCodec, EnvelopeCaps, EnvelopeDecodeError, ENVELOPE_SCHEMA_VERSION, and the bounded-decode entry.
• bytesandbrains/bb-runtime/src/envelope_tests.rs: encode + decode round-trip tests for data-plane and control-plane suffix shapes.
• bytesandbrains/bb-runtime/src/framework/address_book.rs: Protocol, Address, Multiaddress, AddressError, AddressBook, AddressBookError, DEFAULT_ADDRESS_BOOK_CAP, the four mutation methods, and lookup / lookup_first.
• bytesandbrains/bb-runtime/src/engine/poll.rs: route_envelope and deliver_fill, the inbound dispatch entry points; the outbound queue drain that emits EngineStep::SendEnvelope.
• bytesandbrains/bb-runtime/src/engine/step.rs: the EngineStep::SendEnvelope and EngineStep::PeerResolveFailed variants the host observes.
• bytesandbrains/bb-runtime/src/ingress.rs: IngressEvent::EnvelopeFrom, the inbound seam transport adapters push into.
• bytesandbrains/bb-runtime/src/node/mod.rs: Node::poll, Node::deliver_inbound, Node::add_peer, Node::drop_peer, Node::local_addresses, Node::add_local_address, Node::forget_local_address, Node::ingress_handle.
• bytesandbrains/bb-ir/src/slot_value.rs: the universal SlotValue trait, RuntimeTypeBinding, runtime_type_registry, and the register_type_node! macro that drives the type-indexed registry.
• bytesandbrains/bb-dsl/src/graph.rs: Graph::net_out, the single DSL surface for the producer side.
• bytesandbrains/examples/federated_learning.rs, bytesandbrains/examples/multi_target_network.rs: end-to-end examples that exercise net_out, the compiler’s wire-partition passes, and the host-side bus that ferries envelopes between Nodes.
• bytesandbrains/docs/WIRE.md, bytesandbrains/docs/ADDRESSING.md: the bb-private architecture specs.