Architecture

Architecture

ProxAI is a small local compatibility proxy. It accepts local OpenAI-compatible or Anthropic-style requests, normalizes protocol-specific request shapes, forwards them to a configured upstream provider, and translates upstream responses back to the client-facing protocol when needed.

Keep it focused

ProxAI should stay focused: local compatibility proxy, explicit provider/protocol routing, compact diagnostics, and minimal surprises. It is not intended to grow into a generic multi-tenant AI gateway unless explicitly requested.

Related docs

Request lifecycleSource-level path from inbound request through provider transport to outbound response.Module boundariesWhere parsing, routing, translation, provider behavior, HTTP carriers, errors, and observation belong.Config flowHow config.toml, examples, CLI overrides, validation, providers, and runtime state fit together.Error flowHow internal errors become compact client-facing HTTP and SSE errors.ConfigurationRuntime settings, routes, providers, capture, logging, and errors.Protocol overviewPhase axis, protocol axis, and supported runtime paths.Protocol conversionPair-oriented conversion rules and protocol alignment expectations.

Two-axis model

The codebase is easiest to understand as two independent axes.

Phase axis

The phase axis describes where data is in the proxy pipeline:

• inbound_request — the original client request received by ProxAI
• provider_request — the request ProxAI prepares for the upstream provider
• upstream_response — the response returned by the upstream provider
• outbound_response — the response ProxAI returns to the client

Protocol axis

The protocol axis describes the wire protocol used at a given phase:

• openai_responses
• openai_chat_completions
• anthropic_messages

Each phase has its own protocol:

• inbound_request.protocol is what the client sent.
• provider_request.protocol is what ProxAI sends upstream, controlled by the selected provider.
• upstream_response.protocol is what the provider returns.
• outbound_response.protocol is what ProxAI returns to the client.

Provider names are user labels. They are not semantic protocol identifiers.

Top-level source layout

• Directorysrc/

  • main.rs — entry point; delegates to cli::main
  • lib.rs — AppState, axum Router, proxy handler
  • Directorycli/ — CLI parsing and startup

    • …
  • config.rs — config.toml schema and loading
  • paths.rs — app directory resolution
  • request.rs — shared request carrier types
  • sse.rs — SSE parsing and encoding helpers
  • sse_translation.rs — application adapter between SSE bytes and structured translation events
  • formatting.rs — formatting helpers
  • Directoryerror/ — domain errors and rendering

    • …
  • Directoryhttp_support/ — HTTP carrier helpers

    • …
  • Directoryprovider/ — provider request projection/serialization, response observation, and HTTP transport

    • …
  • Directoryupstream/ — upstream response reading

    • …
  • Directorypipeline/ — typed proxy pipeline, including HTTP ingress adaptation

    • …
  • Directoryobserve/ — capture, logging, diagnostics

    • …
  • Directorymcp/ — MCP control surface

    • …
• Directorycrates/

  • Directoryproxai-core/

    • Directorysrc/

      • Directoryingress/ — carrier-independent inbound normalization and validation

        • …
      • Directorypipeline/ — carrier-independent composition across ingress, routing, translation, and provider adaptation

        • …
      • Directoryprotocol/ — protocol wire types and protocol enums

        • …
      • Directoryrouting/ — compiled protocol/model routing and provider-label selection

        • …
      • Directoryprovider/ — carrier-independent provider request preparation and response/error normalization

        • …
      • Directorytranslation/ — carrier-independent cross-protocol conversion

        • …

The application keeps HTTP ingress, provider projections/serialization, transport lookup, and SSE carrier adaptation in src/. proxai-core::pipeline composes structured ingress, provider-label routing, provider request/response adaptation, and translation into a reusable Value/StreamEvent façade.

Request lifecycle

src/lib.rs registers these routes and sends all of them to the same proxy handler:

/v1/responses          /responses
/v1/chat/completions   /chat/completions
/v1/messages           /messages

The simplified inbound path is:

let prepared_provider = inbound_http
    .parse_inbound()?                    // app: path + JSON bytes
    .prepare_provider_request(           // core: ingress + route + translate + adapt
        &state.core_pipeline,
        &state.providers,
    )?;

run_provider_flow(prepared_provider).await

run_provider_flow then handles the provider side:

let provider_http = prepared_provider
    .send_to_upstream().await?           // provider/transport + upstream
    .handle_upstream_response().await?;  // upstream: read body / stream

provider_http.translate_to_outbound().await?  // translation + http_support

Pipeline stages

1. 1
   inbound_request

   Modules

   src/pipeline/inbound.rsproxai-core/pipeline/proxai-core/ingress/

   Responsibility

   Read body bytes, detect protocol, and parse JSON in the app; normalize and validate the structured request through the core pipeline.

2. 2
   Routing

   Modules

   pipeline/provider_request.rsproxai-core/pipeline/proxai-core/routing/

   Responsibility

   Resolve the provider label, protocol, compatibility policy, and upstream model in core, then select the application transport.

3. 3
   provider_request

   Modules

   pipeline/provider_request.rsproxai-core/pipeline/proxai-core/translation/requestproxai-core/provider/requestprovider/request

   Responsibility

   Return a prepared provider value and matching response pipeline from core, then project diagnostics and serialize the body in the application.

4. 4
   Send upstream

   Modules

   pipeline/provider_request.rsprovider/transport

   Responsibility

   Build auth headers, construct upstream URL, send via reqwest.

5. 5
   upstream_response

   Modules

   pipeline/upstream_response.rsupstream/

   Responsibility

   Read status, headers, and body or stream.

6. 6
   outbound_response

   Modules

   pipeline/provider_response.rssse_translation.rsproxai-core/pipeline/proxai-core/translation/responseproxai-core/translation/stream

   Responsibility

   Run structured response normalization and translation through ResponsePipeline, adapt events to/from SSE bytes, observe failures, and rebuild HTTP.

pipeline/ uses a typed ProxyFlow<S> state machine. Each phase consumes one flow state and returns the next one, keeping the phase order explicit.

Module responsibility map

proxai-core/protocol/

Core crate protocol wire shapes and shared protocol enums. Models JSON only; no conversion and no HTTP carrier types.

src/pipeline/inbound.rs

HTTP path detection, JSON byte parsing, and carrier error mapping before structured core preparation.

proxai-core/ingress/

Carrier-independent structured request normalization, validation, model extraction, and typed ingress observations.

proxai-core/pipeline/

Reusable structured façade that composes ingress, routing, request preparation, response normalization, and translation.

proxai-core/observe/

Shared `Observer` contract, closed `Observation` variants, and no-op default; no concrete sinks.

proxai-core/routing/

Carrier-independent provider-label selection from request protocol, model pattern, defaults, and route configuration.

proxai-core/provider/

Carrier-independent provider request value preparation, response compatibility policy, structured response/event and error-payload normalization, and typed adaptation observations.

proxai-core/translation/

Core crate request, response, and structured-event conversion across explicit protocol pairs. Exposes the `Translator` façade and emits typed observations through the shared `Observer`.

provider/

Provider projection/summary extraction, request serialization, response observation, auth headers, upstream URL construction, and transport.

upstream/

Reading upstream status, headers, complete bodies, or streaming byte carriers.

sse_translation.rs

Application carrier adapter: parses SSE bytes into `StreamTranslationInput`, invokes core `ResponsePipeline`, retains raw failure context, and encodes output back to SSE.

http_support/

Protocol-neutral HTTP helpers such as content-type checks, response reconstruction, and boxed byte streams.

observe/

Capture artifacts, structured logging, request hints, and privacy-preserving diagnostics.

error/

Domain-specific error types and client-facing response rendering.

Boundary rules

• proxai-core/protocol/ is low-level wire modeling: JSON shapes only, no conversion.
• proxai-core/ingress/ accepts structured JSON values and owns protocol normalization and validation; it never receives HTTP body bytes.
• src/pipeline/inbound.rs is the application adapter for path detection and JSON byte parsing.
• proxai-core/pipeline/ composes the complete structured request/response path and binds the request-scoped core observer without owning transport.
• src/pipeline/ coordinates HTTP carriers and transport around that core façade while keeping phase order explicit.
• proxai-core/provider/ owns carrier-independent provider request value preparation, response compatibility policy, structured response/event and error-payload normalization, and typed adaptation observations.
• proxai-core/translation/ is carrier-independent: Translator accepts JSON Values or structured StreamEvents and never receives HTTP responses, bodies, SSE bytes, or provider-private structs.
• sse_translation.rs is the application adapter between upstream byte streams and core provider/translation APIs. It owns raw triggering frames and SSE output encoding.
• provider/ owns application-side provider projections/summaries, request serialization, response observation, and transport details such as auth headers, upstream URLs, and idle-read timeout.
• proxai-core/observe/ defines the shared Observer contract and closed Observation enum. Core ingress, provider normalization, and translation emit typed variants; src/observe/ supplies the downstream implementation and owns logging, diagnostics, and capture.
• observe/ cuts across the pipeline for diagnostics, but does not make routing or protocol decisions.
• Semantic stream and HTTP errors should use domain errors instead of being hidden inside std::io::Error.

Dependency direction

flowchart TD
  cli[cli/] --> lib[lib.rs AppState]
  config[config.rs] --> lib
  paths[paths.rs] --> cli
  lib --> pipeline[pipeline/]

  pipeline --> core_pipeline[proxai-core/pipeline/]
  core_pipeline --> core_ingress[proxai-core/ingress/]
  core_pipeline --> core_provider[proxai-core/provider/]
  core_pipeline --> routing[proxai-core/routing/]
  pipeline --> provider[provider/]
  pipeline --> upstream[upstream/]
  pipeline --> sse_translation[sse_translation.rs]
  core_pipeline --> translation[proxai-core/translation/]
  pipeline --> observe[observe/]
  pipeline --> http_support[http_support/]
  pipeline --> error[error/]

  core_ingress --> protocol[proxai-core/protocol/]
  core_provider --> protocol
  translation --> protocol
  provider --> protocol
  upstream --> http_support
  provider --> http_support
  sse_translation --> http_support
  sse_translation --> sse[sse.rs]
  sse_translation --> core_pipeline
  upstream --> sse
  observe --> error
  lib --> observe
  lib --> error

Key rules:

• proxai-core/protocol/ is low-level wire modeling.
• proxai-core/ingress/ is the structured value boundary for inbound normalization and validation; src/pipeline/inbound.rs adapts HTTP bytes and application diagnostics.
• proxai-core/provider/ is the structured provider-response compatibility boundary.
• proxai-core/pipeline/ is the carrier-independent composition root; src/pipeline/ coordinates the HTTP lifecycle and maps the selected provider label to transport.
• proxai-core/translation/ depends only on protocol models, structured values/events, and the core observation contract; it does not depend on HTTP, SSE bytes, provider transport, or concrete application observation.
• sse_translation.rs adapts byte carriers to the core ResponsePipeline; the app pipeline owns diagnostics and client-facing error rendering.
• observe/ implements the core observation trait but does not make protocol or routing decisions.

Translation selection

Translator selects translation from two protocol values:

• inbound request_protocol, detected by pipeline/inbound.rs
• provider protocol, configured on the selected provider

Rules:

• Same protocol: pass through without protocol conversion.
• Different protocols: dispatch to crates/proxai-core/src/translation/<inbound_protocol>/to_<provider_protocol>/.
• Unsupported pairs fail explicitly.

Translator is the single public façade. It directly owns the selected route and observation hook. Each entrypoint derives a phase-bound TranslationScope: request, non-streaming response, or streaming response. The scope is explicit and operation-local—never global, thread-local, or task-local—and is passed only into semantic projection paths that may emit adapted or dropped; pure conversions and outbound constructors remain scope-free. Pair code therefore calls observation methods without separately supplying a phase, so an operation cannot accidentally report telemetry under the wrong pipeline phase. Value translation borrows its derived scope; structured stream translation consumes Translator, moves a streaming-response scope into the returned stream, and lazily creates a private closed PairStreamingState containing exactly one of the six cross-protocol state machines or the identity path. This avoids allocating stream state for value-only translation and prevents pair state from being reused across streams. TranslationRoute and TranslationScope definitions live below the façade in translation/context.rs; top-level dispatch and pair code receive only &TranslationScope, so dispatch, phase, and observation cannot disagree about the active operation. Protocol-neutral event, error, terminal, identity, and lifecycle types live in dependency-free translation/stream.rs. Pair implementations depend only on context.rs and stream.rs, while Translator depends on those foundations plus pair states, so every dependency edge remains one-directional. Carrier adapters call Translator::translate_stream rather than driving pair events or terminal handling directly.

Data type conventions

• Prefer top-level enums keyed by protocol for protocol-specific request/response data.
• Avoid parallel protocol / payload / projection / summary fields that can drift into impossible states.
• Keep core streaming APIs as StreamEvent in/out; use ByteStream only in the application carrier layer.
• Keep provider names separate from protocol names.