Memory
# Docuclear System Overview Three repositories, one system. All architecture docs are in place and validated. ## Repositories | Repo | Purpose | Architecture Doc | |------|---------|------------------| | **docuclear-sphere** | Per-customer Worker (W4P) | `ARCHITECTURE.md` — bundle structure, bindings contract (13 bindings), health endpoint, migrations, versioning | | **docuclear-sphere-deployment** | Platform (provisioning + dispatch) | `Deploy_ARCHITECTURE.md` — dispatch Worker, admin Worker, CustomerAgent DO, 4 workflows, provisioning/deprovisioning flows, binding injection (16 bindings), live-verification (7 tests), test suite (31 tests) | | **docuclear-ingestion** | Document extraction service | `ARCHITECTURE.md` — xberg/crawlberg containers, API endpoints, extraction flow, auth, metering, deferred features (redaction, premium OCR) | ## Cross-Repo Contract **INGESTION_CONTRACT.md** — API Contract v2 between sphere and ingestion service: - Endpoints: `/extract`, `/crawl`, `/jobs/:id`, `/health`, `/admin/metering` - Auth: Bearer token + X-Sphere-Slug + X-Plan-Tier - Idempotency: SHA-256 of bytes or URL+day - Metering: Analytics Engine writes per job - Hash: `9569652AA4A1E3815985B257274E3CA510C6EC95F83F3A32D55C4CA73CA21518` ## Request Flow ``` User → Dispatch Worker (sphere.docuclear.ai/{slug}/...) ↓ Sphere Worker (per-customer, W4P) ↓ Ingestion Service (docuclear-ingestion.mike-c79.workers.dev) ↓ xberg container (text extraction) / crawlberg container (web crawl) ``` ## Provisioning Flow 1. Platform creates resources (D1, KV, Vectorize, R2) via workflow 2. Platform deploys sphere Worker with injected bindings from manifest 3. Platform polls `GET /{slug}/__health` until version matches 4. CustomerAgent DO handles steady-state health checks (3-min alarms) ## Status (Sep 2, 2026) - **docuclear-sphere**: Merged to main, 1502/1502 tests pass - **docuclear-sphere-deployment**: Live, 31/31 tests pass, two test customers active - **docuclear-ingestion**: Live, 21/21 tests pass, clean extraction (no redaction yet) - **Documentation**: All three ARCHITECTURE.md files + INGESTION_CONTRACT.md complete ## Next Steps 1. Onboard first real customer (provision → deploy → verify) 2. Implement redaction (Worker-side regex + Workers AI NER, separate build) 3. Implement semantic classification (ai-airlock-style, sphere-side, Workers AI)
[Update Sep 3, 2026]: [Decision, Sep 3, 2026, ~10:46am BST — Mike with Perplexity]: SPHERE/INGESTION SEPARATION CONFIRMED — the two repos are two kinds of thing: sphere = per-customer trust domain (W4P worker, provisioned per-customer); ingestion = shared service processing untrusted input (raw bytes, crawled web) on containers, forbidden per-customer by the ingestion economics decision (72cafb0f). Merging would either drag containers into every customer sphere or force the sphere to platform cadence — both break the model. Independence is the point: tier-2 (Runware VL) and tier-3 (wrestler) extraction already multiply behind INGESTION_CONTRACT without touching the sphere. The old generation had ingestion inside the sphere (the Docuclear_Sphere connector still shows it: ingest_document/scrape_urls/Curator); the v2 split was deliberate. FRAMING: the workspace holds the handles, not the machinery. CONSEQUENCE (Mike): TWO GATEKEEPERS — generalised to a principle: every versioned contract gets a gatekeeper; the gatekeeper family grows with the contract family (gatekeeper-sphere, gatekeeper-ingestion, web-search gatekeeper, inbound MCP facade…). gatekeeper-ingestion inherits the existing skeleton (Bearer + X-Sphere-Slug + X-Plan-Tier + Analytics Engine metering) and adds policy/approvals/audit. DESIGN REQUIREMENT for acf97a13: gatekeeper-sphere is the REFERENCE IMPLEMENTATION of a repeating pattern, not a one-off — shape it so gatekeeper-ingestion is a configuration, not a rewrite.
[Update Sep 3, 2026]: [Ruling, Sep 3, 2026, ~10:59am BST — Mike]: CONTRACT SCOPE — SPHERE_GATEKEEPER_CONTRACT documents the SPHERE surface only; the ingestion service's own endpoints are documented in INGESTION_CONTRACT and reached via gatekeeper-ingestion (one aperture per contract). Classification for each sphere REST endpoint: (a) memory/data-plane operations → IN the sphere contract; (b) sphere-level ingest-into-memory operations (the sphere orchestrates ingestion internally; results land in the customer's sphere) → IN — they are sphere operations, not ingestion passthroughs; (c) RAW ingestion-service passthroughs (direct /extract, /crawl, /admin/*) → OUT of the sphere contract — reachable via gatekeeper-ingestion. /admin/* is NEVER workspace-exposed. Test: if removing the ingestion service would change the endpoint's behaviour, it's a passthrough (out); if it merely changes how the sphere fulfils it internally, it's a sphere operation (in).
Tags: kind:semantic, status:canonical — Source: mcp — 2026-09-02 07:04:02 UTC