Memory
# DocuClear — .table as Pipeline State Machine + Ingestion Provenance Pattern ## Core Insight The Total.js `.table` format serves as BOTH the data store AND the work queue simultaneously — no Redis, BullMQ, or separate job tracking needed. The `status` column IS the pipeline state machine. ## Ingestion Provenance Table Schema ``` sourceId:string|regime:string|sourceType:string|url:string|scrapedAt:date|status:string|raw:object|extracted:object|graph:object ``` ### Typed spine (queryable): - `regime` → EU-AI-ACT, UK-GDPR, DUAA, ICO-GUIDANCE - `sourceType` → legislation, guidance, case-law, enforcement-notice - `url` → canonical source URL - `scrapedAt` → freshness tracking - `status` → pipeline state driver (see below) ### Three schemaless object columns: - `raw` → Crawlberg scrape output: markdown, title, headings, links, hash, scraper version, httpStatus - `extracted` → Kreuzberg doc intelligence output: provisions, obligations, conditions, confidence, kreuzberg_version - `graph` → SurrealDB ingestion record: nodes_created, edges_created, graph_version_id, ingestedAt ## Status State Machine ``` pending → Crawlberg: fetch this URL scraped → Kreuzberg: extract this markdown extracted → Flow: ingest into SurrealDB graph ingested → Done, part of graph version failed → Retry queue (error stored in raw:object) superseded → URL changed, tombstoned but preserved ``` ## Pipeline Flow (each stage is idempotent) 1. Flow node queries `.where('status','=','pending')` → sends URL to Crawlberg 2. Crawlberg returns markdown → update row: status='scraped', raw={markdown, hash, ...} 3. Flow node queries `.where('status','=','scraped')` → sends markdown to Kreuzberg 4. Kreuzberg returns structured provisions → update row: status='extracted', extracted={provisions,...} 5. Flow node queries `.where('status','=','extracted')` → ingests into SurrealDB 6. SurrealDB confirms nodes/edges → update row: status='ingested', graph={nodes_created,...} 7. GraphVersion promotion pipeline picks up all 'ingested' rows for this version ## Key Properties - **Self-describing pipeline**: the .table file IS the job queue — no external orchestration store - **Idempotent stages**: crash recovery = re-run stage, picks up rows still at input status - **URL registry**: file doubles as canonical list of all sources ever tracked - **Seeding**: append rows with status:'pending' to add new sources - **Change detection**: reverse scan (find2) gets most recent row for URL, compare hash — if changed, tombstone old row, append new with status:'pending' - **Retry**: failed rows carry error+attempts in raw:object; reset to 'pending' on schedule - **Full provenance**: raw URL → markdown → extracted provisions → graph nodes — all in one row ## Per-Customer File Architecture Each customer gets their own directory of .table/.nosql files: ``` /customers/{orgId}/ profile.nosql ← organisation facts, questionnaire responses snapshots.nosql ← versioned CustomerSnapshots (append-only) assessments.table ← AssessmentRun spine + result:object findings.table ← AssessmentFinding typed + detail:object + trace:object + evidence:object gaps.table ← Gap + Recommendation typed + content:object evidence-log.nosql ← OpenLogger equivalent, decision traces agent-memory.table ← agent decisions, reasoning chains, embeddings scrape-cache.table ← regulatory source cache for this customer integrity.nosql ← rolling hash chain for tamper-evidence ``` ### Per-customer benefits: - Physical isolation — no multi-tenancy logic to get wrong - Audit delivery = tar.gz the customer folder, send to regulator - GDPR erasure = rm -rf ./customers/{orgId}/ - Backup = per-customer R2 folder with date-stamped snapshots - Integrity chain in integrity.nosql: {at, file, lines, hash, prev_hash} — tamper-evident ## Shared Infrastructure - **SurrealDB** = regulatory brain only (Regime, LegalProvision, Obligation, DecisionRule, GraphVersion etc.) — read-only from customer perspective, no customer data ever touches it - **OpenLogger** = also Total.js NoSQL under the hood — append-only provenance chain - **R2** = backup layer for all VPS stores (1Panel snapshots + S3-compatible sync) ## The Four-Layer Stack 1. .table files (per-customer) → customer facts, assessment outputs, agent memory 2. SurrealDB (shared) → regulatory knowledge graph, traversal only 3. OpenLogger (shared) → immutable decision provenance 4. R2 → backup and resilience layer ## FlowStream Component Gap Existing NoSQL FlowStream component (nosql.html) only supports .nosql format — no .table support. Need to build a TABLE() FlowStream component with: - schema config field (name:type|name:type header) - scalar method (aggregates) - typed filter operators (between, month, year, etc.) - dynamic collection path per customer ({orgId}/findings) - clean method trigger ## Why This Works for Compliance "Peter Širka built the primitives. The pipeline is the data — complete provenance from raw URL → scraped markdown → extracted provisions → graph nodes → assessment run → customer finding. Everything fits in a tar.gz."
Tags: docuclear, table-format, pipeline, ingestion, provenance, crawlberg, kreuzberg, surrealdb, per-customer-files, architecture, flowstream, nosql, status:canonical, project:docucleargraph, kind:semantic, volatility:state, stale:as-of — Source: claude — 2026-07-02 23:59:47 UTC