Memory
# DocuClear Compliance Engine — System Architecture Blueprint ## Core Design Philosophy The graph is not just a storage model — it is a **decision structure**. Assessments are produced by traversing a versioned regulatory graph against a versioned customer snapshot, with the resulting decision trace stored immutably for replay, audit, and challenge handling. ## Five Optimisation Properties - Deterministic evaluation from fixed inputs and a fixed graph version - Replayability from snapshot IDs and engine version - Explainability through explicit traversed rules, nodes, edges, and fired obligations - Update handling via graph versioning and customer snapshot versioning - Evidence-grade auditability through immutable decision and evidence logs ## Four-Component Architecture | Component | Primary responsibility | Authoritative for | |---|---|---| | **NoSQL** | Customer system of record and immutable business fact snapshots | OrganisationProfile, BusinessAreaRecord, UseCaseRecord, ToolDeclaration, QuestionnaireResponse, CustomerSnapshot | | **SurrealDB** | Canonical regulatory graph and assessment decision structure | Regime, LegalProvision, RoleType, ClassificationScheme, DecisionRule, RuleCondition, Obligation, ControlRequirement, EvidenceRequirement, InteractionLink, GraphVersion | | **Total.js Flow** | Deterministic orchestration and graph traversal | Assessment execution sequence, state transition logic, rule evaluation order | | **OpenLogger** | Immutable evidence and decision provenance store | Decision traces, evidence chain of custody, event chronology, integrity metadata | ## Assessment Lifecycle (7 Stages) 1. **Snapshot input** — Freeze customer facts into versioned CustomerSnapshot → output: `customer_snapshot_id` 2. **Resolve graph** — Select applicable production GraphVersion → output: `graph_version_id` 3. **Build assessment context** — Normalize customer facts into traversal-ready bindings → output: `assessment_context` 4. **Traverse graph** — Walk decision graph, evaluate conditions, assign roles, classify use cases → output: `traversal_trace`, `applicable_obligations` 5. **Determine result** — Aggregate rule outcomes into findings, statuses, recommendations → output: `assessment_result` 6. **Log provenance** — Write full immutable decision record to OpenLogger → output: `decision_trace_id` 7. **Project summaries** — Optionally store customer-facing summaries to NoSQL ## Decision Trace Envelope (minimum fields) assessment_id, customer_snapshot_id, graph_version_id, engine_version, started_at/completed_at, organisation_id, use_case_ids, role_assignments, nodes_visited, edges_traversed, conditions_evaluated, rules_fired, rules_not_fired, classifications, obligations_derived, evidence_refs, findings, integrity_hash ## Three-Level Trace Model - Level 1 Summary: result, fired rules, obligations, final findings (everyday customer reporting) - Level 2 Full decision trace: nodes visited, edges traversed, conditions, classifications, obligations (normal audit) - Level 3 Forensic: every predicate binding, intermediate fact set, edge traversal, evidence check (disputes/regulator) ## Three Mandatory Version Anchors Per Assessment 1. Customer snapshot version (what customer facts were) 2. Graph version (what regulatory graph was traversed) 3. Engine version (what evaluation runtime was used) 4. Optional: Evidence set version ## Immutability Rules - **NoSQL immutable**: CustomerSnapshot, versioned snapshots of org/area/use-case/tool, questionnaire submissions - **SurrealDB immutable**: Released GraphVersion, canonical regulation nodes/edges once promoted to production - **OpenLogger immutable**: Decision trace envelopes, evidence file records and hashes, evidence access/export events ## Graph Design Principle Graph designed for deterministic traversal, not free-form semantic exploration. Requires: clear node types, explicit edge types, stable graph version IDs, mechanically evaluable rule conditions, no hidden procedural logic in graph semantics. It is a **compliance decision graph**, not a general semantic graph. ## Key Failure Modes to Avoid - Running assessments against mutable "latest" customer records (breaks replayability) - Storing only the final result (removes explainability) - Treating graph as generic reference material (makes traversal ambiguous) - Updating graph nodes in place after production use (breaks historical reconstruction) - Mixing evidence payloads into assessment DB (weakens immutability boundaries) - Burying procedural logic inside graph semantics (makes testing difficult) ## Implementation Steps 1. Define NoSQL snapshot model for customer facts 2. Define SurrealDB graph release model (GraphVersion, node versioning, edge versioning) 3. Define OpenLogger decision trace schema 4. Implement Flow stages for snapshot resolution, graph resolution, traversal, result derivation, trace emission 5. Build replay tooling (customer_snapshot_id + graph_version_id + engine_version) 6. Test against law-firm-with-AI-product scenario ## Key Test Scenario: Law Firm with AI Product One customer snapshot, multiple business areas, traversal resolves different role assignments per use case: - Legal practice → controller/processor obligations - AI product → provider obligations - Internal HR screening → deployer/high-risk obligations [Update 7/2/2026]: ## Clarifications confirmed (July 2026) ### NoSQL store Total.js native NoSQL — the append-only, immutable embedded database that ships with the Total.js platform. Not a separate DB product. This is used for CustomerSnapshot, OrganisationProfile, QuestionnaireResponse, and all immutable customer fact records. ### OpenLogger Total.js OpenLogger (MIT, open source) — a lightweight Total.js app backed by the same NoSQL embedded database. Stores logs from 3rd party apps. REST API is a single POST endpoint with fields: type, data, app, url, userid, username, schema, reference, message, ua, ip. This is the immutable decision provenance and evidence audit trail store. GitHub: https://github.com/totaljs/openlogger. Docs: https://docs.totaljs.com/openlogger/ OpenLogger decision trace fields map to the API as follows: - `type` → "info" / "success" / "error" / "warning" (assessment outcome) - `app` → "compliance-engine" - `schema` → entity type (e.g. "AssessmentRun", "EvidenceSubmission") - `reference` → assessment_run_id or decision_trace_id - `userid` / `username` → assessed organisation or user - `data` → full decision trace JSON payload (trace envelope, nodes visited, obligations derived etc.) - `message` → human-readable summary Note: OpenLogger's API is append-only and has no update/delete endpoints — this is structurally immutable by design, which matches the architecture requirement perfectly. ### Infrastructure - Total.js FlowStream runs on UK VPS - Exposed via cloudflared tunnel (Cloudflare Tunnel) - Clients connect via Cloudflare Workers at the edge - Workers act as the edge ingress/auth layer; Flow/FlowStream is the backend runtime ### Evidence Storage Intent confirmed: OpenLogger via the `data` field for evidence metadata and integrity hashes. Actual evidence file payloads — storage destination still to be decided (R2 is candidate, or OpenFiles from Total.js platform). ### Graph Versioning Not yet designed. Needs a separate design pass. [Update 7/2/2026]: ## Evidence & Backup Storage Decision (July 2026) ### Chosen approach: Option 1 — R2 as unified backup/resilience layer for all VPS stores - OpenFiles on VPS as primary evidence file store (token-based access, Total.js FileStorage, robust) - OpenLogger on VPS as primary immutable decision/audit log store - NoSQL (Total.js embedded) on VPS as primary customer fact and snapshot store - R2 as the off-site backup and resilience layer for all three stores (S3-compatible API) - EvidenceSubmission.storage_url points to OpenFiles endpoint (not R2 directly) - R2 bucket structure: single account-level bucket for backup, NOT per-customer buckets ### Why Option 1 over Option 2 (per-customer R2 buckets) - OpenFiles provides token-based access control — customer isolation handled at application layer - Per-customer R2 buckets create operational overhead (bucket proliferation, per-bucket API tokens, harder to manage at scale) - Cloudflare R2 has zero egress fees — backup syncs from VPS to R2 are cost-effective - Single-bucket backup is simpler to audit, replicate, and restore - OpenFiles already handles file serving and access token logic natively [Update 7/2/2026]: ## Backup & Resilience Strategy (July 2026) ### Approach: 1Panel Snapshots to R2 via S3-compatible backup account - All services (Total.js Flow, OpenFiles, OpenLogger, SurrealDB) run as Docker containers on UK VPS managed by 1Panel - 1Panel snapshot captures: Docker daemon.json, installed apps, local backups, 1Panel database, binaries and systemd service - Snapshots cover mounted Docker volumes = includes all NoSQL embedded DB data, OpenLogger data, OpenFiles stored files - 1Panel Backup Accounts supports S3-compatible storage (section 4.6) — R2 connects here via S3-compatible endpoint - Snapshot cadence: TBD (recommend daily at minimum for compliance data) ### Important gap flagged 1Panel snapshot docs list what is captured: Docker daemon.json, installed apps, local backups, 1Panel database, 1panel binary/service. It does NOT explicitly confirm Docker volume data is included in snapshots — this needs verification. If volumes are NOT included, a supplementary volume backup strategy is needed (e.g. docker run --volumes-from + tar to R2, or a 1Panel cron job). Must confirm before relying on snapshots for evidence file resilience.
Tags: docuclear, architecture, compliance-engine, surrealdb, total-js-flow, openlogger, design, kind:semantic, status:canonical, project:docucleargraph — Source: claude — 2026-07-02 15:58:43 UTC