NornicDB

Knowledge-Policy Visibility Layer and Deindex ​

What happens after the scorer marks an entity suppressible: query-time reveal gating, persisted suppression state, deindex cleanup, search/index hiding, and eventual retention-driven deletion.

This document covers the downstream path after SuppressionEligible is computed. It starts where the scoring pipeline hands off to the visibility layer and follows the implementation through pkg/storage/, pkg/cypher/, pkg/search/, and pkg/nornicdb/.

Read this after Knowledge-Policy Scoring Pipeline, which explains how the score itself is computed.

Full Pipeline ​

Legend — Source References ​

Diagram boxSource
hasRevealCall / setRevealOnEnginepkg/cypher/reveal.go
BeginQueryRevealScope / SetRevealAllpkg/storage/badger_decay_filter.go
Tombstone-aware read scanspkg/storage/badger_queries.go
Final entity read filterpkg/storage/badger_decay_filter.go
Property-level visibility filteringpkg/cypher/node_helpers.go, pkg/storage/badger_decay_filter.go
EnqueueDeindexIfSuppressedpkg/storage/badger_deindex_enqueue.go
DeindexCleanupJobpkg/storage/badger_deindex_cleanup.go
IndexEntryCatalog / tombstonespkg/storage/badger_index_catalog.go, pkg/storage/badger_index_tombstone.go
Reconcile and label-change re-scorepkg/storage/badger_decay_reconcile.go, pkg/storage/badger_deindex_enqueue.go
Access flusher recheck callbackpkg/knowledgepolicy/access_flusher.go, pkg/nornicdb/db.go
Search candidate decay filter hookpkg/search/decay_filter.go, pkg/search/search.go
Retention sweep and search-index removalpkg/nornicdb/db_retention.go, pkg/nornicdb/search_services.go

Stages ​

1. Reveal scope is a query-level gate, not a scoring override ​

Cypher detects reveal(...) in the submitted query text and calls setRevealOnEngine, which in turn uses BeginQueryRevealScope on the underlying BadgerEngine.

  • Reveal queries take an exclusive scope and set revealAll = true for that query.
  • Normal queries take a shared scope so they cannot observe another query's reveal mode.
  • This bypass lives at the storage/read layer. It does not rewrite scores or persisted suppression state.

External query result owners use storage.BeginQueryResultProjectionContext(ctx, engine) and defer its returned release. Cypher records its existing reveal mutex ownership with BeginQueryRevealScopeContext, so a nested search service borrows a genuine live lease instead of recursively acquiring a read lock behind a waiting reveal writer. Nested work preserves the owning query's reveal mode. Distinct physical engines have distinct leases; database namespaces and property policies are always resolved from the supplied projection engine, never borrowed from another database. Derived expression contexts inherit only opaque live leases; released contexts cannot impersonate an active scope. Borrowed work must finish before its owner releases the scope.

2. Visibility is enforced twice on reads ​

The visibility layer does not rely on a single check.

  • Index-time hiding: Badger label and adjacency scans skip index keys that have suppression tombstones when decay is enabled and revealAll is false.
  • Entity-time hiding: after decoding a node or edge body, filterNodeByDecay / filterEdgeByDecay re-evaluates whether the entity should be dropped.

That second check matters because it keeps correctness even if a body is reached through a path that did not come from a tombstoned secondary index key.

3. Property suppression is applied after entity visibility ​

If an entity survives the node/edge visibility filter, the Cypher materialization path still removes properties whose property-level rules are currently suppressed.

  • Entity projections, property expressions, properties(), keys(), map projections and entities inside paths share the node and relationship property filters.
  • Projections use detached property maps. Mutation owners retain full entities, so an unrelated SET preserves suppressed stored fields. Direct engine reads and reveal() expose the stored properties for debugging.
  • The property scorer uses the same namespace and access metadata inputs as the entity scorer, but evaluates per-property compiled overrides.

Normal computed HTTP graph neighborhood, expand, and path responses use the same detached node/relationship property projection as Cypher. These operations are query results even though their traversal starts from supplied IDs. Their property filters also evaluate the visible property maps. Direct engine/Go CRUD lookups, explicit debug reads, reveal(), temporal/diff history views, snapshots, backups and GDPR exports retain stored properties. Property suppression is a query presentation policy, not an authorization or storage erasure boundary. The shared search service also projects node and relationship payloads before display fields or property matches are generated, and refreshes durable continuation payloads on each pull. Known matches on hidden properties are omitted without changing retained match slices. A cursor whose grouping property becomes hidden is invalidated rather than emitting a stale group key. Normal search bypasses result caching while decay is enabled because graph revisions cannot represent elapsed visibility. Index construction, ranking scores and trusted Go admission/authorization callbacks retain their existing stored inputs; this is query-result property projection, not index erasure or an authorization boundary. Other direct query integrations are tracked explicitly in issue #1229 until their result owners are covered.

Within a read-only Cypher statement, inline/scalar/result readers reuse detached entity property views. Fresh-pointer index rechecks may reuse a scored map only when ID and the complete scoring input (labels/type, timestamps and stored property body) match a detached snapshot. The cloned-body guard still compares property bodies; this is not a globally constant-time projection claim. Memo state is private to that statement and does not survive another statement. Mixed-write and dynamically non-cacheable statements use uncached projection so mutations immediately observe their new fields and retain hidden stored fields. The disabled path does not discover the backend per value.

While decay is enabled, Cypher bypasses result caching: visibility can change with elapsed time or access metadata without a graph-write revision. Ordinary reads retain result caching when decay is disabled.

4. Persisted suppression state is the fallback truth when no scorer exists ​

filterNodeByDecay and filterEdgeByDecay first try to obtain a namespace scorer. If no scorer exists, the engine falls back to the persisted VisibilitySuppressed flag on the node or edge body.

That makes the persisted flag important even though live scoring usually decides visibility.

5. Becoming suppressed enqueues deindex work, not immediate index deletion ​

When EnqueueDeindexIfSuppressed determines that an entity newly crossed into the suppressed state, it:

  1. rewrites the body with VisibilitySuppressed = true
  2. creates a pending DeindexWorkItem
  3. leaves primary storage intact

The actual secondary-index cleanup is deferred to DeindexCleanupJob.

6. Deindex cleanup writes tombstones over tracked index keys ​

DeindexCleanupJob processes pending work items by consulting the entity's IndexEntryCatalog, which stores the exact secondary-index keys written for that entity.

The cleanup job then:

  1. writes tombstones for those keys
  2. marks the catalog Deindexed = true
  3. deletes the work item

This is why later scans can cheaply hide suppressed entities before they are even decoded.

7. Re-visibility clears tombstones and deindex bookkeeping ​

If a later re-score says the entity is visible again, EnqueueDeindexIfSuppressed takes the opposite branch.

  • It clears stored tombstones for the entity's tracked index keys.
  • It clears the catalog's Deindexed flag.
  • It removes any pending DeindexWorkItem.

That restoration path is what lets a previously suppressed entity re-enter normal query/index visibility without rebuilding the entire database.

8. Suppression can be triggered from multiple reconciliation paths ​

The access flusher is only one trigger.

  • SuppressionRecheck from the access flusher re-runs EnqueueDeindexIfSuppressed after persisted access metadata changes.
  • Schema-policy changes use ReconcileDecaySuppression / ReconcileDecaySuppressionWithChanges to revisit every entity in a namespace.
  • Label changes use rescoreSuppressionAfterLabelChangeInTxn so the stored flag follows the node's new binding.

9. Search has two hiding mechanisms ​

There are two distinct search-layer mechanisms in the repo.

  • The durable mechanism is the deindex/tombstone path above, which removes suppressed entities from the secondary indexes that search builds from.
  • The search service also exposes an optional live nodeDecayFilter hook and applies it through filterDecayedCandidates before RRF fusion.

The second path is a runtime candidate filter; the first path is the persistent storage/index cleanup path.

10. Retention is downstream and independent of suppression ​

Retention is not the same subsystem as decay suppression.

  • Suppression hides entities from normal reads and search.
  • Retention, when enabled, runs a periodic sweep over stored nodes.
  • The sweep skips configured excluded labels.
  • If the retention manager decides a record should be deleted or archived, the DB removes it from search indexes and then processes the record.

So “suppressed for a long time and later deleted” is not a direct hand-off from the scorer. It is an operational sequence where a separate retention subsystem eventually processes the still-stored record.

Relation To The Scoring Page ​

The scoring page ends when SuppressionEligible is determined. This page starts there and explains:

  • how reveal mode changes what queries can see
  • how suppressed entities are hidden from normal reads
  • how deindex work is enqueued and later materialized as tombstones
  • how entities become visible again
  • how retention can eventually delete still-suppressed records

See Also ​

GraphQL node/relationship created and updated events retain their whole-entity notification contract and synchronous publication order. Their detached property payloads use storage.NewNormalEventResultProjection before transport ID prefix removal, so the original namespace policy applies. Event projection deliberately ignores a mutation producer's REVEAL mode without changing that mode or acquiring another query mutex inside the callback. Stored values and the producer's own REVEAL results remain intact. Deleted notifications contain only IDs.

MCP discover related-node title/name fallbacks and Heimdall related-node titles use the same contextual query-result projection as normal search payloads. A discover operation called from an already owned REVEAL query retains its owning scope. Neither integration changes raw storage/debug, history, backup or export APIs.

Analytics privacy