NornicDB Multi-Database Support

Status: Complete Version: 1.2 Last Updated: 2024-12-04

NornicDB supports multiple isolated databases (multi-tenancy) within a single storage backend, providing Neo4j 4.x-compatible multi-database functionality.

Overview
Features
Automatic Migration
Configuration
Usage Examples
Architecture
Implementation Details
Compatibility

Overview

NornicDB supports multiple isolated databases within a single storage backend, enabling multi-tenancy and complete data isolation. This feature is fully compatible with Neo4j 4.x multi-database functionality.

Features

Create and manage databases - CREATE DATABASE, DROP DATABASE, SHOW DATABASES
Complete data isolation - Each database is completely separate
Neo4j 4.x compatibility - Works with existing Neo4j drivers and tools
Automatic migration - Existing data automatically migrated on upgrade
Zero downtime - Migration happens transparently during startup
Backwards compatible - Existing code continues to work without changes

How It Works

NornicDB uses key-prefix namespacing within a single storage backend:

All keys are prefixed with the database name: nornic:node-123
A lightweight wrapper translates between namespaced and user-visible IDs
Single storage engine, multiple logical databases
Complete isolation with no cross-database data leakage

Implemented Features

Database Aliases — CREATE/DROP/SHOW ALIAS commands, alias resolution
Per-Database Resource Limits — ALTER DATABASE ... SET LIMIT max_nodes/max_edges/max_bytes/max_query_time/max_results/max_concurrent_queries/max_connections/max_queries_per_second/max_writes_per_second, enforced at runtime
Composite Databases (Sharding) — CREATE COMPOSITE DATABASE with local and remote constituents, USE <composite>.<alias> Fabric-style routing, many-read/one-write transactional semantics, OIDC credential forwarding for remote constituents
Schema merging for composite databases — constraints and indexes (property, composite, fulltext, vector, range) merged into a unified read-only metadata view

Plain (non-composite) cross-database queries are not supported by design; composite databases are the supported route to query across multiple databases.

Automatic Migration

When you upgrade to NornicDB with multi-database support, existing data is automatically migrated to the default database namespace on first startup.

How It Works

Automatic Detection: On startup, NornicDB checks for data without namespace prefixes
One-Time Migration: All unprefixed nodes and edges are migrated to the default database namespace
Index Updates: All indexes (label, outgoing, incoming, edge type) are automatically updated
Status Tracking: Migration status is persisted in the system database to prevent re-running

Migration Process

The migration runs automatically in NewDatabaseManager(ctx, inner, config):

1. Check if migration already completed (via metadata)
2. If not, detect unprefixed data
3. Migrate nodes: "node-123" → "nornic:node-123"
4. Migrate edges: "edge-456" → "nornic:edge-456"
5. Update all indexes automatically
6. Mark migration as complete

User Experience

Zero downtime: Migration happens during normal startup
Transparent: Users don't need to do anything
Safe: Migration is idempotent and tracked in metadata
Complete: All data, properties, and relationships are preserved

Example:

cypher

// Before upgrade: data stored as "node-123"
// After upgrade: automatically becomes "nornic:node-123"
// You access it the same way - no changes needed!
MATCH (n) RETURN n

Manual Tenant Migration (Optional)

If you want to move data from the default database to tenant-specific databases:

cypher

// 1. Create tenant database
CREATE DATABASE tenant_a

// 2. Use Cypher to copy data (example)
// In default database:
MATCH (n:Customer {database_id: "db_a"})
WITH n
// Switch to tenant database
:USE tenant_a
CREATE (n2:Customer {name: n.name, ...})

Configuration

Default Database

By default, NornicDB uses "nornic" as the default database name (Neo4j uses "neo4j").

Config File:

yaml

database:
  default_database: "custom"

Environment Variable:

bash

export NORNICDB_DEFAULT_DATABASE=custom
# Or Neo4j-compatible:
export NEO4J_dbms_default__database=custom

Configuration Precedence:

CLI arguments (highest priority)
Environment variables
Config file
Built-in defaults ("nornic")

Usage Examples

Cypher Commands

cypher

-- Create database
CREATE DATABASE tenant_a
CREATE DATABASE tenant_a IF NOT EXISTS

-- Drop database  
DROP DATABASE tenant_a
DROP DATABASE tenant_a IF EXISTS

-- List databases
SHOW DATABASES

-- Show specific database
SHOW DATABASE tenant_a

-- Switch database (in session)
:USE tenant_a

-- Database Aliases
CREATE ALIAS main FOR DATABASE tenant_primary_2024
DROP ALIAS main
SHOW ALIASES
SHOW ALIASES FOR DATABASE tenant_primary_2024

-- Resource Limits
ALTER DATABASE tenant_a SET LIMIT max_nodes = 1000000
ALTER DATABASE tenant_a SET LIMIT max_query_time = '60s'
SHOW LIMITS FOR DATABASE tenant_a

Driver Usage

python

# Python
from neo4j import GraphDatabase

driver = GraphDatabase.driver(
    "bolt://localhost:7687",
    auth=("admin", "password"),
    database="tenant_a"  # Specify database
)

session = driver.session()
result = session.run("MATCH (n) RETURN count(n)")

javascript

// JavaScript
const driver = neo4j.driver(
    "bolt://localhost:7687",
    neo4j.auth.basic("admin", "password"),
    { database: "tenant_a" }  // Specify database
);

const session = driver.session();
const result = await session.run("MATCH (n) RETURN count(n)");

HTTP API

POST /db/tenant_a/tx/commit
  - Execute query in specific database

GET /db/tenant_a/stats
  - Get statistics for specific database

Data Isolation Example

cypher

// In tenant_a
CREATE (n:Person {name: "Alice"})

// In tenant_b
CREATE (n:Person {name: "Bob"})

// tenant_a only sees Alice
// tenant_b only sees Bob

Architecture

Multi-Database Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                              Client                                      │
│              (Neo4j Driver with database parameter)                      │
│                                                                          │
│    driver = GraphDatabase.driver("bolt://...", database="tenant_a")      │
└─────────────────────────────────────┬───────────────────────────────────┘
                                      │
                                      ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                           Bolt Server                                    │
│  ┌──────────────────────────────────────────────────────────────────┐   │
│  │                     Connection Handler                            │   │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐               │   │
│  │  │ Conn 1      │  │ Conn 2      │  │ Conn 3      │               │   │
│  │  │ db=tenant_a │  │ db=tenant_b │  │ db=nornic   │               │   │
│  │  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘               │   │
│  │         │                │                │                       │   │
│  └─────────┼────────────────┼────────────────┼───────────────────────┘   │
│            │                │                │                           │
│            ▼                ▼                ▼                           │
│  ┌─────────────────────────────────────────────────────────────────┐    │
│  │                    Database Manager                              │    │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐              │    │
│  │  │ DB: nornic  │  │ DB: tenant_a│  │ DB: tenant_b│              │    │
│  │  │ (default)   │  │             │  │             │              │    │
│  │  │ status: on  │  │ status: on  │  │ status: on  │              │    │
│  │  └─────────────┘  └─────────────┘  └─────────────┘              │    │
│  │                                                                  │    │
│  │  + system (metadata database)                                    │    │
│  └──────────────────────────────────────────────────────────────────┘    │
│                                      │                                   │
└──────────────────────────────────────┼───────────────────────────────────┘
                                       │
                                       ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                      Namespaced Storage Layer                            │
│  ┌─────────────────────────────────────────────────────────────────┐    │
│  │                    NamespacedEngine                              │    │
│  │  Wraps storage.Engine, prefixes all keys with database name      │    │
│  │                                                                  │    │
│  │  CreateNode("123") → inner.CreateNode("tenant_a:123")            │    │
│  │  GetNode("123")    → inner.GetNode("tenant_a:123")               │    │
│  │  AllNodes()        → filter(inner.AllNodes(), "tenant_a:*")    │    │
│  └──────────────────────────────────────────────────────────────────┘    │
│                                      │                                   │
└──────────────────────────────────────┼───────────────────────────────────┘
                                       │
                                       ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                        Storage Engine                                    │
│                    (Single BadgerDB instance)                            │
│                                                                          │
│   Keys: tenant_a:node:123    tenant_b:node:789    nornic:node:001         │
│         tenant_a:edge:456    tenant_b:edge:012    nornic:edge:002         │
│         system:db:tenant_a   system:db:tenant_b   system:db:nornic        │
└─────────────────────────────────────────────────────────────────────────┘

Request Flow

When a client connects with a database parameter:

Bolt HELLO - Client specifies database in connection
Database Validation - Server verifies database exists
Namespaced Storage - Server creates storage view for that database
Query Execution - All queries run against the namespaced storage
Data Isolation - Data is automatically prefixed with database name

This ensures complete isolation - queries in tenant_a can never see data from tenant_b.

Implementation Details

Key Components

NamespacedEngine (pkg/storage/namespaced.go) - Wraps storage with automatic key prefixing
DatabaseManager (pkg/multidb/manager.go) - Manages database lifecycle, metadata, aliases, and limits
Migration (pkg/multidb/migration.go) - Automatic migration of existing unprefixed data
System Commands - CREATE DATABASE, DROP DATABASE, SHOW DATABASES in Cypher executor
Alias Management - CREATE ALIAS, DROP ALIAS, SHOW ALIASES commands with alias resolution
Resource Limits - ALTER DATABASE SET LIMIT, SHOW LIMITS commands with limit storage
Bolt Protocol - Database parameter support in HELLO messages (supports aliases)
HTTP API - Database routing in REST endpoints (supports aliases)

Data Model

Key Format

{database}:{type}:{id}

Examples:
  nornic:node:user-123          # Node in default database
  tenant_a:node:user-456       # Node in tenant_a database
  tenant_a:edge:follows-789    # Edge in tenant_a database
  system:node:databases:meta   # System metadata

Storage Layout

BadgerDB Keys:
├── nornic:node:*              # Default database nodes
├── nornic:edge:*              # Default database edges
├── nornic:idx:*               # Default database indexes
├── tenant_a:node:*           # Tenant A nodes
├── tenant_a:edge:*           # Tenant A edges
├── tenant_b:node:*           # Tenant B nodes
├── tenant_b:edge:*           # Tenant B edges
└── system:node:databases:*   # Database metadata

Protocol Changes

Bolt HELLO Message Extension

Neo4j 4.x compatible database selection:

HELLO {
  "user_agent": "neo4j-python/5.0",
  "scheme": "basic",
  "principal": "admin",
  "credentials": "password",
  "db": "tenant_a"           // Database selection
}

Driver Connection String

python

# Default database
driver = GraphDatabase.driver("bolt://localhost:7687")

# Specific database
driver = GraphDatabase.driver(
    "bolt://localhost:7687",
    database="tenant_a"
)

# Or per-session
session = driver.session(database="tenant_a")

Compatibility

Neo4j 4.x Compatibility Matrix

Feature	Neo4j 4.x	NornicDB v1	Notes
`CREATE DATABASE`	Yes	Yes	Fully implemented
`DROP DATABASE`	Yes	Yes	Fully implemented
`SHOW DATABASES`	Yes	Yes	Fully implemented
`SHOW DATABASE x`	Yes	Yes	Fully implemented
`:USE database`	Yes	Yes	Bolt protocol support
`database` param	Yes	Yes	Bolt protocol support
Default database	`neo4j`	`nornic`	Configurable
Configuration precedence	CLI > Env > File > Default	Yes	Implemented
Backwards compatibility	N/A	Yes	Existing code works
Automatic migration	N/A	Yes	Automatic on upgrade
Database aliases	Yes	Yes	Fully implemented
Composite DBs	Yes	Yes	Fully implemented
Per-DB limits	Yes	Yes	Fully implemented

Backwards Compatibility

Fully backwards compatible:

Existing code without database parameter works with default database
All existing data automatically migrated and accessible in default database namespace
No breaking changes to existing APIs
Legacy "nornicdb" name supported (via config)

Testing

unit tests verify:

Database creation and deletion
Data isolation between databases
Configuration precedence
Backwards compatibility
Automatic migration of existing unprefixed data
Metadata persistence
Namespaced storage operations
Migration idempotency (doesn't run twice)

Test Coverage: 84.7% for multidb package

NornicDB Multi-Database Support

Table of Contents

Overview

Features

How It Works

Implemented Features

Automatic Migration

How It Works

Migration Process

User Experience

Manual Tenant Migration (Optional)

Configuration

Default Database

Usage Examples

Cypher Commands

Driver Usage

HTTP API

Data Isolation Example

Architecture

Multi-Database Architecture

Request Flow

Implementation Details

Key Components

Data Model

Key Format

Storage Layout

Protocol Changes

Bolt HELLO Message Extension

Driver Connection String

Compatibility

Neo4j 4.x Compatibility Matrix

Backwards Compatibility

Testing

See Also

NornicDB Multi-Database Support ​

Table of Contents ​

Overview ​

Features ​

How It Works ​

Implemented Features ​

Automatic Migration ​

How It Works ​

Migration Process ​

User Experience ​

Manual Tenant Migration (Optional) ​

Configuration ​

Default Database ​

Usage Examples ​

Cypher Commands ​

Driver Usage ​

HTTP API ​

Data Isolation Example ​

Architecture ​

Multi-Database Architecture ​

Request Flow ​

Implementation Details ​

Key Components ​

Data Model ​

Key Format ​

Storage Layout ​

Protocol Changes ​

Bolt HELLO Message Extension ​

Driver Connection String ​

Compatibility ​

Neo4j 4.x Compatibility Matrix ​

Backwards Compatibility ​

Testing ​

See Also ​

NornicDB Multi-Database Support

Table of Contents

Overview

Features

How It Works

Implemented Features

Automatic Migration

How It Works

Migration Process

User Experience

Manual Tenant Migration (Optional)

Configuration

Default Database

Usage Examples

Cypher Commands

Driver Usage

HTTP API

Data Isolation Example

Architecture

Multi-Database Architecture

Request Flow

Implementation Details

Key Components

Data Model

Key Format

Storage Layout

Protocol Changes

Bolt HELLO Message Extension

Driver Connection String

Compatibility

Neo4j 4.x Compatibility Matrix

Backwards Compatibility

Testing

See Also