Skip to content

Scaling up with ShardedNphdIndex

This tutorial builds on the Getting Started and Variable-Length Vectors guides. You will create a sharded index that handles large datasets with consistent insert throughput and bounded memory usage.

When to use ShardedNphdIndex

Use ShardedNphdIndex instead of NphdIndex when:

  • Your dataset exceeds available RAM.
  • Insert throughput degrades as the index grows.
  • You need persistent storage with automatic shard rotation.

ShardedNphdIndex combines variable-length NPHD support with transparent sharding. The API is nearly identical to NphdIndex -- you add, remove, upsert, and search vectors without managing shards manually.

Create a sharded index

import numpy as np
from iscc_usearch import ShardedNphdIndex

index = ShardedNphdIndex(
    max_dim=256,
    path="./my_index",
    shard_size=512 * 1024 * 1024,  # 512 MB per shard
)

The path directory is created automatically. Shard files appear as the index grows.

Add mixed-resolution vectors

Just like NphdIndex, you can mix 64-bit, 128-bit, and 256-bit vectors. Batch insertion is the idiomatic high-performance pattern:

# Batch add -- 64-bit vectors (8 bytes each)
keys_64 = list(range(100))
vecs_64 = np.random.randint(0, 256, size=(100, 8), dtype=np.uint8)
index.add(keys_64, vecs_64)

# Batch add -- 128-bit vectors (16 bytes each)
keys_128 = list(range(100, 200))
vecs_128 = np.random.randint(0, 256, size=(100, 16), dtype=np.uint8)
index.add(keys_128, vecs_128)

# Batch add -- 256-bit vectors (32 bytes each)
keys_256 = list(range(200, 300))
vecs_256 = np.random.randint(0, 256, size=(100, 32), dtype=np.uint8)
index.add(keys_256, vecs_256)

When the active shard exceeds shard_size, it is saved to disk and reopened as a read-only memory-mapped view. A fresh active shard takes its place. This happens automatically.

Search across all shards

Queries fan out across all shards and results are merged:

query = np.random.randint(0, 256, size=8, dtype=np.uint8)
matches = index.search(query, count=10)

for key, dist in zip(matches.keys, matches.distances):
    print(f"Key {key}: distance = {dist:.4f}")

NPHD compares only the common prefix, so a 64-bit query finds nearest neighbors among vectors of any stored resolution.

Retrieve vectors

Retrieve the original (unpadded) vector by key:

vec = index.get(0)
print(vec)  # Original 64-bit vector

# Batch retrieval
vecs = index.get([0, 100, 200])

The bloom filter provides O(1) rejection of non-existent keys, so lookups stay fast regardless of shard count.

Remove and update vectors

Remove vectors by key. Active shard entries are deleted immediately; view shard entries are tombstoned and filtered from search results:

# Remove a single vector
index.remove(0)

# Remove a batch
index.remove([100, 101, 102])

# Upsert — insert-or-update
vec_updated = np.random.randint(0, 256, size=8, dtype=np.uint8)
index.upsert(200, vec_updated)

After many removals or upserts, call compact() to rebuild view shards and reclaim disk space:

removed = index.compact()
print(f"Compaction removed {removed} stale entries")

Save and reopen

# Save current state (active shard + bloom filter)
index.save()

# Reopen later -- auto-detects existing shards and max_dim
index = ShardedNphdIndex(path="./my_index")

# Verify
matches = index.search(query, count=5)
print(matches.keys)

Inspect the index

print(index.size)  # Logical vector count (excludes tombstoned entries)
print(index.shard_count)  # Number of shard files
print(index.max_dim)  # Maximum bits per vector
print(index.dirty)  # Unsaved key mutations since last save

The dirty property tracks unsaved mutations. Use it to implement flush policies like "save every N writes". It resets to 0 on save() and reset().

Shard directory layout

After adding data and saving, the directory looks like this:

my_index/
    shard_000.usearch   # view shard (memory-mapped, read-only)
    shard_001.usearch   # view shard (memory-mapped, read-only)
    shard_002.usearch   # active shard (RAM, read-write)
    bloom.isbf          # bloom filter for key lookups
    tombstones.npy      # deletion/dedup state (if any removals or upserts pending)

Completed shards are read-only. Only the highest-numbered shard is the active shard.

Open for read-only access

Once you have a populated index, you can open it in read-only mode. All shards are memory-mapped and write operations are blocked:

reader = ShardedNphdIndex(path="./my_index", read_only=True)

# Search and retrieval work normally
matches = reader.search(query, count=5)
vec = reader.get(0)

# Writes raise RuntimeError
# reader.add(999, vec)  # RuntimeError: index is read-only

This is useful for serving search queries from a pre-built index without risk of accidental modification.

Next steps