Skip to content

Scaling up with ShardedNphdIndex

This tutorial builds on the Getting Started and Variable-Length Vectors guides. You will create a sharded index that handles large datasets with consistent insert throughput and bounded memory usage.

When to use ShardedNphdIndex

Use ShardedNphdIndex instead of NphdIndex when:

  • Your dataset exceeds available RAM.
  • Insert throughput degrades as the index grows.
  • You need persistent storage with automatic shard rotation.

ShardedNphdIndex combines variable-length NPHD support with transparent sharding. The API is nearly identical to NphdIndex -- you add, remove, upsert, and search vectors without managing shards manually.

Create a sharded index

import numpy as np
from iscc_usearch import ShardedNphdIndex

index = ShardedNphdIndex(
    max_dim=256,
    path="./my_index",
    shard_size=512 * 1024 * 1024,  # 512 MB per shard
)

The path directory is created automatically. Shard files appear as the index grows.

Add mixed-resolution vectors

Just like NphdIndex, you can mix 64-bit, 128-bit, and 256-bit vectors. Batch insertion is the idiomatic high-performance pattern:

# Batch add -- 64-bit vectors (8 bytes each)
keys_64 = list(range(100))
vecs_64 = np.random.randint(0, 256, size=(100, 8), dtype=np.uint8)
index.add(keys_64, vecs_64)

# Batch add -- 128-bit vectors (16 bytes each)
keys_128 = list(range(100, 200))
vecs_128 = np.random.randint(0, 256, size=(100, 16), dtype=np.uint8)
index.add(keys_128, vecs_128)

# Batch add -- 256-bit vectors (32 bytes each)
keys_256 = list(range(200, 300))
vecs_256 = np.random.randint(0, 256, size=(100, 32), dtype=np.uint8)
index.add(keys_256, vecs_256)

When the active shard exceeds shard_size, it is saved to disk and reopened as a read-only memory-mapped view. A fresh active shard takes its place. This happens automatically.

Search across all shards

Queries fan out across all shards and results are merged:

query = np.random.randint(0, 256, size=8, dtype=np.uint8)
matches = index.search(query, count=10)

for key, dist in zip(matches.keys, matches.distances):
    print(f"Key {key}: distance = {dist:.4f}")

NPHD compares only the common prefix, so a 64-bit query finds nearest neighbors among vectors of any stored resolution.

Retrieve vectors

Retrieve the original (unpadded) vector by key:

vec = index.get(0)
print(vec)  # Original 64-bit vector

# Batch retrieval
vecs = index.get([0, 100, 200])

The bloom filter provides O(1) rejection of non-existent keys, so lookups stay fast regardless of shard count.

Remove and update vectors

Remove vectors by key. Active shard entries are deleted immediately; view shard entries are tombstoned and filtered from search results:

# Remove a single vector
index.remove(0)

# Remove a batch
index.remove([100, 101, 102])

# Upsert — insert-or-update
vec_updated = np.random.randint(0, 256, size=8, dtype=np.uint8)
index.upsert(200, vec_updated)

After many removals or upserts, call compact() to rebuild view shards and reclaim disk space:

removed = index.compact()
print(f"Compaction removed {removed} stale entries")

Save and reopen

# Save current state (active shard + bloom filter)
index.save()

# Reopen later -- auto-detects existing shards and max_dim
index = ShardedNphdIndex(path="./my_index")

# Verify
matches = index.search(query, count=5)
print(matches.keys)

Inspect the index

print(index.size)  # Logical vector count (excludes tombstoned entries)
print(index.shard_count)  # Number of shard files
print(index.max_dim)  # Maximum bits per vector
print(index.dirty)  # Unsaved key mutations since last save

stats = index.stats()
print(stats["total_vectors"])
print(stats["view_shards"])
print(stats["active_shard_vectors"])
print(stats["memory_usage"])

The dirty property tracks unsaved mutations. Use it to implement flush policies like "save every N writes". It resets to 0 on save() and reset().

Use stats() for monitoring and diagnostics when you need one structured snapshot instead of reading individual properties.

Shard directory layout

After adding data and saving, the directory looks like this:

my_index/
    shard_000.usearch   # view shard (memory-mapped, read-only)
    shard_001.usearch   # view shard (memory-mapped, read-only)
    shard_002.usearch   # active shard (RAM, read-write)
    bloom.isbf          # bloom filter for key lookups
    tombstones.npy      # deletion/dedup state (if any removals or upserts pending)

Completed shards are read-only. Only the highest-numbered shard is the active shard.

Open for read-only access

Once you have a populated index, you can open it in read-only mode. All shards are memory-mapped and write operations are blocked:

reader = ShardedNphdIndex(path="./my_index", read_only=True)

# Search and retrieval work normally
matches = reader.search(query, count=5)
vec = reader.get(0)

# Writes raise RuntimeError
# reader.add(999, vec)  # RuntimeError: index is read-only

This is useful for serving search queries from a pre-built index without risk of accidental modification.

Next steps