For Coding Agents¶

Dense, prescriptive reference for AI agents working on or integrating with iscc-usearch. Tables and code over prose. Terminology matches the codebase exactly.

Architecture map¶

File layout¶

Class hierarchy¶

usearch.index.Index
└── Index                            (index.py — adds upsert, internal base)
    └── NphdIndex                    (nphd.py — variable-length + NPHD metric)

ShardedIndex                         (sharded.py — composition-based sharding, uint64 keys)
├── ShardedIndex128                  (sharded.py — 128-bit UUID keys via _UuidKeyMixin)
└── ShardedNphdIndex                 (sharded_nphd.py — variable-length + NPHD metric)
    └── ShardedNphdIndex128          (sharded_nphd.py — 128-bit UUID keys via _UuidKeyMixin)

ScalableBloomFilter                  (bloom.py — standalone, used by ShardedIndex)

Import dependency flow¶

utils.py
  ↑
bloom.py ← sharded.py ← sharded_nphd.py
              ↑
index.py ← nphd.py

Public API exports (__all__)¶

__all__ = [
    "NphdIndex",
    "ShardedIndex",
    "ShardedIndex128",
    "ShardedNphdIndex",
    "ShardedNphdIndex128",
    "ScalableBloomFilter",
    "timer",
]

Index (from index.py) is not exported — it is an internal base class.

Decision dispatch¶

Which index class?¶

Which persistence method?¶

load/view/restore are on NphdIndex. Sharded indexes handle persistence internally via save()/constructor — they auto-load/view shards from the path directory.

Which add method?¶

For sharded indexes, add() only checks duplicate keys within the active shard. It does not scan view shards. Use add_once() when keys may already exist across shard boundaries and first-write-wins semantics are required.

Which read method?¶

None of these register completed background rotations. Call drain_rotations() before reads that require visibility of recently rotated shards.

Unsupported operations (ShardedIndex)¶

These raise NotImplementedError: rename(), join(), cluster(), pairwise_distance(), copy(), clear().

Constraints and invariants¶

Concurrency rules¶

• Single-process only. No file locking on .usearch files.
• Multiple processes against the same index → data corruption.
• Within one process: async/await is safe for concurrent reads.
• Threads require an external lock — index objects are not thread-safe.
• Shard rotation is not atomic — concurrent writes must be serialized.
• background_rotation=True uses a single background thread for shard serialization. The detached shard has exclusive ownership in the background thread. Pending rotations serialize via a ThreadPoolExecutor(max_workers=1).
• Key-dependent writes (upsert, add_once, remove, save, compact, reset) drain pending rotations first. add() registers completed rotations non-blocking.

Vector constraints¶

Key constraints¶

• 128-bit variants: keys=None (auto-generation) raises ValueError.
• _bloom_key() / _tombstone_key() convert raw keys to canonical Python types.

Dirty counter semantics¶

Bloom filter behavior¶

• Enabled by default (bloom_filter=True on ShardedIndex).
• Updated on add() — keys are added via add_batch().
• Never updated on remove() — bloom filter is append-only.
• contains() / get() / remove() use bloom for fast rejection of non-existent keys.
• False positives are expected (probabilistic). False negatives do not occur.
• rebuild_bloom() creates a fresh filter from all existing keys.
• Automatically rebuilt on load if the file is missing or corrupt (writable indexes only).
• compact() calls rebuild_bloom() internally.
• Bloom filter file: bloom.isbf in the shard directory.
• Always loaded from disk if file exists, regardless of _use_bloom setting.

Tombstone lifecycle¶

remove(key) → key added to _tombstones (in-memory set)
    ↓
save() → bloom persisted, then shard, then _tombstones as tombstones.npy
    ↓
compact() → view shards rebuilt excluding tombstoned entries
    ↓
_tombstones.clear(), tombstones.npy deleted

• _needs_compact flag: set when add() clears a tombstone (creating cross-shard duplicates).
• tombstones.npy file existence signals _needs_compact=True on next load.
• Tombstones are never written to bloom filter — bloom has no "remove" operation.
• Tombstones are persisted after shard data so that tombstone removals only become visible once the shard they depend on is durable.

Side effects catalog¶

Task recipes¶

Add vectors to a sharded index¶

from iscc_usearch import ShardedNphdIndex
import numpy as np

idx = ShardedNphdIndex(max_dim=256, path="my_index")

# Single vector
key = 1
vec = np.random.randint(0, 256, size=32, dtype=np.uint8)
idx.add(key, vec)

# Batch of uniform-length vectors
keys = np.arange(100, 200, dtype=np.uint64)
vecs = np.random.randint(0, 256, size=(100, 32), dtype=np.uint8)
idx.add(keys, vecs)

# Batch of variable-length vectors
keys2 = np.arange(200, 205, dtype=np.uint64)
vecs2 = [np.random.randint(0, 256, size=s, dtype=np.uint8) for s in [8, 16, 24, 32, 8]]
idx.add(keys2, vecs2)

idx.save()

Search with mixed bit-lengths¶

# Query vectors can differ in length from stored vectors.
# NPHD metric normalizes by the shorter prefix.
q8 = np.random.randint(0, 256, size=8, dtype=np.uint8)  # 64-bit query
q32 = np.random.randint(0, 256, size=32, dtype=np.uint8)  # 256-bit query

matches_8 = idx.search(q8, count=10)  # compares prefix of stored vectors
matches_32 = idx.search(q32, count=10)  # full comparison with 256-bit vectors

# Batch search with variable-length queries
queries = [q8, q32]
batch_matches = idx.search(queries, count=5)

Flush based on dirty counter¶

FLUSH_THRESHOLD = 10_000

idx.add(keys, vecs)
if idx.dirty >= FLUSH_THRESHOLD:
    idx.save()

Upsert then compact¶

# Upsert replaces existing vectors (remove + add).
# After many upserts, view shards accumulate stale entries.
idx.upsert(key, new_vec)

# Compact rebuilds view shards to reclaim space.
# Call periodically, not after every upsert.
if idx.tombstone_count > 1000:
    removed = idx.compact()

Rebuild bloom filter for existing index¶

# For indexes created without bloom filter, or after corruption:
idx = ShardedNphdIndex(max_dim=256, path="existing_index")
count = idx.rebuild_bloom()  # Scans all shards, saves bloom.isbf

Background rotation with context manager¶

from iscc_usearch import ShardedNphdIndex

with ShardedNphdIndex(
    max_dim=256,
    path="my_index",
    background_rotation=True,
    max_pending_rotations=2,
) as idx:
    for key, vec in data_stream:
        idx.add(key, vec)
# close() called automatically: drains rotations, saves, releases resources

Reopen existing index (max_dim auto-detected)¶

# max_dim is auto-detected from existing shard metadata when omitted
idx = ShardedNphdIndex(path="existing_index")
assert idx.max_dim == 256  # read from shard_*.usearch files

Iterate keys and vectors lazily¶

# keys and vectors are lazy iterators — no full materialization
for key in idx.keys:
    print(key)

for vec in idx.vectors:
    print(vec.shape)  # variable-length, unpadded

# Indexing, slicing, len(), and numpy conversion supported
first_key = idx.keys[0]
last_vec = idx.vectors[-1]
all_keys = np.asarray(idx.keys)

Inspect index with stats()¶

info = idx.stats()
# Returns: total_vectors, dimensions, metric, dtype, connectivity,
#   view_shards, active_shard_vectors, shard_size, dirty, tombstones,
#   bloom_filter, memory_usage, path, read_only

Open existing index read-only¶

# All shards memory-mapped, no active shard. Write operations raise RuntimeError.
idx = ShardedNphdIndex(path="existing_index", read_only=True)
matches = idx.search(query_vec)
# idx.add(...)  # RuntimeError: index is read-only

Change playbook¶

If modifying padding logic¶

Files: nphd.py (pad_vectors, unpad_vectors)

Also update:

• sharded_nphd.py — add(), search(), get(), vectors iterator (all call pad_vectors/unpad_vectors)
• tests/test_nphd.py — padding round-trip tests
• docs/explanation/architecture.md — "Length-prefixed padding" section

If modifying shard rotation¶

Files: sharded.py (_rotate_shard, _rotate_shard_sync, _rotate_shard_background, _background_rotate_task, _schedule_next_size_check, _adds_until_size_check)

Also update:

• sharded.py — add() (calls rotation), _load_existing() (loads rotated shards), drain_rotations(), close(), _enforce_pending_limit(), _register_completed_rotations()
• sharded_nphd.py — _create_shard(), _restore_shard() (shard creation/restore hooks)
• tests/test_sharded_background_rotation.py — background rotation tests
• docs/explanation/sharding-design.md
• docs/howto/sharding.md

If modifying bloom filter format¶

Files: bloom.py (save, load, MAGIC, VERSION)

Also update:

• sharded.py — _load_bloom_if_exists(), save(), rebuild_bloom(), compact()
• Bump VERSION constant in bloom.py if format changes
• tests/test_bloom.py
• docs/howto/bloom-filters.md

If modifying key handling¶

Files: sharded.py (_UuidKeyMixin, key hook methods on ShardedIndex)

Also update:

• sharded_nphd.py — ShardedNphdIndex128 inherits _UuidKeyMixin
• index.py — Index.upsert() has its own key normalization
• tests/test_sharded.py, tests/test_sharded_nphd.py — key-related tests
• docs/howto/uuid-keys.md

If adding a new index class¶

Also update:

• src/iscc_usearch/__init__.py — add to __all__ and import
• docs/reference/api.md — add mkdocstrings directive (or handled via gen_llms_full.py)
• scripts/gen_llms_full.py — add to _API_CLASSES list
• docs/explanation/architecture.md — update class hierarchy and "Choosing an index class" table
• This page — update architecture map, class hierarchy, and decision dispatch tables
• Add test file in tests/

Common mistakes¶

NEVER open the same index directory from multiple processes. Use a single process with async/await for concurrent access.

NEVER mutate an index opened with view(). Viewed indexes are memory-mapped and read-only. Use load() or restore() for mutable access. Sharded indexes use read_only=True for the same purpose.

NEVER pass ndim, metric, or dtype to NphdIndex. These are computed from max_dim. Passing them raises TypeError.

# WRONG
idx = NphdIndex(ndim=264, metric=MetricKind.NPHD)

# CORRECT
idx = NphdIndex(max_dim=256)

NEVER skip compact() after sustained upsert/remove workloads. View shards accumulate tombstoned and duplicate entries that waste disk space and slow searches.

NEVER assume len(sharded_index) is exact after upserts without compaction. Cross-shard duplicates cause size to overcount. Call compact() for an exact count.

ALWAYS call close() or use a context manager before process exit. close() drains pending background rotations, saves unsaved state, and releases mmap/executor resources. save() is sufficient for durability (it also drains pending rotations) but does not release mmap or executor resources. Unsaved data in the active shard is lost.

ALWAYS use np.uint8 dtype for vectors. Other dtypes cause silent data corruption in the NPHD metric computation.

ALWAYS pass explicit keys to upsert() and add_once(). Both raise ValueError on keys=None. For 128-bit variants, add() also requires explicit keys.

ALWAYS serialize concurrent writes within a single process (e.g., via asyncio.Lock). Shard rotation during concurrent writes can cause data loss.