iscc-usearch¶

Larger-than-RAM writable HNSW indexes, and variable-length binary vector search.

Introduction¶

iscc-usearch is a Python library that extends USearch - a high-performance HNSW library adopted by ClickHouse, LangChain, and others - with three independent capabilities:

Sharded HNSW indexes (ShardedIndex) keep a single active shard in RAM for writes while completed shards are memory-mapped for reads. Works with any vector type and metric USearch supports, including user-defined distance functions. Insert throughput stays consistent and memory stays bounded as the index grows to billions of vectors.

Normalized Prefix Hamming Distance (NphdIndex, ShardedNphdIndex) compares binary vectors of mixed bit-lengths - a 64-bit query finds nearest neighbors among 256-bit vectors with comparable distances. Purpose-built for ISCC (ISO 24138) content fingerprints, also applicable to Matryoshka embeddings, perceptual hashes, and locality-sensitive hashing.

128-bit UUID keys (ShardedIndex128, ShardedNphdIndex128) extend the key space from 64-bit integers to 128-bit bytes(16) keys. Useful when your identifiers are UUIDs, 128-bit hashes, or structured multi-part keys that don't fit in a uint64.

Key features:

• Bounded memory - only one shard in RAM at a time, the rest memory-mapped
• Billions of vectors - sharded indexes scale well beyond single-machine RAM
• Full CRUD - add, remove, upsert, compact, and dirty-counter-driven flush across all index variants
• Mixed bit-lengths - 64-bit and 256-bit vectors coexist in the same index
• 128-bit keys - bytes(16) UUID keys when 64-bit integers are not enough
• Any distance metric - user-defined metrics via USearch's plugin system
• Fast - inherits USearch's HNSW engine, benchmarked at 10x the throughput of FAISS

Which index class?¶

See the architecture overview for the full class hierarchy.

Quick start¶

pip install iscc-usearch

import numpy as np
from iscc_usearch import NphdIndex

index = NphdIndex(max_dim=256)

# Mix 64-bit and 128-bit vectors in the same index
index.add(1, np.array([255, 128, 64, 32, 16, 8, 4, 2], dtype=np.uint8))
index.add(2, np.array([255, 128, 64, 32, 16, 8, 4, 2, 1, 0, 255, 128, 64, 32, 16, 8], dtype=np.uint8))

# Search with a 64-bit query - NPHD compares the common prefix
query = np.array([255, 128, 64, 32, 16, 8, 4, 2], dtype=np.uint8)
matches = index.search(query, count=2)

print(matches.keys)  # Nearest neighbor keys
print(matches.distances)  # NPHD distances in [0.0, 1.0]

import numpy as np
from iscc_usearch import ShardedIndex

# Shards are stored in a directory on disk
index = ShardedIndex(ndim=64, path="my_index", dtype="f32")

# Add vectors - shards rotate automatically when size limit is reached
keys = list(range(1000))
vectors = np.random.rand(1000, 64).astype(np.float32)
index.add(keys, vectors)

# Search across all shards
matches = index.search(vectors[0], count=10)

print(matches.keys)  # Nearest neighbor keys
print(matches.distances)  # Cosine distances

Architecture¶

Documentation¶

Development & Contributing - Dev setup, testing, and contribution guidelines.

Source code on GitHub