API reference

The typed public surface of the lexindex Python package.

StringIndex

lexindex.StringIndex

Ordered string↔id index (FST) with prefix / range / fuzzy / subsequence queries.

ID_DTYPE class-attribute

ID_DTYPE: str = 'uint64'

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.

MISSING_ID class-attribute

MISSING_ID: int = 18446744073709551615

int([x]) -> integer int(x, base=10) -> integer

Convert a number or string to an integer, or return 0 if no arguments are given. If x is a number, return x.int(). For floating-point numbers, this truncates towards zero.

If x is not a number or if base is given, then x must be a string, bytes, or bytearray instance representing an integer literal in the given base. The literal can be preceded by '+' or '-' and be surrounded by whitespace. The base defaults to 10. Valid bases are 0 and 2-36. Base 0 means to interpret the base from the string as an integer iteral.

int('0b100', base=0) 4

__doc__ class-attribute

__doc__ = "Ordered string↔id index (FST) with prefix / range / fuzzy / subsequence queries."

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.

__module__ class-attribute

__module__ = 'lexindex._core'

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.

__contains__ method descriptor

__contains__(key: str) -> bool

Return bool(key in self).

__getitem__ method descriptor

__getitem__(key: str) -> int

Return self[key].

__iter__ method descriptor

__iter__() -> Iterator[tuple[str, int]]

Implement iter(self).

__len__ method descriptor

__len__() -> int

Return len(self).

__new__ builtin

__new__(*args, **kwargs) -> StringIndex

Create and return a new object. See help(type) for accurate signature.

__reduce__ method descriptor

__reduce__()

Pickle support: the blob, and the loader that reads it back.

An index pickles by copying its bytes, including one opened with load_mmap, whose pages are borrowed from a file the unpickling process may not have — a path would not survive a spawned worker on another machine, and a borrowed mapping would not survive the file changing. Pickling a large index therefore costs its serialised size in the pickle; save + load_mmap is what to use when both ends can see the same file.

build_sorted_to_file staticmethod

build_sorted_to_file(
    items: Iterable[str], path: str | PathLike[str]
) -> int

from_sorted streamed straight to path, so neither the corpus nor the finished index has to fit in memory. Returns the number of keys written.

build_to_file staticmethod

build_to_file(
    items: Iterable[str], path: str | PathLike[str]
) -> int

The constructor for a corpus that does not fit in memory, written straight to path: the keys, in any order, are sorted in runs that spill beside the output and merged into the transducer, so neither the corpus nor the index is ever held whole. Returns the number of distinct keys written.

common_prefix method descriptor

common_prefix(query: str) -> list[tuple[str, int]]

Every key that is a prefix of query, shortest first -- the reverse of prefix, which returns the keys query starts. This is the dictionary-matching query: given a vocabulary and a position in a sentence, it returns the entries that start there. The empty key, if the index holds it, is a prefix of everything and comes first.

contains method descriptor

contains(key: str) -> bool

Whether key is present.

from_bytes staticmethod

from_bytes(data: bytes) -> StringIndex

Reconstruct from a to_bytes blob.

from_sorted staticmethod

from_sorted(items: Iterable[str]) -> StringIndex

Build from an iterable of strings that is already in ascending byte order, without materialising it — the constructor has to hold the whole corpus to sort it, this does not. Adjacent duplicates are dropped exactly as the constructor drops them after sorting; input that is not ascending raises ValueError rather than producing an index that answers wrongly.

Keys composed from sorted parts are not themselves sorted unless the separator is below every byte that can follow a part — joining a sorted word list to itself with "." puts 'tween-decks.&c before 'tween.ARU, because - is below ..

from_untrusted_bytes staticmethod

from_untrusted_bytes(data: bytes) -> StringIndex

Reconstruct from a blob someone else wrote, validating the transducer before any query can reach it — slower than from_bytes, and total where that one is not.

from_bytes documents the one exception to "arbitrary bytes raise ValueError": the blob is an fst transducer whose node decoder is safe but not total, and the checksum in front of it is public, so bytes crafted to carry a matching one panic — pyo3_runtime.PanicException, not ValueError. This loader checks the transducer as a graph — every reachable node once, in time proportional to nodes and transitions rather than to the keys they spell — so that its values are ranks and its keys are UTF-8, and catches the panic at the load boundary, so a crafted blob raises ValueError like any other bad input.

It costs two decodes of every node and the checksum: 22.9 ms against 0.7 ms for from_bytes on the 479 823-word /usr/share/dict/words, 32×. Worth paying once for a blob from a stranger, not worth paying for one of your own.

Two things it cannot promise. The panic runs the process-wide hook on its way out, so the rejection normally prints a panic message to stderr before ValueError is raised — nothing suppresses it, because the hook is global. And a build with panic = "abort" has no unwinding to catch, which the published wheels do not use.

fuzzy method descriptor

fuzzy(
    query: str, max_distance: int, limit: int | None = None
) -> list[tuple[str, int]]

(key, id) pairs within Levenshtein edit distance max_distance of query.

get method descriptor

get(key: str, default: _T | None = None) -> int | _T | None

Dense id of key, or default (None unless given) — idx.get("apple", -1).

id method descriptor

id(key: str) -> int | None

Id of key, or None if absent.

ids_into method descriptor

ids_into(keys: Sequence[str], out: Buffer) -> None

ids_of_bytes written into memory the caller owns instead of a fresh bytes per call: out is any writable C-contiguous buffer of ID_DTYPE items — np.empty(len(keys), dtype=index.ID_DTYPE) is the usual one — so a hot loop can reuse one array. The first len(keys) items are written; the rest are left as they were. With no keys nothing is written and out need only be a writable buffer.

A read-only, strided or mistyped buffer is a BufferError (a uint32 array handed to this index is refused rather than half-filled); one shorter than keys is a ValueError.

ids_into_arrow method descriptor

ids_into_arrow(column: object, out: Buffer) -> None

ids_of_arrow written into memory the caller owns, as ids_into does for a list: out is a writable C-contiguous buffer of ID_DTYPE items at least as long as the column.

ids_of method descriptor

ids_of(keys: Sequence[str]) -> list[int | None]

Batched id: one call for many keys, looping in Rust to amortise the Python↔Rust boundary. Returns a list aligned with keys, None where a key is absent. (Named ids_of, not ids/keys, so the class is not mistaken for a mapping by dict(index).)

ids_of_arrow method descriptor

ids_of_arrow(column: object) -> bytes

Batched id over an Arrow utf8 / large_utf8 column — a pyarrow Array or ChunkedArray, a pandas column of ArrowDtype, a polars Series — packed like ids_of_bytes: one ID_DTYPE item per element, MISSING_ID for an absent key and for a null. The keys are read from the column's offset and data buffers, so no Python string exists per key — building and borrowing those was half to two thirds of what the list forms cost.

ids_of_bytes method descriptor

ids_of_bytes(keys: Sequence[str]) -> bytes

Batched id packed into a bytes buffer instead of a list, for callers who hand the result to numpy or array rather than reading it item by item.

One 8-byte native-endian item per key, aligned with keys, MISSING_ID where a key is absent. np.frombuffer(buf, dtype=index.ID_DTYPE) shares the memory rather than copying it; ids_of has to build one Python int per key, which is what this avoids.

Native endianness — unlike the blobs, which are little-endian everywhere — because the buffer is meant for np.frombuffer on the machine that produced it, not for the wire.

key method descriptor

key(id: int) -> str | None

Key for id, or None if out of range.

keys_of method descriptor

keys_of(ids: Sequence[int]) -> list[str | None]

Batched key: one call for many ids. Returns a list aligned with ids, None where an id is out of range.

load staticmethod

load(path: str | PathLike[str]) -> StringIndex

Load an index previously written with save.

load_mmap staticmethod

load_mmap(path: str | PathLike[str]) -> StringIndex

Zero-copy load: memory-map the file and borrow the index from it — no read into RAM, so a multi-gigabyte index is ready instantly and its pages are shared across processes.

The mapped file must not be modified or truncated by any process while the index is alive: the bytes are borrowed, not copied, so a concurrent write is undefined behaviour rather than a stale answer. Python cannot express that obligation in the type system the way the Rust API does (where this is an unsafe fn), so it is the caller's contract. Use load if the file may change.

load_mmap_untrusted staticmethod

load_mmap_untrusted(
    path: str | PathLike[str],
) -> StringIndex

load_mmap for a file someone else wrote and too large to copy: the validation of from_untrusted_bytes over the mapping, pages shared, nothing copied.

The same obligation as load_mmap, and it weighs more here: the validation reads the mapping once and trusts what it saw, so a file that changes afterwards — the stranger's, if they can still write it — is exactly what the obligation forbids. Map a copy you own.

load_mmap_verified staticmethod

load_mmap_verified(
    path: str | PathLike[str],
) -> StringIndex

load_mmap plus the checksum load makes: one pass over the mapping at load, pages still shared, nothing copied. For a file you wrote but did not carry yourself.

The same obligation as load_mmap: the file must not change while the index is alive. The checksum is computed once, at load, and says nothing about later.

load_untrusted staticmethod

load_untrusted(path: str | PathLike[str]) -> StringIndex

load for a file someone else wrote: the bytes go through from_untrusted_bytes, whose validation and cost this inherits. For a file too large to read into memory there is load_mmap_untrusted, under load_mmap's obligation.

longest_prefix method descriptor

longest_prefix(query: str) -> tuple[str, int] | None

The longest key that is a prefix of query, or None -- the match a longest-match tokeniser takes. See common_prefix.

lower_bound method descriptor

lower_bound(query: str) -> int

How many keys sort below query -- equivalently the id of the first key >= query, and the position query would be inserted at. Defined for any string, present or not, and never larger than len.

occurrences method descriptor

occurrences(text: str) -> list[tuple[int, int, int]]

Every key that occurs in text, as (start, end, id) with text[start:end] the key: starts ascending, and within a start shortest first. The whole-text form of common_prefix that a dictionary segmenter runs at every character, in one call a text rather than one a character, which is where a loop over characters spends its time. Offsets count characters, as a str is indexed; the empty key, if the index holds it, is not reported.

predecessor method descriptor

predecessor(query: str) -> tuple[str, int] | None

The largest (key, id) with key <= query, or None if every key is larger.

prefix method descriptor

prefix(
    prefix: str, limit: int | None = None
) -> list[tuple[str, int]]

(key, id) pairs whose key starts with prefix, lexicographically ordered. limit stops after that many matches, walking no further — what autocomplete wants, since it needs ten of them, not every match.

prefix_count method descriptor

prefix_count(prefix: str) -> int

How many keys start with prefix.

prefix_id_range method descriptor

prefix_id_range(prefix: str) -> tuple[int, int]

The contiguous (start, end) id range of the keys starting with prefix, half-open. Ids follow lexicographic order, so a prefix is a slice of the id space rather than a set of ids to test one at a time -- usable directly as a range() or a bitset window.

range method descriptor

range(
    lo: str, hi: str, limit: int | None = None
) -> list[tuple[str, int]]

(key, id) pairs with lo <= key < hi, lexicographically ordered. limit stops after that many matches.

range_count method descriptor

range_count(lo: str, hi: str) -> int

How many keys satisfy lo <= key < hi, without decoding any of them.

save method descriptor

save(path: str | PathLike[str]) -> None

Write the index to path.

serialized_len method descriptor

serialized_len() -> int

Length of the to_bytes blob in bytes, without producing it.

subsequence method descriptor

subsequence(
    query: str, limit: int | None = None
) -> list[tuple[str, int]]

(key, id) pairs whose key contains query as a subsequence. limit stops after that many matches.

successor method descriptor

successor(query: str) -> tuple[str, int] | None

The smallest (key, id) with key >= query, or None if every key is smaller.

to_bytes method descriptor

to_bytes() -> bytes

Serialise to a bytes blob.

DictIndex

lexindex.DictIndex

Ordered dictionary with the key stored for every id: exact string ↔ rank, 2.64 B/key on real words.

ID_DTYPE class-attribute

ID_DTYPE: str = 'uint64'

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.

MISSING_ID class-attribute

MISSING_ID: int = 18446744073709551615

int([x]) -> integer int(x, base=10) -> integer

Convert a number or string to an integer, or return 0 if no arguments are given. If x is a number, return x.int(). For floating-point numbers, this truncates towards zero.

If x is not a number or if base is given, then x must be a string, bytes, or bytearray instance representing an integer literal in the given base. The literal can be preceded by '+' or '-' and be surrounded by whitespace. The base defaults to 10. Valid bases are 0 and 2-36. Base 0 means to interpret the base from the string as an integer iteral.

int('0b100', base=0) 4

__doc__ class-attribute

__doc__ = "Ordered dictionary with the key stored for every id: exact `string ↔ rank`, 2.64 B/key on real\nwords."

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.

__module__ class-attribute

__module__ = 'lexindex._core'

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.

block property

block: int

Keys per block, as given at build time.

__contains__ method descriptor

__contains__(key: str) -> bool

Return bool(key in self).

__getitem__ method descriptor

__getitem__(key: str) -> int

Return self[key].

__iter__ method descriptor

__iter__() -> Iterator[tuple[str, int]]

Implement iter(self).

__len__ method descriptor

__len__() -> int

Return len(self).

__new__ builtin

__new__(*args, **kwargs) -> DictIndex

Create and return a new object. See help(type) for accurate signature.

__reduce__ method descriptor

__reduce__()

Pickle support: the blob, and the loader that reads it back.

build_to_file staticmethod

build_to_file(
    items: Iterable[str],
    path: str | PathLike[str],
    block: int
    | Literal["fast", "balanced", "compact"] = 256,
) -> int

The constructor for a corpus that does not fit in memory, written straight to path: the keys, in any order, are sorted in runs that spill beside the output and merged back, and the block data goes into the file as it is encoded. The bytes are exactly what the constructor followed by save would have written. Returns the number of distinct keys.

What is still held is the block heads, the per-block arrays and one start a microblock, about (mean head length + 20) / block + 8 / micro bytes per key -- so a bigger block is what a corpus whose heads crowd memory wants. block takes a number or a name, as the constructor's does. If the iterable raises, the build is abandoned with path untouched.

common_prefix method descriptor

common_prefix(query: str) -> list[tuple[str, int]]

Every key that is a prefix of query, shortest first -- the reverse of prefix, which returns the keys query starts. This is the dictionary-matching query: given a vocabulary and a position in a sentence, it returns the entries that start there. The empty key, if the index holds it, is a prefix of everything and comes first.

from_bytes staticmethod

from_bytes(data: bytes) -> DictIndex

Reconstruct from a to_bytes blob. Every length, both checksums, the symbol table and the per-block arrays are validated, so arbitrary input raises rather than misbehaving.

get method descriptor

get(key: str, default: _T | None = None) -> int | _T | None

Rank of key, or default (None unless given).

id method descriptor

id(key: str) -> int | None

Rank of key, or None if absent.

ids_into method descriptor

ids_into(keys: Sequence[str], out: Buffer) -> None

ids_of_bytes written into memory the caller owns: out is any writable C-contiguous buffer of ID_DTYPE items at least len(keys) long; the first len(keys) items are written and the rest left as they were. A read-only, strided or mistyped buffer is a BufferError; one shorter than keys is a ValueError.

ids_into_arrow method descriptor

ids_into_arrow(column: object, out: Buffer) -> None

ids_of_arrow written into memory the caller owns, as ids_into does for a list: out is a writable C-contiguous buffer of ID_DTYPE items at least as long as the column.

ids_of method descriptor

ids_of(keys: Sequence[str]) -> list[int | None]

Batched id: one call for many keys, aligned with keys, None where a key is absent.

ids_of_arrow method descriptor

ids_of_arrow(column: object) -> bytes

Batched id over an Arrow utf8 / large_utf8 column — a pyarrow Array or ChunkedArray, a pandas column of ArrowDtype, a polars Series — packed like ids_of_bytes: one ID_DTYPE item per element, MISSING_ID for an absent key and for a null. The keys are read from the column's offset and data buffers, so no Python string exists per key.

ids_of_bytes method descriptor

ids_of_bytes(keys: Sequence[str]) -> bytes

Batched id packed into a bytes buffer instead of a list: one 8-byte native-endian item per key, aligned with keys, MISSING_ID where a key is absent, for np.frombuffer(buf, dtype=index.ID_DTYPE).

key method descriptor

key(id: int) -> str | None

Key at rank id, or None past the end.

keys_of method descriptor

keys_of(ids: Sequence[int]) -> list[str | None]

Batched key: a list aligned with ids, None where an id is out of range. Ids that ascend within one block are answered by a single walk of it, so the range prefix_id_range returns costs one pass rather than one per id.

load staticmethod

load(path: str | PathLike[str]) -> DictIndex

Load a file written with save, validated like from_bytes.

load_mmap staticmethod

load_mmap(path: str | PathLike[str]) -> DictIndex

Zero-copy load: memory-map the file and borrow the keys, the block data and the two offset arrays from it — the header, the symbol table and the per-block samples (eight bytes a block, built from the heads rather than stored) are what the load holds.

The mapped file must not be modified or truncated by any process while the index is alive — the bytes are borrowed, so a concurrent write is undefined behaviour. See StringIndex.load_mmap for the full contract; use load if the file may change.

load_mmap_verified staticmethod

load_mmap_verified(path: str | PathLike[str]) -> DictIndex

load_mmap plus the checks load makes — the payload checksum and the walk over the per-block arrays — one pass over the mapping at load, the keys and the block data still borrowed. For a file you wrote but did not carry yourself.

The same obligation as load_mmap: the file must not change while the index is alive. The checks run once, at load, and say nothing about later.

longest_prefix method descriptor

longest_prefix(query: str) -> tuple[str, int] | None

The longest key that is a prefix of query, or None -- the match a longest-match tokeniser takes. See common_prefix.

lower_bound method descriptor

lower_bound(key: str) -> int

The rank of the first key not below key: its own id if it is a member, otherwise the id it would have, len(self) past every key. Two of these bound a range of keys as a range of ids.

predecessor method descriptor

predecessor(query: str) -> tuple[str, int] | None

The largest (key, id) with key <= query, or None if every key is larger.

prefix method descriptor

prefix(
    prefix: str, limit: int | None = None
) -> list[tuple[str, int]]

(key, id) pairs whose key starts with prefix, lexicographically ordered. limit stops after that many matches, walking no further -- what autocomplete wants. A sorted dictionary answers this as a range, so nothing is scanned that is not returned.

prefix_count method descriptor

prefix_count(prefix: str) -> int

How many keys start with prefix, without decoding any of them.

prefix_id_range method descriptor

prefix_id_range(prefix: str) -> tuple[int, int]

The contiguous (start, end) id range of the keys starting with prefix, half-open. Ids follow lexicographic order, so a prefix is a slice of the id space rather than a set of ids to test one at a time -- usable directly as a range() or a bitset window.

range method descriptor

range(
    lo: str, hi: str, limit: int | None = None
) -> list[tuple[str, int]]

(key, id) pairs with lo <= key < hi, lexicographically ordered. limit stops after that many matches.

range_count method descriptor

range_count(lo: str, hi: str) -> int

How many keys satisfy lo <= key < hi, without decoding any of them.

route_microblocks method descriptor

route_microblocks() -> int

Derive the words that send a lookup straight to its microblock, once, and return the bytes they take.

Every lookup -- id, ids_of, lower_bound and the order queries -- walks one block's restart run to the microblock it scans. Routed, it compares eight bytes of every restart at once and walks the run only where the query's eight bytes tie one. Against the same index unrouted, at the default block, id took 19-31 % less time on twelve corpora of thirteen at a million keys and 13-22 % on six at ten million; paths, whose restarts tie, did not move.

The words are 8 / micro bytes a key -- 0.5 at the default block, 0.25 at 1024 -- held beside the index and in no blob, so a routed index takes that much more memory than serialized_len() says. Deriving them reads the start of every block, which a load_mmap index otherwise leaves on disk until a query needs it. A second call derives nothing.

save method descriptor

save(path: str | PathLike[str]) -> None

Write the index to path.

serialized_len method descriptor

serialized_len() -> int

Length of the to_bytes blob in bytes, without producing it.

successor method descriptor

successor(query: str) -> tuple[str, int] | None

The smallest (key, id) with key >= query, or None if every key is smaller.

to_bytes method descriptor

to_bytes() -> bytes

Serialise to a bytes blob.

HashedDictIndex

lexindex.HashedDictIndex

A DictIndex with a hash sidecar answering id(key): the dictionary's ranks at a hash index's lookup cost, and its ordered queries through .dict.

ID_DTYPE class-attribute

ID_DTYPE: str = 'uint64'

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.

MISSING_ID class-attribute

MISSING_ID: int = 18446744073709551615

int([x]) -> integer int(x, base=10) -> integer

Convert a number or string to an integer, or return 0 if no arguments are given. If x is a number, return x.int(). For floating-point numbers, this truncates towards zero.

If x is not a number or if base is given, then x must be a string, bytes, or bytearray instance representing an integer literal in the given base. The literal can be preceded by '+' or '-' and be surrounded by whitespace. The base defaults to 10. Valid bases are 0 and 2-36. Base 0 means to interpret the base from the string as an integer iteral.

int('0b100', base=0) 4

__doc__ class-attribute

__doc__ = "A `DictIndex` with a hash sidecar answering `id(key)`: the dictionary's ranks at a hash index's\nlookup cost, and its ordered queries through `.dict`."

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.

__module__ class-attribute

__module__ = 'lexindex._core'

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.

dict property

dict: DictIndex

The dictionary the sidecar rides on, sharing its memory: key, prefix, lower_bound, range, iteration and the exact id, with this index's ids.

fingerprint_bits property

fingerprint_bits: int

Width of the stored fingerprints in bits: from one up, the false-positive rate of id is 2 ** -fingerprint_bits; at zero, id is exact.

__contains__ method descriptor

__contains__(key: str) -> bool

Return bool(key in self).

__getitem__ method descriptor

__getitem__(key: str) -> int

Return self[key].

__len__ method descriptor

__len__() -> int

Return len(self).

__reduce__ method descriptor

__reduce__()

Pickle support: the blob, and the loader that reads it back.

contains method descriptor

contains(key: str) -> bool

Whether key is present, under the same contract as id.

from_bytes staticmethod

from_bytes(data: bytes) -> HashedDictIndex

Reconstruct from a to_bytes blob. Every length and every checksum, the dictionary's included, is validated, so arbitrary input raises rather than misbehaving.

from_dict staticmethod

from_dict(
    dict: DictIndex, fingerprint_bits: int
) -> HashedDictIndex

Build the sidecar over dict, storing fingerprint_bits (0..=32) of a second hash beside each key's rank. From one bit up, id answers a non-member with some rank at 2 ** -fingerprint_bits; at zero it is the dictionary's own exact search, and the sidecar serves id_unchecked. The dictionary is shared with dict, not copied.

get method descriptor

get(key: str, default: _T | None = None) -> int | _T | None

Rank of key, or default (None unless given).

id method descriptor

id(key: str) -> int | None

Rank of key, or None: through the sidecar from one fingerprint bit up, and a non-member answered with a rank at 2 ** -fingerprint_bits; exact at zero bits.

id_unchecked method descriptor

id_unchecked(key: str) -> int

Rank of key without any membership check -- key must be a member, or the result is some rank below len(). The fastest lookup for a closed vocabulary, at any width.

ids_into method descriptor

ids_into(keys: Sequence[str], out: Buffer) -> None

ids_of_bytes written into memory the caller owns: out is any writable C-contiguous buffer of ID_DTYPE items at least len(keys) long. A read-only, strided or mistyped buffer is a BufferError; one shorter than keys is a ValueError.

ids_into_arrow method descriptor

ids_into_arrow(column: object, out: Buffer) -> None

ids_of_arrow written into memory the caller owns, as ids_into does for a list.

ids_of method descriptor

ids_of(keys: Sequence[str]) -> list[int | None]

Batched id: one call for many keys, aligned with keys, None where a key is absent.

ids_of_arrow method descriptor

ids_of_arrow(column: object) -> bytes

Batched id over an Arrow utf8 / large_utf8 column, packed like ids_of_bytes: one ID_DTYPE item per element, MISSING_ID for an absent key and for a null. The keys are read from the column's offset and data buffers, so no Python string exists per key.

ids_of_bytes method descriptor

ids_of_bytes(keys: Sequence[str]) -> bytes

Batched id packed into a bytes buffer instead of a list: one 8-byte native-endian item per key, aligned with keys, MISSING_ID where a key is absent, for np.frombuffer(buf, dtype=index.ID_DTYPE).

load staticmethod

load(path: str | PathLike[str]) -> HashedDictIndex

Load a file written with save, validated like from_bytes.

load_mmap staticmethod

load_mmap(path: str | PathLike[str]) -> HashedDictIndex

Zero-copy load: memory-map the file and borrow the rank table and the dictionary from it, as DictIndex.load_mmap does; the perfect hash and the dictionary's per-block samples are what the load holds.

The mapped file must not be modified or truncated by any process while the index is alive -- the bytes are borrowed, so a concurrent write is undefined behaviour. See StringIndex.load_mmap for the full contract; use load if the file may change.

load_mmap_verified staticmethod

load_mmap_verified(
    path: str | PathLike[str],
) -> HashedDictIndex

load_mmap plus the checks load makes: one pass over the mapping at load, the bulk still borrowed. For a file you wrote but did not carry yourself.

The same obligation as load_mmap: the file must not change while the index is alive. The checks run once, at load, and say nothing about later.

save method descriptor

save(path: str | PathLike[str]) -> None

Write the index to path.

serialized_len method descriptor

serialized_len() -> int

Length of the to_bytes blob in bytes, without producing it.

to_bytes method descriptor

to_bytes() -> bytes

Serialise to a bytes blob, the dictionary's own blob inside it.

CompactHashIndex

lexindex.CompactHashIndex

Fingerprint minimal-perfect-hash dictionary: the smallest string -> dense id map that can reject a non-member. Membership is probabilistic (false-positive rate 2 ** -fingerprint_bits) and there is no reverse id -> key.

ID_DTYPE class-attribute

ID_DTYPE: str = 'uint32'

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.

MISSING_ID class-attribute

MISSING_ID: int = 4294967295

int([x]) -> integer int(x, base=10) -> integer

Convert a number or string to an integer, or return 0 if no arguments are given. If x is a number, return x.int(). For floating-point numbers, this truncates towards zero.

If x is not a number or if base is given, then x must be a string, bytes, or bytearray instance representing an integer literal in the given base. The literal can be preceded by '+' or '-' and be surrounded by whitespace. The base defaults to 10. Valid bases are 0 and 2-36. Base 0 means to interpret the base from the string as an integer iteral.

int('0b100', base=0) 4

__doc__ class-attribute

__doc__ = "Fingerprint minimal-perfect-hash dictionary: the smallest `string -> dense id` map that can\nreject a non-member. Membership is\nprobabilistic (false-positive rate `2 ** -fingerprint_bits`) and there is no reverse `id -> key`."

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.

__module__ class-attribute

__module__ = 'lexindex._core'

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.

fingerprint_bits property

fingerprint_bits: int

Width of the stored fingerprints in bits; the false-positive rate is 2 ** -fingerprint_bits.

__contains__ method descriptor

__contains__(key: str) -> bool

Return bool(key in self).

__getitem__ method descriptor

__getitem__(key: str) -> int

Return self[key].

__len__ method descriptor

__len__() -> int

Return len(self).

__new__ builtin

__new__(*args, **kwargs) -> CompactHashIndex

Create and return a new object. See help(type) for accurate signature.

__reduce__ method descriptor

__reduce__()

Pickle support: the blob, and the loader that reads it back.

An index pickles by copying its bytes, including one opened with load_mmap, whose pages are borrowed from a file the unpickling process may not have — a path would not survive a spawned worker on another machine, and a borrowed mapping would not survive the file changing. Pickling a large index therefore costs its serialised size in the pickle; save + load_mmap is what to use when both ends can see the same file.

build_to_file staticmethod

build_to_file(
    items: Iterable[str],
    path: str | PathLike[str],
    fingerprint_bytes: int = 1,
    *,
    fingerprint_bits: int | None = None,
) -> int

The constructor for a corpus that does not fit in memory, written straight to path: the keys, in any order, are hashed as they come and their 16-byte pairs sorted in runs that spill beside the output; the perfect hash is built from the merged runs a chunk at a time and the fingerprints written at their slots, so neither the corpus, its hashes nor the finished table is ever held whole. The file is byte for byte what the constructor and save write. Returns the number of distinct keys written. Widths as in the constructor.

contains method descriptor

contains(key: str) -> bool

Whether key is present (subject to the false-positive rate).

from_bytes staticmethod

from_bytes(data: bytes) -> CompactHashIndex

Reconstruct from a to_bytes blob.

Every length the index will read is validated against the bytes present, so arbitrary input raises rather than misbehaving. A blob written before 1.0 is refused: its perfect hash came from a crate this version no longer links, and this index stores no keys, so the only route is to rebuild it from them.

get method descriptor

get(key: str, default: _T | None = None) -> int | _T | None

Dense id of key, or default (None unless given) — idx.get("apple", -1).

id method descriptor

id(key: str) -> int | None

Dense id of key (membership checked against the fingerprint), or None.

id_unchecked method descriptor

id_unchecked(key: str) -> int

Dense id of key without the fingerprint check — key must be a member, or the result is an arbitrary valid slot. Fastest lookup for a fixed vocabulary.

ids_into method descriptor

ids_into(keys: Sequence[str], out: Buffer) -> None

ids_of_bytes written into memory the caller owns instead of a fresh bytes per call: out is any writable C-contiguous buffer of ID_DTYPE items — np.empty(len(keys), dtype=index.ID_DTYPE) is the usual one — so a hot loop can reuse one array. The first len(keys) items are written; the rest are left as they were. With no keys nothing is written and out need only be a writable buffer.

A read-only, strided or mistyped buffer is a BufferError (a uint64 array handed to this index is refused rather than half-filled); one shorter than keys is a ValueError.

ids_into_arrow method descriptor

ids_into_arrow(column: object, out: Buffer) -> None

ids_of_arrow written into memory the caller owns, as ids_into does for a list: out is a writable C-contiguous buffer of ID_DTYPE items at least as long as the column.

ids_of method descriptor

ids_of(keys: Sequence[str]) -> list[int | None]

Batched id: one call for many keys, aligned with keys (None where absent).

ids_of_arrow method descriptor

ids_of_arrow(column: object) -> bytes

Batched id over an Arrow utf8 / large_utf8 column — a pyarrow Array or ChunkedArray, a pandas column of ArrowDtype, a polars Series — packed like ids_of_bytes: one ID_DTYPE item per element, MISSING_ID for an absent key and for a null. The keys are read from the column's offset and data buffers, so no Python string exists per key — building and borrowing those was half to two thirds of what the list forms cost.

ids_of_bytes method descriptor

ids_of_bytes(keys: Sequence[str]) -> bytes

Batched id packed into a bytes buffer instead of a list, for callers who hand the result to numpy or array rather than reading it item by item.

One 4-byte native-endian item per key, aligned with keys, MISSING_ID where a key is absent. np.frombuffer(buf, dtype=index.ID_DTYPE) shares the memory rather than copying it; ids_of has to build one Python int per key, which is what this avoids.

Native endianness — unlike the blobs, which are little-endian everywhere — because the buffer is meant for np.frombuffer on the machine that produced it, not for the wire.

load staticmethod

load(path: str | PathLike[str]) -> CompactHashIndex

Load a dictionary previously written with save. Validated like from_bytes.

load_mmap staticmethod

load_mmap(path: str | PathLike[str]) -> CompactHashIndex

Zero-copy load: memory-map the file and borrow the fingerprint table.

The mapped file must not be modified or truncated by any process while the dictionary is alive — the bytes are borrowed, so a concurrent write is undefined behaviour. See StringIndex.load_mmap for the full contract; use load if the file may change.

load_mmap_verified staticmethod

load_mmap_verified(
    path: str | PathLike[str],
) -> CompactHashIndex

load_mmap plus the payload checksum load makes: one pass over the mapping at load, the fingerprint table still borrowed. For a file you wrote but did not carry yourself.

The same obligation as load_mmap: the file must not change while the index is alive. The checksum is computed once, at load, and says nothing about later.

save method descriptor

save(path: str | PathLike[str]) -> None

Write the dictionary to path.

serialized_len method descriptor

serialized_len() -> int

Length of the to_bytes blob in bytes, without producing it.

to_bytes method descriptor

to_bytes() -> bytes

Serialise to a bytes blob.

ClosedHashIndex

lexindex.ClosedHashIndex

Minimal perfect hash and nothing else: string -> dense id for a vocabulary known to be closed. id never says "absent" -- a member's id, or some id in [0, n) for any other string -- and the index is the perfect hash alone, about 0.24 bytes per key.

ID_DTYPE class-attribute

ID_DTYPE: str = 'uint32'

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.

__doc__ class-attribute

__doc__ = 'Minimal perfect hash and nothing else: `string -> dense id` for a vocabulary known to be closed.\n`id` never says "absent" -- a member\'s id, or some id in `[0, n)` for any other string -- and the\nindex is the perfect hash alone, about 0.24 bytes per key.'

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.

__module__ class-attribute

__module__ = 'lexindex._core'

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.

__len__ method descriptor

__len__() -> int

Return len(self).

__new__ builtin

__new__(*args, **kwargs) -> ClosedHashIndex

Create and return a new object. See help(type) for accurate signature.

__reduce__ method descriptor

__reduce__()

Pickle support: the blob, and the loader that reads it back.

from_bytes staticmethod

from_bytes(data: bytes) -> ClosedHashIndex

Reconstruct from a to_bytes blob. Every length the index will read is validated against the bytes present, so arbitrary input raises rather than misbehaving.

id method descriptor

id(key: str) -> int

Dense id of key if it is a member; some id in [0, n) otherwise, and 0 for an empty index. Nothing stored can tell the two apart, so nothing tries: every query must be a member by construction. There is no __contains__ and no __getitem__ -- a membership test this index cannot make would be a lie in the dict spelling.

ids_into method descriptor

ids_into(keys: Sequence[str], out: Buffer) -> None

ids_of_bytes written into memory the caller owns: out is any writable C-contiguous buffer of ID_DTYPE items at least len(keys) long; the first len(keys) items are written and the rest left as they were. A read-only, strided or mistyped buffer is a BufferError; one shorter than keys is a ValueError.

ids_into_arrow method descriptor

ids_into_arrow(column: object, out: Buffer) -> None

ids_of_arrow written into memory the caller owns, as ids_into does for a list: out is a writable C-contiguous buffer of ID_DTYPE items at least as long as the column.

ids_of method descriptor

ids_of(keys: Sequence[str]) -> list[int]

Batched id: one call for many keys, aligned with keys.

ids_of_arrow method descriptor

ids_of_arrow(column: object) -> bytes

Batched id over an Arrow utf8 / large_utf8 column — a pyarrow Array or ChunkedArray, a pandas column of ArrowDtype, a polars Series — packed like ids_of_bytes: one ID_DTYPE item per element, a ValueError for an absent key and for a null. The keys are read from the column's offset and data buffers, so no Python string exists per key — building and borrowing those was half to two thirds of what the list forms cost.

ids_of_bytes method descriptor

ids_of_bytes(keys: Sequence[str]) -> bytes

Batched id packed into a bytes buffer instead of a list: one 4-byte native-endian item per key, aligned with keys, for np.frombuffer(buf, dtype=index.ID_DTYPE). No item stands for an absent key, because this index never reports one.

load staticmethod

load(path: str | PathLike[str]) -> ClosedHashIndex

Load a file written with save. Validated like from_bytes. There is no load_mmap: the whole blob is the perfect hash, which is read into memory whichever way it is loaded.

save method descriptor

save(path: str | PathLike[str]) -> None

Write the index to path.

serialized_len method descriptor

serialized_len() -> int

Length of the to_bytes blob in bytes, without producing it.

to_bytes method descriptor

to_bytes() -> bytes

Serialise to a bytes blob.

PerfectHashIndex

lexindex.PerfectHashIndex

Minimal-perfect-hash dictionary: exact string → dense id with reverse lookup and persistence.

ID_DTYPE class-attribute

ID_DTYPE: str = 'uint32'

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.

MISSING_ID class-attribute

MISSING_ID: int = 4294967295

int([x]) -> integer int(x, base=10) -> integer

Convert a number or string to an integer, or return 0 if no arguments are given. If x is a number, return x.int(). For floating-point numbers, this truncates towards zero.

If x is not a number or if base is given, then x must be a string, bytes, or bytearray instance representing an integer literal in the given base. The literal can be preceded by '+' or '-' and be surrounded by whitespace. The base defaults to 10. Valid bases are 0 and 2-36. Base 0 means to interpret the base from the string as an integer iteral.

int('0b100', base=0) 4

__doc__ class-attribute

__doc__ = "Minimal-perfect-hash dictionary: exact `string → dense id` with reverse lookup and persistence."

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.

__module__ class-attribute

__module__ = 'lexindex._core'

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.

__contains__ method descriptor

__contains__(key: str) -> bool

Return bool(key in self).

__getitem__ method descriptor

__getitem__(key: str) -> int

Return self[key].

__len__ method descriptor

__len__() -> int

Return len(self).

__new__ builtin

__new__(*args, **kwargs) -> PerfectHashIndex

Create and return a new object. See help(type) for accurate signature.

__reduce__ method descriptor

__reduce__()

Pickle support: the blob, and the loader that reads it back.

An index pickles by copying its bytes, including one opened with load_mmap, whose pages are borrowed from a file the unpickling process may not have — a path would not survive a spawned worker on another machine, and a borrowed mapping would not survive the file changing. Pickling a large index therefore costs its serialised size in the pickle; save + load_mmap is what to use when both ends can see the same file.

build_to_file staticmethod

build_to_file(
    source: Callable[[], Iterable[str]],
    path: str | PathLike[str],
    *,
    fingerprints: bool = False,
) -> int

Build straight to path without ever holding the keys, for a corpus that does not fit in memory. Returns the number of keys written.

source is a zero-argument callable returning an iterable of str, and it is called twice: the build hashes every key first and can only place a key once the perfect hash exists, so a one-shot generator cannot serve — hand it lambda: open(path)-style factories, never the iterable itself (that is a TypeError). Keys must be distinct; a repeated key, or a second pass that yields different keys, is refused with ValueError and path is left untouched. Runs with the GIL held throughout: every key comes from a Python iterator.

contains method descriptor

contains(key: str) -> bool

Whether key is present.

from_bytes staticmethod

from_bytes(data: bytes) -> PerfectHashIndex

Reconstruct from a to_bytes blob.

Every length the index will read is validated against the bytes present, so arbitrary input raises rather than misbehaving. A blob written before 1.0 is refused: its perfect hash came from a crate this version no longer links, and the index has to be rebuilt from its keys.

get method descriptor

get(key: str, default: _T | None = None) -> int | _T | None

Dense id of key, or default (None unless given) — idx.get("apple", -1).

has_fingerprints method descriptor

has_fingerprints() -> bool

Whether the index was built with fingerprints=True.

id method descriptor

id(key: str) -> int | None

Dense id of key (membership verified), or None if absent.

id_unchecked method descriptor

id_unchecked(key: str) -> int

Dense id of key without membership verification — key must be in the dictionary, or the result is an arbitrary valid slot. Fastest lookup for a fixed vocabulary.

ids_into method descriptor

ids_into(keys: Sequence[str], out: Buffer) -> None

ids_of_bytes written into memory the caller owns instead of a fresh bytes per call: out is any writable C-contiguous buffer of ID_DTYPE items — np.empty(len(keys), dtype=index.ID_DTYPE) is the usual one — so a hot loop can reuse one array. The first len(keys) items are written; the rest are left as they were. With no keys nothing is written and out need only be a writable buffer.

A read-only, strided or mistyped buffer is a BufferError (a uint64 array handed to this index is refused rather than half-filled); one shorter than keys is a ValueError.

ids_into_arrow method descriptor

ids_into_arrow(column: object, out: Buffer) -> None

ids_of_arrow written into memory the caller owns, as ids_into does for a list: out is a writable C-contiguous buffer of ID_DTYPE items at least as long as the column.

ids_of method descriptor

ids_of(keys: Sequence[str]) -> list[int | None]

Batched id: one call for many keys, aligned with keys (None where absent).

ids_of_arrow method descriptor

ids_of_arrow(column: object) -> bytes

Batched id over an Arrow utf8 / large_utf8 column — a pyarrow Array or ChunkedArray, a pandas column of ArrowDtype, a polars Series — packed like ids_of_bytes: one ID_DTYPE item per element, MISSING_ID for an absent key and for a null. The keys are read from the column's offset and data buffers, so no Python string exists per key — building and borrowing those was half to two thirds of what the list forms cost.

ids_of_bytes method descriptor

ids_of_bytes(keys: Sequence[str]) -> bytes

Batched id packed into a bytes buffer instead of a list, for callers who hand the result to numpy or array rather than reading it item by item.

One 4-byte native-endian item per key, aligned with keys, MISSING_ID where a key is absent. np.frombuffer(buf, dtype=index.ID_DTYPE) shares the memory rather than copying it; ids_of has to build one Python int per key, which is what this avoids.

Native endianness — unlike the blobs, which are little-endian everywhere — because the buffer is meant for np.frombuffer on the machine that produced it, not for the wire.

key method descriptor

key(id: int) -> str | None

Key for id, or None if out of range.

keys_of method descriptor

keys_of(ids: Sequence[int]) -> list[str | None]

Batched key: one call for many ids, aligned with ids (None where out of range).

load staticmethod

load(path: str | PathLike[str]) -> PerfectHashIndex

Load a dictionary previously written with save. Validated like from_bytes.

load_mmap staticmethod

load_mmap(path: str | PathLike[str]) -> PerfectHashIndex

Memory-map the file and borrow the key arena zero-copy (only the small MPH is read into RAM).

The mapped file must not be modified or truncated by any process while the dictionary is alive — the bytes are borrowed, so a concurrent write is undefined behaviour. See StringIndex.load_mmap for the full contract; use load if the file may change.

load_mmap_verified staticmethod

load_mmap_verified(
    path: str | PathLike[str],
) -> PerfectHashIndex

load_mmap plus the payload checksum load makes: one pass over the mapping at load, the key arena still borrowed. For a file you wrote but did not carry yourself.

The same obligation as load_mmap: the file must not change while the index is alive. The checksum is computed once, at load, and says nothing about later.

save method descriptor

save(path: str | PathLike[str]) -> None

Write the dictionary to path.

serialized_len method descriptor

serialized_len() -> int

Length of the to_bytes blob in bytes, without producing it.

to_bytes method descriptor

to_bytes() -> bytes

Serialise to a bytes blob.

DoubleArrayIndex

lexindex.DoubleArrayIndex

Character-wise double-array trie for dictionary matching: common_prefix, longest_prefix and occurrences at one load a character, with the ids StringIndex assigns to the same keys. No reverse lookup: a StringIndex or DictIndex over the same keys spells an id back.

__doc__ class-attribute

__doc__ = "Character-wise double-array trie for dictionary matching: `common_prefix`, `longest_prefix`\nand `occurrences` at one load a character, with the ids `StringIndex` assigns to the same keys.\nNo reverse lookup: a `StringIndex` or `DictIndex` over the same keys spells an id back."

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.

__module__ class-attribute

__module__ = 'lexindex._core'

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to 'utf-8'. errors defaults to 'strict'.

__contains__ method descriptor

__contains__(key: str) -> bool

Return bool(key in self).

__getitem__ method descriptor

__getitem__(key: str) -> int

Return self[key].

__len__ method descriptor

__len__() -> int

Return len(self).

__new__ builtin

__new__(*args, **kwargs) -> DoubleArrayIndex

Create and return a new object. See help(type) for accurate signature.

__reduce__ method descriptor

__reduce__()

Pickle support: the blob, and the loader that reads it back.

common_prefix method descriptor

common_prefix(query: str) -> list[tuple[str, int]]

Every key that is a prefix of query, shortest first. This is the dictionary-matching query: given a vocabulary and a position in a sentence, it returns the entries that start there. The empty key, if the index holds it, is a prefix of everything and comes first.

contains method descriptor

contains(key: str) -> bool

Whether key is present.

from_bytes staticmethod

from_bytes(data: bytes) -> DoubleArrayIndex

Reconstruct from a to_bytes blob. The framing and both checksums are validated and every slot is walked, so arbitrary input raises rather than misbehaving; a blob crafted to carry matching checksums answers wrong ids, never out-of-range ones.

get method descriptor

get(key: str, default: _T | None = None) -> int | _T | None

Dense id of key, or default (None unless given).

id method descriptor

id(key: str) -> int | None

Id of key, or None if absent.

ids_of method descriptor

ids_of(keys: Sequence[str]) -> list[int | None]

Batched id: one call for many keys, aligned with keys, None where a key is absent.

load staticmethod

load(path: str | PathLike[str]) -> DoubleArrayIndex

Load a file written with save, validated like from_bytes.

load_mmap staticmethod

load_mmap(path: str | PathLike[str]) -> DoubleArrayIndex

Zero-copy load: memory-map the file and read the slots and the code table where they lie. The payload checksum load verifies is skipped; the walk over every slot is not, since it is what lets a query read without a bounds check, so the load still reads the array once.

The mapped file must not be modified or truncated by any process while the index is alive: the bytes are borrowed, not copied, so a concurrent write is undefined behaviour rather than a stale answer. Python cannot express that obligation in the type system the way the Rust API does (where this is an unsafe fn), so it is the caller's contract. Use load if the file may change.

longest_prefix method descriptor

longest_prefix(query: str) -> tuple[str, int] | None

The longest key that is a prefix of query, or None -- the match a longest-match tokeniser takes. See common_prefix.

occurrences method descriptor

occurrences(text: str) -> list[tuple[int, int, int]]

Every key that occurs in text, as (start, end, id) with text[start:end] the key: starts ascending, and within a start shortest first. The whole-text form of common_prefix that a dictionary segmenter runs at every character, in one call a text rather than one a character, which is where a loop over characters spends its time. Offsets count characters, as a str is indexed; the empty key, if the index holds it, is not reported.

save method descriptor

save(path: str | PathLike[str]) -> None

Write the index to path.

serialized_len method descriptor

serialized_len() -> int

Length of the to_bytes blob in bytes, without producing it.

to_bytes method descriptor

to_bytes() -> bytes

Serialise to a bytes blob.

Overlay

lexindex.Overlay

Add and remove keys on top of an index that is expensive to rebuild.

An index is built once from the whole key set, so adding one key has always meant rebuilding for the whole corpus. An overlay wraps one with the keys added since and the ids retired from it, leaving the base untouched and still usable.

Ids are stable: an id is never reissued, removing a key does not renumber anything, and re-adding a removed key revives its original id. compact is the one operation that renumbers.

id_space

id_space() -> int

How many ids have ever been issued; key is None at or above this.

__getitem__

__getitem__(key: str) -> int

Dense id of key, raising KeyError if it is absent — the dict spelling of id. There is no __setitem__ and no keys / values / items: this is an immutable str -> int lookup, not a mapping.

get

get(key: str, default: _T | None = None) -> int | _T | None

Dense id of key, or default (None unless given).

add

add(key: str) -> int

Add key and return its id, reviving the id a removed key used to have.

remove

remove(key: str) -> bool

Remove key, returning whether it was there.

Over a CompactHashIndex base this inherits that index's false-positive rate: a contains that was never true of a real key can retire an id. Remove by a key you know is present, or by the id itself with retire_id.

retire_id

retire_id(id: int) -> bool

Retire id itself, returning whether it was live.

The same tombstone remove sets, without the key lookup in front of it -- which over a CompactHashIndex base is the part a false positive corrupts. An application that stored the id it was given when the key was added has the exact id to retire and needs no membership test. An id at or above id_space, or one already retired, is False and changes nothing.

key

key(id: int) -> str | None

Key for id. Raises TypeError on a CompactHashIndex base, which stores no keys.

keys

keys() -> list[str]

Every live key. Raises TypeError on a CompactHashIndex base.

compact

compact() -> Overlay

Fold the edits into a fresh base. This renumbers: ids do not survive it. Raises TypeError on a CompactHashIndex base.

compact_to_file

compact_to_file(path: str | PathLike[str]) -> int

compact written straight to path as the base's own blob, without the live keys ever being held in memory at once; load it with the base class's load or load_mmap and wrap it in a new Overlay. Returns how many keys the file holds. Raises TypeError on a CompactHashIndex base.

compact_with_remap

compact_with_remap() -> tuple[Overlay, bytes]

compact, and the renumbering it did: one native-endian uint64 per id the overlay had issued (np.frombuffer(remap, dtype="uint64")), the new id of each old one, 2**64 - 1 where the id was retired. Raises TypeError on a CompactHashIndex base.

base

base() -> StringIndex | PerfectHashIndex | CompactHashIndex

The index underneath, unchanged and shared with this overlay.

save

save(path: str | PathLike[str]) -> None

Write to_bytes to path, atomically: a crash or a full disk leaves the previous file intact rather than a truncated one under the real name.

from_bytes staticmethod

from_bytes(
    data: bytes,
    base: type[StringIndex]
    | type[PerfectHashIndex]
    | type[CompactHashIndex],
) -> Overlay

Read a blob, rebuilding the base with base's own loader — pass the class itself.

The blob records which base wrote it and a mismatch is refused, so the wrong class is an error rather than an unchecked read of bytes meant for something else.

Validated and checksummed throughout. The header carries a check of its own and a hash of everything after it, verified before any of it is read, so a flipped bit anywhere raises ValueError rather than loading as a different key or a revived id. The framing is checked past the checksums too — the lengths, the additions and their UTF-8, the tombstones against the id space — and so is the base region, by that class's own loader.

A blob written by 0.12 still loads, without the two checksums it does not carry; saving it again writes the current format and it gains them.

from_untrusted_bytes staticmethod

from_untrusted_bytes(
    data: bytes,
    base: type[StringIndex]
    | type[PerfectHashIndex]
    | type[CompactHashIndex],
) -> Overlay

from_bytes for a blob someone else wrote.

The overlay's own framing is checked identically either way. What changes is the loader the embedded base is handed to, and it matters for exactly one base: a StringIndex region can panic from_bytes (see StringIndex.from_untrusted_bytes), and an overlay frame passes every check it makes for itself before that region is reached. Over the two hash bases this is the same work as from_bytes, whose loaders are already total.

load staticmethod

load(
    path: str | PathLike[str],
    base: type[StringIndex]
    | type[PerfectHashIndex]
    | type[CompactHashIndex],
) -> Overlay

from_bytes from a file: checksummed and validated the same way.

load_untrusted staticmethod

load_untrusted(
    path: str | PathLike[str],
    base: type[StringIndex]
    | type[PerfectHashIndex]
    | type[CompactHashIndex],
) -> Overlay

from_untrusted_bytes from a file.

lexindex.OverlayInfo

Bases: TypedDict

An overlay's own sections, and its base inspected in turn.

plan

lexindex.plan builtin

plan(
    keys: Iterable[str],
    *,
    reverse: bool = False,
    ordered: bool = False,
    prefix: bool = False,
    fuzzy: bool = False,
    exact: bool = False,
    objective: Literal["memory", "latency", "balanced"]
    | Workload = Ellipsis,
) -> Plan

What each index would cost on these keys, and which of them answer the caller's questions.

The keys are sorted once and priced from the statistics the formats are actually paid in. The three numbers no statistic gives -- what the symbol table squeezes a suffix into, the bytes an fst spends a key, the bits the perfect hash spends per key -- come from builds of two 100 000-key draws, so past that size every entry is a model and measured is False; below it the candidates are built and bytes is what they weigh.

A dense id space is the one corpus the fst estimate still misses badly: ten million decimal ids merge to 356 bytes whole, which no draw of a hundred thousand of them can see, and the number reads about a hundredfold high. It is still far under every other index, so the ranking picks the fst -- but thin is set, and a byte count under that flag is not one to quote.

The keyword arguments say what the index has to be able to do. One that cannot is left out of the ranking rather than ranked last, so estimates holds only usable answers and best is always one of them. close marks a pair the estimate cannot separate and thin a corpus whose suffixes are too short for a sampled compression ratio to carry; either one means build rather than trust the number. text is the whole thing as the paragraph the Rust Plan prints.

lexindex.Plan

Bases: TypedDict

What plan priced: the ranking, the shape of the corpus and the caveats.

lexindex.Estimate

Bases: TypedDict

What one index would weigh on the keys plan was given.

lexindex.Workload

Bases: TypedDict

How often each operation is asked of the index, for plan to rank by.

inspect

lexindex.inspect builtin

inspect(blob: str | PathLike[str] | bytes) -> BlobInfo

What a blob is, from its header alone: its kind, its format and the sizes a caller would otherwise have to load it to learn. blob is a path or bytes; over a path only the header and the footer are read, so an index of gigabytes inspects in microseconds (an overlay's tombstone words are read too, to count its retired ids). Nothing is decoded or verified -- a blob that inspects cleanly may still fail to load, and the sizes are what the header claims. A blob from before 1.0 is a ValueError naming the type to rebuild.

lexindex.BlobInfo

Bases: TypedDict

What inspect reads out of a blob's header; nothing in it is verified.