tokenizer

Tokenizer module for Lightning IR.

This module contains the main tokenizer class for the Lightning IR library.

Classes

LightningIRTokenizer(*args[, query_length, ...])

Base class for Lightning IR tokenizers.