tokenizer

Tokenizer module for bi-encoder models.

This module contains the tokenizer class bi-encoder models.

Classes

BiEncoderTokenizer(*args[, query_expansion, ...])