Class Tokenizer

Implementation of a tokenizer derived from here.

Used to convert a string into a sequence of numbers, where smaller numbers indicate more frequently occurring tokens.

internal

Hierarchy

Tokenizer

Index

Constructors

constructor

Properties

Methods

Constructors

constructor

new Tokenizer(vocabSize: number): Tokenizer

- Defined in packages/backend-ml/src/tokenizer.ts:59
Parameters
- vocabSize: number
  
  The maximum size of the vocabulary.
Returns Tokenizer

Properties

wordCounts

wordCounts: Map<string, number> = ...

A mapping between words (tokens) and their corresponding frequencies.

wordIndex

wordIndex: Map<string, number> = ...

A mapping between words (tokens) and their corresponding index.

Methods

fitOnTexts

fitOnTexts(texts: string[]): void

- Defined in packages/backend-ml/src/tokenizer.ts:112
Fits the tokenizer to a given set of strings. The frequency that tokens appear in these model strings will be used when generating sequences for unknown strings.

Parameters
- texts: string[]
  
  The strings to fit the tokenizer with.
Returns void

fromJSON

fromJSON(wordIndex: StringNumberMapping): void

- Defined in packages/backend-ml/src/tokenizer.ts:152
Loads a tokenizer from a given object.

Parameters
- wordIndex: StringNumberMapping
  
  The object to loads the tokenizer from.
Returns void

textToSequence

textToSequence(text: string): Sequence

- Defined in packages/backend-ml/src/tokenizer.ts:142
Tokenizes a given string.

Parameters
- text: string
  
  The string to tokenize.
Returns Sequence

The tokenized string.

toJSON

toJSON(): StringNumberMapping

- Defined in packages/backend-ml/src/tokenizer.ts:162
Converts the tokenizer to an object that can be JSON encoded.

Returns StringNumberMapping

The JSON encodable tokenizer object.

Static tokenize

tokenize(text: string): string[]

- Defined in packages/backend-ml/src/tokenizer.ts:81
Converts a string into an array of tokens.

The following rules are followed:
- Multiple spaces are collapsed into a single space.
- Special characters are removed.
- Numbers are replaced with a single token.
- URLs are replaced with a single token.
- Tokens are split at white space.
Parameters
- text: string
  
  The raw string.
Returns string[]

An array of tokens.

Generated using TypeDoc