C D E F G M N R V

C

com.knuddels.jtokkit - package com.knuddels.jtokkit

com.knuddels.jtokkit.api - package com.knuddels.jtokkit.api

countTokens(String) - Method in interface com.knuddels.jtokkit.api.Encoding

Encodes the given text into a list of token ids and returns the amount of tokens.

countTokensOrdinary(String) - Method in interface com.knuddels.jtokkit.api.Encoding

Encodes the given text into a list of token ids and returns the amount of tokens.

D

decode(List<Integer>) - Method in interface com.knuddels.jtokkit.api.Encoding

Decodes the given list of token ids into a text.

decodeBytes(List<Integer>) - Method in interface com.knuddels.jtokkit.api.Encoding

Decodes the given list of token ids into a byte array.

E

encode(String) - Method in interface com.knuddels.jtokkit.api.Encoding

Encodes the given text into a list of token ids.

encodeOrdinary(String) - Method in interface com.knuddels.jtokkit.api.Encoding

Encodes the given text into a list of token ids, ignoring special tokens.

Encoding - Interface in com.knuddels.jtokkit.api

EncodingRegistry - Interface in com.knuddels.jtokkit.api

The EncodingRegistry is used to register custom encodings and to retrieve encodings by name or type.

Encodings - Class in com.knuddels.jtokkit

EncodingType - Enum in com.knuddels.jtokkit.api

encodingType - Variable in enum com.knuddels.jtokkit.api.ModelType

F

fromName(String) - Static method in enum com.knuddels.jtokkit.api.ModelType

Returns a ModelType for the given name, or Optional.empty() if no such model type exists.

G

getEncoder() - Method in class com.knuddels.jtokkit.api.GptBytePairEncodingParams

getEncoding(String) - Method in interface com.knuddels.jtokkit.api.EncodingRegistry

Returns the encoding with the given name, if it exists.

getEncoding(EncodingType) - Method in interface com.knuddels.jtokkit.api.EncodingRegistry

Returns the encoding with the given type.

getEncodingForModel(String) - Method in interface com.knuddels.jtokkit.api.EncodingRegistry

Returns the encoding that is used for the given model type, if it exists.

getEncodingForModel(ModelType) - Method in interface com.knuddels.jtokkit.api.EncodingRegistry

Returns the encoding that is used for the given model type.

getEncodingType() - Method in enum com.knuddels.jtokkit.api.ModelType

getName() - Method in interface com.knuddels.jtokkit.api.Encoding

Returns the name of this encoding.

getName() - Method in enum com.knuddels.jtokkit.api.EncodingType

getName() - Method in class com.knuddels.jtokkit.api.GptBytePairEncodingParams

getName() - Method in enum com.knuddels.jtokkit.api.ModelType

getPattern() - Method in class com.knuddels.jtokkit.api.GptBytePairEncodingParams

getSpecialTokensEncoder() - Method in class com.knuddels.jtokkit.api.GptBytePairEncodingParams

GptBytePairEncodingParams - Class in com.knuddels.jtokkit.api

Parameter for the byte pair encoding used to tokenize for the OpenAI GPT models.

GptBytePairEncodingParams(String, Pattern, Map<byte[], Integer>, Map<String, Integer>) - Constructor for class com.knuddels.jtokkit.api.GptBytePairEncodingParams

Creates a new instance of GptBytePairEncodingParams.

M

ModelType - Enum in com.knuddels.jtokkit.api

N

name - Variable in enum com.knuddels.jtokkit.api.ModelType

newDefaultEncodingRegistry() - Static method in class com.knuddels.jtokkit.Encodings

Creates a new EncodingRegistry with the default encodings found in the EncodingType enum.

R

registerCustomEncoding(Encoding) - Method in interface com.knuddels.jtokkit.api.EncodingRegistry

Registers a new custom encoding with the given name.

registerGptBytePairEncoding(GptBytePairEncodingParams) - Method in interface com.knuddels.jtokkit.api.EncodingRegistry

Registers a new byte pair encoding with the given name.

V

valueOf(String) - Static method in enum com.knuddels.jtokkit.api.EncodingType

Returns the enum constant of this type with the specified name.

valueOf(String) - Static method in enum com.knuddels.jtokkit.api.ModelType

Returns the enum constant of this type with the specified name.

values() - Static method in enum com.knuddels.jtokkit.api.EncodingType

Returns an array containing the constants of this enum type, in the order they are declared.

values() - Static method in enum com.knuddels.jtokkit.api.ModelType

Returns an array containing the constants of this enum type, in the order they are declared.

C D E F G M N R V