com.knuddels.jtokkit.api

Interface Encoding

public interface Encoding

Method Summary

Modifier and Type	Method and Description
int	countTokens(java.lang.String text) Encodes the given text into a list of token ids and returns the amount of tokens.
int	countTokensOrdinary(java.lang.String text) Encodes the given text into a list of token ids and returns the amount of tokens.
java.lang.String	decode(java.util.List<java.lang.Integer> tokens) Decodes the given list of token ids into a text.
byte[]	decodeBytes(java.util.List<java.lang.Integer> tokens) Decodes the given list of token ids into a byte array.
java.util.List<java.lang.Integer>	encode(java.lang.String text) Encodes the given text into a list of token ids.
java.util.List<java.lang.Integer>	encodeOrdinary(java.lang.String text) Encodes the given text into a list of token ids, ignoring special tokens.
java.lang.String	getName() Returns the name of this encoding.

Method Detail

encode

java.util.List<java.lang.Integer> encode(java.lang.String text)

Encodes the given text into a list of token ids.

Special tokens are artificial tokens used to unlock capabilities from a model, such as fill-in-the-middle. There is currently no support for parsing special tokens in a text, so if the text contains special tokens, this method will throw an UnsupportedOperationException.

If you want to encode special tokens as ordinary text, use encodeOrdinary(String).

 Encoding encoding = EncodingRegistry.getEncoding(EncodingType.CL100K_BASE);
 encoding.encode("hello world");
 // returns [15339, 1917]

 encoding.encode("hello <|endoftext|> world");
 // raises an UnsupportedOperationException
 

Parameters:

text - the text to encode

Returns:

the list of token ids

Throws:

java.lang.UnsupportedOperationException - if the text contains special tokens which are not supported for now

encodeOrdinary

java.util.List<java.lang.Integer> encodeOrdinary(java.lang.String text)

Encodes the given text into a list of token ids, ignoring special tokens.

This method does not throw an exception if the text contains special tokens, but instead encodes them as if they were ordinary text.

 Encoding encoding = EncodingRegistry.getEncoding(EncodingType.CL100K_BASE);
 encoding.encodeOrdinary("hello world");
 // returns [15339, 1917]

 encoding.encodeOrdinary("hello <|endoftext|> world");
 // returns [15339, 83739, 8862, 728, 428, 91, 29, 1917]
 

Parameters:

text - the text to encode

Returns:

the list of token ids

countTokens

int countTokens(java.lang.String text)

Encodes the given text into a list of token ids and returns the amount of tokens. This is a convenience method for encode(String), if all you want is to know the amount of tokens. It is not more performant than encode(String), so prefer to use encode(String) if you actually need the tokens.

 Encoding encoding = EncodingRegistry.getEncoding(EncodingType.CL100K_BASE);
 encoding.countTokens("hello world");
 // returns 2

 encoding.countTokens("hello <|endoftext|> world");
 // raises an UnsupportedOperationException
 

Parameters:

text - the text to count tokens for

Returns:

the amount of tokens

Throws:

java.lang.UnsupportedOperationException - if the text contains special tokens which are not supported for now

countTokensOrdinary

int countTokensOrdinary(java.lang.String text)

Encodes the given text into a list of token ids and returns the amount of tokens. This is a convenience method for encodeOrdinary(String), if all you want is to know the amount of tokens. It is not more performant than encodeOrdinary(String), so prefer to use encodeOrdinary(String) if you actually need the tokens.

 Encoding encoding = EncodingRegistry.getEncoding(EncodingType.CL100K_BASE);
 encoding.countTokensOrdinary("hello world");
 // returns 2

 encoding.countTokensOrdinary("hello <|endoftext|> world");
 // returns 8
 

Parameters:

text - the text to count tokens for

Returns:

the amount of tokens

Throws:

java.lang.UnsupportedOperationException - if the text contains special tokens which are not supported for now

decode

java.lang.String decode(java.util.List<java.lang.Integer> tokens)

Decodes the given list of token ids into a text.

 Encoding encoding = EncodingRegistry.getEncoding(EncodingType.CL100K_BASE);
 encoding.decode(List.of(15339, 1917));
 // returns "hello world"

 encoding.decode(List.of(15339, 1917, Integer.MAX_VALUE));
 // raises an IllegalArgumentException
 

Parameters:

tokens - the list of token ids

Returns:

the decoded text

Throws:

java.lang.IllegalArgumentException - if the list contains invalid token ids

decodeBytes

byte[] decodeBytes(java.util.List<java.lang.Integer> tokens)

Decodes the given list of token ids into a byte array.

 Encoding encoding = EncodingRegistry.getEncoding(EncodingType.CL100K_BASE);
 encoding.decodeBytes(List.of(15339, 1917));
 // returns [104, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100]

 encoding.decodeBytes(List.of(15339, 1917, Integer.MAX_VALUE));
 // raises an IllegalArgumentException
 

Parameters:

tokens - the list of token ids

Returns:

the decoded byte array

Throws:

java.lang.IllegalArgumentException - if the list contains invalid token ids

getName

java.lang.String getName()

Returns the name of this encoding. This is the name which is used to identify the encoding and must be unique for registration in the EncodingRegistry.

Returns:

the name of this encoding