Package com.yahoo.language.huggingface.config

Class HuggingFaceTokenizerConfig

java.lang.Object

com.yahoo.config.Node

com.yahoo.config.InnerNode

com.yahoo.config.ConfigInstance

com.yahoo.language.huggingface.config.HuggingFaceTokenizerConfig

public final class HuggingFaceTokenizerConfig
extends com.yahoo.config.ConfigInstance

This class represents the root node of hugging-face-tokenizer Copyright Vespa.ai. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final class	HuggingFaceTokenizerConfig.Builder
static final class	HuggingFaceTokenizerConfig.Model	This class represents hugging-face-tokenizer.model[]
static final class	HuggingFaceTokenizerConfig.Padding	This class represents hugging-face-tokenizer.padding Padding strategy.
static interface	HuggingFaceTokenizerConfig.Producer
static final class	HuggingFaceTokenizerConfig.Truncation	This class represents hugging-face-tokenizer.truncation Truncation strategy.

Field Summary

Fields

Modifier and Type	Field	Description
static final String	CONFIG_DEF_MD5
static final String	CONFIG_DEF_NAME
static final String	CONFIG_DEF_NAMESPACE
static final String[]	CONFIG_DEF_SCHEMA

Constructor Summary

Constructors

Constructor	Description
HuggingFaceTokenizerConfig(HuggingFaceTokenizerConfig.Builder builder)

Method Summary

Modifier and Type	Method	Description
boolean	addSpecialTokens()
static String	getDefMd5()
static String	getDefName()
static String	getDefNamespace()
int	maxLength()
List<HuggingFaceTokenizerConfig.Model>	model()
HuggingFaceTokenizerConfig.Model	model(int i)
HuggingFaceTokenizerConfig.Padding.Enum	padding()
HuggingFaceTokenizerConfig.Truncation.Enum	truncation()

Methods inherited from class com.yahoo.config.ConfigInstance

getDefName, getDefNamespace, serialize, serialize

Methods inherited from class com.yahoo.config.InnerNode

equals, getAllDescendantLeafNodes, getChildren, getChildrenWithVectorsFlattened, hashCode, postInitialize, toString

Methods inherited from class com.yahoo.config.Node

clone

Methods inherited from class java.lang.Object

finalize, getClass, notify, notifyAll, wait, wait, wait

Field Details

CONFIG_DEF_MD5

public static final String CONFIG_DEF_MD5

See Also:

• Constant Field Values

CONFIG_DEF_NAME

public static final String CONFIG_DEF_NAME

See Also:

• Constant Field Values

CONFIG_DEF_NAMESPACE

public static final String CONFIG_DEF_NAMESPACE

See Also:

• Constant Field Values

CONFIG_DEF_SCHEMA

public static final String[] CONFIG_DEF_SCHEMA

Constructor Details

HuggingFaceTokenizerConfig

public HuggingFaceTokenizerConfig(HuggingFaceTokenizerConfig.Builder builder)

Method Details

getDefMd5

public static String getDefMd5()

getDefName

public static String getDefName()

getDefNamespace

public static String getDefNamespace()

model

public List<HuggingFaceTokenizerConfig.Model> model()

Returns:

hugging-face-tokenizer.model[]

model

public HuggingFaceTokenizerConfig.Model model(int i)

Parameters:

i - the index of the value to return

Returns:

hugging-face-tokenizer.model[]

addSpecialTokens

public boolean addSpecialTokens()

Returns:

hugging-face-tokenizer.addSpecialTokens

maxLength

public int maxLength()

Returns:

hugging-face-tokenizer.maxLength

truncation

public HuggingFaceTokenizerConfig.Truncation.Enum truncation()

Returns:

hugging-face-tokenizer.truncation

padding

public HuggingFaceTokenizerConfig.Padding.Enum padding()

Returns:

hugging-face-tokenizer.padding