Module org.elasticsearch.xcore

Package org.elasticsearch.xpack.core.ml.inference.trainedmodel

Class ByteLevelBpeTokenization

java.lang.Object

org.elasticsearch.xpack.core.ml.inference.trainedmodel.Tokenization

org.elasticsearch.xpack.core.ml.inference.trainedmodel.ByteLevelBpeTokenization

All Implemented Interfaces:

NamedWriteable, Writeable, ToXContent, ToXContentObject, NamedXContentObject

public class ByteLevelBpeTokenization
extends Tokenization

Tokenization settings for byte-level BPE models (for example GPT-2 style vocabularies) that ship a merge table alongside the vocabulary. The in-process tokenizer applies UTF-8 byte mapping and BPE merges using the same engine as the RoBERTa tokenizer, but special token strings are configurable so deployments can align with Hugging Face tokenizer metadata. do_lower_case is not applied by the byte-level BPE analyzer and must not be set to true; requests that set it to true are rejected like RobertaTokenization. When with_special_tokens is true, the configured BOS and EOS strings must be present in the vocabulary; that requirement is enforced when the inference tokenizer is constructed for the model (not only in validateVocabulary(org.elasticsearch.xpack.core.ml.action.PutTrainedModelVocabularyAction.Request)), consistent with RobertaTokenization. The configured mask string is optional for encoding-only inference, and the mask token id may be unset if that token is missing from the vocabulary (masking-oriented tasks should ensure it exists).

Nested Class Summary

Nested classes/interfaces inherited from class org.elasticsearch.xpack.core.ml.inference.trainedmodel.Tokenization

Tokenization.SpanSettings, Tokenization.Truncate

Nested classes/interfaces inherited from interface org.elasticsearch.xcontent.ToXContent

ToXContent.DelegatingMapParams, ToXContent.MapParams, ToXContent.Params

Nested classes/interfaces inherited from interface org.elasticsearch.common.io.stream.Writeable

Writeable.Reader<V>, Writeable.Writer<V>

Field Summary

Fields

Modifier and Type	Field	Description
static final TransportVersion	ML_BYTE_LEVEL_BPE_TOKENIZATION_ADDED	Tokenization wire format for byte-level BPE; do not send to nodes that do not support this version.
static final String	NAME	Name used in inference configuration XContent and named writeables.

Fields inherited from class org.elasticsearch.xpack.core.ml.inference.trainedmodel.Tokenization

DEFAULT_MAX_SEQUENCE_LENGTH, DO_LOWER_CASE, doLowerCase, MAX_SEQUENCE_LENGTH, maxSequenceLength, span, SPAN, truncate, TRUNCATE, UNSET_SPAN_VALUE, WITH_SPECIAL_TOKENS, withSpecialTokens

Fields inherited from interface org.elasticsearch.xcontent.ToXContent

EMPTY, EMPTY_PARAMS

Constructor Summary

Constructors

Constructor	Description
ByteLevelBpeTokenization(Boolean doLowerCase, Boolean withSpecialTokens, Integer maxSequenceLength, Tokenization.Truncate truncate, Integer span, Boolean addPrefixSpace, String unkToken, String padToken, String bosToken, String eosToken, String maskToken)
ByteLevelBpeTokenization(StreamInput in)

Method Summary

Modifier and Type	Method	Description
static ConstructingObjectParser<ByteLevelBpeTokenization,Void>	createParser(boolean ignoreUnknownFields)
boolean	equals(Object o)
static ByteLevelBpeTokenization	fromXContent(XContentParser parser, boolean lenient)
String	getBosToken()
String	getEosToken()
String	getMaskToken()
String	getName()
String	getPadToken()
String	getUnkToken()
String	getWriteableName()
int	hashCode()
boolean	isAddPrefixSpace()
void	validateVocabulary(PutTrainedModelVocabularyAction.Request request)
void	writeTo(StreamOutput out)

Methods inherited from class org.elasticsearch.xpack.core.ml.inference.trainedmodel.Tokenization

createDefault, doLowerCase, getMaxSequenceLength, getSpan, getTruncate, maxSequenceLength, toXContent, updateWindowSettings, validateSpanAndMaxSequenceLength, validateSpanAndTruncate, withSpecialTokens

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.elasticsearch.xcontent.ToXContentObject

isFragment

Field Details

NAME

public static final String NAME

Name used in inference configuration XContent and named writeables.

See Also:

• Constant Field Values

ML_BYTE_LEVEL_BPE_TOKENIZATION_ADDED

public static final TransportVersion ML_BYTE_LEVEL_BPE_TOKENIZATION_ADDED

Tokenization wire format for byte-level BPE; do not send to nodes that do not support this version.

Constructor Details

ByteLevelBpeTokenization

public ByteLevelBpeTokenization(@Nullable
 Boolean doLowerCase,
 @Nullable
 Boolean withSpecialTokens,
 @Nullable
 Integer maxSequenceLength,
 @Nullable
 Tokenization.Truncate truncate,
 @Nullable
 Integer span,
 @Nullable
 Boolean addPrefixSpace,
 @Nullable
 String unkToken,
 @Nullable
 String padToken,
 @Nullable
 String bosToken,
 @Nullable
 String eosToken,
 @Nullable
 String maskToken)

ByteLevelBpeTokenization

public ByteLevelBpeTokenization(StreamInput in)
                         throws IOException

Throws:

IOException

Method Details

createParser

public static ConstructingObjectParser<ByteLevelBpeTokenization,Void> createParser(boolean ignoreUnknownFields)

fromXContent

public static ByteLevelBpeTokenization fromXContent(XContentParser parser,
 boolean lenient)

isAddPrefixSpace

public boolean isAddPrefixSpace()

getUnkToken

public String getUnkToken()

getPadToken

public String getPadToken()

getBosToken

public String getBosToken()

getEosToken

public String getEosToken()

getWriteableName

public String getWriteableName()

writeTo

public void writeTo(StreamOutput out)
             throws IOException

Specified by:

writeTo in interface Writeable

Overrides:

writeTo in class Tokenization

Throws:

IOException

getMaskToken

public String getMaskToken()

Specified by:

getMaskToken in class Tokenization

getName

public String getName()

Returns:

The name of the XContentObject that is to be serialized

validateVocabulary

public void validateVocabulary(PutTrainedModelVocabularyAction.Request request)

Overrides:

validateVocabulary in class Tokenization

equals

public boolean equals(Object o)

Overrides:

equals in class Tokenization

hashCode

public int hashCode()

Overrides:

hashCode in class Tokenization