Module org.elasticsearch.xcore

Package org.elasticsearch.xpack.core.ml.inference.trainedmodel

Class RobertaTokenization

java.lang.Object

org.elasticsearch.xpack.core.ml.inference.trainedmodel.Tokenization

org.elasticsearch.xpack.core.ml.inference.trainedmodel.RobertaTokenization

All Implemented Interfaces:

NamedWriteable, Writeable, ToXContent, ToXContentObject, NamedXContentObject

public class RobertaTokenization
extends Tokenization

Nested Class Summary

Nested classes/interfaces inherited from class org.elasticsearch.xpack.core.ml.inference.trainedmodel.Tokenization

Tokenization.SpanSettings, Tokenization.Truncate

Nested classes/interfaces inherited from interface org.elasticsearch.xcontent.ToXContent

ToXContent.DelegatingMapParams, ToXContent.MapParams, ToXContent.Params

Nested classes/interfaces inherited from interface org.elasticsearch.common.io.stream.Writeable

Writeable.Reader<V>, Writeable.Writer<V>

Field Summary

Fields

Modifier and Type	Field	Description
static final String	MASK_TOKEN
static final String	NAME

Fields inherited from class org.elasticsearch.xpack.core.ml.inference.trainedmodel.Tokenization

DEFAULT_MAX_SEQUENCE_LENGTH, DO_LOWER_CASE, doLowerCase, MAX_SEQUENCE_LENGTH, maxSequenceLength, span, SPAN, truncate, TRUNCATE, UNSET_SPAN_VALUE, WITH_SPECIAL_TOKENS, withSpecialTokens

Fields inherited from interface org.elasticsearch.xcontent.ToXContent

EMPTY, EMPTY_PARAMS

Constructor Summary

Constructors

Constructor	Description
RobertaTokenization(Boolean withSpecialTokens, Boolean addPrefixSpace, Integer maxSequenceLength, Tokenization.Truncate truncate, Integer span)
RobertaTokenization(StreamInput in)

Method Summary

Modifier and Type	Method	Description
static ConstructingObjectParser<RobertaTokenization,Void>	createParser(boolean ignoreUnknownFields)
static RobertaTokenization	fromXContent(XContentParser parser, boolean lenient)
String	getMaskToken()
String	getName()
String	getWriteableName()
boolean	isAddPrefixSpace()
void	validateVocabulary(PutTrainedModelVocabularyAction.Request request)
void	writeTo(StreamOutput out)

Methods inherited from class org.elasticsearch.xpack.core.ml.inference.trainedmodel.Tokenization

createDefault, doLowerCase, equals, getMaxSequenceLength, getSpan, getTruncate, hashCode, maxSequenceLength, toXContent, updateWindowSettings, validateSpanAndMaxSequenceLength, validateSpanAndTruncate, withSpecialTokens

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.elasticsearch.xcontent.ToXContentObject

isFragment

Field Details

NAME

public static final String NAME

See Also:

• Constant Field Values

MASK_TOKEN

public static final String MASK_TOKEN

See Also:

• Constant Field Values

Constructor Details

RobertaTokenization

public RobertaTokenization(@Nullable
 Boolean withSpecialTokens,
 @Nullable
 Boolean addPrefixSpace,
 @Nullable
 Integer maxSequenceLength,
 @Nullable
 Tokenization.Truncate truncate,
 @Nullable
 Integer span)

RobertaTokenization

public RobertaTokenization(StreamInput in)
                    throws IOException

Throws:

IOException

Method Details

createParser

public static ConstructingObjectParser<RobertaTokenization,Void> createParser(boolean ignoreUnknownFields)

fromXContent

public static RobertaTokenization fromXContent(XContentParser parser,
 boolean lenient)

isAddPrefixSpace

public boolean isAddPrefixSpace()

getWriteableName

public String getWriteableName()

writeTo

public void writeTo(StreamOutput out)
             throws IOException

Specified by:

writeTo in interface Writeable

Overrides:

writeTo in class Tokenization

Throws:

IOException

getMaskToken

public String getMaskToken()

Specified by:

getMaskToken in class Tokenization

getName

public String getName()

Returns:

The name of the XContentObject that is to be serialized

validateVocabulary

public void validateVocabulary(PutTrainedModelVocabularyAction.Request request)

Overrides:

validateVocabulary in class Tokenization