Module org.elasticsearch.xcore
Class XLMRobertaTokenization
java.lang.Object
org.elasticsearch.xpack.core.ml.inference.trainedmodel.Tokenization
org.elasticsearch.xpack.core.ml.inference.trainedmodel.XLMRobertaTokenization
- All Implemented Interfaces:
NamedWriteable,Writeable,org.elasticsearch.xcontent.ToXContent,org.elasticsearch.xcontent.ToXContentObject,NamedXContentObject
-
Nested Class Summary
Nested classes/interfaces inherited from class org.elasticsearch.xpack.core.ml.inference.trainedmodel.Tokenization
Tokenization.SpanSettings, Tokenization.TruncateNested classes/interfaces inherited from interface org.elasticsearch.xcontent.ToXContent
org.elasticsearch.xcontent.ToXContent.DelegatingMapParams, org.elasticsearch.xcontent.ToXContent.MapParams, org.elasticsearch.xcontent.ToXContent.ParamsNested classes/interfaces inherited from interface org.elasticsearch.common.io.stream.Writeable
Writeable.Reader<V>, Writeable.Writer<V> -
Field Summary
FieldsFields inherited from class org.elasticsearch.xpack.core.ml.inference.trainedmodel.Tokenization
DEFAULT_MAX_SEQUENCE_LENGTH, DO_LOWER_CASE, doLowerCase, MAX_SEQUENCE_LENGTH, maxSequenceLength, span, SPAN, truncate, TRUNCATE, UNSET_SPAN_VALUE, WITH_SPECIAL_TOKENS, withSpecialTokensFields inherited from interface org.elasticsearch.xcontent.ToXContent
EMPTY, EMPTY_PARAMS -
Constructor Summary
ConstructorsConstructorDescriptionXLMRobertaTokenization(Boolean withSpecialTokens, Integer maxSequenceLength, Tokenization.Truncate truncate, Integer span) -
Method Summary
Modifier and TypeMethodDescriptionprotected TokenizationbuildWindowingTokenization(int maxSeqLength, int span) static org.elasticsearch.xcontent.ConstructingObjectParser<XLMRobertaTokenization, Void> createParser(boolean ignoreUnknownFields) static XLMRobertaTokenizationfromXContent(org.elasticsearch.xcontent.XContentParser parser, boolean lenient) getName()voidvoidwriteTo(StreamOutput out) Methods inherited from class org.elasticsearch.xpack.core.ml.inference.trainedmodel.Tokenization
createDefault, doLowerCase, equals, getMaxSequenceLength, getSpan, getTruncate, hashCode, maxSequenceLength, toXContent, updateWindowSettings, validateSpanAndMaxSequenceLength, validateSpanAndTruncate, withSpecialTokensMethods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, toString, wait, wait, waitMethods inherited from interface org.elasticsearch.xcontent.ToXContentObject
isFragment
-
Field Details
-
NAME
- See Also:
-
MASK_TOKEN
- See Also:
-
-
Constructor Details
-
XLMRobertaTokenization
public XLMRobertaTokenization(@Nullable Boolean withSpecialTokens, @Nullable Integer maxSequenceLength, @Nullable Tokenization.Truncate truncate, @Nullable Integer span) -
XLMRobertaTokenization
- Throws:
IOException
-
-
Method Details
-
createParser
public static org.elasticsearch.xcontent.ConstructingObjectParser<XLMRobertaTokenization,Void> createParser(boolean ignoreUnknownFields) -
fromXContent
public static XLMRobertaTokenization fromXContent(org.elasticsearch.xcontent.XContentParser parser, boolean lenient) -
buildWindowingTokenization
-
getWriteableName
-
writeTo
- Specified by:
writeToin interfaceWriteable- Overrides:
writeToin classTokenization- Throws:
IOException
-
getMaskToken
- Specified by:
getMaskTokenin classTokenization
-
getName
- Returns:
- The name of the XContentObject that is to be serialized
-
validateVocabulary
- Overrides:
validateVocabularyin classTokenization
-