Module org.elasticsearch.xcore
Class RobertaTokenization
java.lang.Object
org.elasticsearch.xpack.core.ml.inference.trainedmodel.Tokenization
org.elasticsearch.xpack.core.ml.inference.trainedmodel.RobertaTokenization
- All Implemented Interfaces:
NamedWriteable,Writeable,org.elasticsearch.xcontent.ToXContent,org.elasticsearch.xcontent.ToXContentObject,NamedXContentObject
-
Nested Class Summary
Nested classes/interfaces inherited from class org.elasticsearch.xpack.core.ml.inference.trainedmodel.Tokenization
Tokenization.SpanSettings, Tokenization.TruncateNested classes/interfaces inherited from interface org.elasticsearch.xcontent.ToXContent
org.elasticsearch.xcontent.ToXContent.DelegatingMapParams, org.elasticsearch.xcontent.ToXContent.MapParams, org.elasticsearch.xcontent.ToXContent.ParamsNested classes/interfaces inherited from interface org.elasticsearch.common.io.stream.Writeable
Writeable.Reader<V>, Writeable.Writer<V> -
Field Summary
FieldsFields inherited from class org.elasticsearch.xpack.core.ml.inference.trainedmodel.Tokenization
DEFAULT_MAX_SEQUENCE_LENGTH, DO_LOWER_CASE, doLowerCase, MAX_SEQUENCE_LENGTH, maxSequenceLength, span, SPAN, truncate, TRUNCATE, UNSET_SPAN_VALUE, WITH_SPECIAL_TOKENS, withSpecialTokensFields inherited from interface org.elasticsearch.xcontent.ToXContent
EMPTY, EMPTY_PARAMS -
Constructor Summary
ConstructorsConstructorDescriptionRobertaTokenization(Boolean withSpecialTokens, Boolean addPrefixSpace, Integer maxSequenceLength, Tokenization.Truncate truncate, Integer span) -
Method Summary
Modifier and TypeMethodDescriptionstatic org.elasticsearch.xcontent.ConstructingObjectParser<RobertaTokenization, Void> createParser(boolean ignoreUnknownFields) static RobertaTokenizationfromXContent(org.elasticsearch.xcontent.XContentParser parser, boolean lenient) getName()booleanvoidvoidwriteTo(StreamOutput out) Methods inherited from class org.elasticsearch.xpack.core.ml.inference.trainedmodel.Tokenization
createDefault, doLowerCase, equals, getMaxSequenceLength, getSpan, getTruncate, hashCode, maxSequenceLength, toXContent, updateWindowSettings, validateSpanAndMaxSequenceLength, validateSpanAndTruncate, withSpecialTokensMethods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, toString, wait, wait, waitMethods inherited from interface org.elasticsearch.xcontent.ToXContentObject
isFragment
-
Field Details
-
NAME
- See Also:
-
MASK_TOKEN
- See Also:
-
-
Constructor Details
-
RobertaTokenization
public RobertaTokenization(@Nullable Boolean withSpecialTokens, @Nullable Boolean addPrefixSpace, @Nullable Integer maxSequenceLength, @Nullable Tokenization.Truncate truncate, @Nullable Integer span) -
RobertaTokenization
- Throws:
IOException
-
-
Method Details
-
createParser
public static org.elasticsearch.xcontent.ConstructingObjectParser<RobertaTokenization,Void> createParser(boolean ignoreUnknownFields) -
fromXContent
public static RobertaTokenization fromXContent(org.elasticsearch.xcontent.XContentParser parser, boolean lenient) -
isAddPrefixSpace
public boolean isAddPrefixSpace() -
getWriteableName
-
writeTo
- Specified by:
writeToin interfaceWriteable- Overrides:
writeToin classTokenization- Throws:
IOException
-
getMaskToken
- Specified by:
getMaskTokenin classTokenization
-
getName
- Returns:
- The name of the XContentObject that is to be serialized
-
validateVocabulary
- Overrides:
validateVocabularyin classTokenization
-