Module org.elasticsearch.xcore

Package org.elasticsearch.xpack.core.ml.inference.trainedmodel

Class BertTokenization

java.lang.Object

org.elasticsearch.xpack.core.ml.inference.trainedmodel.Tokenization

org.elasticsearch.xpack.core.ml.inference.trainedmodel.BertTokenization

All Implemented Interfaces:

NamedWriteable, Writeable, ToXContent, ToXContentObject, NamedXContentObject

public class BertTokenization
extends Tokenization

Nested Class Summary

Nested classes/interfaces inherited from class org.elasticsearch.xpack.core.ml.inference.trainedmodel.Tokenization

Tokenization.SpanSettings, Tokenization.Truncate

Nested classes/interfaces inherited from interface org.elasticsearch.xcontent.ToXContent

ToXContent.DelegatingMapParams, ToXContent.MapParams, ToXContent.Params

Nested classes/interfaces inherited from interface org.elasticsearch.common.io.stream.Writeable

Writeable.Reader<V>, Writeable.Writer<V>

Field Summary

Fields

Modifier and Type	Field	Description
static final String	MASK_TOKEN
static final ParseField	NAME

Fields inherited from class org.elasticsearch.xpack.core.ml.inference.trainedmodel.Tokenization

DEFAULT_MAX_SEQUENCE_LENGTH, DO_LOWER_CASE, doLowerCase, MAX_SEQUENCE_LENGTH, maxSequenceLength, span, SPAN, truncate, TRUNCATE, UNSET_SPAN_VALUE, WITH_SPECIAL_TOKENS, withSpecialTokens

Fields inherited from interface org.elasticsearch.xcontent.ToXContent

EMPTY, EMPTY_PARAMS

Constructor Summary

Constructors

Constructor	Description
BertTokenization(Boolean doLowerCase, Boolean withSpecialTokens, Integer maxSequenceLength, Tokenization.Truncate truncate, Integer span)
BertTokenization(StreamInput in)

Method Summary

Modifier and Type	Method	Description
static ConstructingObjectParser<BertTokenization,Void>	createParser(boolean ignoreUnknownFields)
boolean	equals(Object o)
static BertTokenization	fromXContent(XContentParser parser, boolean lenient)
String	getMaskToken()
String	getName()
String	getWriteableName()
int	hashCode()
void	writeTo(StreamOutput out)

Methods inherited from class org.elasticsearch.xpack.core.ml.inference.trainedmodel.Tokenization

createDefault, doLowerCase, getMaxSequenceLength, getSpan, getTruncate, maxSequenceLength, toXContent, updateWindowSettings, validateSpanAndMaxSequenceLength, validateSpanAndTruncate, validateVocabulary, withSpecialTokens

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.elasticsearch.xcontent.ToXContentObject

isFragment

Field Details

NAME

public static final ParseField NAME

MASK_TOKEN

public static final String MASK_TOKEN

See Also:

• Constant Field Values

Constructor Details

BertTokenization

public BertTokenization(@Nullable
 Boolean doLowerCase,
 @Nullable
 Boolean withSpecialTokens,
 @Nullable
 Integer maxSequenceLength,
 @Nullable
 Tokenization.Truncate truncate,
 @Nullable
 Integer span)

BertTokenization

public BertTokenization(StreamInput in)
                 throws IOException

Throws:

IOException

Method Details

createParser

public static ConstructingObjectParser<BertTokenization,Void> createParser(boolean ignoreUnknownFields)

fromXContent

public static BertTokenization fromXContent(XContentParser parser,
 boolean lenient)

writeTo

public void writeTo(StreamOutput out)
             throws IOException

Specified by:

writeTo in interface Writeable

Overrides:

writeTo in class Tokenization

Throws:

IOException

getMaskToken

public String getMaskToken()

Specified by:

getMaskToken in class Tokenization

getWriteableName

public String getWriteableName()

getName

public String getName()

Returns:

The name of the XContentObject that is to be serialized

equals

public boolean equals(Object o)

Overrides:

equals in class Tokenization

hashCode

public int hashCode()

Overrides:

hashCode in class Tokenization