Module org.elasticsearch.xcore

Package org.elasticsearch.xpack.core.ml.inference.preprocessing

Class CustomWordEmbedding

java.lang.Object

org.elasticsearch.xpack.core.ml.inference.preprocessing.CustomWordEmbedding

All Implemented Interfaces:

org.apache.lucene.util.Accountable, NamedWriteable, Writeable, ToXContent, ToXContentObject, LenientlyParsedPreProcessor, PreProcessor, StrictlyParsedPreProcessor, NamedXContentObject

public class CustomWordEmbedding
extends Object
implements LenientlyParsedPreProcessor, StrictlyParsedPreProcessor

This is a pre-processor that embeds text into a numerical vector. It calculates a set of features based on script type, ngram hashes, and most common script values. The features are then concatenated with specific quantization scales and weights into a vector of length 80. This is a fork and a port of: https://github.com/google/cld3/blob/06f695f1c8ee530104416aab5dcf2d6a1414a56a/src/embedding_network.cc

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	CustomWordEmbedding.StringLengthAndEmbedding

Nested classes/interfaces inherited from interface org.elasticsearch.xpack.core.ml.inference.preprocessing.PreProcessor

PreProcessor.PreProcessorParseContext

Nested classes/interfaces inherited from interface org.elasticsearch.xcontent.ToXContent

ToXContent.DelegatingMapParams, ToXContent.MapParams, ToXContent.Params

Nested classes/interfaces inherited from interface org.elasticsearch.common.io.stream.Writeable

Writeable.Reader<V>, Writeable.Writer<V>

Field Summary

Fields

Modifier and Type	Field	Description
static final ParseField	DEST_FIELD
static final ParseField	EMBEDDING_QUANT_SCALES
static final ParseField	EMBEDDING_WEIGHTS
static final ParseField	FIELD
static final int	MAX_STRING_SIZE_IN_BYTES
static final ParseField	NAME

Fields inherited from interface org.apache.lucene.util.Accountable

NULL_ACCOUNTABLE

Fields inherited from interface org.elasticsearch.xcontent.ToXContent

EMPTY, EMPTY_PARAMS

Constructor Summary

Constructors

Constructor	Description
CustomWordEmbedding(short[][] embeddingsQuantScales, byte[][] embeddingsWeights, String fieldName, String destField)
CustomWordEmbedding(StreamInput in)

Method Summary

Modifier and Type	Method	Description
boolean	equals(Object o)
static CustomWordEmbedding	fromXContentLenient(XContentParser parser)
static CustomWordEmbedding	fromXContentStrict(XContentParser parser)
String	getName()
String	getOutputFieldType(String outputField)
String	getWriteableName()
int	hashCode()
List<String>	inputFields()	The expected input fields
boolean	isCustom()
List<String>	outputFields()
void	process(Map<String,Object> fields)	Process the given fields and their values and return the modified map.
long	ramBytesUsed()
Map<String,String>	reverseLookup()
XContentBuilder	toXContent(XContentBuilder builder, ToXContent.Params params)
void	writeTo(StreamOutput out)

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.lucene.util.Accountable

getChildResources

Methods inherited from interface org.elasticsearch.xcontent.ToXContentObject

isFragment

Field Details

MAX_STRING_SIZE_IN_BYTES

public static final int MAX_STRING_SIZE_IN_BYTES

See Also:

• Constant Field Values

NAME

public static final ParseField NAME

FIELD

public static final ParseField FIELD

DEST_FIELD

public static final ParseField DEST_FIELD

EMBEDDING_WEIGHTS

public static final ParseField EMBEDDING_WEIGHTS

EMBEDDING_QUANT_SCALES

public static final ParseField EMBEDDING_QUANT_SCALES

Constructor Details

CustomWordEmbedding

public CustomWordEmbedding(StreamInput in)
                    throws IOException

Throws:

IOException

CustomWordEmbedding

public CustomWordEmbedding(short[][] embeddingsQuantScales,
 byte[][] embeddingsWeights,
 String fieldName,
 String destField)

Method Details

fromXContentStrict

public static CustomWordEmbedding fromXContentStrict(XContentParser parser)

fromXContentLenient

public static CustomWordEmbedding fromXContentLenient(XContentParser parser)

inputFields

public List<String> inputFields()

Description copied from interface: PreProcessor

The expected input fields

Specified by:

inputFields in interface PreProcessor

outputFields

public List<String> outputFields()

Specified by:

outputFields in interface PreProcessor

Returns:

The resulting output fields. It is imperative that the order is consistent between calls.

process

public void process(Map<String,Object> fields)

Description copied from interface: PreProcessor

Process the given fields and their values and return the modified map. NOTE: The passed map object is mutated directly

Specified by:

process in interface PreProcessor

Parameters:

fields - The fields and their values to process

reverseLookup

public Map<String,String> reverseLookup()

Specified by:

reverseLookup in interface PreProcessor

Returns:

Reverse lookup map to match resulting features to their original feature name

isCustom

public boolean isCustom()

Specified by:

isCustom in interface PreProcessor

Returns:

Is the pre-processor a custom one provided by the user, or automatically created? This changes how feature importance is calculated, as fields generated by custom processors get individual feature importance calculations.

getOutputFieldType

public String getOutputFieldType(String outputField)

Specified by:

getOutputFieldType in interface PreProcessor

ramBytesUsed

public long ramBytesUsed()

Specified by:

ramBytesUsed in interface org.apache.lucene.util.Accountable

getWriteableName

public String getWriteableName()

Specified by:

getWriteableName in interface NamedWriteable

writeTo

public void writeTo(StreamOutput out)
             throws IOException

Specified by:

writeTo in interface Writeable

Throws:

IOException

getName

public String getName()

Specified by:

getName in interface NamedXContentObject

Returns:

The name of the XContentObject that is to be serialized

toXContent

public XContentBuilder toXContent(XContentBuilder builder,
 ToXContent.Params params)
                           throws IOException

Specified by:

toXContent in interface ToXContent

Throws:

IOException

equals

public boolean equals(Object o)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object