Module org.elasticsearch.xcore

Package org.elasticsearch.xpack.core.inference.chunking

Class SentenceBoundaryChunker

java.lang.Object

org.elasticsearch.xpack.core.inference.chunking.SentenceBoundaryChunker

All Implemented Interfaces:

Chunker

public class SentenceBoundaryChunker
extends Object
implements Chunker

Split text into chunks aligned on sentence boundaries. The maximum chunk size is measured in words and controlled by maxNumberWordsPerChunk. Sentences are combined greedily until adding the next sentence would exceed maxNumberWordsPerChunk, at which point a new chunk is created. If an individual sentence is longer than maxNumberWordsPerChunk it is split on word boundary with overlap.

Nested Class Summary

Nested classes/interfaces inherited from interface org.elasticsearch.xpack.core.inference.chunking.Chunker

Chunker.ChunkOffset

Constructor Summary

Constructors

Constructor	Description
SentenceBoundaryChunker()

Method Summary

Modifier and Type	Method	Description
List<Chunker.ChunkOffset>	chunk(String input, int maxNumberWordsPerChunk, boolean includePrecedingSentence)	Break the input text into small chunks on sentence boundaries.
List<Chunker.ChunkOffset>	chunk(String input, ChunkingSettings chunkingSettings)	Break the input text into small chunks on sentence boundaries.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

SentenceBoundaryChunker

public SentenceBoundaryChunker()

Method Details

chunk

public List<Chunker.ChunkOffset> chunk(String input,
 ChunkingSettings chunkingSettings)

Break the input text into small chunks on sentence boundaries.

Specified by:

chunk in interface Chunker

Parameters:

input - Text to chunk

chunkingSettings - Chunking settings that define maxNumberWordsPerChunk

Returns:

The input text chunked

chunk

public List<Chunker.ChunkOffset> chunk(String input,
 int maxNumberWordsPerChunk,
 boolean includePrecedingSentence)

Break the input text into small chunks on sentence boundaries.

Parameters:

input - Text to chunk

maxNumberWordsPerChunk - Maximum size of the chunk

includePrecedingSentence - Include the previous sentence

Returns:

The input text offsets