Vector Embedding Preprocessor#

The vector preprocessor sends entry data to an OpenAI-compatible embeddings API and attaches the resulting vector to the entry as an enumerated value named embeddings. This allows entries to carry a semantic representation of their contents alongside the original data, which can then be used for similarity comparisons and other vector-based analysis at search time.

The entry payload is never modified; the embedding is attached as additional metadata.

Every entry in a processing block is embedded in a single request to the configured endpoint, which keeps the number of API round trips low.

The Vector Embedding preprocessor Type is vector.

Supported Options#

  • Model (string, required): The name of the embedding model to request, for example text-embedding-3-small or nomic-embed-text. The model must be available on the configured endpoint.

  • Endpoint (string, required): The full URL of the OpenAI-compatible embeddings endpoint, for example http://localhost:11434/v1/embeddings. The URL must specify either the http or https scheme and include a host.

  • Token (string, optional): An API token to present to the endpoint. When set, it is sent as an Authorization: Bearer header. Omit this option for endpoints that do not require authentication, such as a local model server.

  • Timeout (integer, optional): The HTTP timeout in seconds for each embedding request. If omitted, the default is 30 seconds.

  • Retry-Attempts (integer, optional): The total number of times to attempt an embedding request before giving up. If omitted, the default is 3. Retries use an exponential backoff starting at 250 milliseconds and capped at 5 seconds.

  • Passthrough-On-Error (boolean, optional): When set to true, entries which cannot be embedded are passed through the preprocessor chain unmodified. When false (the default), a failed embedding request causes the entire block of entries to be dropped and the error to be logged by the ingester.

Only failures which may be transient are retried: network and timeout errors, HTTP 429 responses, and HTTP 5xx responses. Other failures, such as a malformed response body or an HTTP 401, are treated as permanent and fail immediately without consuming the remaining attempts.

Note

Entries with an empty or whitespace-only payload are passed through untouched and are not sent to the embeddings endpoint. An embedding of an empty payload carries no semantic content, and some endpoints reject empty input outright.

Note

Embeddings are generated by making a synchronous network request while the entry block is in flight, so the throughput of the data flow is bounded by the throughput of the embeddings endpoint. Be conservative about applying this preprocessor to high-volume data flows, and consider whether the endpoint you are pointing at meters requests or charges per token.

Enumerated Value Format#

The attached embeddings enumerated value is a string containing the JSON encoding of the vector, for example:

[0.0023064255,-0.009327292,0.015797347, ... ]

Common Use Cases#

The vector preprocessor is commonly used to:

  • Enrich log or document data with embeddings at ingest time so that vectors are available immediately at search time rather than being computed on demand.

  • Attach embeddings to LLM prompt and response data so that conversations can be compared and clustered.

  • Build a semantic index over a corpus of text data such as support tickets, commit messages, or documentation.

Example: Embedding With a Local Model Server#

A local model server which does not require authentication needs only the model name and endpoint:

[Listener "documents"]
	Bind-String="0.0.0.0:7700"
	Tag-Name=documents
	Preprocessor=embed

[Preprocessor "embed"]
	Type=vector
	Model=nomic-embed-text
	Endpoint=http://127.0.0.1:11434/v1/embeddings

Every entry arriving on port 7700 with a non-empty payload will be sent to the local embeddings endpoint and will carry an embeddings enumerated value when it reaches the indexer. If the model server is unreachable, the block of entries is dropped and the ingester logs the failure.

Example: Embedding With a Remote API and Passthrough#

When using a hosted embeddings API, supply a token and consider whether losing data is preferable to losing embeddings. The following configuration keeps the raw data flowing even when the API is unavailable:

[Preprocessor "embed"]
	Type=vector
	Model=text-embedding-3-small
	Endpoint=https://api.example.com/v1/embeddings
	Token=`sk-example-token`
	Timeout=15
	Retry-Attempts=5
	Passthrough-On-Error=true

With Passthrough-On-Error=true, an entry which cannot be embedded after five attempts is still sent to the indexer, just without the embeddings enumerated value. Note that this means the presence of the enumerated value is not guaranteed on any given entry, so downstream queries should be prepared to handle entries which lack it.

Example: Embedding Only Selected Data#

Because embedding is comparatively expensive, it is often worth narrowing the data flow before the vector preprocessor runs. Preprocessors are applied in the order they are listed on the data consumer, so a filtering preprocessor placed ahead of the vector preprocessor will reduce the number of entries sent to the embeddings endpoint:

[Listener "endpoint_logs"]
	Bind-String="0.0.0.0:7701"
	Tag-Name=endpoint
	Preprocessor=onlyalerts
	Preprocessor=embed

[Preprocessor "onlyalerts"]
	Type=regexdrop
	Regex=`"severity":"(critical|high)"`
	Invert=true

[Preprocessor "embed"]
	Type=vector
	Model=text-embedding-3-small
	Endpoint=https://api.example.com/v1/embeddings
	Token=`sk-example-token`

Only entries which survive the regexdrop preprocessor are passed to the vector preprocessor for embedding.