Skip to main content

Embedding Models

Generate Embeddings

Embeddings are automatically generated when ingesting a document. Currently, this is part of the job of the vector store implementation (chromem-go).

Choosing an Embedding Model Provider

The knowledge tool supports multiple embedding model providers, which you can configure via the global config file or via environment variables. You can choose which of your configured providers to use by setting the KNOW_EMBEDDING_MODEL_PROVIDER environment variable or using the --embedding-model-provider flag.

note

The default selected provider is OpenAI

OpenAI + Azure

OpenAI and Azure are configured via a single provider configuration to make the configuration similar to the one used by GPTScript.

Environment VariableConfig KeyDefaultNotes
OPENAI_BASE_URLbaseURLhttps://api.openai.com/v1---
OPENAI_API_KEYapiKeysk-foorequired
OPENAI_EMBEDDING_MODELembeddingModeltext-embedding-ada-002---
OPENAI_API_TYPEapiTypeOPEN_AIone of OPEN_AI, AZURE, AZURE_AD
OPENAI_API_VERSIONapiVersion2024-02-01for Azure
OPENAI_AZURE_DEPLOYMENT``required for Azure

OpenAI Compatible Providers

We have some first-class supported providers that are compatible with OpenAI API, that you can find on this page. If yours is not in the list, you can still try to configure it using the OpenAI provider configuration as shown below for LM-Studio and Ollama.

LM-Studio

LM-Studio failed to return any embeddings if requested concurrently, so we set the parallel threads to 1. This may change in the future. Tested with LM-Studio v0.2.27.

export OPENAI_BASE_URL=http://localhost:1234/v1
export OPENAI_API_KEY=lm-studio
export OPENAI_EMBEDDING_MODEL="CompendiumLabs/bge-large-en-v1.5-gguf"
export VS_CHROMEM_EMBEDDING_PARALLEL_THREAD="1"
note

Running with VS_CHROMEM_EMBEDDING_PARALLEL_THREAD="1" may be really really slow for a large amount of files (or just really large files).

Ollama

Tested with Ollama v0.2.6 (pre-release that introduced OpenAI API compatibility).

export OPENAI_BASE_URL=http://localhost:11434/v1
export OPENAI_EMBEDDING_MODEL="mxbai-embed-large"

Cohere

Environment VariableConfig KeyDefaultNotes
COHERE_API_KEYapiKey---required
COHERE_MODELmodelembed-english-v3.0---

Jina

Environment VariableConfig KeyDefaultNotes
JINA_API_KEYapiKey---required
JINA_MODELmodeljina-embeddings-v2-base-en---

LocalAI

Environment VariableConfig KeyDefaultNotes
LOCALAI_MODELmodelbert-cpp-minilm-v6---

Mistral

Environment VariableConfig KeyDefaultNotes
MISTRAL_API_KEYapiKey---required

Mixedbread

Environment VariableConfig KeyDefaultNotes
MIXEDBREAD_API_KEYapiKey---required
MIXEDBREAD_MODELmodelall-MiniLM-L6-v2---

Ollama

Requires Ollama v0.2.6 or later.

Environment VariableConfig KeyDefaultNotes
OLLAMA_BASE_URLbaseURLhttp://localhost:11434/v1---
MIXEDBREAD_MODELmodelmxbai-embed-large---