Semantic Index with Qdrant

Semantic Index with Qdrant

Qdrant is an open‑source vector database built in Rust that’s optimized for high‑performance similarity search. It stores embeddings (vectors) alongside rich metadata (payload), supports powerful filtering, and provides both HTTP and gRPC APIs. Under the hood, it uses modern ANN algorithms (HNSW), optional quantization/compression, and on‑disk persistence—perfect for RAG and semantic search scenarios.

In this post you’ll:

  • Run Qdrant locally with a prebuilt Docker image
  • Create a collection with the right vector settings (size, distance)
  • Connect Qdrant to Semantic Kernel (C#) via the vector store connector
  • Index (upsert) data and run vector searches with optional filters

Qdrant in one minute

Key facts:

  • Built for vectors first: approximate nearest neighbor (HNSW), re‑scoring, and filtering
  • Rich payload (JSON) attached to each point for faceting and security filters
  • Distance functions: cosine, dot, and Euclidean
  • Durable by default with snapshots and write‑ahead logging (WAL)
  • Easy deployment: single Docker container, binary, or managed cloud
  • APIs: HTTP (default port 6333) and gRPC (6334)

Run it with Docker (local dev)

docker run \
  -p 6333:6333 \
  -p 6334:6334 \
  -v $(pwd)/qdrant_storage:/qdrant/storage \
  --name qdrant \
  qdrant/qdrant:latest

Notes:

  • The volume persists your data on restarts
  • Default HTTP endpoint: http://localhost:6333
  • You can configure auth, snapshots, and performance via environment variables

Create a collection (schema)

In Qdrant a “collection” is like a table for vectors. You define the vector size (dimensions) and distance. For example, for text-embedding-3-small you’d often use 1536 dimensions.

PUT http://localhost:6333/collections/docs
Content-Type: application/json

{
  "name": "docs",
  "vectors": { "size": 1536, "distance": "Cosine" }
}

You can also add payload indexing for fields you plan to filter on (e.g., tags, customerId).

Connect Qdrant to Semantic Kernel (C#)

Semantic Kernel provides a Qdrant vector store connector so you can use the same abstractions (IVectorStore, ICollection) across different databases.

// dotnet add package Microsoft.SemanticKernel
// dotnet add package Microsoft.SemanticKernel.Connectors.Qdrant --prerelease
using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.Connectors.Qdrant;
using Microsoft.Extensions.DependencyInjection;

var builder = Kernel.CreateBuilder();

// Register a Qdrant vector store pointing to local Docker instance
builder.Services.AddQdrantVectorStore(options =>
{
    options.Endpoint = new Uri("http://localhost:6333");
    // options.ApiKey = "<if-enabled>"; // Qdrant auth is optional in local dev
});

var kernel = builder.Build();

// Resolve the vector store and target a collection (will create if needed)
var vectorStore = kernel.GetRequiredService<QdrantVectorStore>();
var collection = vectorStore.GetCollection<string, MyChunk>("docs");

// Ensure collection exists with correct dimensions & distance
await collection.CreateCollectionIfNotExistsAsync(new()
{
    Dimensions = 1536,
    DistanceMetric = QdrantDistance.Cosine
});

public sealed class MyChunk
{
    public string id { get; set; } = default!;
    public string content { get; set; } = default!;
    public float[] contentVector { get; set; } = default!;
    public string[]? tags { get; set; }
}

Tip: Keep dimensions identical to the embedding model you use at indexing and query time.

Upsert documents with embeddings

Compute embeddings (for example, with Azure OpenAI) and push points to Qdrant via the SK collection.

using Microsoft.SemanticKernel.Embeddings;

// Assume you registered ITextEmbeddingGenerationService in the Kernel
var embedder = kernel.GetRequiredService<ITextEmbeddingGenerationService>();

var text = "Qdrant is a fast, open-source vector database built in Rust.";
var vector = await embedder.GenerateEmbeddingAsync(text); // IReadOnlyList<float>

var doc = new MyChunk
{
    id = Guid.NewGuid().ToString("N"),
    content = text,
    contentVector = vector.ToArray(),
    tags = new[] { "intro", "qdrant" }
};

await collection.UpsertAsync(doc.id, doc.contentVector, doc, cancellationToken: default);

Under the hood, the connector writes a point with your vector and stores the rest of the properties as payload, so you can filter later.

Vector search (k‑NN) with optional filters

To search, embed the query text and ask for the top‑K nearest vectors. You can also pass a filter (e.g., limit to tags that include intro).

var query = "What is Qdrant?";
var qVec = await embedder.GenerateEmbeddingAsync(query);

var results = await collection.VectorizedSearchAsync(
    qVec.ToArray(),
    top: 5,
    filter:
        QdrantFilter.ContainsAny("tags", new[] { "intro" })
);

foreach (var r in results)
{
    Console.WriteLine($"score={r.Score:F4} id={r.Key} text={r.Record.content}");
}

How it works:

  • SK generates an embedding for your query
  • Qdrant executes a nearest‑neighbors search using the configured distance
  • Optional filters trim candidates using payload fields (fast and scalable)
  • Results include the score and your payload (for rendering / grounding)

C# program: Embed Markdown files from a local directory

The following end‑to‑end sample scans a local folder for .md/.mdx files, normalizes Markdown to plain text, chunks content, generates embeddings (OpenAI or Azure OpenAI), and upserts the chunks into a Qdrant collection via Semantic Kernel’s Qdrant connector.

Prerequisites (NuGet):

dotnet add package Microsoft.SemanticKernel
dotnet add package Microsoft.SemanticKernel.Connectors.Qdrant --prerelease

Configuration (environment variables):

  • Qdrant
    • QDRANT_ENDPOINT (default: http://localhost:6333)
    • QDRANT_API_KEY (optional)
    • QDRANT_COLLECTION (default: docs)
  • Embeddings (choose one)
    • OpenAI: OPENAI_API_KEY, optional OPENAI_EMBEDDING_MODEL (default: text-embedding-3-small, 1536 dims)
    • Azure OpenAI: AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_API_KEY, AZURE_OPENAI_EMBEDDING_DEPLOYMENT
  • Source folder and dimensions
    • SOURCE_DIR (default: ./docs)
    • EMBEDDING_DIM (default: 1536; must match your embedding model)
using System.Text;
using System.Text.RegularExpressions;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.Connectors.Qdrant;
using Microsoft.SemanticKernel.Embeddings;

// Minimal, self-contained example
// 1) Set env vars per the post, 2) build & run, 3) browse Qdrant UI or query via SK

var sourceDir = Environment.GetEnvironmentVariable("SOURCE_DIR") ?? "./docs";
var qdrantEndpoint = Environment.GetEnvironmentVariable("QDRANT_ENDPOINT") ?? "http://localhost:6333";
var qdrantApiKey = Environment.GetEnvironmentVariable("QDRANT_API_KEY");
var collectionName = Environment.GetEnvironmentVariable("QDRANT_COLLECTION") ?? "docs";
var embeddingDim = int.TryParse(Environment.GetEnvironmentVariable("EMBEDDING_DIM"), out var dim) ? dim : 1536;

var builder = Kernel.CreateBuilder();

// Embedding service: Azure OpenAI (if configured) or OpenAI
var azureEndpoint = Environment.GetEnvironmentVariable("AZURE_OPENAI_ENDPOINT");
var azureKey = Environment.GetEnvironmentVariable("AZURE_OPENAI_API_KEY");
var azureDeployment = Environment.GetEnvironmentVariable("AZURE_OPENAI_EMBEDDING_DEPLOYMENT");

var openAiKey = Environment.GetEnvironmentVariable("OPENAI_API_KEY");
var openAiModel = Environment.GetEnvironmentVariable("OPENAI_EMBEDDING_MODEL") ?? "text-embedding-3-small"; // 1536 dims

if (!string.IsNullOrWhiteSpace(azureEndpoint) && !string.IsNullOrWhiteSpace(azureKey) && !string.IsNullOrWhiteSpace(azureDeployment))
{
  builder.AddAzureOpenAITextEmbeddingGeneration(azureDeployment, azureEndpoint, azureKey);
}
else if (!string.IsNullOrWhiteSpace(openAiKey))
{
  builder.AddOpenAITextEmbeddingGeneration(openAiModel, openAiKey);
}
else
{
  throw new InvalidOperationException("No embedding provider configured. Set Azure OpenAI or OpenAI environment variables.");
}

// Qdrant vector store
builder.Services.AddQdrantVectorStore(options =>
{
  options.Endpoint = new Uri(qdrantEndpoint);
  if (!string.IsNullOrEmpty(qdrantApiKey)) options.ApiKey = qdrantApiKey;
});

var kernel = builder.Build();
var embedder = kernel.GetRequiredService<ITextEmbeddingGenerationService>();
var vectorStore = kernel.GetRequiredService<QdrantVectorStore>();
var collection = vectorStore.GetCollection<string, DocumentChunk>(collectionName);

await collection.CreateCollectionIfNotExistsAsync(new()
{
  Dimensions = embeddingDim,
  DistanceMetric = QdrantDistance.Cosine,
});

Console.WriteLine($"Indexing Markdown from: {Path.GetFullPath(sourceDir)}");

var files = Directory.EnumerateFiles(sourceDir, "*.*", SearchOption.AllDirectories)
  .Where(p => p.EndsWith(".md", StringComparison.OrdinalIgnoreCase) || p.EndsWith(".mdx", StringComparison.OrdinalIgnoreCase))
  .ToList();

int fileCount = 0, chunkCount = 0;
foreach (var file in files)
{
  var raw = await File.ReadAllTextAsync(file);
  var plain = NormalizeMarkdown(raw);
  var chunks = ChunkText(plain, maxChars: 1200);

  var relPath = Path.GetRelativePath(sourceDir, file);
  for (int i = 0; i < chunks.Count; i++)
  {
    var text = chunks[i];
    if (string.IsNullOrWhiteSpace(text)) continue;

    var vec = await embedder.GenerateEmbeddingAsync(text);

    var rec = new DocumentChunk
    {
      id = $"{relPath.Replace('\\', '/')}#{i}",
      path = relPath.Replace('\\', '/'),
      chunkIndex = i,
      title = Path.GetFileNameWithoutExtension(file),
      content = text,
      contentVector = vec.ToArray(),
      tags = new[] { "markdown" }
    };

    await collection.UpsertAsync(rec.id, rec.contentVector, rec);
    chunkCount++;
  }
  fileCount++;
}

Console.WriteLine($"Done. Files: {fileCount}, Chunks: {chunkCount} -> Collection: {collectionName}");

// --- helpers ---
static string NormalizeMarkdown(string md)
{
  // Remove YAML frontmatter
  md = Regex.Replace(md, "^---[\\s\\S]*?---\\s*", string.Empty, RegexOptions.Multiline);
  // Remove fenced code blocks
  md = Regex.Replace(md, "```[\\s\\S]*?```", string.Empty, RegexOptions.Multiline);
  // Images ![alt](url)
  md = Regex.Replace(md, "!\\[[^\\]]*\\]\\([^\\)]*\\)", string.Empty);
  // Links [text](url) -> text
  md = Regex.Replace(md, "\\[([^\\]]+)\\]\\([^\\)]*\\)", "$1");
  // Inline code `code` -> code
  md = Regex.Replace(md, "`([^`]*)`", "$1");
  // Strip heading markers
  md = Regex.Replace(md, "^\\s*#+\\s*", string.Empty, RegexOptions.Multiline);
  // Collapse whitespace
  md = Regex.Replace(md, "\\s{2,}", " ");
  return md.Trim();
}

static List<string> ChunkText(string text, int maxChars)
{
  var chunks = new List<string>();
  var paras = text.Split(new[] { "\r\n\r\n", "\n\n" }, StringSplitOptions.RemoveEmptyEntries);
  var sb = new StringBuilder();

  foreach (var p in paras)
  {
    var para = p.Trim();
    if (para.Length > maxChars)
    {
      for (int i = 0; i < para.Length; i += maxChars)
      {
        chunks.Add(para.Substring(i, Math.Min(maxChars, para.Length - i)));
      }
      continue;
    }

    if (sb.Length + para.Length + 2 > maxChars)
    {
      if (sb.Length > 0)
      {
        chunks.Add(sb.ToString().Trim());
        sb.Clear();
      }
    }

    if (sb.Length > 0) sb.AppendLine();
    sb.AppendLine(para);
  }

  if (sb.Length > 0)
  {
    chunks.Add(sb.ToString().Trim());
  }

  return chunks;
}

public sealed class DocumentChunk
{
  public string id { get; set; } = default!;
  public string path { get; set; } = default!;          // relative file path
  public int chunkIndex { get; set; }
  public string title { get; set; } = default!;
  public string content { get; set; } = default!;
  public float[] contentVector { get; set; } = default!; // embedding
  public string[]? tags { get; set; }
}

TL;DR

  • Qdrant is a fast, open‑source vector DB with rich filtering and simple Docker deployment
  • Use the SK Qdrant connector to create collections, upsert documents, and query with embeddings
  • Keep vector dimensions consistent with your embedding model and use payload filters to keep results relevant

Useful links: