Neura Market
Llama2 Embedding Server logo

Llama2 Embedding Server

Free

Llama2 Embeddings FastAPI Service using LangChain ![GitHub Repo stars](https://img.shields.io/github/stars/Dicklesworthstone/llama_embeddings_fastapi_service?style=social)

FreeFree tier
Inputs: text
Type
Open Source
Visit Website GitHub (40 stars)

About Llama2 Embedding Server

Swiss Army Llama is a FastAPI service that provides REST endpoints for semantic text search using precomputed embeddings. It supports embedding extraction from PDFs (including OCR), Word files, audio (via Whisper transcription), and plain text. Embeddings are cached in SQLite to avoid recomputation, and optional RAM disks speed up model loading. The service leverages the high-performance Rust library fast_vector_similarity for advanced similarity measures (spearman_rho, kendall_tau, etc.) and FAISS for vector search. It also includes endpoints for text completions using local LLMs via llama_cpp. Designed for easy setup and integration with a Swagger UI.

Key Features

FastAPI REST endpoints for embedding generation and semantic search
Support for PDFs (including OCR), Word files, audio (Whisper transcription), and plain text
Automatic caching of embeddings in SQLite to save computation
Optional RAM disks for faster LLM loading
Advanced similarity measures via Rust library: spearman_rho, kendall_tau, distance correlation, etc.
FAISS vector search with optional second-pass advanced similarity measures
Integration with local LLMs via llama_cpp for text completions
Swagger UI for easy testing and integration

Pros & Cons

Pros
  • Comprehensive file type support (PDF, Word, audio, text)
  • Advanced similarity measures beyond cosine similarity
  • Caching reduces redundant computation
  • Fast vector search with FAISS
  • Automatic OCR for scanned PDFs
  • Easy setup with Docker and install scripts
Cons
  • Requires local setup and LLM model files (not a hosted service)
  • Performance depends on local hardware (GPU recommended for LLMs)
  • Documentation may be limited beyond README
  • Repo name (swiss_army_llama) might cause confusion with embedding-specific expectations

Best For

Semantic text search across document collectionsDocument embedding and analysisTranscribing audio and embedding the resultComputing semantic similarity between text stringsBuilding RAG pipelines with local LLMs

FAQ

What file types are supported for embedding?
The service supports PDFs (including OCR), Word files, audio (via Whisper), and plain text. It uses textract for most document types.
How are embeddings cached?
Embeddings are stored in a SQLite database, so if the same text is submitted again, the cached embedding is returned without recomputation.
What similarity measures are available?
Besides cosine similarity via FAISS, the service offers advanced measures like spearman_rho, kendall_tau, approximate_distance_correlation, jensen_shannon_dependency_measure, and hoeffding_d using fast_vector_similarity.
Is this service free and open-source?
Yes, the project is open-source and free to use. You can find the code on GitHub and run it locally.