Using large language models (LLMs) on our own (local) data

Goal

Use large language models (LLMs) to extract information from our own (local) data.

Typical use case: a company has private documents that were never used to train the model. Therefore, we need to enhance the interaction with the model to get results from these documents.

Possible solutions:

1. Fine-tuning: fine-tune a pre-trained model on our own data.
   1. Train the model on our own data.
2. Zero-shot learning: use a pre-trained model without fine-tuning it on our own data.
   1. Extract information with a vector (embedding) of our own data.
   2. Feed that vector to the pre-trained model as part of the prompt.

What to consider when selecting a solution

Items to consider when selecting a solution:

1. License
   1. Commercial vs. non-commercial use.
   2. GPLv3 vs. others.
2. Time to make the solution available.
   1. Fine-tuning: time to train the model.
   2. Zero-shot learning: time to create a vector database that will be used to extract information from our own data.
3. Time to update the solution.
   1. Fine-tuning: time to re-train the model on changed data.
   2. Zero-shot learning: time to update the vector database.
4. Cost
   1. Fine-tuning: cost of training the model.
   2. Zero-shot learning: cost of creating the vector database and (potential) costs of larger prompts (pay per token).
5. Accuracy
   1. Fine-tuning: potentially less accurate than zero-shot learning because the trained model is probabilistic, not deterministic.
   2. Zero-shot learning: accuracy of the model on our own data.
6. Response time
   1. Fine-tuning: potentially faster response time because of smaller prompts.
   2. Zero-shot: Potentially larger response time because of vector database lookup and larger prompts.
7. Data privacy
   1. Fine-tuning: data privacy is a concern because the model is trained on our own data.
   2. Zero-shot learning: data privacy is less of a concern because the model is not trained on our own data (but we still expose the prompt to the model).
8. Security
   1. How to filter out private documents from the model?
   2. How to protect individuals' privacy?
   3. What regulations apply, e.g. GDPR, CCPA, HIPAA, etc.?
   4. What audit and compliance requirements apply?
   5. Resistance to poisoning attacks.
   6. Resistance to backdoor attacks.
   7. Resistance to model inversion attacks (extract data from the model).
   8. Resistance to membership inference attacks (determine if a data point was used to train the model).
   9. Resistance to offensive language attacks (generate offensive language).
   10. Resistance to model stealing attacks (steal the model).

Vendors, frameworks

Azure

Azure requires a corporate email to sign up for the service.

• Main reference
• Customizing
• Document search
• Pricing

Technical references:

• ChatGPT + Enterprise data with Azure OpenAI and Cognitive Search
  • Companion blog post.
  • Unofficial implementation (refer to the blog post above).

Google

• Vertex AI matching engine

OpenAI

NOTE: OpenAI removed all examples under the apps directory (explanation). The links below take you to the repository state before the apps directory was removed. Removing them also means they are no longer maintained. As they get older, newer techniques may appear, so use these examples as a starting point and look for newer ones.

• The "query your data" tutorial.
  • See slides first.
  • This is a good conceptual example - a good place to start learning (starting with slides, then code).
• Enterprise Knowledge Retrieval.
  • "The notebook is the best place to start, and takes you through an end-to-end workflow for setting up and evaluating a simple back-end knowledge retrieval service"
  • This is a very good notebook to start with. It shows not only the code but also the thought process of putting a good solution in place and evaluating it.
  • It also covers agents, i.e. invoke external tools to complement the LLM.
• File Q&A: "[U]pload files and ask questions related to their content, and the app will use embeddings and GPT to generate answers from the most relevant files".
  • Similar to the "query your data" and "enterprise knowledge retrieval" tutorials, but includes a front-end and backend.
  • Seems to be older than the "knowledge retrieval" tutorial.
• Using Vector Databases for Embeddings Search.
  • Shows how to use different embedding databases.

Haystack

• haystack: "Apply the latest NLP technology to your own data with the use of Haystack's pipeline architecture".

LangChain

• How-To Guides
  • The ones related to answering questions on documents:
    • Question Answering: "how to use LangChain for question answering over a list of documents. It covers four different types of chains: stuff, map_reduce, refine, map_rerank".
    • Retrieval Question/Answering: "showcases question answering over an index."

References

Projects that run GPT locally

• privateGPT. Based on GPT4All.
  • Shows how to ingest data from different file formats with LangChain.
• Run ChatGPT-Style Questions Over Your Own Files Using the OpenAI API and LangChain!.
• GPT-4 & LangChain: "Create a ChatGPT Chatbot for Your PDF Files".

Projects that create/publish models

• GPT4All
• GPT-J
• MPT-7B: "provide a commercially-usable, open-source model that matches (and - in many ways - surpasses) LLaMA-7B."

Other projects with LLMs

• AutoGPT: Similar to "agents" in LangChain. It may be interesting to see how it is implemented behind the scenes (LangChain is also open source, but it has a lot more than agents -- this one is more focused).

Vector database and similarity search

• Faiss: the missing manual: Covers the basic concepts in the context of Faiss.

Fine-tuning

• OpenAI fine-tuning guide

Prompt techniques to improve information retrieval

• OpenAI's Techniques to improve reliability: Illustrates techniques to improve answers, including "Let's think step by step" (Large Language Models are Zero-Shot Reasoners - zero shot) and "chain of thought" (Language Models Perform Reasoning via Chain of Thought - few shot).
• Unit test writing using a multi-step prompt: Shows how to use a sequence of prompts to get a response.

Document parsers/loaders

• Llama Hub
• LangChain Document Loaders
  • Some loaders seem to be based on the ones from Llama Hub.

Improving search results

• Question answering using a search API and re-ranking (OpenAI)
• Search re-ranking in more details (OpenAI)
• Hypothetical Document Embeddings: Using hallucinations productively (OpenAI): demonstrates how to use HyDE to improve retrieval
• Fine-tuning a Classifier to Improve Truthfulness (OpenAI): Discusses how to fine-tune a classifier to remove false results.
• Dynamically chosing prompts and retrievers.
• Moderating the results.

Other

• Reddit: The best way to train an LLM on company data (March 2023): open-ended discussion.
• psychic: "extract and transform unstructured data from SaaS applications like Notion, Slack, Zendesk, Confluence, and Google Drive".
• MLC LLM: "a universal solution that allows any language models to be deployed natively on a diverse set of hardware backends and native applications".
• Prompt auto-evaluator: Use GPT-4 to evaluate prompts.
• How write prompts that invoke external tools (agents): This is the same as the link added above for OpenAI knowledge retrieval. I added it here again to remmeber it also has this interesting piece about agents.