What is Retrieval Augmented Generation, and how to build an AI RAG system on Instill Cloud?

Retrieval-Augmented Generation (RAG) is an AI framework that combines information retrieval systems with large language models (LLMs) to generate more accurate and relevant text. By combining the strengths of LLMs with the ability to access external knowledge, RAG offers a way to generate responses that are both contextually aware and grounded in real-time, relevant information. This dual functionality enhances the accuracy of AI systems, reducing hallucinations and making them more reliable in dynamic environments.

In this tutorial, we will explore the fundamentals of RAG, its practical benefits, and guide you through building a basic RAG system on Instill Cloud for assisting doctors and medical professionals with the diagnoisis of urinary tract infections. By the end, you'll have a functional API-ready RAG pipeline to test, and will have the knowledge to build and integrate RAG into your own applications, empowering them with real-time retrieval capabilities.

#So What Actually is RAG?

RAG is a natural language processing framework designed to improve the responses of language models by combining two powerful techniques: retrieval and generation. Instead of relying solely on the pre-trained knowledge of a model, RAG allows AI systems to retrieve relevant information from external sources in real time and generate responses that are enriched by this external data.

At a high level, this process involves:

1. Retrieval: Searching a knowledge base or data catalog to find information related to the user's query.
2. Generation: Using the retrieved information to craft a response that is accurate, contextually relevant, and factually grounded.

This method allows AI models to provide more reliable and nuanced responses, especially in fields where information changes frequently or where accuracy is paramount.

#How to Build a RAG System?

At a lower level, building a RAG system typically involves two key phases.

#Indexing Phase (Offline):

The indexing phase occurs offline during the setup of the RAG system, where you prepare the knowledge base for future retrieval.

RAG Indexing Phase (Offline): Building a data catalog for future retrieval

1. Extract text from files: Gather textual data from documents, websites, or other sources that will serve as the knowledge base.
2. Split the text into smaller chunks: Break the extracted text into manageable pieces (chunks) to ensure more efficient retrieval and better context handling.
3. Convert text chunks into vectors using an embedding model: Use a pre-trained embedding model to convert each text chunk into vector representations, capturing the semantic meaning of the text.
4. Store vectors and their corresponding text as metadata in a vector database: Save the vectors in a vector database, alongside metadata like the original text, to enable fast similarity-based searches during the online phase.

#Retrieving and Generating Phase (Online):

The retrieving and generating phase happens online, during real-time interactions when the system is serving user queries.

RAG Retrieving and Generating Phase (Online): Retrieving relevant text chunks and generating responses

1. Convert user queries into vectors: Transform the user's input query into a vector using the same embedding model.
2. Retrieve relevant text chunks from the vector database based on similarity: Search the vector database to find the most semantically similar chunks of text based on the query vector.
3. Generate a response to the query using the retrieved chunks as context: Feed the retrieved text chunks into the language model to generate a response, grounding it in the retrieved information.

#Why is RAG so Useful?

RAG offers a range of benefits that significantly enhance AI performance, making it a go-to approach for a variety of applications:

💡 Minimizes Hallucinations
Large language models sometimes generate responses that seem plausible but are factually incorrect or nonsensical. RAG mitigates this issue by grounding responses in real, retrieved data, thus reducing the chances of hallucinations and improving the reliability of AI systems.

📚 Incorporates External Knowledge
Unlike standalone models limited by their training data, RAG systems can access and utilize up-to-date information from external sources. This is especially valuable in fast-moving domains like news, finance, law, or medicine, where data relevance can shift rapidly.

🔧 Reduces the Need for Expensive Retraining
Traditional models require frequent retraining to incorporate new information, which is time-consuming and costly. For certain applications, RAG offers a more efficient alternative - by simply updating the external knowledge base, you can maintain access to the latest information without the need to retrain the underlying model.

💰 Cost-Efficiency and Flexibility
RAG is ideal for applications like question-answering, text summarization, and dialogue systems, where it can help avoid the high computational costs of training models from scratch. Instead of fine-tuning large models on new datasets, RAG integrates external data seamlessly into the generation process.

🔐 Data Control and Security
With RAG, organizations have complete control over the data sources their AI models access. Sensitive data can be managed according to authorization levels, ensuring compliance with privacy policies and regulations.

📈 Enhanced Performance in QA and Dialogue Systems
RAG excels in environments where accuracy is critical, such as in QA systems, by grounding answers in factual documents. For dialogue systems, RAG enhances conversation by maintaining relevant context throughout the interaction, making the experience more coherent and dynamic.

#Building a Basic RAG System on ☁️ Instill Cloud

Let's dive in and create a RAG system for assisting doctors and medical professionals with the diagnoisis of urinary tract infections. We'll follow these steps:

1. Create an Instill Catalog using raw documents (a mix of docx and pdf files) to serve as the knowledge base
2. Build the basic-rag-anthropic pipeline in the editor
3. Test the performance of the pipeline on real-world queries
4. Call the pipeline via API

#Try it Yourself!

Click the button below to view and try the pipeline yourself — for free — on the Instill Cloud platform!

Please note that to use this pipeline, you need to first sign up for an Instill Cloud account, and create, upload and process files to your own Instill Catalog.

#Step-by-Step Tutorial

We'll first use Instill Artifact to handle the Indexing Phase and build our catalog - this will serve as our RAG knowledge base and encompasses converting the documents to markdown text, chunking the text, embedding the chunks, and storing the embeddings in a vector database.

Next, we'll tackle the Retrieving and Generating Phase using Instill VDP to build a simple and generalisable RAG pipeline. Although we will be focusing on building a medical RAG system in this tutorial, the pipeline we will build is fully generalisable to other domains.

#Prerequisites

• Please create an Instill Cloud account first by following the steps outlined in the Quickstart Guide.
• Download the medical documents we will be using to build our data catalog from this link. Once downloaded, extract the contents. The files we will be using are a mix of docx and pdf file formats, and contain medical information related to urinary tract infections and their diagnosis.

#Step 1: Create a Catalog with 💾 Instill Artifact

To create a new Catalog from the console:

1. Launch Instill Console on Instill Cloud.
2. Navigate to the Artifacts page using the navigation bar.
3. Click the + Create Catalog button.
4. Select the Owner (your namespace ID).
5. Enter a name for your Catalog, e.g. "medical-documents".
6. (Optional) Enter a description and tags for your Catalog.
7. Click the Create button.

#Step 2: Index Files into the Catalog

To upload files to a Catalog via the Console, follow these steps:

1. Click the Catalog card for the new Catalog.
2. Select Upload Documents in the left panel.
3. Drag and drop the rag-tutorial-files files into the blue box or click browse computer to upload files.
4. Click the Process Files button.

The processing status of your files appears in the Files tab. When the status is Completed, you can view your processed Files and Chunks.

This catalog will serve as the knowledge base for the RAG pipeline we will now build.

#Step 3: Create a New Pipeline

1. Navigate to the Pipelines page using the navigation bar on Instill Cloud.
2. Click + Create Pipeline in the top right to enter the pipeline editor.
3. Select the Owner (your namespace ID).
4. Enter a name for your Pipeline, e.g. "basic-rag".
5. (Optional) Enter a description for your Pipeline.
6. Click the Create button, and you will be redirected to the pipeline editor for your new pipeline.

Pipeline editor (left), pipeline preview & input/output (right)

#Step 4. Write the RAG Pipeline Recipe

In the pipeline editor, we will now write the recipe for the Retrieving and Generating Phase of the RAG system.

#Input Variables

In the pipeline recipe, add the necessary input variables required from the user:

#Pipeline Components

To enhance retrieval accuracy, we'll now add a query revisor that refines the user's original query to the pipeline recipe:

By refining the user's query before searching the catalog, we can improve the accuracy of the retrieval process and, in turn, the quality of the generated response. For this step, its often preferable to use a fast, lightweight model as it typically won't need to perform complex reasoning. We choose Anthropic's low-latency claude-3-haiku-20240307 model for this component.

Now, let's add the retrieval component to the pipeline recipe:

This component searches your catalog and retrieves relevant documents based on the revised query.

To generate an answer to the user's query using the retrieved documents, we can now add the following code to the recipe:

For response generation, it can be preferable to use a more powerful model, particularly if the response requires complex reasoning or detailed explanations. In this case, we choose Anthropic's claude-3-5-sonnet-20240620 model for this component.

As you can see, this component also involves some prompt engineering. In particular, the system message includes chain-of-thought instructions for the LLM to follow when generating the response. This helps ensure the response is accurate, relevant, and contextually grounded.

#Define the Outputs

Finally, let's add outputs to capture and display the assistant's response, the revised query, and the retrieved document chunks:

#Step 5: Run the Pipeline

There are a few ways you can run a pipeline, see Run Pipeline docs.

#1. Run via Pipeline Editor

In the Input form on the bottom right, fill in the corresponding values:

Click Run to execute and observe the response in the output panel, alongside the retrieved chunks used to formulate the response.

Here are some further real-world queries you can try out:

• "Is 'fever attack' considered a symptom related to urinary tract infection?"
• "What does 'LUTS' signify in medical terms?"
• "If a patient doesn't have a catheter or has had a catheter for less than two days but shows UTI symptoms like dysuria or Lower Urinary Tract Symptoms, is this considered a CAUTI?"
• "If a patient without a catheter has a fever with UTI symptoms, is this a CAUTI? Why?"
• "A patient has had an indwelling catheter for three calendar days and still has it one day before the UTI diagnosis date, showing symptoms like lower abdominal pain and pelvic discomfort. Is this a CAUTI? Please explain why."
• "Can mixed bacterial flora be considered a pathogen for healthcare-associated infections? Why?"
• "If a patient has multiple bed transfers the day before the infection date, how should the infection department be determined?"

#2. Run via API

To generate your Instill API Token, see the API Token Management docs.

Here is an example response, showing the response, retrieved-chunks, and revised-query outputs:

#Conclusion

Congratulations on building a generalisable and customizable Retrieval-Augmented Generation (RAG) system on Instill Cloud! 🎉

Through this tutorial, you have:

1. Created an Instill Catalog to serve as a knowledge base.
2. Built a generalizable RAG pipeline with components for query revision, retrieval, and response generation for assisting doctors with UTI-related queries.
3. Successfully tested the pipeline and learned how to make API calls for seamless integration.

#Next Steps

Now that you've built a simple RAG system, here's what you can explore next:

🍎 Try the new Instill App AI Assistant Seamlessly interact and converse with an out-of-the-box RAG-based AI Assistant with conversation history that is grounded in the context of a chosen Instill Catalog.

📄 Process Complex Documents Build bespoke indexing pipelines to handle complex documents with tables, complex formatting and multi-language content.

🎨 Expand to Multimodal RAG Incorporate image, video and audio processing to handle diverse data queries and adjust your pipeline for multimodal-based information.

⛁ Integrate with Other Vector Databases Connect to vector databases like Pinecone or Weaviate to leverage existing knowledge bases.

📚 Explore API-First Documentation Dive into our documentation and Python SDK to further simplify catalog creation and seamlessly integrate with your existing tech stack.

Thank you for following along with this tutorial, and stay tuned for more AI content where we tackle advanced RAG topics (and more) coming soon! 🚀