Sign in Subscribe

Building a Personal Knowledge Assistant: A Local RAG Pipeline

Building a Personal Knowledge Assistant: A Local RAG Pipeline
Photo by Niklas Ohlrogge (niamoh.de) / Unsplash

Ever wanted to build your own AI assistant that can answer questions about your personal knowledge base? I recently built a local RAG (Retrieval-Augmented Generation) system that does exactly that - and keeps everything private by running entirely on my machine.

The Challenge

I subscribe to a weekly newsletter that's packed with valuable insights on marketing analytics. After accumulating dozens of PDFs (of such newsletter) over time, I found myself constantly searching through them to find specific information or references. That's when I thought: why not build an AI assistant that could instantly answer questions about this content?

The Technical Approach

Stack Selection

  • LangChain for document processing and RAG orchestration
  • PyPDF for extracting text from PDF files
  • HuggingFace Embeddings (sentence-transformers/all-MiniLM-L6-v2) for local embeddings
  • Chroma as the vector database
  • OpenAI GPT-4o-mini for the language model
  • Gradio for a simple web interface

The Pipeline

1. Document Ingestion

# Load all PDFs from local directory
loader = DirectoryLoader(folder, glob="**/*.pdf", loader_cls=PyPDFLoader)
documents = loader.load()

# Split into manageable chunks
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
chunks = text_splitter.split_documents(documents)

2. Local Embeddings

Instead of relying on external APIs for embeddings, I used HuggingFace's sentence transformers locally:

embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")

3. Vector Storage

vectorstore = Chroma.from_documents(
    documents=chunks, 
    embedding=embeddings, 
    persist_directory=db_name
)

4. Conversational Interface
Built a conversational retrieval chain that maintains context across queries.

The Results

The system successfully processed 1,836 text chunks with 384-dimensional embeddings. I even created a t-SNE visualization to see how the content clusters in vector space - pretty satisfying to see related topics naturally grouping together!

How the Vector Groupings Look Like!

Why This Approach Works

Privacy First: The only external API call is to OpenAI for the final response generation. All document processing, embedding, and storage happens locally.

Contextual Responses: Unlike simple keyword search, the system understands semantic meaning and can find relevant information even when you don't use exact terms from the source material.

Conversation Memory: The assistant remembers your conversation context, so you can ask follow-up questions naturally.

Key Learnings

  1. Chunk Size Matters: Finding the right balance between context and specificity in your chunk size is crucial. I settled on 1000 characters with 200 character overlap.
  2. Local Embeddings: HuggingFace's sentence transformers work surprisingly well for this use case and keep everything private.
  3. Vector Visualization: Adding t-SNE visualization helped me understand how well the content was being embedded and clustered.

What's Next?

This is just the beginning. I'm thinking about:

  • Adding support for other document types
  • Implementing better chunk strategies
  • Building a more sophisticated UI
  • Adding citation tracking to show exact sources

The beauty of this approach is that it's completely adaptable - you could use it for research papers, company documents, personal notes, or any collection of text-based knowledge you want to make searchable and conversational.

Want to build something similar? The core components are surprisingly accessible, and the privacy benefits of running everything locally make it worth the setup effort.