Local LLM Setup Guide for Personal Portfolio Chatbot

Overview

This guide will help you set up a local LLM (Large Language Model) that runs on your computer and knows everything about you from your resume and portfolio.

Why Local LLM?

âś… Privacy: All data stays on your machine âś… No API costs: Completely free to run âś… No rate limits: Unlimited conversations âś… Offline capable: Works without internet âś… Customizable: Fine-tune on your specific data

Step 1: Install Ollama

macOS (Your System)

# Download and install from website
# Visit: https://ollama.ai/download

# Or use Homebrew
brew install ollama

Verify Installation

ollama --version

Step 2: Start Ollama Service

# Start the Ollama service (runs in background)
ollama serve

Keep this terminal window open, or run it as a background service.

Step 3: Download a Model

Option 1: Llama 3.2 (3B) - Fast & Good Quality

ollama pull llama3.2:3b
  • Size: ~2GB
  • RAM needed: 4GB
  • Speed: Very fast
  • Quality: Excellent for Q&A

Option 2: Llama 3.1 (8B) - Best Balance

ollama pull llama3.1:8b
  • Size: ~4.7GB
  • RAM needed: 8GB
  • Speed: Fast
  • Quality: Excellent

Option 3: Mistral (7B) - Alternative

ollama pull mistral:7b
  • Size: ~4.1GB
  • RAM needed: 8GB
  • Speed: Fast
  • Quality: Very good

Option 4: Phi-3 (3.8B) - Lightweight

ollama pull phi3:3.8b
  • Size: ~2.3GB
  • RAM needed: 4GB
  • Speed: Very fast
  • Quality: Good

Check Downloaded Models

ollama list

Step 4: Test the Model

# Test basic conversation
ollama run llama3.2:3b

# Try asking: "Hello, who are you?"
# Type /bye to exit

Step 5: Create Custom Model with Your Data

Create a Modelfile

I’ve already created Modelfile in your project with your resume data embedded.

Build Your Custom Model

# Navigate to your project directory
cd /path/to/your/project

# Create the custom model
ollama create darya-assistant -f Modelfile

# This creates a model specifically trained on your information

Test Your Custom Model

ollama run darya-assistant

# Try questions like:
# "What are Darya's technical skills?"
# "Tell me about her research experience"
# "What projects has she worked on?"

Step 6: Verify Chatbot Configuration

The chatbot is already configured to use your local LLM. Just verify the settings in chatbot.js:

const CHATBOT_CONFIG = {
    provider: 'ollama',
    model: 'darya-assistant',  // Your custom model
    endpoint: 'http://localhost:11434',
    temperature: 0.7,
    maxTokens: 500
};

Step 7: Start Your Website

# Option 1: Simple Python server
python3 -m http.server 8000

# Option 2: Node.js server (if you have it)
npx http-server -p 8000

# Option 3: PHP server
php -S localhost:8000

Then open: http://localhost:8000

Testing Your Chatbot

Questions to Test:

  1. “What programming languages does Darya know?”
  2. “Tell me about Darya’s research on Graph Neural Networks”
  3. “What is Darya’s GPA?”
  4. “What projects has Darya built with C++?”
  5. “Where can I contact Darya?”
  6. “What leadership roles has Darya held?”
  7. “Does Darya speak any foreign languages?”
  8. “What certifications does Darya have?”
  9. “Tell me about Darya’s quantitative trading project”
  10. “Is Darya looking for internships?”

Performance Optimization

If Responses Are Slow:

  1. Use a smaller model (llama3.2:3b or phi3)
  2. Reduce maxTokens in config
  3. Lower temperature for faster, more focused responses
  4. Close other heavy applications

If Running Out of Memory:

# Use quantized models (smaller, faster)
ollama pull llama3.2:3b-q4_0

Advanced: Fine-Tuning (Optional)

For even better responses, you can fine-tune the model:

Create Training Data

Create a file training-data.jsonl:

{"prompt": "What are Darya's main skills?", "completion": "Darya's main technical skills include C++ with STL and multi-threading, Python with PyTorch, pandas, and NumPy, as well as JavaScript, Bash, and C. She's experienced with machine learning frameworks, cloud platforms like AWS and Google Cloud, and tools like Docker and Git."}
{"prompt": "Tell me about Darya's research", "completion": "Darya is a Research Assistant at FSU's Computer Science Department, focusing on adversarial machine learning and Graph Neural Networks. She's reproduced state-of-the-art GNN fingerprinting frameworks using PyTorch and is developing statistical defenses for model ownership verification."}

Fine-tune with Ollama

# This requires additional setup
ollama create darya-finetuned -f Modelfile-finetuned

Troubleshooting

Ollama Not Starting

# Check if already running
ps aux | grep ollama

# Kill existing process
killall ollama

# Restart
ollama serve

Model Not Found

# List available models
ollama list

# Pull the model again
ollama pull llama3.2:3b

CORS Issues

If you get CORS errors, start Ollama with:

OLLAMA_ORIGINS=* ollama serve

Connection Refused

Make sure Ollama is running:

# Check if service is up
curl http://localhost:11434/api/tags

Monitoring

Check Model Performance

# View running models
ollama ps

# Check system resources
ollama show darya-assistant

View Logs

# macOS logs location
tail -f ~/Library/Logs/Ollama/server.log

Deployment Options

Option 1: Local Development Only

  • Run Ollama on your machine
  • Website works only on localhost
  • Perfect for testing and demos

Option 2: Deploy with Backend Server

  • Set up a VPS (DigitalOcean, AWS, etc.)
  • Install Ollama on server
  • Configure your website to call server endpoint
  • Add authentication and rate limiting

Option 3: Hybrid Approach

  • Use local LLM for development
  • Switch to cloud API (OpenAI) for production
  • Best of both worlds

Cost Comparison

Local LLM (Ollama)

  • Initial: $0 (free)
  • Running: $0 (uses your computer)
  • Electricity: ~$0.01/hour (negligible)

Cloud API (OpenAI)

  • GPT-3.5: ~$0.002 per 1K tokens
  • GPT-4: ~$0.03 per 1K tokens
  • Estimated: $10-50/month for moderate traffic

Next Steps

  1. âś… Install Ollama
  2. âś… Download a model (llama3.2:3b recommended)
  3. âś… Create custom model with your data
  4. âś… Test the chatbot on your website
  5. âś… Fine-tune responses if needed
  6. âś… Deploy to production (optional)

Resources

  • Ollama Documentation: https://github.com/ollama/ollama
  • Model Library: https://ollama.ai/library
  • Community Discord: https://discord.gg/ollama

Support

If you encounter issues:

  1. Check Ollama is running: ollama ps
  2. Verify model is loaded: ollama list
  3. Test API: curl http://localhost:11434/api/tags
  4. Check browser console for errors
  5. Review chatbot.js configuration

Ready to go! Once Ollama is installed and running, your chatbot will work completely offline with no API costs.