LLM Playground

Manual Crawling

Targeted collection from specific high-quality sources like Wikipedia, academic papers, books, and curated websites.

# Example: Simple web crawler
import requests
from bs4 import BeautifulSoup

def crawl_page(url):
    response = requests.get(url)
    soup = BeautifulSoup(response.text, 'html.parser')
    text = soup.get_text()
    return clean_text(text)

Common Crawl

A massive open repository of web crawl data, containing petabytes of raw web page content collected over years.

RefinedWeb

High-quality filtered web data using strict deduplication and quality filtering.

• URL filtering
• Text extraction
• Language identification
• Quality scoring

Dolma

Open corpus for language model pre-training with documented curation pipeline.

• Multi-source mixing
• Deduplication
• Content filtering
• Reproducible pipeline

FineWeb

15 trillion token dataset with aggressive deduplication and filtering.

• MinHash deduplication
• Quality classifiers
• Educational content boost
• Open and reproducible

What is SFT?

SFT involves training the model on a curated dataset of (instruction, response) pairs to learn to follow instructions and provide helpful responses.

SFT Data Examples

Reward Models

Reward models learn to predict human preferences from comparison data.

# Reward model training objective
def reward_loss(preferred, rejected):
    # Bradley-Terry model
    return -log(sigmoid(r(preferred) - r(rejected)))

PPO (Proximal Policy Optimization)

PPO optimizes the policy while preventing too large updates.

• Clipped objective prevents large updates
• KL penalty to stay close to SFT model
• Value function for advantage estimation

Verifiable Tasks

Tasks with objectively verifiable outputs enable automatic reward signals.

DPO (Direct Preference Optimization)

A simpler alternative to PPO that directly optimizes on preferences without a reward model.

Perplexity

Measures how well the model predicts the test data. Lower is better.

BLEU Score

Measures n-gram overlap with reference text. Used for translation.

ROUGE Score

Recall-oriented measure for summarization tasks.

• ROUGE-N: N-gram overlap
• ROUGE-L: Longest common subsequence
• ROUGE-W: Weighted LCS

F1 Score

Harmonic mean of precision and recall for QA tasks.

Human Evaluation Methods

• Absolute Rating

  Rate outputs on a Likert scale (1-5)

• Pairwise Comparison

  Compare two outputs and pick the better one

• Ranking

  Rank multiple outputs from best to worst

Chatbot Arena (LMSYS)

Crowdsourced platform for blind model comparisons using Elo ratings.

Ratings are approximate and change over time

Conversation Memory

Managing context across multiple turns.

• Buffer Memory (recent messages)
• Summary Memory (compressed history)
• Entity Memory (key information extraction)
• Vector Memory (semantic retrieval)

RAG (Retrieval Augmented Generation)

Enhance LLM responses with external knowledge.

Function/Tool Calling

Enable LLMs to interact with external systems.

{
  "name": "get_weather",
  "parameters": {
    "location": "San Francisco",
    "unit": "celsius"
  }
}

Safety & Guardrails

Ensuring safe and appropriate responses.

• Input validation
• Content filtering
• Output moderation
• Rate limiting