
Different Approaches to Train or Fine-Tune a Large Language Model (LLM)
Large Language Models (LLMs) such as GPTs are powerful tools that can perform a wide variety of natural language processing (NLP) tasks. However, the true power of LLMs lies in their ability to be customized for specific use cases. There are several approaches to train or fine-tune LLMs to align them with particular applications, goals, or domains. Below, we explore the key methods used to customize LLMs: RAG (Retrieval-Augmented Generation) and fine-tuning.
1. Retrieval-Augmented Generation (RAG) or Grounding
Overview
RAG is a technique where the LLM leverages external data sources, such as custom databases or knowledge repositories, during inference. Instead of modifying the model itself, RAG enhances the model’s responses by grounding its answers in domain-specific knowledge.
How It Works
- The LLM is paired with an external data retrieval system (e.g., a database, document store, or search engine).
- During inference, the system retrieves the most relevant information from the external data source based on the input query.
- The retrieved data is provided to the LLM as context, enabling it to generate responses that are accurate and grounded in the specific domain.
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from haystack.document_store.memory import InMemoryDocumentStore
from haystack.nodes import BM25Retriever
# Load the pre-trained LLM
model_name = "facebook/bart-large"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
# Set up the document store and retriever
document_store = InMemoryDocumentStore()
retriever = BM25Retriever(document_store=document_store)
def retrieve_and_generate(query, context_docs):
# Retrieve relevant documents
retrieved_docs = retriever.retrieve(query, top_k=3)
context = "\n".join([doc.content for doc in retrieved_docs])
# Generate grounded response
inputs = tokenizer(f"{context}\n\n{query}", return_tensors="pt", truncation=True)
outputs = model.generate(**inputs)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
return response
# Example usage
query = "Explain the benefits of solar energy."
print(retrieve_and_generate(query, document_store))
2. Fine-Tuning
Overview
Fine-tuning involves training the LLM on a smaller, domain-specific dataset to modify its behavior and align it with a particular task or field. Unlike RAG, fine-tuning adjusts the model’s weights, creating a new version of the LLM tailored to the specific use case.
How It Works
- A custom dataset is curated to align with the desired task or domain.
- The LLM is trained on this dataset using gradient descent techniques to optimize its weights.
- The resulting fine-tuned model is then deployed for use.
from transformers import AutoTokenizer, AutoModelForCausalLM, Trainer, TrainingArguments
from datasets import load_dataset
# Load base model and tokenizer
model_name = "gpt2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
# Load and preprocess dataset
dataset = load_dataset("path_to_custom_dataset")
def tokenize_function(examples):
return tokenizer(examples["text"], truncation=True, padding="max_length")
tokenized_datasets = dataset.map(tokenize_function, batched=True)
def preprocess_logits_for_metrics(logits, labels):
predictions = logits.argmax(dim=-1)
return predictions, labels
# Training arguments
training_args = TrainingArguments(
output_dir="./results",
evaluation_strategy="epoch",
learning_rate=5e-5,
per_device_train_batch_size=4,
num_train_epochs=3,
save_strategy="epoch",
save_total_limit=2,
weight_decay=0.01,
logging_dir="./logs",
)
# Train the model
trainer = Trainer(
model=model,
args=training_args,
train_dataset=tokenized_datasets["train"],
eval_dataset=tokenized_datasets["validation"],
tokenizer=tokenizer,
)
trainer.train()
# Save the fine-tuned model
model.save_pretrained("./fine_tuned_model")
Comparison of RAG and Fine-Tuning
Aspect | RAG/Grounding | Fine-Tuning |
---|---|---|
Customization Method | Leverages external knowledge sources | Modifies model weights |
Resource Requirements | Low (no retraining needed) | High (requires computational power) |
Adaptability | Highly flexible with dynamic data | Specialized for static domains/tasks |
Latency | May introduce delays during inference | Typically faster after deployment |
Risk of Overfitting | Low (depends on retrieval system) | High if dataset is not well-curated |
3. Other Approaches to Model Customization
In addition to RAG and fine-tuning, there are other emerging techniques for customizing LLMs:
- Prompt Engineering: Crafting specific prompts to elicit desired behavior without altering the model. This is quick and resource-efficient but limited in scope.
- In-Context Learning: Providing examples within the input prompt to guide the model’s responses. Useful for one-off tasks.
- Adapters and Plug-ins: Modular approaches where external layers or modules are added to the model without changing its core architecture.

import os
import pandas as pd
from IPython.display import Markdown, HTML, display
from langchain.schema import HumanMessage
from langchain_openai import AzureChatOpenAI
model = AzureChatOpenAI(
openai_api_version="2023-05-15",
azure_deployment="gpt-4-1106",
azure_endpoint=os.getenv("AZURE_OPENAI_ENDPOINT"),
)
df = pd.read_csv("./data/anyfile.csv").fillna(value = 0)
#prepare langchain agent
from langchain.agents.agent_types import AgentType
from langchain_experimental.agents.agent_toolkits import create_pandas_dataframe_agent
agent = create_pandas_dataframe_agent(llm=model,df=df,verbose=True)
agent.invoke("how many rows are there?")
# Take help from prompt
CSV_PROMPT_PREFIX = """
First set the pandas display options to show all the columns,
get the column names, then answer the question.
"""
CSV_PROMPT_SUFFIX = """
- **ALWAYS** before giving the Final Answer, try another method.
Then reflect on the answers of the two methods you did and ask yourself
if it answers correctly the original question.
If you are not sure, try another method.
- If the methods tried do not give the same result,reflect and
try again until you have two methods that have the same result.
- If you still cannot arrive to a consistent result, say that
you are not sure of the answer.
- If you are sure of the correct answer, create a beautiful
and thorough response using Markdown.
- **DO NOT MAKE UP AN ANSWER OR USE PRIOR KNOWLEDGE,
ONLY USE THE RESULTS OF THE CALCULATIONS YOU HAVE DONE**.
- **ALWAYS**, as part of your "Final Answer", explain how you got
to the answer on a section that starts with: "\n\nExplanation:\n".
In the explanation, mention the column names that you used to get
to the final answer.
"""
QUESTION = "How may patients were hospitalized during July 2020"
"in Texas, and nationwide as the total of all states?"
"Use the hospitalizedIncrease column"
agent.invoke(CSV_PROMPT_PREFIX + QUESTION + CSV_PROMPT_SUFFIX)