A practical fine-tuning experiment that adapts Microsoft's Phi-3 Mini language model for payment intent classification using QLoRA, 4-bit quantization, and LoRA adapters.
The model is trained to classify payment-related user requests into 7 predefined intents, including money transfer, balance checking, transaction history, mobile recharge, FASTag recharge, account change, and PIN change.
| Component | Details |
|---|---|
| Base Model | microsoft/Phi-3-mini-4k-instruct |
| Fine-Tuning | QLoRA + LoRA |
| Quantization | 4-bit NF4 |
| Task | Payment Intent Classification |
| Intents | 7 |
| Final Dataset | 200 examples |
| Train / Validation / Test | 160 / 20 / 20 |
| Trainable Parameters | 4.72M / 3.83B (0.1233%) |
| Baseline Accuracy | 95.00% |
| Fine-Tuned Accuracy | 100.00% |
| Improvement | +5 percentage points |
| GPU | NVIDIA Tesla T4 |
| Frameworks | PyTorch, Transformers, PEFT, TRL |
Result: QLoRA fine-tuning improved Phi-3 Mini from 95% to 100% accuracy on the 20-example held-out test set used in this experiment.
Important: The reported accuracy is based on a small held-out test set and should not be interpreted as production-level model performance.
In this project, I fine-tune microsoft/Phi-3-mini-4k-instruct for a payment intent classification task.
The model receives a user request such as:
"I want to recharge my phone."
and should return only:
mobile_recharge
The model supports seven payment-related intents:
| Intent | Meaning |
|---|---|
send |
Send, transfer, or pay money to another person or account |
check_balance |
Check the current available account balance |
transaction_history |
View previous transactions, payments, transfers, or account activity |
mobile_recharge |
Recharge or top up a mobile phone/number |
fastag_recharge |
Recharge or add money to a FASTag account |
change_account |
Change, switch, replace, or update a linked bank account |
change_pin |
Change, reset, update, or create a new PIN |
The main question behind the experiment was:
Does fine-tuning improve Phi-3's performance on these payment intents?
To answer this, I first evaluate the original Phi-3 model and then compare it with the QLoRA fine-tuned version using the same held-out test set.
The complete workflow is:
Create seed dataset
↓
Add synthetic examples
↓
Remove duplicate examples
↓
Check dataset quality
↓
Train / Validation / Test split
↓
Create prompt-completion examples
↓
Load Phi-3 Mini
↓
Evaluate original Phi-3
↓
Add LoRA adapters
↓
Fine-tune using QLoRA
↓
Save LoRA adapter
↓
Evaluate fine-tuned model
↓
Compare baseline vs fine-tuned model
↓
Analyze errors
↓
Test fresh user requests
Phi-3 is already a capable language model, but this project is focused on a specific task and a fixed set of payment intents.
Instead of asking the model to understand the task only from the prompt, I provide examples showing:
User request → Correct intent
For example:
"I want to send money to my brother."
→ send
"How much money do I have in my account?"
→ check_balance
"Show me my recent transactions."
→ transaction_history
"Please recharge my phone."
→ mobile_recharge
Fine-tuning allows the model to learn this task-specific mapping from examples.
However, training all of Phi-3's parameters would require much more memory.
That is why I use QLoRA.
QLoRA combines two ideas:
- Quantization
- LoRA (Low-Rank Adaptation)
The base Phi-3 model is loaded in 4-bit precision using bitsandbytes.
This reduces the memory required to load the model.
The notebook uses:
BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=torch.float16,
)The important settings are:
load_in_4bit=True— load the base model using 4-bit weightsbnb_4bit_quant_type="nf4"— use NF4 quantizationbnb_4bit_use_double_quant=True— use double/nested quantizationbnb_4bit_compute_dtype=torch.float16— use FP16 for computation
The purpose is to make the model practical to run on the available Colab GPU.
LoRA adds a small number of trainable parameters to selected parts of the model while the original model weights remain frozen.
This means we do not need to update the entire 3.8B parameter model.
LoRA stands for Low-Rank Adaptation.
Instead of updating all of the original model parameters, LoRA adds small trainable matrices to selected parts of the model.
The original Phi-3 weights remain frozen.
Only the LoRA parameters are updated during training.
Conceptually:
Original Phi-3
│
├── Original weights → Frozen
│
└── LoRA adapters → Trainable
In this experiment:
Total parameters: 3,825,798,144
Trainable parameters: 4,718,592
Trainable percentage: 0.1233%
So only a very small fraction of the model parameters are trained.
The LoRA configuration targets these two modules:
target_modules=["qkv_proj", "o_proj"]These modules are part of the model's attention mechanism.
This is the projection used to produce the combined Query, Key, and Value representations used by self-attention.
This is the output projection of the attention block.
So the experiment adapts selected attention projections rather than adding LoRA to every layer.
Conceptually:
Attention block
Input
│
├── qkv_proj ← LoRA
│
│ Attention
│
└── o_proj ← LoRA
│
↓
Output
The project uses:
LoraConfig(
r=8,
lora_alpha=16,
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
target_modules=["qkv_proj", "o_proj"],
)This is the rank of the low-rank LoRA matrices.
A smaller rank means fewer trainable parameters.
Here:
r = 8
This controls the scaling applied to the LoRA update.
Here:
alpha = 16
A small dropout is applied to the LoRA path during training.
dropout = 5%
No additional bias parameters are trained.
Phi-3 is being used as a causal language model.
Only these attention-related projections receive LoRA adapters:
qkv_proj
o_proj
The project uses a small synthetic dataset created specifically for this experiment.
No real customer or payment information is used.
Each example contains:
text
intent
For example:
text:
"I want to transfer money to my friend."
intent:
send
The initial seed dataset contains:
35 examples
5 examples per intent
Additional synthetic examples are then added.
The expanded dataset contains:
210 examples
After removing:
10 exact duplicates
the final dataset contains:
200 examples
| Intent | Examples |
|---|---|
change_account |
29 |
change_pin |
27 |
check_balance |
29 |
fastag_recharge |
29 |
mobile_recharge |
29 |
send |
30 |
transaction_history |
27 |
| Total | 200 |
The class distribution is reasonably balanced for this small experiment.
Before training, the dataset is checked for:
- required columns
- missing values
- invalid intent labels
- duplicate rows
- missing intents
The final dataset passed these checks.
The notebook verifies that:
text
intent
are present and that all seven intents are represented.
The final 200 examples are split into three groups:
Training: 160 examples
Validation: 20 examples
Test: 20 examples
The split is stratified so that the seven intents remain represented across the splits.
The test set is kept separate and is used for the final model comparison.
The notebook also checks for text overlap between the three splits.
The result was:
Train ∩ Validation: 0
Train ∩ Test: 0
Validation ∩ Test: 0
This helps avoid accidentally evaluating on the same text used for training.
Phi-3 is a generative language model, so instead of using a traditional classification head, the classification task is formulated as a text generation task.
The prompt tells the model the available intents and asks it to return only one intent name.
Example:
You are a payment intent classification assistant.
Classify the user's request into exactly one of these intents:
- send
- check_balance
- transaction_history
- mobile_recharge
- fastag_recharge
- change_account
- change_pin
Return only the intent name. Do not provide an explanation.
User request:
Please top up my mobile number.
Intent:
The expected completion is:
mobile_recharge
This same prompt structure is used during training and inference.
The base model used is:
microsoft/Phi-3-mini-4k-instruct
It is loaded using Transformers.
The model is loaded in 4-bit precision using bitsandbytes.
base_model = AutoModelForCausalLM.from_pretrained(
BASE_MODEL,
quantization_config=bnb_config,
device_map="auto",
)The original base model is evaluated before LoRA is attached.
This gives us the baseline result.
Before fine-tuning, the original Phi-3 model is tested on the 20-example test set.
The baseline accuracy was:
95.00%
The baseline model made one incorrect prediction.
The incorrect example was:
Request:
Please show my previous account activity.
True intent:
transaction_history
Predicted:
invalid
This baseline is important because without it there would be no way to measure whether fine-tuning actually helped.
The training examples are converted into a prompt-completion format.
Each example contains:
prompt
completion
For example:
prompt:
You are a payment intent classification assistant.
...
User request:
Please top up my mobile number.
Intent:
completion:
mobile_recharge
The completion is the target intent.
This lets the language model learn to generate the correct intent after the prompt.
The project uses TRL's SFTTrainer for supervised fine-tuning.
Main settings:
| Setting | Value |
|---|---|
| Epochs | 3 |
| Batch size | 2 |
| Gradient accumulation | 4 |
| Effective batch size | 8 |
| Learning rate | 2e-4 |
| Optimizer | paged_adamw_8bit |
| Weight decay | 0.01 |
| Maximum sequence length | 128 |
| Completion-only loss | Enabled |
| Gradient checkpointing | Enabled |
| FP16 | Disabled |
| BF16 | Disabled |
The effective batch size is:
2 × 4 = 8
because four gradient accumulation steps are used for every optimizer update.
The GPU can process only a limited number of examples at once.
Instead of increasing the actual batch size, the project uses:
Batch size = 2
Gradient accumulation = 4
The model processes four mini-batches before updating the trainable parameters.
So the effective batch size becomes:
2 × 4 = 8
This gives a larger effective batch without requiring the GPU to hold eight examples simultaneously.
Gradient checkpointing is another memory-saving technique.
Normally, the model stores many intermediate activations during the forward pass so they can be used during backpropagation.
Gradient checkpointing stores fewer activations and recomputes some of them when needed.
This trades some extra computation for lower memory usage.
For this experiment, reducing GPU memory usage was more important than minimizing training time.
The actual fine-tuning is performed using:
training_result = trainer.train()The model was trained for three epochs.
Training completed successfully.
The recorded losses were:
| Epoch | Training Loss | Validation Loss |
|---|---|---|
| 1 | 0.005175 | 0.000094 |
| 2 | 0.000042 | 0.000030 |
| 3 | 0.000025 | 0.000026 |
The training output also reports token-level metrics such as mean token accuracy.
These should not be confused with the final intent classification accuracy.
The final classification accuracy is measured separately on the held-out test set.
After training, the project saves the trained LoRA adapter using:
trainer.save_model(ADAPTER_DIR)
tokenizer.save_pretrained(ADAPTER_DIR)The output directory is:
phi3-payment-intent-lora/
The saved adapter contains files such as:
adapter_config.json
adapter_model.safetensors
tokenizer.json
tokenizer_config.json
chat_template.jinja
training_args.bin
The important idea is that the project does not save another full copy of the Phi-3 model.
Instead, it saves the smaller set of LoRA parameters learned during fine-tuning.
The saved adapter depends on the original Phi-3 base model.
Conceptually:
Phi-3 Mini base model
+
LoRA adapter
↓
Fine-tuned Phi-3
The base model provides the original language understanding.
The LoRA adapter contains the task-specific changes learned during fine-tuning.
For later inference, the base model can be loaded again and the saved adapter can be attached to it.
Conceptually, the PEFT loading workflow is:
from peft import PeftModel
base_model = AutoModelForCausalLM.from_pretrained(...)
model = PeftModel.from_pretrained(
base_model,
"phi3-payment-intent-lora"
)The important point is:
The adapter is not a complete standalone copy of Phi-3. The original base model is also required.
In the current notebook, the new-query inference is performed using the already fine-tuned model object in memory after training. The notebook saves the adapter for later reuse, but it does not separately reload the saved adapter from disk before running those new-query examples.
After fine-tuning, the model is evaluated on the same 20-example test set used for the baseline.
This makes the comparison fair.
The fine-tuned model achieved:
100.00% accuracy
All 20 test examples were classified correctly.
| Model | Accuracy |
|---|---|
| Original Phi-3 | 95.00% |
| Phi-3 + QLoRA | 100.00% |
Improvement:
+5 percentage points
The comparison is also saved to:
results/model_comparison.csv
The baseline model made one error:
Input:
Please show my previous account activity.
Expected:
transaction_history
Baseline:
invalid
After fine-tuning:
Fine-tuned errors:
No fine-tuned errors.
This suggests that fine-tuning helped the model on the examples in this test set.
However, because the test set contains only 20 examples, this result should not be treated as production-level accuracy.
After evaluation, I also tested the fine-tuned model on several fresh requests.
Examples include:
I need to transfer some money to my mother.
→ send
Can you tell me how much balance is left?
→ check_balance
Show me the payments I made yesterday.
→ transaction_history
Please recharge my mobile number.
→ mobile_recharge
I need to add funds to my FASTag.
→ fastag_recharge
I want to use another linked bank account.
→ change_account
How can I reset my PIN?
→ change_pin
These examples are used as a quick sanity check of the model's behavior on fresh requests.
They are not included in the reported test accuracy.
The experiment was run in Google Colab using a Tesla T4 GPU.
Main environment:
PyTorch: 2.11.0+cu128
Transformers: 5.16.1
Datasets: 5.0.1
PEFT: 0.20.0
TRL: 1.12.0
bitsandbytes: 0.50.2
CUDA available: True
GPU: Tesla T4
CUDA version: 12.8
phi3-payment-intent-finetuning/
│
├── README.md
├── requirements.txt
├── .gitignore
│
├── notebook/
│ └── phi3_payment_intent_finetuning.ipynb
│
├── data/
│ ├── payment_intents_seed.csv
│ ├── train.csv
│ ├── validation.csv
│ └── test.csv
│
├── results/
│ ├── baseline_predictions.csv
│ ├── fine_tuned_predictions.csv
│ └── model_comparison.csv
│
└── screenshots/
├── 01_environment.png
├── 02_dataset.png
├── 03_data_split.png
├── 04_lora_parameters_training.png
├── 05_fine_tune.png
├── 06_baseline_finetune_errors.png
└── 07_comparison.png
Contains the complete experiment from dataset creation to evaluation.
Contains the dataset artifacts used in the experiment.
payment_intents_seed.csv— original seed examplestrain.csv— training examplesvalidation.csv— validation examplestest.csv— held-out test examples
Contains the model evaluation results.
baseline_predictions.csv— predictions made by the original Phi-3fine_tuned_predictions.csv— predictions made by the fine-tuned modelmodel_comparison.csv— accuracy comparison
Contains screenshots of important experiment outputs.
This project is intentionally a small learning experiment.
There are several limitations.
The final dataset contains only:
200 examples
This is enough for demonstrating the fine-tuning workflow, but not enough to represent the wide variety of language used by real users.
The test set contains:
20 examples
Therefore, the reported 100% accuracy should be interpreted only as performance on this test set.
It should not be interpreted as general production accuracy.
The examples were created specifically for this project.
Real payment systems would require much more diverse user language, including:
- spelling mistakes
- short messages
- incomplete requests
- regional language variations
- mixed-language requests
- speech-to-text errors
- ambiguous requests
The model only has seven available intents.
There is no separate:
out_of_scope
class.
Therefore, an unrelated request could potentially be forced into one of the seven payment intents.
This makes the project a closed-set intent classifier, not a general-purpose intent detector.
This project focuses on the fine-tuning experiment.
It does not include:
- production API deployment
- authentication
- monitoring
- model serving infrastructure
- real-time traffic testing
- production-scale evaluation
This project helped me understand the complete fine-tuning workflow rather than only running a training command.
The main concepts covered are:
- causal language models
- supervised fine-tuning
- prompt-completion datasets
- 4-bit quantization
- QLoRA
- LoRA adapters
- LoRA rank and scaling
- attention projection layers
- trainable vs frozen parameters
- gradient accumulation
- gradient checkpointing
- validation vs test data
- baseline evaluation
- classification metrics
- error analysis
- saving and reusing LoRA adapters
The most important idea from this project is that a large pretrained model does not always need to be fully retrained for a specialized task.
A small number of trainable adapter parameters can be used to adapt the model while keeping the original model frozen.
Base model:
microsoft/Phi-3-mini-4k-instruct
Final dataset:
200 examples
Train / Validation / Test:
160 / 20 / 20
Fine-tuning method:
QLoRA
Quantization:
4-bit NF4
LoRA target modules:
qkv_proj
o_proj
LoRA rank:
8
Trainable parameters:
4,718,592
Total parameters:
3,825,798,144
Trainable percentage:
0.1233%
Baseline accuracy:
95.00%
Fine-tuned accuracy:
100.00%
Improvement:
5.00 percentage points
This project demonstrates a complete parameter-efficient fine-tuning workflow for adapting Phi-3 Mini to a small payment intent classification task.
The original model achieved 95% accuracy on the held-out 20-example test set.
After applying QLoRA fine-tuning, the model achieved 100% accuracy on the same test set.
The experiment shows how LoRA can adapt a large language model by training only a small fraction of its parameters while keeping the base model frozen.
The result is promising for this small experiment, but larger and more diverse datasets would be required before making claims about real-world performance.