QLoRA Without Guesswork: Start With Evals, Then Scale the Data

QLoRA makes fine-tuning look deceptively easy. Quantize a base model to four bits, train a small adapter on one GPU, and export a file that may be a few hundred megabytes.

The mechanics are easy now. The difficult part is deciding whether the result improved the product.

A good QLoRA project starts with an evaluation, not a training run. The evaluation defines the behaviour you want, the errors you can accept, and the cases the model must handle. Once that exists, choices about quantization and dataset size become experiments with clear outcomes.

Four-bit QLoRA is a strong default

QLoRA keeps the base model frozen in a quantized form and trains low-rank adapter weights. Four-bit quantization reduces memory enough to fit models and training batches on a modest GPU. Libraries such as Unsloth have made this workflow accessible to small teams.

Use four-bit QLoRA as the first serious run unless your hardware or model family gives you a reason to choose differently. Then test eight-bit or a higher-precision path against the same held-out cases. The winning configuration is the cheapest one that meets the quality target.

Avoid choosing precision from a general rule about model size. Quantization interacts with the base model, adapter rank, training data, optimiser, sequence length, and serving runtime. Treat the full configuration as a versioned recipe.

Training data should grow from the evaluation

Teams often ask for a universal minimum number of examples. There is none. A stable classification task may improve with a few thousand examples. A broader generation task can need much more variety.

The session guidance put the practical range for many bounded tasks at a few thousand to roughly 10,000 examples. That range is useful for budgeting, but your learning curve should decide when to stop.

Begin with a small, clean training set built around the same failure categories as the evaluation. Run the fine-tune. Plot held-out quality as you add data. If another 1,000 examples barely move the score, more data of the same kind will not rescue the project. Inspect the failures and add cases that cover the missing behaviour.

Synthetic data can help. Start from human-reviewed training cases, perturb their wording and structure, and generate related examples with a stronger model. Keep the held-out set separate. Synthetic variants derived from the test cases must never leak back into training.

Measure domain gain and general loss

Fine-tuning creates a specialist. That can reduce quality outside the target domain.

The presentation included a German legal retrieval experiment using a 568 million parameter Arctic embedding model. A 114 MB LoRA adapter trained in 41 minutes on one NVIDIA L4. On the in-domain GerDaLIRSmall benchmark, nDCG@10 rose from 0.2613 to 0.3087, an 18.1% relative gain. Legal transfer performance improved by 8%.

General German retrieval dropped by 2.1%, while a general question-answering score fell by 0.6%. Mixing legal and general training data at a 2:1 ratio kept most of the legal gain and reduced the wider damage.

That is the right shape of analysis. Report the target gain and the transfer cost. A single improved benchmark can hide a model that became less useful everywhere else.

Use short runs to eliminate bad recipes

Every part of QLoRA can be tuned, but a team does not need to explore every combination at full scale.

Run small experiments first. Vary one setting, compare on the same held-out cases, and discard weak configurations. Give longer runs and larger datasets to the few recipes that survive. Save the winning recipe with the base model revision, data version, prompt format, adapter settings, and serving profile.

Automation can help here. A research agent can launch small training runs, collect eval results, and propose the next configuration. It should receive raw scores and failures, not a single aggregate number. Human review still decides whether the apparent gain reflects the behaviour the product needs.

Fine-tuning becomes routine once the evaluation is trustworthy. Before that, it is expensive guesswork with a convincing loss curve.

QLoRA questions from engineering teams

What is QLoRA?

QLoRA freezes a quantized base model and trains small low-rank adapter weights. Four-bit quantization cuts memory use enough to fine-tune many models on a single GPU.

Should QLoRA use four-bit or eight-bit quantization?

Four-bit is a practical starting point. Compare it with higher precision on the same held-out evaluation, then choose the lowest-cost configuration that meets the quality target and works with the serving runtime.

How much training data does a LoRA fine-tune need?

Many bounded tasks show useful gains with a few thousand to about 10,000 examples. The right amount depends on task variety and label quality. Use a learning curve and failure analysis instead of a fixed target.

Can LoRA fine-tuning hurt general performance?

Yes. Domain training can reduce quality on general or transfer tasks. Evaluate both, and mix some general data into training when the application needs to preserve wider behaviour.