Qwen2.5-1.5B-Instruct, fine-tuned with QLoRA (4-bit NF4) on TREC DL 2020 graded
relevance judgments, bucketed into irrelevant, partial, relevant. The adapter
implements a GraderInterface, so it slots into any RAG pipeline as a
drop-in reranker.
Grader OFF (naive RAG): every retrieved passage reaches the generator.
Grader ON (corrective RAG): passages graded irrelevant are
dropped. If none survive, the pipeline returns an explicit no grounded answer
rather than inventing text.
This page is a Static Space: it is generated from the repo's JSON artifacts and cannot run a model. The interactive version lives in the Gradio/ZeroGPU Space.
| metric | zero-shot baseline |
|---|---|
| accuracy | 0.3695 |
| macro-F1 | 0.2672 |
| F1 — irrelevant | 0.4735 |
| F1 — partial | 0.0131 |
| F1 — relevant | 0.3149 |
Baseline only: n=2122 · n_test_queries=10 · n_test_qrels=2122 · model=Qwen/Qwen2.5-1.5B-Instruct · mode=zero-shot. The fine-tuned column appears once the evaluation notebook writes results/finetuned_results.json; the success bar is +10 percentage points macro-F1. The holdout is grouped by query, so read it as directional evidence.
results/demo_trace.json. Record one on a machine with a working model:python -m src.record_demo_trace --corpus colab-t4Eight notes about fitting a fine-tune into a free-tier GPU. Four answer the query; four talk about GPUs and plans that are not the T4.
Example query: How much VRAM does a free Colab T4 have, and what do I trade away to fit a fine-tune into it?
Gate design intent — Keep the 16 GB T4, QLoRA 4-bit and gradient-checkpointing notes; the A100, L4, Colab Pro and W&B notes are decoys that match on vocabulary.
colab-t4/p1 — Free Colab sessions are assigned an NVIDIA T4 with 16 GB of VRAM. That number, not the parameter count of your model, is what decides the largest per-device batch size you can use, and it is why tutorials that fit on a paid instance OOM on the free tier.colab-t4/p2 — QLoRA keeps the base weights frozen and loads them in 4-bit NF4, so a 1.5B-parameter model occupies roughly a third of its fp16 footprint. Only the LoRA adapters are trained, and those stay in full precision, which is where most of the memory savings come from.colab-t4/p3 — Gradient checkpointing trades compute for memory: activations are thrown away during the forward pass and recomputed during the backward pass. It slows training down by roughly a third but is the single biggest lever when both batch size and sequence length are already large.colab-t4/p4 — The A100 available on paid Colab carries 80 GB of HBM, so scripts that OOM on a T4 will run unquantized with large batches on that card. The hardware is a different generation entirely, so timings do not carry over between the two.colab-t4/p5 — The NVIDIA L4 is the newer Ada Lovelace card that appears on some Colab Pro tiers. It offers 24 GB of memory and much higher bandwidth than the T4, which shifts the memory budget in a way that has to be re-measured rather than assumed from T4 numbers.colab-t4/p6 — A Colab Pro subscription buys more GPU-hours and shorter idle timeouts. It does not change the hardware that free sessions receive, so the memory ceiling a free session runs under stays exactly the same.colab-t4/p7 — The T4 is a Turing-generation card with 16 GB of GDDR6 memory. It has no tensor cores, so bf16 matrix multiplies are emulated and can run slower than fp16 rather than faster on this hardware.colab-t4/p8 — Weights & Biases logging adds negligible overhead to a training loop, but the sweeps feature needs a paid plan once you want more than a handful of concurrent runs tracked against the same project.Eight notes from a home barista's log. Five bear on a thin, fast shot; three are maintenance tasks that share words with the query.
Example query: My espresso comes out thin and gushes out far too fast. What should I change?
Gate design intent — Keep grind, pressure, temperature and channeling; descaling, gasket and milk-steaming notes are maintenance tasks, not answers.
espresso/p1 — Grind size sets flow rate. A coarse grind leaves large gaps for water to rush through, so the shot pours in seconds, the crema looks pale, and the cup tastes thin and sour at the same time. Finer by one notch is usually the whole fix.espresso/p2 — Brew pressure should reach about nine bars at the group head. If the pump never gets there the puck is too coarse to resist the water, and if the gauge sits pinned at zero the puck may be too tight or the basket blocked.espresso/p3 — Water temperature belongs between 92 and 94 degrees. Cooler water under-extracts and gives a sharp, salty cup; hotter water pushes bitterness out of the grounds and speeds up the pour noticeably.espresso/p4 — Channeling means part of the puck erodes early and water escapes down one side instead of through the bed. The signs are a fast pour with one channel in the stream and a spent puck that comes out with a hole in it.espresso/p5 — Steaming milk is a separate skill from pulling espresso: introduce air during the first third of the pitcher, then drop the wand below the surface to roll the milk until it reaches about 60 degrees. Pushing the steam any harder just makes large fast bubbles you cannot texture.espresso/p6 — Descale the machine every one to three months depending on how hard your water is. Symptoms of scale are a slow flow rate and a rising temperature reading, not changes in how fast the shot itself comes out.espresso/p7 — A burr grinder matters more than most upgrades because blade grinders produce a wide range of particle sizes. Fines from blades clog the puck and stall the shot; a consistent grind keeps the pressure steady.espresso/p8 — A leak from the group head is a gasket problem, not a taste problem. Replace the gasket when water drips from under the portafilter after the shot is pulled, and check that the basket is seated before blaming the grinder.Eight notes from a travel desk. Four are about trains and paying; four are about opening hours and sights, which retrieve well on the word 'hours'.
Example query: How late do trains run in Amsterdam, and can I pay with a bank card?
Gate design intent — Keep the night-train, tap-to-pay and OVpay notes; the museum hours and cruise notes are decoys that match on 'hours' and 'run'.
amsterdam/p1 — Night trains from Utrecht, Rotterdam and Den Haag reach Amsterdam Centraal between roughly 00:30 and 02:30. The last direct service is around 02:00, so a flight the next morning usually needs a hotel near the station rather than a night bus.amsterdam/p2 — You can tap a contactless bank card on the platform gates and buy a one-hour ticket without an account. Each tap starts a new hour, and checking in and out with the same card keeps the fare calculated per journey.amsterdam/p3 — The OV-chipkaart is being replaced by OVpay: a debit card or phone wallet works in the meantime, and OV-fiets bikes unlock with the same tap. Paper tickets are still sold at the machine inside the station.amsterdam/p4 — The Rijksmuseum opens daily at 09:00 and closes at 17:00, with last entry an hour before closing. The Anne Frank House sells timed-entry tickets online only and is often booked out weeks ahead.amsterdam/p5 — Canal cruises leave from behind the station every thirty minutes in season. The evening departures run until about 22:00 and are the quieter way to see the lit bridges than walking the canal ring.amsterdam/p6 — OV-fiets rental is included in most rail passes, otherwise it costs a few euros per day. Availability drops sharply during the evening commute when everyone is cycling home at the same time.amsterdam/p7 — Amsterdam Centraal is about fifteen minutes by direct train from Schiphol airport, and every twelve minutes or so outside the peaks. Night buses also run from the airport but take close to an hour.amsterdam/p8 — Trams 26 and 27 leave Centraal for the museums quarter, while tram 5 runs along the canal ring past the Anne Frank House. Trams stop for traffic, so the metro is faster when a schedule matters.