Understand experiments · Lesson 08
Compare the two recorded fraud Runs
The recording keeps the notebook, forest settings and seed fixed while switching between two synthetic cohorts. Their feature rows match but their fraud-label rules differ. This is a comparison-reading exercise, not an unchanged benchmark proving a better model.
By the end of this lesson
- Read material evidence separately from observed result changes.
- Preserve unknown evidence in self-comparison.
- Predict B-minus-A deltas and their reverse direction.
Watch the lesson
English · 7:15 · Recorded with Apizr 0.4.6 development candidate.
Chapters in this recording
| Time | Chapter |
|---|---|
| 00:00 | A different score is the start of an investigation |
| 00:24 | Write the comparison question first |
| 00:54 | Record baseline A |
| 01:56 | Change the cohort; retain B |
| 03:01 | Separate material evidence from observed results |
| 04:00 | Read the actual metric trade-off |
| 04:39 | Variant: compare a Run with itself |
| 05:34 | Your turn: reverse the comparison |
| 05:54 | Check the prediction against the real reverse diff |
| 06:38 | From a score to a review question |
| 07:04 | Continue with the written guide |
Read the recording transcript and commands
This is the retained English narration, in the original chapter order. Commands keep the recorded version and prerequisites; setup additions in the lesson are labelled separately.
00:00 — A different score is the start of an investigation
Your colleague sees another AUC and asks whether the new model is better. We will compare two actual fraud detection Runs. The notebook, forest settings and seed stay fixed; the active synthetic data changes. Our goal is to find the recorded differences, interpret the metric direction, and state what further evidence would be needed before choosing a model.
00:24 — Write the comparison question first
Before running anything, make the comparison question explicit. We expect the input content identity to change, while the source and declared parameters should agree. Observed scores and the saved model may change as a consequence of execution. That expectation is a hypothesis to check against the records. The two synthetic cohorts have identical feature rows but different fraud-label rules, so they are not an unchanged benchmark.
00:54 — Record baseline A
We begin with the first cohort in a fresh local project and retain a real baseline Run. The metric values are measured from the held-out split, rather than assigned as constants in a toy script. Read the model and input fingerprints alongside the numbers. This is the evidence we will compare with B. Writing A down before changing data prevents the current working directory from becoming our only memory of the experiment.
apizr experiment run fraud_detection.ipynb --output model=artifacts/model.joblib --format json > run-a.json
python lab.py run run-a.json
01:56 — Change the cohort; retain B
Replace only the active CSV, then run the same notebook with the same output selection. The active model path is reused, but the two retained Runs keep their recorded identities. Before looking at the diff, notice the experimental limitation: changing the label rule changes the task being evaluated. Even if one score rises, this setup does not establish an improvement against a fixed real-world evaluation target.
cp data/transactions-b.csv data/transactions.csv
apizr experiment run fraud_detection.ipynb --output model=artifacts/model.joblib --format json > run-b.json
python lab.py run run-b.json
03:01 — Separate material evidence from observed results
We select the exact two Run digests and ask for the canonical diff. Read material evidence first: does the input reference match while its content changes? Do intended and observed parameter values agree? Then read the results: metric values and their B minus A deltas. Unknown is its own state. It does not mean same, and it should not be silently counted as unchanged. The projection shows actual diff fields; the full report remains available.
RUN_A=$(python -c 'import json; print(json.load(open("run-a.json"))["run_digest"])'); RUN_B=$(python -c 'import json; print(json.load(open("run-b.json"))["run_digest"])')
apizr experiment diff "$RUN_A" "$RUN_B" --format json > comparison.json
python lab.py comparison comparison.json
04:00 — Read the actual metric trade-off
Read AUC, precision and recall side by side in the actual retained comparison. Each row shows A, B and the B minus A delta. A metric direction is not a universal quality ranking: precision and recall answer different error questions, while AUC summarizes ranking across thresholds. More importantly, these cohorts use different label rules. A defensible conclusion names those changed inputs and observed results; a model-quality decision still needs a stable evaluation protocol, an appropriate test population and error analysis.
04:39 — Variant: compare a Run with itself
An advanced comparison must preserve unknown evidence even in a self-comparison. Predict what happens when A is compared with A. Known numeric metric deltas should be zero. But the absence of a runtime seed observation does not become evidence that the seed was observed merely because the two record identities match. Read the actual unknown facet count and the parameter states. Identity equality and field-level knowledge are different concepts; your review should preserve both.
RUN_A=$(python -c 'import json; print(json.load(open("run-a.json"))["run_digest"])')
apizr experiment diff "$RUN_A" "$RUN_A" --format json > self-comparison.json
python lab.py comparison self-comparison.json
05:34 — Your turn: reverse the comparison
Pause and predict. If we swap A and B, what should happen to a numeric delta? What should happen to a material content difference? Would two missing observations suddenly become known? Write the expected result, then watch the same canonical comparison in the reverse direction.
05:54 — Check the prediction against the real reverse diff
The reverse comparison is a useful reading check. Comparable numeric deltas change sign because the subtraction direction changes. Different recorded content still differs. Missing evidence stays unknown rather than becoming a match. Read the actual result before accepting your prediction. This simple check teaches you how to read a comparison rather than memorizing a particular score from this machine.
RUN_A=$(python -c 'import json; print(json.load(open("run-a.json"))["run_digest"])'); RUN_B=$(python -c 'import json; print(json.load(open("run-b.json"))["run_digest"])')
apizr experiment diff "$RUN_B" "$RUN_A" --format json > reverse.json
python lab.py comparison reverse.json
06:38 — From a score to a review question
We have compared two measured model Runs, separated material and result changes, and checked the direction of the diff. Keep this habit: state the question, hold an appropriate baseline, inspect evidence origins and explain the remaining uncertainty. The next lesson makes another explicit choice: which retained prediction and which model bytes should be included in an interface?
07:04 — Continue with the written guide
The current written guide remains authoritative. Continue there for the detailed steps. Thank you for following Apizr Academy.
1. Retain A and B from the same notebook
Start a fresh fraud lab from the recorded materials. Use the same trusted scientific environment as lesson 7.
cp data/transactions-a.csv data/transactions.csv
apizr experiment run fraud_detection.ipynb --output model=artifacts/model.joblib --format json > run-a.json
cp data/transactions-b.csv data/transactions.csv
apizr experiment run fraud_detection.ipynb --output model=artifacts/model.joblib --format json > run-b.json
RUN_A=$(python -c 'import json; print(json.load(open("run-a.json"))["run_digest"])')
RUN_B=$(python -c 'import json; print(json.load(open("run-b.json"))["run_digest"])')
apizr experiment diff "$RUN_A" "$RUN_B" --format json > comparison.json
python lab.py comparison comparison.json
Read whether a reference matches while its content changes, whether intended and observed parameters agree, then read metric values and deltas. Unknown is a separate state, not another word for unchanged.
2. Interpret the trade-off actually shown
The retained recording displays these observations, rounded here exactly as its teaching projection prints them. Your Run identities and results should be read from your own environment.
| Metric | A | B | B minus A |
|---|---|---|---|
| Precision | 0.48571 | 0.58333 | +0.09762 |
| Recall | 0.35417 | 0.625 | +0.27083 |
| ROC AUC | 0.68504 | 0.65485 | −0.03019 |
Precision and recall rise while AUC falls. More fundamentally, the changed label rule changes the evaluated task. These values cannot by themselves rank generalization on a fixed real-world fraud population.
3. Compare A with itself, then reverse the pair
apizr experiment diff "$RUN_A" "$RUN_A" --format json > self-comparison.json
python lab.py comparison self-comparison.json
apizr experiment diff "$RUN_B" "$RUN_A" --format json > reverse.json
python lab.py comparison reverse.json
The recorded self-comparison has zero known metric deltas, while seven material facets remain unknown. Identical record identity does not manufacture an absent runtime observation.
Practice
Use the video's pause question: predict what reversing A and B does to a numeric delta, an input-content difference and a missing observation. Then check the actual reverse diff.
Check your reasoning
Comparable numeric deltas change sign. Different recorded input bytes still differ. Missing evidence stays unknown. Describe the observed trade-off and changed cohort definition; selecting a model needs a suitable evaluation protocol and error analysis beyond this diff.