Track and review · Lesson 10
Capture the fraud holdout with sessions and decorators
The recorded lab evaluates 120 held-out synthetic rows, using probabilities from the actual trained fraud forest. It records precision, recall and flagged volume, first for a block and then for repeated function calls.
By the end of this lesson
- Choose a session or decorator for the unit of work you want to review.
- Retain the evaluation file, threshold and observed metrics.
- Check that tracking preserves normal returns and the original exception.
Watch the lesson
English · 6:29 · Recorded with Apizr 0.4.6 development candidate.
Chapters in this recording
| Time | Chapter |
|---|---|
| 00:00 | The same model can make different review queues |
| 00:27 | Decide what one Run should mean |
| 00:56 | Inspect the held-out sample and metric calculation |
| 01:57 | Record a block with named evidence |
| 02:51 | Why report precision and recall together? |
| 03:22 | Compare two calls; keep their normal results |
| 04:24 | Variant: failed capture preserves the exception |
| 05:20 | Your turn: an in-memory array replaces the file |
| 05:46 | Capture the evidence your decision needs |
| 06:18 | Continue with the written guide |
Read the recording transcript and commands
This is the retained English narration, in the original chapter order. Commands keep the recorded version and prerequisites; setup additions in the lesson are labelled separately.
00:00 — The same model can make different review queues
The fraud model emits a probability. Your team must choose when to flag a transaction for review. A lower threshold can flag more transactions, while precision and recall describe different mistakes. In this lesson we evaluate a real held-out synthetic sample from our trained forest. We will capture one evaluation block with a session, then compare two ordinary function calls with a decorator.
00:27 — Decide what one Run should mean
Choose the unit you want to review. A session suits a block with several operations and observations. A decorator suits repeated function calls, where each call becomes a Run. In both cases explicitly select parameters, inputs and metrics. Tracking should preserve your function return and exception behavior. It does not infer every meaningful variable in your program or decide which metric the business should optimize.
00:56 — Inspect the held-out sample and metric calculation
The preparation step has trained the same example in this fresh copy and retained its baseline Run. Now we derive the exact held-out rows using the notebook split and saved model. Evaluation dot CSV contains labels and the actual predicted probabilities. Our small evaluator counts true positives, false positives and false negatives, then computes precision and recall. Unlike a hard-coded score, these values depend on the selected sample and threshold. The sample remains synthetic and is not a production benchmark.
python lab.py prepare-holdout
python lab.py source evaluate
01:57 — Record a block with named evidence
Read the session code first. The input call fingerprints the actual evaluation file. The parameter call records threshold zero point five, and the loop records the observed metric values. Run the code, then read the returned result and retained Run identity. The distinction matters: the return is still an ordinary Python value used by your program; the record is the evidence you chose to retain for later review.
python lab.py source session_example
python thresholds.py session
02:51 — Why report precision and recall together?
Precision asks how many flagged rows were actually labelled fraudulent. Recall asks how many labelled fraudulent rows were found. The flagged count describes review volume in this particular sample. Changing a threshold can trade one concern against another, so a single best score is not enough to define a review policy. These metrics are observed values over selected data, and the data identity is part of their interpretation.
03:22 — Compare two calls; keep their normal results
The decorator selects threshold from the function arguments. Each call logs the same input file and its observed metrics. We call thresholds zero point three five and zero point six five, and verify that tracking leaves their ordinary return values unchanged. Before reading the output, predict which call should flag at least as many rows. Compare the recorded precision and recall rather than assuming that either threshold is automatically preferable.
python lab.py source tracked_evaluate
python thresholds.py calls
python lab.py history
04:24 — Variant: failed capture preserves the exception
A useful SDK integration must survive failures without changing program behavior. This evaluation deliberately logs a partial observation, then raises a synthetic ValueError. The caller catches that same exception outside the decorated function. We check the retained failed Run and verify that its exception message is absent from the record. Failure status and explicitly logged observations can be useful evidence; a complete traceback is not silently made part of the experiment. This is also why successful function returns alone are an incomplete qualification of a tracking integration.
python lab.py source failed_review
python thresholds.py failure
python lab.py failure-check
05:20 — Your turn: an in-memory array replaces the file
Pause and explain what changes if the program logs only an input name for an in-memory array, instead of identifying evaluation dot CSV. Does that input name fingerprint its contents? Would an unchanged name establish that two evaluations used the same values? Then decide what evidence you would record for a review that must compare actual input content.
05:46 — Capture the evidence your decision needs
An input name alone does not establish an in-memory content identity. A selected local file can be fingerprinted, as in this lesson; other unknowns must stay visible. Choose a session for a block or a decorator for repeated calls, and keep the same input definition when comparing their outcomes. The SDK makes these observations reviewable without selecting your threshold policy. Next we use these same evaluation ideas in a real live kernel.
06:18 — Continue with the written guide
The current written guide remains authoritative. Continue there for the detailed steps. Thank you for following Apizr Academy.
1. Prepare the recorded holdout
Start with a fresh fraud lab from the recorded materials. As in the video, train cohort A in the selected scientific environment before deriving the held-out sample:
cp data/transactions-a.csv data/transactions.csv
apizr experiment run fraud_detection.ipynb --output model=artifacts/model.joblib --format json > run-a.json
python lab.py prepare-holdout
python lab.py source evaluate
evaluation.csv contains held-out labels and actual predicted probabilities. thresholds.py counts true positives, false positives and false negatives to calculate precision, recall and flagged volume. These scores depend on the sample and threshold; they are not assigned constants.
2. Record a block, then two ordinary calls
python lab.py source session_example
python thresholds.py session
python lab.py source tracked_evaluate
python thresholds.py calls
python lab.py history
The session explicitly fingerprints evaluation.csv, records threshold 0.5 and logs the observed metrics. The decorator selects each call's threshold and preserves its ordinary return. The recording then calls 0.35 and 0.65.
| Recorded threshold | Precision | Recall | Flagged rows |
|---|---|---|---|
| 0.5, session | 0.4857 | 0.3542 | 35 |
| 0.35, decorated call | 0.55 | 0.6875 | 60 |
| 0.65, decorated call | 0.5714 | 0.25 | 21 |
These are observations of the recorded synthetic sample. Review the full Run JSON for parameter origins rather than relying only on the teaching projection. Neither threshold is automatically the right business policy.
3. Check the failed capture
python lab.py source failed_review
python thresholds.py failure
python lab.py failure-check
The decorated function logs a partial observation and raises ValueError. The caller receives the same exception. The retained capture is failed and excludes the synthetic exception message. Tracking does not turn failure into success or silently retain a complete traceback.
Practice
Use the video's exercise: replace the selected file with an in-memory array and log only an input name. Does the name identify the array's contents? What would you explicitly retain for a meaningful comparison?
Check your reasoning
A name alone does not fingerprint in-memory values. The recorded lesson selects a real local file and its content identity. Keep the input definition consistent across calls, choose relevant parameters and metrics explicitly, and preserve the function's normal return and exception behavior.