Pipelines and Reproducible Experiments

Goal

By the end of this lesson, you can build a preprocessing-and-model pipeline, explain how it protects the training boundary during validation, and record enough experiment settings for another person to repeat the comparison.

An ML experiment is more than a model object

Suppose someone tells you:

“I used logistic regression and got 84%.”

You still do not know enough to repeat or judge the experiment.

You need to know things such as:

• which dataset revision and features were used;
• how train and test data were split;
• which random seed controlled the split;
• whether features were standardized;
• where the scaler was fitted;
• the model hyperparameters;
• the evaluation metric.

If these steps live in scattered notebook cells, it is easy to run them in a different order the next time.

A pipeline is an ordered fitting procedure

Think of a pipeline as an assembly line.

For a scaled logistic-regression model, the intended order is:

raw features -> fit/transform scaler -> fit classifier

When the pipeline is fitted on training data, each learned step receives only that training data.

During prediction, the scaler reuses its learned training statistics and the classifier reuses its learned parameters. Neither should refit itself on the test example.

This makes the data boundary part of the code structure rather than a rule you must remember manually in every notebook cell.

Why pipelines matter during cross-validation

Without a pipeline, a common mistake is:

1. fit a scaler on all rows;
2. transform all rows;
3. run cross-validation on the transformed data.

The validation folds influenced the scaler before cross-validation even began.

With a pipeline, each fold does this instead:

1. take that fold's training portion;
2. fit the scaler on that portion;
3. transform the training portion;
4. fit the classifier;
5. transform and score the held-out fold using the training-fitted scaler.

That keeps learned preprocessing inside the correct boundary.

Run the same experiment twice

The Lab builds make_pipeline(StandardScaler(), LogisticRegression(...)) and uses a fixed split.

1. Click Run and record the held-out result: test accuracy: 0.8.
2. Click Run again without changing anything.
3. Confirm that you get 0.8 again. The split seed (random_state=42) and data seed are fixed, so the experiment repeats exactly.
4. Now change one model setting on purpose. Find LogisticRegression(max_iter=500, random_state=42) and add a stronger regularization setting: LogisticRegression(max_iter=500, random_state=42, C=0.05). (Smaller C pushes the model toward smaller weights; you will study this idea later.)
5. Before running, write the change down in your own notes: “changed C from the default 1.0 to 0.05.”
6. Click Run. The result becomes test accuracy: 0.833 on the same split and metric.
7. Now look at experiment record:. It does not mention C. If you saved only this record, nobody could reproduce the 0.833 result. Add "C": 0.05, to the experiment_record dictionary and run once more.
8. Press Reset afterward.

Loading lab…

Reproducibility makes a comparison auditable. It does not make a poor methodology correct.

What a minimal experiment record should contain

For this level, a useful record includes at least:

Item	Example
Data	source/revision and feature definitions
Split	strategy, proportions, seed or grouping rule
Preprocessing	transformations and where they are fitted
Model	class and important hyperparameters
Metric	exact held-out or validation measure
Result	score plus important failure details

Later projects add package versions, hardware, model checkpoints, and other reproducibility metadata.

A common misconception

“If a pipeline runs reproducibly, the experiment is trustworthy.”

A pipeline cannot detect that leaky_answer literally contains the target or that the dataset was collected after the event you claim to predict.

A reproducible broken experiment is still broken. You need both repeatable procedure and sound methodology.

Quick Check

1. What problem does a pipeline help prevent?

Preprocessing being fit outside the intended training boundaryA model having more parameters than training examplesA future production distribution differing from the evaluation data

2. What should stay fixed for a fair model comparison?

Only the random seed while allowing the split and preprocessing to changeThe model class while allowing a different evaluation setThe data split and evaluation procedure

3. Does reproducibility guarantee correctness?

Yes, if the same code and random seed reproduce the same metricsNo; a repeatable procedure can still use a leaky feature or an invalid splitYes, if the pipeline logs every package version

0 of 3 questions answered.

Key Takeaways

• An ML result includes data, splitting, preprocessing, model settings, and evaluation choices.
• Pipelines keep learned preprocessing and model fitting in a fixed order.
• Cross-validation should refit the whole pipeline inside each training fold.
• Reproducibility makes evidence auditable but does not guarantee sound methodology.

Next Lesson

Next, you will use the entire Level 1 workflow in a model review: allowed features, fair split, baseline, pipeline, multiple metrics, and failure analysis.

References

• scikit-learn, Pipeline.
• scikit-learn, Common pitfalls and recommended practices.