Neural Network Debugging Workshop

Goal

By the end of this lesson, you can diagnose neural-network failures by moving through observable boundaries—data, shapes, forward values, loss, gradients, updates, and curves—and propose one evidence-driven repair experiment.

Debugging is easier when you inspect boundaries in order

A neural network is a numerical pipeline.

When the final result is wrong, jumping directly to “change the optimizer” or “add another layer” skips many places where the failure may actually begin.

Use an evidence ladder:

1. Data and targets — Are the examples and labels what you think they are?
2. Shapes — Do dimensions represent batch, features, and neurons correctly?
3. Forward pass — Are weighted sums and activations finite and sensible?
4. Loss — Is the objective finite and using the intended representation?
5. Gradients — Do values exist, have plausible signs, and avoid NaN/Inf?
6. Updates — Do parameters actually move by sensible amounts?
7. Curves — Does learning improve, stall, diverge, or overfit?

This order moves from basic correctness toward higher-level modeling choices.

Follow the evidence, not a favorite fix

Consider three examples.

Case A: shape mismatch before the first layer

Do not tune the learning rate. The computation cannot even represent the intended operation yet. Name the dimensions and fix the data/layer contract.

Case B: finite loss and nonzero gradients, but parameters never change

The backward path exists. Inspect the optimizer/update boundary: Are parameters registered? Is step() called? Are updates overwritten or immediately restored?

Case C: training loss falls while validation loss rises

The forward and optimization paths are at least functioning. Now investigate generalization: capacity, data coverage, regularization, leakage, and split quality.

The same symptom word “model failed” leads to very different next checks.

Diagnose the four cases in the Lab

The Lab contains four named evidence records: shape, dead_relu, no_update, and overfit.

1. Click Run and read all four name -> diagnosis lines.
2. Focus on the no_update case. It starts with gradients_nonzero: True and parameters_changed: False, so the diagnosis should be inspect_optimizer_update.
3. In cases["no_update"], change only "parameters_changed": False to "parameters_changed": True.
4. Before running, reason through the evidence ladder: shapes are valid, loss is finite, gradients exist, parameters now move, and validation is not rising. Predict that the diagnosis should move past the optimizer boundary.
5. Click Run. Confirm no_update now reports continue_training_or_collect_more_evidence while the other three cases keep their original diagnoses.
6. Restore "parameters_changed": False.

Loading lab…

After this guided pass, choose a different case and change one evidence flag. Predict the first boundary that should become decisive before running again.

The diagnostic function is a practice aid. In a real system, your evidence—not the label returned by a helper—is what justifies the next action.

Keep a short experiment log

For every debugging attempt, record:

Field	Example
Symptom	validation loss rises after epoch 8
Evidence	train loss continues falling
Hypothesis	model capacity is fitting training noise
One change	reduce hidden width
Fixed	data split, seed, optimizer, metric
Result	validation curve improves/worsens/unchanged
Interpretation	hypothesis supported or not supported

Revert failed hypotheses instead of stacking unexplained changes on top of one another.

A common mistake: applying several repairs at once

Suppose you change learning rate, batch size, initialization, and architecture together and the model improves.

You have an improved run but weak understanding. You do not know which change mattered or whether one change merely compensated for another mistake.

Controlled debugging produces slower-looking individual experiments but faster understanding.

The Level 2 mental model

At this point, you should be able to explain a neural network without treating it as a black box:

inputs -> weighted sums -> activations -> later layers -> prediction -> loss -> local derivatives -> gradients -> optimizer updates -> new parameters

Training curves then show what repeated updates do over time.

If you can trace that chain and identify where evidence first goes wrong, you have the main first-principles skill this level is designed to teach.

Quick Check

1. Gradients exist but parameters do not move. What boundary should you inspect next?

The forward activations again, even though gradients are already presentOptimizer/update pathThe loss definition, because any nonzero gradient proves the optimizer ran

2. Why change one thing at a time?

Because validation evidence becomes invalid after two changesSo each result can be traced to one specific changeBecause one model parameter may legally change per training step

3. What makes a failure easier to debug?

Changing several settings until one run succeedsLooking at the final metric so intermediate behavior cannot distract youReproducible evidence and a logged hypothesis

0 of 3 questions answered.

Key Takeaways

• Debug neural networks by observable boundaries from data to evaluation.
• Use the earliest wrong evidence to choose the next check.
• Keep symptom, evidence, hypothesis, controlled change, and result together.
• Reproducible failures are easier to fix than unexplained intermittent ones.
• Level 2 mastery means you can both build a small network and explain why it succeeds or fails.

Next Lesson

You are ready for the Level 2 project, Neural Network From Scratch. After that, Level 3 moves from tiny first-principles networks to learned representations across images, sequences, and pretrained models.

References

• PyTorch, Autograd.
• PyTorch, torch.optim.
• PyTorch, Reproducibility.