Two models each get an example with true label 1 wrong: model A gives \(\lambda = 0.4\) and model B gives \(\lambda = 0.01\). Use \(\ln 0.4 \approx -0.92\), \(\ln 0.6 \approx -0.51\), \(\ln 0.01 \approx -4.61\) and \(\ln 0.99 \approx -0.01\).
Which statement correctly compares their losses on this example?
A. B’s loss is about 5 times A’s.
B. B’s loss is about 1.65 times A’s.
C. The two losses are about equal.
D. A’s loss is about 50 times B’s.
Sample: a given formula, answer
The label is 1, so the second term vanishes and \(\ell = -\ln \lambda\)
A: 4.61 against 0.92, a ratio of about 5
B: assumes the loss grows in proportion to the miss
the logarithm makes a confident miss cost far more than its distance suggests
C: reads the loss as the error rate, where both models are simply wrong
D: keeps the wrong term, \(-\ln(1 - \lambda)\)
Sample: a short case
A residual block computes \(h_k = h_{k-1} + f_k[h_{k-1}]\); a plain block computes \(h_k = f_k[h_{k-1}]\). Backpropagation multiplies the gradient by each block’s Jacobian \(\partial h_k / \partial h_{k-1}\) in turn.
Early in training the branches \(f_k\) have last layers initialised close to zero, so each branch’s own Jacobian \(J_k\) is close to zero.
Compared with the gradient at the output, what is the gradient with respect to \(h_1\) like in a deep residual stack, and in a plain stack built from the same branches?
A. Residual: far smaller. Plain: far smaller.
B. Residual: roughly unchanged. Plain: far smaller.
C. Residual: roughly unchanged. Plain: roughly unchanged.
D. Residual: far larger. Plain: roughly unchanged.
Sample: a short case, answer
A residual block’s Jacobian is \(I + J_k\), which is close to \(I\) here
A plain block’s Jacobian is \(J_k\), which is close to zero
B: a product of near-identities stays near the identity; a product of near-zeros vanishes
A: leaves the skip out of the backward pass
C: treats the gradient as independent of the architecture
D: reads the chain rule as adding, when it multiplies
Written questions
Structure
Each question opens with a Given: the formulas and facts you’ll need
Then three parts, climbing
(a), 2 marks: explain something
(b), 3 marks: apply it to a case
(c), 5 marks: evaluate a claim or justify a choice
The parts are independent: a slip in (a) can’t cost you (b)
No part needs a calculation
About twenty minutes each, and a few sentences per part is enough
How they are marked
One mark for one claim, stated plainly
the question tells you how many marks, so it tells you how many claims
Prose is fine: you never need to type an equation to earn a mark
State a claim and then contradict it in the same part: no mark for that claim
Asked for one change and you give three: the first is the one marked
Nothing is required that the question didn’t ask for
Sample written question
Given: at initialisation, the variance of a ReLU layer’s pre-activations is the previous layer’s variance multiplied by a gain that depends on the scale of that layer’s weights. Gradients are scaled the same way on the backward pass. The variances of uncorrelated quantities add. A residual block computes \(h_k = h_{k-1} + f_k[h_{k-1}]\).
(a) [2 marks] Explain why a deep network is far more sensitive than a shallow one to the scale of its initial weights.
(b) [3 marks] A 30-layer plain ReLU network is initialised with weights that are too small. Describe what you would expect to see in the activations of its last layers, in the gradients reaching its first layers, and in the training loss over the first few epochs.
(c) [5 marks] A colleague chooses the weight scale so that each layer’s gain is exactly one, then wraps every layer in a residual connection, keeping that scale. They claim the signal is still stable through depth “because each branch preserves variance”.
Assume a block’s input and its branch output are uncorrelated. Evaluate the claim (3 marks), then propose one change that restores stability and explain why it works (2 marks).
Model answer: (a) and (b)
(a)
each layer multiplies the variance by its gain
the gains compound through depth, so a gain slightly off one is harmless over a few layers and extreme over many
(b)
the last layers’ activations shrink towards zero, and look alike for every input
the gradients reaching the first layers are vanishingly small
the training loss stays near its starting value and barely falls
Five marks, five claims, each a single sentence
Model answer: (c)
Each branch does preserve variance, but a block outputs its input plus its branch
The two variances add, so every block roughly doubles the variance
That growth compounds through depth, so the signal is not stable and the claim is wrong
One change: scale each block’s output down after the addition
It works because halving the doubled variance returns it to the input’s level
The marks for (c)
1: a block’s output is the sum of its input and its branch
1: the variances add, so the output has more variance than the input
1: the growth compounds through depth, so the claim is wrong
1: one change that restores stability
rescaling after the addition, normalising within each block, and starting each branch near zero all earn it
1: an explanation that matches the change you chose
“The claim is wrong” with no reason earns nothing
“Lower the learning rate” earns nothing: the problem exists before any step is taken
Writing an answer
Lead with the claim, then give the reason
One claim per sentence makes your marks easy to find
Use the vocabulary of the Given; you don’t need to restate it
If a part asks you to evaluate, say whether the claim holds and why
Move on at twenty minutes: the first marks of the next question are cheaper than the last mark of this one