Week 7 · revision
| When | Where | |
|---|---|---|
| Final exam | Tuesday 20 October, 09:00–11:00 | MSc Teaching Space |
| Lab submission | Wednesday 21 October, 12:00 | Brightspace and email |
| Project submission | Thursday 22 October, 12:00 | Brightspace and email |
| Project interviews | Friday 23 October, your group’s slot | MSc Teaching Space |
eoin+advanced-ai-labs@eoin.aiAdvanced AI labs: Firstname Lastname, StudentIDLabs_Firstname_Lastname_StudentID.zip
Labs_Aoife_Murphy_24123456.zipMac-Giolla7_2_Backpropagation.ipynb, not Copy of 7_2_Backpropagation.ipynbcheck_labs.py from the landing page’s Labs sectioneoin+advanced-ai-project@eoin.ai, copying every member of the groupAdvanced AI project: Team Name| Section | Questions | Time guide | Marks |
|---|---|---|---|
| Multiple choice | 30, four options each | 50 minutes | 30 |
| Written | 3, in three parts each | 60 minutes | 30 |
Two hours in total, so about ten minutes are left for reading and checking.
Why does averaging the predictions of several separately trained models usually give lower error than a typical one of them?
A. The combined model has more capacity, so it fits the training data more closely.
B. The models’ errors partly differ, so some of them cancel in the average.
C. Each model corrects mistakes that the model trained before it had made.
D. The label noise each model had fitted is removed from the average.
With momentum, what is each gradient-descent step computed from?
A. The current gradient, scaled by a learning rate that grows over time.
B. The current gradient, divided by its recent typical magnitude.
C. The current gradient, corrected by the curvature of the loss there.
D. A weighted average of the current gradient and earlier ones.
For one example with true label \(y \in \{0, 1\}\) and predicted probability \(\lambda = \Pr(y = 1 \mid x)\), binary cross-entropy is
\[ \ell = -y \ln \lambda - (1 - y) \ln(1 - \lambda). \]
Two models each get an example with true label 1 wrong: model A gives \(\lambda = 0.4\) and model B gives \(\lambda = 0.01\). Use \(\ln 0.4 \approx -0.92\), \(\ln 0.6 \approx -0.51\), \(\ln 0.01 \approx -4.61\) and \(\ln 0.99 \approx -0.01\).
Which statement correctly compares their losses on this example?
A. B’s loss is about 5 times A’s.
B. B’s loss is about 1.65 times A’s.
C. The two losses are about equal.
D. A’s loss is about 50 times B’s.
A residual block computes \(h_k = h_{k-1} + f_k[h_{k-1}]\); a plain block computes \(h_k = f_k[h_{k-1}]\). Backpropagation multiplies the gradient by each block’s Jacobian \(\partial h_k / \partial h_{k-1}\) in turn.
Early in training the branches \(f_k\) have last layers initialised close to zero, so each branch’s own Jacobian \(J_k\) is close to zero.
Compared with the gradient at the output, what is the gradient with respect to \(h_1\) like in a deep residual stack, and in a plain stack built from the same branches?
A. Residual: far smaller. Plain: far smaller.
B. Residual: roughly unchanged. Plain: far smaller.
C. Residual: roughly unchanged. Plain: roughly unchanged.
D. Residual: far larger. Plain: roughly unchanged.
Given: at initialisation, the variance of a ReLU layer’s pre-activations is the previous layer’s variance multiplied by a gain that depends on the scale of that layer’s weights. Gradients are scaled the same way on the backward pass. The variances of uncorrelated quantities add. A residual block computes \(h_k = h_{k-1} + f_k[h_{k-1}]\).
(a) [2 marks] Explain why a deep network is far more sensitive than a shallow one to the scale of its initial weights.
(b) [3 marks] A 30-layer plain ReLU network is initialised with weights that are too small. Describe what you would expect to see in the activations of its last layers, in the gradients reaching its first layers, and in the training loss over the first few epochs.
(c) [5 marks] A colleague chooses the weight scale so that each layer’s gain is exactly one, then wraps every layer in a residual connection, keeping that scale. They claim the signal is still stable through depth “because each branch preserves variance”.
Assume a block’s input and its branch output are uncorrelated. Evaluate the claim (3 marks), then propose one change that restores stability and explain why it works (2 marks).