Autocoder

Deep Learning Interview Questions and Answers

Last updated 2026-11-19

Deep learning interviews test whether you understand what happens inside a network: how gradients flow, why training fails and how layer shapes and parameter counts work out. Many questions are small calculations, so they reward practice with pen and paper.

Each answer below gives the idea, the formula and a worked example. For the classical algorithms that sit under these ideas read machine learning interview questions, and for convolution-specific questions see computer vision interview questions.

Typical roundsTheory and calculations, PyTorch coding, project deep dive
Core skillsBackpropagation, CNNs, RNNs, transformers, optimisation
Questions here14 worked questions

Calculations that come up repeatedly

Debugging a network that will not train

Overfit one small batch first: if the loss cannot reach near zero, there is a bug in the model, the loss or the labels. Then check the learning rate, input scaling, the loss function and the output activation pairing. Only then tune regularisation. Say this order aloud in an interview, because it shows you debug systematically.

Deep Learning Interview Questions and Answers: questions with worked answers

  1. 1

    Do backpropagation by hand for one linear neuron: x = 2, w = 3, b = 1, target y = 1, loss = (z - y)^2.

    easy

    Forward: z = w x + b = 3 x 2 + 1 = 7, loss = (7 - 1)^2 = 36. Backward uses the chain rule: dL/dz = 2 (z - y) = 12, dz/dw = x = 2 and dz/db = 1. So dL/dw = 24 and dL/db = 12. With learning rate 0.01 the update gives w = 3 - 0.24 = 2.76 and b = 1 - 0.12 = 0.88. Backpropagation is just this chain rule applied layer by layer, reusing the stored forward values.

  2. 2

    Compute softmax and cross-entropy for logits [2, 1, 0] with the true class at index 0.

    easy

    Exponentials are e^2 = 7.389, e^1 = 2.718, e^0 = 1, sum 11.107. Probabilities are 0.665, 0.245, 0.090. The cross-entropy loss is -ln(0.665) = 0.408. The gradient of the loss with respect to the logits is simply probability minus one-hot: [-0.335, 0.245, 0.090]. In code, subtract the maximum logit before exponentiating to avoid overflow, or use a fused loss function.

    python
    import torch, torch.nn.functional as F
    logits = torch.tensor([[2.0, 1.0, 0.0]])
    print(F.softmax(logits, dim=1))                 # 0.665, 0.245, 0.090
    print(F.cross_entropy(logits, torch.tensor([0])))   # 0.408
  3. 3

    Why do gradients vanish, and what fixes it?

    medium

    The sigmoid derivative is at most 0.25, so backpropagating through 10 sigmoid layers multiplies gradients by at most 0.25^10, about 9.5 x 10^-7, and early layers barely learn. Fixes: ReLU-type activations whose derivative is 1 for positive inputs, careful initialisation (He for ReLU, Xavier for tanh), batch or layer normalisation, residual connections that give gradients a direct path, and gated units such as LSTM in recurrent networks. Exploding gradients are handled with gradient clipping.

  4. 4

    Compare sigmoid, tanh, ReLU and GELU.

    easy

    Sigmoid maps to (0, 1) and saturates, so it is used for binary outputs and gates, not hidden layers. Tanh maps to (-1, 1) and is zero-centred but still saturates. ReLU is max(0, x): cheap, no saturation for positive values, but neurons with permanently negative input stop learning (dying ReLU), which Leaky ReLU softens. GELU is a smooth ReLU-like function used in transformers. Softmax is the output activation for multi-class problems.

  5. 5

    How do you calculate the output size and parameter count of a convolution layer? Input 32 x 32 x 3, 16 filters of 5 x 5, stride 1, no padding.

    medium

    Output size = (W - K + 2P) / S + 1 = (32 - 5 + 0) / 1 + 1 = 28, so the output is 28 x 28 x 16. Parameters = (5 x 5 x 3 + 1) x 16 = 76 x 16 = 1,216, where the 1 is the bias per filter. The count does not depend on the image size, which is why convolutions are efficient compared with a dense layer: connecting 32 x 32 x 3 = 3,072 inputs to 28 x 28 x 16 = 12,544 outputs densely would need over 38 million weights.

  6. 6

    Count the parameters of a 784-128-10 fully connected network.

    easy

    The first layer has 784 x 128 weights plus 128 biases = 100,352 + 128 = 100,480. The second has 128 x 10 + 10 = 1,290. The total is 101,770 parameters. In 32-bit floats this is about 407 KB for the weights. Adam stores two extra values per parameter, so optimiser state triples the memory for trainable weights.

  7. 7

    Write Adam's update and compute the first step for gradient 2 with learning rate 0.001.

    medium

    m = b1 m + (1 - b1) g and v = b2 v + (1 - b2) g^2, with b1 = 0.9, b2 = 0.999. At step 1, m = 0.2 and v = 0.004. Bias correction divides by (1 - b^t): m_hat = 0.2 / 0.1 = 2 and v_hat = 0.004 / 0.001 = 4. The update is lr x m_hat / (sqrt(v_hat) + eps) = 0.001 x 2 / 2 = 0.001. So the first step has size about the learning rate regardless of the gradient scale. Adam adapts the step per parameter; SGD with momentum often generalises slightly better when tuned.

  8. 8

    Explain batch normalisation with a numeric example and its train versus eval behaviour.

    medium

    For a batch, compute the mean and variance, normalise x_hat = (x - mean) / sqrt(var + eps), then scale and shift with learnable gamma and beta. For the batch [1, 2, 3, 4], the mean is 2.5 and the population variance is 1.25, so the standard deviation is 1.118 and the value 1 becomes -1.342. In training it uses batch statistics and updates running averages; in eval it uses those running averages. Forgetting model.eval() is a common bug. Layer norm normalises across features per sample, so it does not depend on batch size and is used in transformers.

  9. 9

    How does dropout work, and why is the output scaled?

    easy

    During training each activation is set to zero with probability p, which prevents units co-adapting. With inverted dropout the surviving activations are divided by (1 - p), so for p = 0.5 they are doubled, which keeps the expected activation the same and means nothing needs to change at inference. At inference dropout is turned off. Dropout is less common after batch norm and in convolutional layers, where data augmentation and weight decay are usually used instead.

  10. 10

    How do He and Xavier initialisation differ?

    medium

    Both choose the weight variance to keep activation variance stable across layers. Xavier (Glorot) uses variance 2 / (fan_in + fan_out) and suits tanh or sigmoid. He uses variance 2 / fan_in and suits ReLU, which zeroes about half of the activations and so needs twice the variance. Initialising all weights to zero makes every neuron in a layer compute the same thing, so symmetry is never broken.

  11. 11

    Write a standard PyTorch training loop.

    medium

    The loop zeroes gradients, runs the forward pass, computes the loss, backpropagates and steps the optimiser. Switch between train and eval mode, and disable gradients during validation. The most common bug is forgetting optimizer.zero_grad(), which accumulates gradients across batches.

    python
    model.train()
    for xb, yb in train_loader:
        xb, yb = xb.to(device), yb.to(device)
        optimizer.zero_grad()
        loss = criterion(model(xb), yb)
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        optimizer.step()
    
    model.eval()
    with torch.no_grad():
        val_loss = sum(criterion(model(x.to(device)), y.to(device)).item()
                       for x, y in val_loader) / len(val_loader)
  12. 12

    How does an LSTM avoid the vanishing gradient problem?

    hard

    An LSTM keeps a cell state updated additively: c_t = f_t * c_(t-1) + i_t * g_t, where f is the forget gate, i the input gate and g the candidate. Because the cell state carries information forward by elementwise multiplication with the forget gate rather than through repeated matrix multiplications and squashing, gradients can flow across many steps when the forget gate is near 1. The output gate decides how much of the cell state is exposed as the hidden state. GRUs merge gates to save parameters.

  13. 13

    Explain scaled dot-product attention and why we divide by the square root of d_k.

    hard

    Attention(Q, K, V) = softmax(Q K^T / sqrt(d_k)) V. Each query is compared with every key and the resulting weights mix the values. If query and key components have zero mean and unit variance, their dot product has variance d_k, so for d_k = 64 the standard deviation is 8. Large scores push softmax into regions with tiny gradients, so dividing by sqrt(d_k) restores unit variance. Cost is O(n^2 d) in sequence length n, which is why long contexts are expensive.

  14. 14

    Your validation loss is rising while training loss falls. What do you do?

    easy

    That is overfitting. Add weight decay, dropout or data augmentation, reduce model size, use early stopping on validation loss, or get more data. For pretrained models, freeze more layers or lower the learning rate. Verify the validation set is representative and free of leakage. Plot both curves per epoch and stop at the minimum validation loss.

Frequently asked questions

Do I need to know both PyTorch and TensorFlow?

Know one well. PyTorch is more common in research-style roles, and the concepts transfer easily between frameworks.

How deep should my transformer knowledge be?

Be able to write the attention formula, explain multi-head attention, positional encoding and the encoder versus decoder difference. See the NLP and LLM pages for the language side.

Are deep learning questions asked for non-research roles?

For data science and ML engineer roles, yes, at a conceptual level: backpropagation, regularisation, CNN basics and transfer learning.

What is the best way to practise?

Train a small model on a public dataset end to end, then answer the questions above without looking. Compute parameter counts and shapes by hand.

Related guides

More in Role-wise Interview Questions by Job and Tech Stack

See all in this section