Università degli Studi di Pavia

Facoltà di Ingegneria

Deep Learning

A.A. 2021-2022

Second Semester

Fri: 11:00 a.m. - 1:00 p.m., Aula E2

Fri: 2:00 p.m. - 4:00 p.m., Aula B2

Lectures & Suggested Readings:

  • Reports of errors in the resources below are always welcome
    1. 2022.03.11 (theory)

      Introduction [pdf]
      AI spring? Artificial Intelligence, Machine Learning, Deep Learning: facts, myths and a few reflections.

    2. 2022.03.11 (theory)

      Fundamentals: Artificial Neural Networks [pdf]
      Foundations of machine learning: dataset, representation, evaluation, optimization. Feed-forward neural networks as universal approximators.

    3. 2022.03.18 (theory)

      Flow Graphs and Automatic Differentiation [pdf]
      Tensorial representation, flow graphs. Automatic differentiation: graphs from graphs.

      Aside 1: Tensor Broadcasting [pdf]
      From theoretical tensor algebra to actual computation: operators and automatic broadcasting.

    4. 2022.03.25 (theory)

      Deep Networks [pdf]
      Deeper networks: potential advantages and new challenges. Tensorial layerwise representation. Softmax and cross-entropy.

      Shannon Entropy (Wikipedia)

      Cross Entropy (Wikipedia)

    5. 2022.04.01 (theory)

      Learning as Optimization [pdf]
      Vanishing and exploding gradients. First and second order optimization, approximations, optimizers. Further tricks.

      Aside 2: Exponential Moving Average [pdf]

      Aside 3: Predictors [pdf]
      From in-sample optimization to out-of-sample generalization.

    6. 2022.04.22 (theory)

      Convolutional Networks [pdf]
      Convolutional filter, filter banks, feature maps, pooling, layerwise gradients.

      Aside 4: Hardware for Deep Learning [pdf]
      Main differences bewtween CPUs and GPUs, SIMT parallelism, bus-oriented communication, a few caveats.

    7. 2022.04.29 (theory)

      Deep Convolutional Neural Networks and Beyond [pdf]
      Some insight into what happens in convolution layers. DCNN architectures. Transfer learning. Working in reverse: image generation. Generative adversarial networks. Autoencoders and segmentation. Object detection.

    8. 2022.05.06 (theory)

      Deep Learning and Time Series [pdf]
      Recurrent Neural Networks (RNN), temporal unfolding, LSTM Cells, GRU cells, encoder / decoder, convolution, time series analysis-

    9. 2022.05.13 (theory)

      Reinforcement Learning [pdf]
      A short recap about RL foundations, Markov decision process, state value function, policy, optimality, action value function, Q-learning.

    10. 2022.05.20 (theory)

      Deep Reinforcement Learning [pdf]
      Integrating DNNs into the RL paradigm, DQN algorithm, policy gradient, actor-critic method, NAF algorithm-

    11. 2022.05.27 (theory)

      Aside 5: Word Embedding [pdf]
      Skip-grams, probability distribution of context and center words, training and results, continuous bag of words (CBOW) model.

    12. 2022.06.10 (theory)

      Monte Carlo Tree Search [pdf]
      Game trees, Monte Carlo strategy, Monte Carlo Tree Search (MCTS), Upper Confidence Bounds applied to Trees (UCT).

      Alpha Zero [pdf]
      MCTS + DNN, network architecture, replacing MCTS rollout with estimation, network training, AlphaZero in continuous spaces (hints).

    13. 2022.06.13 (theory)

      Attention and Transformers [pdf]
      Attention as a kernel, attention maps, queries, key and values, attention-based encoder and decoder, transformer architecture, translator.


    1. Marco Piastra

    2. Contact: marco.piastra@unipv.it


    1. Course info


    1. See Faculty website

    Further resources:

    All Colab notebooks used in the course are available on Kiro


    1. Artificial Intelligence Reading Group

    1. Deep Learning, A.A. 2020-2021