# Online Course: Machine Learning for Physicists 2021 :::info **Lecture Series by Florian Marquardt** This is the website for the online lecture series on machine learning (April 12-July 12, 2021). Here we will collect the course overview and links to the forum, code, etc. This online lecture series takes place every Monday 6pm CET (Western European summer time), until July 12. ::: :::success **Repeat Exam** The repeat exam has now officially been scheduled for **Tuesday, October 12, lecture hall E, 15:00-17:00**. Note that this has shifted by one day from an earlier announcement, due to room availability. You should be able to register in meinCampus now. ::: :::success **Exam July 28** For the FAU students registered for the exam: The exam will take place July 28 (Wednesday), 10:00-12:00. It will take place inside lecture halls D and F in the physics/biology lecture hall building at Staudtstrasse **(update: we might also be using halls H and E if needed)**. Please arrive 15 minutes before and wait outside the lecture hall building. Follow the university's official COVID rules, and check in with your cell phone on your marked seat in the lecture hall. You are allowed to bring a handwritten cheat sheet (1 sheet, written by yourself, front and back page). We will supply paper. **The exam has now been graded and the grades should be visible in meinCampus** You can come to my office at the Max Planck Institute for the Science of Light (Staudtstr. 2) on **Thursday, August 5, from 14:00 to 17:00,** to have a look at the exam. When you enter the building, please register at the front office and wear a mask. It would be good to send an email beforehand to me, so I know to expect you. Here is the solution: [Machine Learning Exam 2021 Solution](https://pad.gwdg.de/c9_oYrlKQYyhjk7dFbybzw?view) ::: :::success Please **register** for the zoom online meetings at https://fau.zoom.us/meeting/register/u5YvdumpqD8rHNI5-UfwGcmIslkaHVEvD-Jq (free of charge; you only need to register once for the whole series; I like to keep track of the numbers of expected participants and collect a bit of statistics like where people are from and their field of study) ::: ## Some of the questions we will address in this lecture series - How do artificial neural networks look like? - How do you train them efficiently? - How do you recognize images? - How do you learn a compact representation of data, without providing any extra information? - How do you analyze time series and sentences? - How do you discover strategies from scratch without a teacher? - What are some modern applications in science/physics? ## Most important info The course is **inverted-classroom style**. This means you watch one of the pre-recorded video lectures (about 90min) on your own, and then we use a live zoom meeting to: discuss the lecture, do live tutorials, and discuss homework problems! The live meetings take place every week on Monday, at 6pm German time (CEST = Central European Summer Time). ## Certificates For the students at FAU Erlangen-Nürnberg, please register in meinCampus, and there will be a written exam in the end. For all other students from anywhere in the world: Just like last year, I will offer you the possibility of doing a 'mini-project', i.e. work on a small project of your own choice. In the end, you will hand in a short write-up plus the code, and you will deliver a short online presentation of the results in front of your peers in the course. If you fulfill these requirements, I will send you a certificate stating that you have participated in this way in the online course. It is up to you to find out whether some dean of studies at your university would even accept this in some official manner - in any case, you could add it to your CV, if you like. ## Some extra info This course has been delivered twice before 'in person', in the summers of 2017 and 2019, at the [physics department](https://www.physics.nat.fau.eu) of the [university in Erlangen](https://www.fau.eu) (Bavaria/Germany). Both versions have been recorded on video, and it is the 2019 recordings that we will use here (see below for the links). Please disregard any organizational announcements made on the recorded video, as they of course relate to the 2019 course. We will make any up-to-date announcements in the live sessions. The original website for the course is https://machine-learning-for-physicists.org, but the detailed up-to-date materials for the present online version will be found here! I (Florian Marquardt) am a theoretical physicist working at the intersection of nanophysics and quantum optics, with a strong recent focus on applications of machine learning. My group is located at the [Max Planck Institute for the Science of Light](https://mpl.mpg.de/divisions/marquardt-division/), and I am also affiliated with the university in Erlangen. On my low-volume twitter account [FMarquardtGroup](https://twitter.com/FMarquardtGroup), you can find announcements of our most recent research papers as well as job openings (the latter also on the [group website](https://mpl.mpg.de/divisions/marquardt-division/)). If you want to reach me with questions regarding the course, it is most efficient to use the forum. My email account is usually overflowing with administrative stuff, so I would likely miss your email in my inbox. Finally, I also would like to thank Thomas Fösel, Leopoldo Sarra, Riccardo Porotti, and Victor Lopez-Pastor for their help in the earlier lectures. ## Link Map (literature and video links) Visit the [Deep Learning Basics Link Map](https://florianmarquardt.github.io/deep_learning_basics_linkmap.html) with links to a few selected original articles, as well as to the video lectures (YouTube links). ![](https://pad.gwdg.de/uploads/b29c10fe-42cd-4f67-b070-e653f3a1b7ab.jpg) ## Videos The lecture videos are available on: [Lecture videos playlist on YouTube](https://www.youtube.com/playlist?list=PLemsnf33Vij4eFWwtoQCrt9AHjLe3uo9_) [Lecture videos on FAU Erlangen platform (originals)](https://www.fau.tv/course/id/778) [Lecture videos on iTunes University](https://podcasts.apple.com/us/podcast/id1490099216) ## Lecture Notes Although there are no extended lecture notes that cover all the material of this series, I have produced a condensed version (40 pages) of lecture notes, on the occasion of the Les Houches Summer School 2019. You can download a draft here (including discussions of applications to quantum physics, and a bit about quantum machine learning): [Les Houches 2019 Machine Learning Lecture Notes](https://scipost.org/SciPostPhysLectNotes.29) If you find these lecture notes useful for your research, please cite them, e.g. like this: *F. Marquardt, "Machine Learning and Quantum Devices", SciPost Phys. Lect. Notes 29 (2021)* ## Slides Here are the slides used in the lectures, split up in three parts (first one goes up to and including t-SNE; third one covers some applications to science). [Machine Learning for Physicists, Slides, Part One (pdf, 25 MB)](https://owncloud.gwdg.de/index.php/s/qetLJgXMW6u3FwC) [Machine Learning for Physicists, Slides, Part Two (pdf, 13 MB)](https://owncloud.gwdg.de/index.php/s/Sc89k0zzFlaw0g9) [Machine Learning for Physicists, Slides, Part Three (pdf, 4 MB)](https://owncloud.gwdg.de/index.php/s/iLpJmXvDI2I7zpz) ## Material from previous series [Link to the old 2020 lecture series website](https://pad.gwdg.de/Machine_Learning_For_Physicists_2020) ## Discussion Forum https://groups.google.com/g/machine-learning-for-physicists This is a forum that we already used for the 2020 version of this lecture series. I think some of the old posts may still be useful, so we continue there. ## Github code repository All the python codes for tutorials, homework, and examples in the lectures (as jupyter notebooks and also as pure python code): https://github.com/FlorianMarquardt/machine-learning-for-physicists ## Software needed First of all, you want to install the [python](https://www.python.org) programming language. Also, I recommend [JupyterLab](https://jupyter.org) as a convenient notebook interface for python programming. Depending on your taste and your system, you might want to download these individually or as part of a full distribution like [Anaconda](https://www.anaconda.com). An alternative, completely online solution is the [Colaboratory](https://colab.research.google.com/notebooks/intro.ipynb) platform by Google. This is a web-based jupyter notebook interface to python that comes with tensorflow & keras pre-installed and allows you to run your code for free on their GPU servers. A third alternative is [deepnote](https://deepnote.com/). This is also a free online platform with a jupyter interface (and tensorflow & keras). But in contrast to colaboratory, several people can edit the same notebook (and execute cells) at the same time! We use it internally in our group for joint programming sessions to explore topics (discussing via a zoom session). Starting in the 3rd lecture, we will use [keras](https://keras.io), the convenient high-level python neural-network software package. This is included automatically in every recent [tensorflow](https://www.tensorflow.org) installation, so I recommend installing tensorflow (after having python) and then getting access to keras commands in the form ```python import tensorflow as tf from tensorflow.keras.layers import Dense ``` Note: In the examples shown in the course, I assumed you have a keras installation (and, below that, as support for keras, there would be tensorflow or other similar packages), so the syntax for these imports looked slightly different. Adapt as needed. If you have trouble installing the software (all of which is free!), you may use the forum (link above) to ask other students in the course who might help you. If you have suggestions for useful software or platforms etc., please post them on the forum! This is a community effort (since I will not be able to provide individual help with installations). ## Course overview: Lectures and online sessions :::info Note: All notebooks herein are distributed under the terms of the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license, with the exception of the few notebooks that contain code by other people (which is clear inside the notebook): https://creativecommons.org/licenses/by-sa/4.0/ ::: ### Session 0: Introduction There is no video lecture to watch before this course, this is just the introduction. :::success :calendar: Online session introducing the course: **April 12, 6pm** (all times CEST, i.e. time in Germany) :movie_camera: [video recording](https://owncloud.gwdg.de/index.php/s/iadCsoIXbPgenRs) ::: We will discuss the lecture series program, prerequisites, and the format of the online sessions. I will also say some words about the pre-history of neural networks. [Old 2020 Slides for online session (PDF, 14 MB)](https://owncloud.gwdg.de/index.php/s/icQRtIwFtwPvXfh) Suggested historical reading: [Alan Turing's "Computing Machinery and Intelligence" (1950)](https://academic.oup.com/mind/article/LIX/236/433/986238), introducing what is now known as the "Turing test" for artificial intelligence. A wonderful (and very nontechnical) paper! Some recent 'Turing test chat' with the most powerful neural language model (GPT-3): https://kirkouimet.medium.com/turing-test-2669daffae38 (beware: in the description he writes that he sometimes reruns the model's probabilistic responses until he finds something suitable, so this might be a bit of 'cheating'...) Further historical reading: [Rosenblatt's concept of a "Perceptron" (1957)](http://blogs.umass.edu/brain-wars/files/2016/03/rosenblatt-1957.pdf) ### Lecture 1: Basic Structure of Neural Networks :::success :cinema: Recorded video: [Lecture 1](https://www.video.uni-erlangen.de/clip/id/10611) (slides: see above in section "Slides") :calendar: Online Q&A session and tutorials about this material: **April 19, 6pm** (please watch the lecture video before). [Recording](https://owncloud.gwdg.de/index.php/s/cVJXAIwAifv0jlk) ::: Contents: Introduction (the power of deep neural networks in applications), brief discussion of the lecture outline, structure of a neural network and information processing steps, very brief introduction to python and jupyter, implementing a deep neural network efficiently in basic python (without any additional packages like tensorflow), illustration: complicated functions generated by deep neural networks After this lecture, you will know the basic structure of a neural network and how to implement the 'forward-pass' in python, but you don't yet know how to adapt the network weights (training). :::info **Code (jupyter notebook):** [01_MachineLearning_Basics_NeuralNetworksPython.ipynb](https://owncloud.gwdg.de/index.php/s/Unl2Yru1HsqwQNK) (or download [code as pure python script](https://owncloud.gwdg.de/index.php/s/WLfHIv2YXhq0Z29)) ::: This notebook shows how to calculate the forward-pass through a neural network in pure python, and how to illustrate the results for randomly initialized deep neural networks (as shown in the lecture). :::info **Notebooks for tutorials:** [Tutorial: Network Visualization notebook](https://owncloud.gwdg.de/index.php/s/lugGF9eCxClOk56) (also as [pure python](https://owncloud.gwdg.de/index.php/s/SgogI8iVzi60owR)) [Tutorial: Curve Fitting notebook](https://owncloud.gwdg.de/index.php/s/63ok2nUCuTvYwb7) (also as [pure python](https://owncloud.gwdg.de/index.php/s/SgogI8iVzi60owR)) ::: The "Network Visualization" notebook shows how to visualize arbitrary multilayer networks (with two inputs and 1 output neuron, so the result can be displayed in the plane), including a visualization of the network structure. The "Curve Fitting" notebook visualizes nonlinear curve fitting for a 1D function with a few parameters, via stochastic gradient descent. This is useful to understand what is going on in the higher-dimensional case of neural networks, where essentially the same concept is applied. :::success **Homework for Lecture 1** (The solutions will be briefly discussed in the online session for lecture 2. In order to follow the lecture series, please do at least two of these problems – I suggest the ones with the *. And of course, if you do more, you will get more practice and quickly become a master!) (1)* Implement a network that computes XOR (arbitrary number of hidden layers); meaning: the output should be +1 for y1 y2<0 and 0 otherwise! (2) Implement a network that approximately or exactly computes XOR, with just 1 hidden layer(!) (3)* Visualize the results of intermediate layers in a multi-layer randomly initialized NN (meaning: take a fixed randomly initialized multi-layer network, and then throw away the layers above layer n; and directly connect layer n to the output layer; see how results change when you vary n; you can start from the notebook [01_MachineLearning_Basics_NeuralNetworksPython.ipynb](https://owncloud.gwdg.de/index.php/s/Unl2Yru1HsqwQNK)) (4) What happens when you change the spread of the random weights? Smart weight initialization is an important point for NN training. (5) Explore cases of curve fitting where there are several (non-equivalent) local minima. Is sampling noise helpful (i.e. the noise that comes about because of the small number of x samples)? ::: ### Lecture 2: Training a Neural Network :::success :cinema: Recorded video: [Lecture 2](https://www.video.uni-erlangen.de/clip/id/11034) :calendar: Online session: **Apr 26, 6pm** [Recording](https://owncloud.gwdg.de/index.php/s/5LYS5TDUbnBJ4tQ) ::: Contents: Batch processing of many input samples, efficient implementation in python, neural networks can generate arbitrary functions, training a network as (high-dimensional) nonlinear curve fitting, cost function, stochastic gradient descent, backpropagation algorithm, full implementation in python, relation to physics (path integral), summary: basic ingredients of neural networks (and hyperparameters) After this lecture, you will in principle be able to train a deep neural network on arbitrary tasks (using pure python code that we provide in the lecture). But you don't yet know how to choose a smart representation of the data in more complicated cases, how best to choose the parameters during training, how to accelerate the gradient descent, etc. :::info **Code shown in lecture:** [Notebook: backpropagation with python](https://owncloud.gwdg.de/index.php/s/1xIsExzz03f0e5b) (also as [pure python script](https://owncloud.gwdg.de/index.php/s/wLg15kOdNFrc4WF)) ::: This notebook demonstrated how to implement backpropagation in python, in a small amount of code. For a more cleaned-up version of this implementation and much better visualization, see the tutorial notebook below! :::info **Notebook for online tutorials:** [Tutorial: Network Training Visualization notebook](https://owncloud.gwdg.de/index.php/s/ME4Gxe0XLskNPK1) (as [pure python script](https://owncloud.gwdg.de/index.php/s/QFiiS41PlJFpw8V)) ::: This notebook visualizes training of multilayer neural networks! :::success **Homework for Lecture 2** (1)* Carefully study the backpropagation algorithm, on paper and in the program (2)* Visualize the training of a multi-layer network for some interesting function! Explore reLU vs sigmoid! (3) Analyze the evolution of the slope w during stochastic gradient descent on a cost function given by C=(1/2) sum_j (w x_j - wtilde x_j)^2, where x_j are the N samples drawn from a Gaussian distribution in a single training step. (this is an advanced exercise) ::: (following items still under construction/revision) ### Lecture 3: Training and analyzing networks, Keras package, Image recognition :::success :cinema: Recorded video: [Lecture 3](https://www.video.uni-erlangen.de/clip/id/11243) :calendar: Online session: **May 3, 6pm** [Recording](https://owncloud.gwdg.de/index.php/s/PwHn0yD8of42Ubf) ::: Contents: extended review of previous material, training a 2D function (reproducing an arbitrary image), first steps towards analyzing and interpreting a neural network, influence of batch size and learning rate, brief introduction to the keras package and its basic neural network routines, image recognition (one-hot-encoding, softmax, cross-entropy), the danger of overfitting, training vs. validation vs. test data After this lecture, you will have more experience in training, understand the dangers, and know what the most elementary hyperparameters such as batch size or learning rate will do. You will also have a first taste of [keras](https://keras.io), the convenient high-level neural-network software package (included in every [tensorflow](https://www.tensorflow.org) installation). You will know the basics of how image recognition works, which is one of the most important applications of neural networks. But you don't yet know about the more advanced network structures one can build, such as the convolutional neural networks important for image recognition. Hyperparameter optimization: Some of you asked about how to choose hyperparameters. There is a variety of automatic approaches, and interestingly the [Wikipedia page on hyperparameter optimization](https://en.wikipedia.org/wiki/Hyperparameter_optimization) provides a nice overview and a good set of links. But do not worry about this too early! :::info **Notebooks shown in lectures:** [Notebook: Training on an Image and Interpreting Neurons](https://owncloud.gwdg.de/index.php/s/S6VfIxnaEuBGorv) (also as [python script](https://owncloud.gwdg.de/index.php/s/S6VfIxnaEuBGorv)); you also need to download [the smiley image](https://owncloud.gwdg.de/index.php/s/MNSDa0GclF9i85h)! This is the training on the smiley image and inspecting the network by switching off neurons in the last layer (slightly modified from the lecture). Of course, you can replace the smiley by an image of your choice! [Notebook: Handwritten Digits (MNIST) Recognition using keras](https://owncloud.gwdg.de/index.php/s/UfQRfNPtdmAYUuA) This will require you to download the freely available MNIST images (Update: it seems these are included with keras installations nowadays). You can also adapt this notebook to work on other images, of course, including algorithmically generated ones. ::: :::info **Notebook for online tutorials:** [Tutorial Notebook: Minimal Keras example](https://owncloud.gwdg.de/index.php/s/I6b9ElLLngufWCF) (also as [python script](https://owncloud.gwdg.de/index.php/s/GUhLTBaMiVArHWe)) This is really a minimal example, which shows how to build a neural network in a few lines using keras, and how to train and evaluate it (with one line each). ::: :::info **Further notebook:** [Tutorial Notebook: Network Training Visualization using Keras](https://owncloud.gwdg.de/index.php/s/ehGzaEavwb7DDvr) (also as [python script](https://owncloud.gwdg.de/index.php/s/l2DLevxQc8PCCkn)) This is essentially a keras-version of our visualization notebook from lecture 2! ::: :::success **Homework for Lecture 3** (will be discussed in online session for lecture 4; hint: use the forum to discuss and post your ideas and solutions!) (1)* Train on some function or image and explore the role of various neurons (by switching weights on/off like in the lecture); use the keras training visualization notebook above! (2)* Produce a simple network with keras that can distinguish a 1D plot of a random Gaussian from a 1D plot of a random Lorentzian! (or other classes of functions) (3) Produce a simple network with keras that can distinguish a 2D picture of a randomly placed square from that of a randomly placed circle! What happens if you add noise to the images? ::: ### Lecture 4: Convolutional Neural Networks, Autoencoders, Principal Component Analysis :::success :cinema: Recorded video: [Lecture 4](https://www.video.uni-erlangen.de/clip/id/11266) :calendar: Online session: **May 10, 6pm** [Recording](https://owncloud.gwdg.de/index.php/s/bIYKJDEt96M82u9) ::: Contents: A bit more on image recognition, convolutional neural networks as an efficient way to process images and other data with translational invariance, (kernels, channels, and so on), autoencoders for unsupervised learning and information compression, principal component analysis as a simple linear way to extract the main (linear) features of data sets After this lecture, you will know the workhorse behind image recognition networks, i.e. convolutional layers, you will understand the beautiful concept of autoencoders which automatically extract the essential features of data (and can turn those back into the data), and you will know about the widely used tool of principal component analysis. But there is more to come! :::info **Notebook for online tutorials:** [Tutorial Notebook: Visualizing the training of convolutional autoencoders](https://owncloud.gwdg.de/index.php/s/WaTnoRBSF9UxWoN) (also as [pure python script](https://owncloud.gwdg.de/index.php/s/GFe4zAy02vSiItM)) See how an autoencoder operates! And take the **Grand Autoencoder Challenge** in the homework problem afterwards! ::: :::success **Homework for Lecture 4** (will be discussed in online session for lecture 5; hint: use the forum to discuss and post your ideas and solutions!) *(1) Compete in the ***Grand Autoencoder Challenge!*** This is described in the notebook above: try to optimize an autoencoder that can efficiently encode randomly placed circles (post your intermediate cost values in the [forum](https://groups.google.com/d/forum/machine-learning-for-physicists); specify which version of the challenge this refers to: MEDIUM/HARD, NORMAL/LONG-TRAINING) (2) Train a network to turn images that contain multiple random circles into squares of the same size and placement! ::: ### Lecture 5: Principal Component Analysis, t-SNE and unsupervised dimensionality reduction, Advanced Gradient Techniques, Introduction to Recurrent Neural Networks :::success :cinema: Recorded video: [Lecture 5](https://www.video.uni-erlangen.de/clip/id/11487) :calendar: Online session: **May 17, 6pm** [Recording](https://owncloud.gwdg.de/index.php/s/C3PtCFYbYeQz8cd) ::: Contents: More about the principal component analysis, unsupervised dimensionality reduction techniques for clustering and other applications (t-SNE etc.), advanced gradient descent techniques (like 'adam' and its siblings), first introduction to recurrent neural networks After this lecture, you will know how to take a high-dimensional data cloud and reduce it down to low dimensions without losing the important clusters of data points, you will know what's behind 'adam' and similar gradient descent techniques that are adaptive and can accelerate training, and you will know what's the motivation for recurrent networks which can be used to analyze time series and sentences. However, you do not yet know exactly how efficient recurrent networks are implemented, which will be the subject of lecture 6! :::info **Notebook for online tutorials:** [Tutorial Notebook: t-SNE for dimensionality reduction](https://owncloud.gwdg.de/index.php/s/6EwPmdqcHUWOkAt) (also as [pure python script](https://owncloud.gwdg.de/index.php/s/dtsLRWKvkaCIjPP)) See how t-SNE works for producing 2D representations of high-dimensional data! ::: :::success **Homework for Lecture 5** (will be discussed in online session for lecture 6; hint: use the forum to discuss and post your ideas and solutions!) *(1) Train an autoencoder and then visualize the neuron activations in the bottleneck layer via t-SNE! (i.e. for each of many input images extract the bottleneck activations, which defines a vector in a high-dimensional space. Plot all these vectors in a 2D projection produced by t-SNE) Can you observe any meaningful clusters? ::: ### Lecture 6: Recurrent Neural Networks (LSTM), Word Vectors, Introduction to Reinforcement Learning :::success :cinema: Recorded video: [Lecture 6](https://www.video.uni-erlangen.de/clip/id/11537) :calendar: Online session: **May 31, 6pm** [Recording](https://owncloud.gwdg.de/index.php/s/YrW8JFfdh29WD69) ::: Contents: The inner workings of a long short-term memory (LSTM) recurrent network, simple examples for recurrent network applications, word vector encodings, first introduction to reinforcement learning After this lecture, you will know how to set up a recurrent network to process sequences, you will have learned about "word vectors" that encode conceptual similarities between words, and you will know what is the main goal of reinforcement learning. But you do not yet know how reinforcement learning really works in practice! :::info **Notebook with solutions for lecture 5 homework** (discussed in online session 6) [Notebook: Applying tSNE to the bottleneck neurons of an autoencoder](https://owncloud.gwdg.de/index.php/s/FOYEqZ9eRynfH0c) (also as [pure python script](https://owncloud.gwdg.de/index.php/s/ieNTFFLLiaYYSde)) ::: :::info [Notebook: Simple LSTM examples](https://owncloud.gwdg.de/index.php/s/4rwfuMpKZow6V6m) (also as [pure python script](https://owncloud.gwdg.de/index.php/s/iuPohyfCrf8cgFG)) From the lecture: the recall network and the countdown network! ::: :::info **Notebook for online tutorials:** [Tutorial Notebook: Recurrent neural network for predicting physical time evolution](https://owncloud.gwdg.de/index.php/s/ewRZOw8B1sRYaWJ) (also as [pure python script](https://owncloud.gwdg.de/index.php/s/wZNzWTPeFzzasG5)) See how an LSTM network can help to predict the time evolution of a particle that is observed only indirectly through its effects on another particle. This task requires memory! ::: :::success **Homework for Lecture 6** (will be discussed in online session for lecture 7; hint: use the forum to discuss and post your ideas and solutions!) *(1) Apply a recurrent network to predict the time-evolution of a physical system during t=T..2T after observing it during t=0..T (for random initial conditions); try to treat the case of 'partial' observation (i.e. observing only one of the particles). Hint: Give as input a trajectory that is nonzero for 0..T (according to the evolution), and zero during T..2T. Give as the correct output the full trajectory for all times. ::: ### Lecture 7: Reinforcement Learning: Policy Gradient, Baseline, Simple Examples :::success :cinema: Recorded video: [Lecture 7](https://www.video.uni-erlangen.de/clip/id/11571) :calendar: Online session: **June 7, 6pm** [Recording](https://owncloud.gwdg.de/index.php/s/7K2bQHVvtxwqJrP) ::: Contents: Reinforcement learning (agent, environment, state, action, reward, policy), policy gradient, baseline, simple example: random walker (with analytical solution) After this lecture, you will understand how the important "policy gradient" algorithm for reinforcement learning is derived, how a baseline for the rewards can help, and you will have seen in a very simple example how this all works (not yet with neural networks). However, you still need to see other techniques of reinforcement learning, and how "deep reinforcement learning" with neural networks works. :::info **Notebook with solutions for lecture 6 homework** (discussed in online session 7) [Notebook: LSTM for predicting time evolution during second half of an interval](https://owncloud.gwdg.de/index.php/s/nAJnfaEKuXPBkxN) (also as [pure python script](https://owncloud.gwdg.de/index.php/s/reBBgeN95LCr5Yx)) Given the trajectory of a single particle (coupled to others!) during [0,T], predict its evolution during [T,2T]. This requires implicitly understanding the initial conditions for all the particles, as well as the coupled equations of motion! ::: :::info **Notebook for online tutorials:** [Tutorial Notebook: Reinforcement Learning with Policy Gradient: Treasures in a Maze](https://owncloud.gwdg.de/index.php/s/TbJU0Qgw4r76Mlt) (also as [pure python script](https://owncloud.gwdg.de/index.php/s/fXl6ChrMLLTNnl3)) A little robot wanders through a maze, searching for treasure chests! Will it discover the optimal strategy to get the maximum sum of rewards? This shows table-based reinforcement learning. ::: ### Lecture 8: Reinforcement Learning: Policy Gradient with Neural Networks, Q Learning :::success :cinema: Recorded video: [Lecture 8](https://www.video.uni-erlangen.de/clip/id/11621) :calendar: Online session: **June 14, 6pm** [Recording](https://owncloud.gwdg.de/index.php/s/8leyfDD8p7Fh0dh) ::: Contents: more examples for policy gradient, neural-network-based policy gradient ("deep policy gradient"), AlphaGo, Q learning :::info **Notebook for online tutorials:** [Notebook: Improved deep policy gradient for solving mazes](https://owncloud.gwdg.de/index.php/s/yZQQjmeeVK6hi74) (also as [pure python script](https://owncloud.gwdg.de/index.php/s/Mo1J79RoeMRP5pu)) This shows neural-network-based reinforcement learning, converting an input image (a map showing the location of the robot) into a map of action probabilities. ::: For much(!) more information on deep reinforcement learning, please see the expert lectures at: [2017 Deep RL Bootcamp](https://sites.google.com/view/deep-rl-bootcamp/lectures) ### Lecture 9: Boltzmann machines :::success :cinema: Recorded video: [Lecture 9](https://www.video.uni-erlangen.de/clip/id/11647) :calendar: Online session: **June 21, 6pm** [Recording](https://owncloud.gwdg.de/index.php/s/irwQtzgdoYb4aLE) [**Updated version of all tutorial slides** (11 MB)](https://owncloud.gwdg.de/index.php/s/7uc5XCRYWcDfwD7) ::: Contents: Boltzmann machines as a way to generate samples from an observed probability distribution (e.g. images that look like images that have been observed), connection to statistical physics and Monte Carlo sampling, applications in physics :::info **Notebook with solutions for lecture 8 homework** (discussed in online session 9) [Notebook: Improved deep policy gradient for solving mazes](https://owncloud.gwdg.de/index.php/s/yZQQjmeeVK6hi74) (also as [pure python script](https://owncloud.gwdg.de/index.php/s/Mo1J79RoeMRP5pu)) That now works very nicely. ::: :::info **Notebook for online tutorials:** [Tutorial Notebook: Boltzmann Machine](https://owncloud.gwdg.de/index.php/s/om63z9vknk195Bi) (also as [pure python script](https://owncloud.gwdg.de/index.php/s/qsZJxJFqPu5bLTD))) Train a Boltzmann machine to sample from a probability distribution that it has observed via training samples! (this is really just a few lines of code) ::: :::success **Homework for Lecture 9** (will be discussed in online session for lecture 10; hint: use the forum to discuss and post your ideas and solutions!) *(1) Apply the Boltzmann machine to the MNIST images! (explore the role of learning rate, batchsize, number of hidden units, etc.) (2) **Challenge**: Who can get the highest return in the maze reinforcement learning? [simulation parameters: 15x15 maze, 10 treasure chests, 40 time steps, chests deleted; like in the last example in the 'improved' deep policy gradient notebook above, see there for typical values for the return!] You can try to experiment with the network, with the learning rate and procedure, with the batchsize, etc. ::: ### Lecture 10: Neural Networks in Science, Artificial Scientific Discovery :::success :cinema: Recorded video: [Lecture 10](https://www.video.uni-erlangen.de/clip/id/11735) :calendar: Online session: **June 28, 6pm** [Recording](https://owncloud.gwdg.de/index.php/s/N6kc5QFLzOAwMm4) ::: Contents: General considerations in applying neural networks to science, example applications in various domains (like chemistry and medicine) and especially in physics (e.g. statistical physics), artificial scientific discovery as a long-term goal: what would be needed? :::info **Notebook with solutions for lecture 9 homework** (discussed in online session 10) [Notebook: Boltzmann Machine applied to MNIST images](https://owncloud.gwdg.de/index.php/s/FsqAqiS8a4D15e0) (also as [pure python script](https://owncloud.gwdg.de/index.php/s/2RjCREAfbFz6drx)) Learn to reproduce images of digits, by training a restricted Boltzmann machine on the MNIST images (supplied with every keras installation). ::: ### Lecture 11: Applications to Quantum Science and Technology, Reinforcement Learning for Quantum Error Correction :::success In the tutorial, I will present a different take than in the recorded video (Old recorded video: [Lecture 11](https://www.video.uni-erlangen.de/clip/id/11761)) :calendar: Online session: **July 5, 6pm** [Recording](https://owncloud.gwdg.de/index.php/s/xjwDoIdl0KgRnOP) ::: [Slides of Talk on Deep Reinforcement Learning for Quantum Technologies](https://owncloud.gwdg.de/index.php/s/8B6GgiPRUMINhwP) Overview of some applications of machine learning to challenges in quantum science and technology, detailed example from our own research: applying deep reinforcement learning to quantum error correction Thomas Fösel, Petru Tighineanu, Talitha Weiss, and Florian Marquardt, "Reinforcement Learning with Neural Networks for Quantum Feedback", [Physical Review X, 031084 (2018)](https://journals.aps.org/prx/abstract/10.1103/PhysRevX.8.031084) *This 2018 article was actually the first to apply deep reinforcement learning anywhere in quantum physics, together with a qubit control paper from another team (August and Hernandez-Lobato) that appeared practically simultaneously.* ### Lecture 12: General question session & discussion of practice exam :::success :calendar: Online session: **July 12, 6pm** [Recording](https://owncloud.gwdg.de/index.php/s/pUV1zOq6NdHcchM) ::: We will first take about half an hour to discuss any remaining general questions you might have regarding the content of the lectures. Then we will discuss an exam, which is relevant for the FAU students. In preparation for this session, please try to solve the [2020 exam](https://owncloud.gwdg.de/index.php/s/jSKKfZi1pvF5onx). The students had 2 hours for that. Solution of the [2020 exam](https://pad.gwdg.de/2020_MachineLearningPhysicists_Exam_Solution#). [2019 exam](https://owncloud.gwdg.de/index.php/s/QUBPdc5Bv8fOyeU)...sorry, no written solution here. ### Mini project presentations :::success :calendar: Online session: **July 19, 6pm** ::: ### Exam for Erlangen students :::success :calendar: **Wednesday July 28, 10:00-12:00, lecture hall HD & HF** :::