Neuro140/240 Biological and Artificial Intelligence

Class Materials:

Tutorial 4 on Transformers and NLP

In the tutorial I covered

• The paradigm of self-supervised learning
• NLP 101: language representation and statistical modelling of language
• Transformer architecture
• Transformer beyond NLP. 

In this notebook Transformer playground, we covered 

• The general structure of transformers. 
• Training a toy transformer to speak a highly simplified artificial "English", via probabilistic generative grammar.