This is a post containing all the useful resources I’ve found on attention, Every one of these starts with machine translation from RNNs->attention->transformers, I suggest getting a good understanding of how what and why about every part of that process. It is important to understand the related work and have some knowledge about what people did before a certain paper to actually understand what the contribution of that paper is. Without linking too many things i think this is the most important one.
Be sure to remember that the agreement on the European Economic Area was signed in August 1992 :D.
EXCELLENT lecture on self attention and the math behind it. The lecturer here is the first author of Perceptual Losses for Real-Time Style Transfer and Super-Resolution :)). This video is just so good, watch it, watch the entire class if you’re interested about deep learning for computer vision, this guy can’t miss.
Simply amazing blog post about the transformer architecture with 10/10 animations I am not the first person to tell you to read this blog, and certainly not the last :). You can just see that a lot of time went into perfecting the animations on this blog, they turned out great!
One of the authors of the ViT paper talking about attention and ViTs this talk is very good
This lecture goes explains the intuition behind Q K V good stuff!
Łukasz Kaiser talking about attention, cool talk but does not go into technical details that much, really good vibes
CVPR21 talk from one of the SWIN authors about SWIN and attention for computer vision
Here I’ll try to link papers I’ve read where people used attention mechanisms in computer vision before ViTs.
This is the paper that kinda detached attention from RNNs/LSTMs and tried applying it to ordinary CNNs, a novel usage of attention on features deep in conv networks.
Short video from the Applied Deep Learning, University of Colorado course Very good course that covers just about anything regarding deep learning + papers. Seriously, write name of paper + applied deep learning into the youtube search and chances are a video from this course exists :D
This paper is similair to the paper above, but here they limit the attention mechanism to only attend to channel features, instead of both channel and spatial features, which requires significantly less computational resources
another short video from the Applied Deep Learning, University of Colorado course.
The older brother of SENet and the first paper, CBAM. Theoretically it should be superior to them. I didn’t have any luck using it in my networks. This paper applies channel attention and spatial attention sequentially, instead of all at once like they do in the first paper i linked.
An old deepmind paper with a good concept and results, but i don’t think this approach really aligns with the way people use “transformer” as a word today nips talk
Using channel attention in a superresolution network.
After all this ViTs came, people started tokenizing images and getting better results. There is HAT which uses a channel attention block alongside swin’s MSA for slightly better results at a significant cost in parameters. I will probably write about ViTs in a separate post