The Unreasonable Effectiveness of Recurrent Neural Networks (Karpathy)
June 21, 2024
-RNNs are special because they allow machines to operate on sequences, not constrained to a fix # of steps
1.Vanilla mode of processing without RNN, from fixed-sized input to fixed-sized output (e.g. image classification).
2.Sequence output (e.g. image captioning takes an image and outputs a sentence of words).
3.Sequence input (e.g. sentiment analysis where a given sentence is classified as expressing positive or negative sentiment).
4.Sequence input and sequence output (e.g. Machine Translation: an RNN reads a sentence in English and then outputs a sentence in French).
5.Synced sequence input and output (e.g. video classification where we wish to label each frame of the video).
-RNN combine an input vector, state vector, and a learned function to produce a new state
RNNs are Turing-Complete (can simulate arbitrary programs)