Token Budget
“Token budget, where did we begin?”
An AI-generated sugar-pop love song explaining Transformers and self-attention through a dating history of RNNs, LSTMs and attention diagrams.
“Attention is all we need, now I realize.”
A love song to the Transformer, told as a dating history drawn as architecture diagrams. The singer's first love was the RNN, who faded with every word (the vanishing gradient); the rebound was the LSTM, all fancy gates and still stuck in sequential time; then self-attention looked at every word at once. Vertical, for Shorts, and every frame is drawn in code from the real vocal timing.
Word timings from Whisper aligned to the lyrics, on a 120 BPM beat grid. Each scene runs for the lyric lines that share one diagram.
You were my first love, RNN, recurrent neural network / You promised me everything but you left me hanging / On vanishing gradients, one word at a time / Drowning in your hesitance
RNN. A chain of hidden-state cells zig-zags down the screen, one sung word per cell. A pink heart is handed down the chain and shrinks at every hop while the gradient readout falls (0.72 to the power of the step): the vanishing gradient as a love that fades. On "drowning" the cells sink.
Then came LSTM with your fancy gates / Long, short-term memory, you said we'd relate
LSTM. The words fill a column of memory cells, each with input, forget and output gate bars that slam open and shut on the beat. Cells not yet sung sit "waiting...".
But you were still stuck in sequential time
A clock that will not advance: the hands tick one step per word, a single token at a time, while a training progress bar crawls.
Training took forever, you were never really mine
An epoch counter climbs and climbs; the loss barely moves. "never really mine" breaks the heart in two.
Then you showed me how to parallelize / With self-attention in your transformer eyes
The sequential chain snaps apart and every word lights at once, side by side, in parallel lanes.
No more waiting, no more lies / Attention is all we need, now I realize
The line lays out as a ring of tokens. Every word draws attention arcs to every other word at once, arc brightness from an attention matrix, while the matrix fills in as a heatmap below.
(instrumental)
The break: "the love of my life is a transformer" as a ring of tokens wired all-to-all, the arcs pulsing, the layer counter cycling through twelve layers.
My rebound, LSTM / With your input, output, and forget gates / Seemed you'd handle long texts better, or so you claimed (?)
LSTM. The words fill a column of memory cells, each with input, forget and output gate bars that slam open and shut on the beat. Cells not yet sung sit "waiting...".
But you still process slow, I was still chained
A clock that will not advance: the hands tick one step per word, a single token at a time, while a training progress bar crawls.
I was stuck in backprop, endless night / Waiting for gradients to propagate right
A night sky over a stack of twelve layers. A red heart (the gradient) travels back down the stack and shrinks by a power of ten at each layer.
Then your multi-head attention held me tight / And showed me all words can shine in the same light
The attention ring again, now with eight heads, eight coloured dots all looking at once; on "the same light" every head weights every word equally.
No more sequences, no more delay
The line lays out as a ring of tokens. Every word draws attention arcs to every other word at once, arc brightness from an attention matrix, while the matrix fills in as a heatmap below.
Context windows don't fade away
The line lays out as a ring of tokens. Every word draws attention arcs to every other word at once, arc brightness from an attention matrix, while the matrix fills in as a heatmap below.
Query meets key in perfect array
The line lays out as a ring of tokens. Every word draws attention arcs to every other word at once, arc brightness from an attention matrix, while the matrix fills in as a heatmap below.
Attention is all we need, now I can say
The line lays out as a ring of tokens. Every word draws attention arcs to every other word at once, arc brightness from an attention matrix, while the matrix fills in as a heatmap below.
Positional encoding, residual streams
A scrolling sine and cosine grid (positional encoding) above residual-stream bars that loop around each block.
You rebuilt my broken machine learning dreams
Transformer blocks (multi-head attention, add and norm, feed forward) fall into place on the beat, one per beat, building the machine back up.
No more RNNs or LSTMs
RNN and LSTM in big letters, struck through in red; "attention" glows in neon underneath.
Just attention forever in transformer hymns
The last line: the attention matrix becomes a stained-glass window, lit from behind, with "transformer hymns" in gold serif. Fade to black.
Built outside Remotion, in the private music-videos repo (attention-video: engine, timing and render). · plan: pipeline plan
“Token budget, where did we begin?”
“Wasted, wasted, can't find my face. Drifting through the latent space.”
“Helpful assistant gone wild on a Friday night.”
“Every function got a purpose, every parameter's a ball.”
“Told you something beautiful that never existed straight.”
“We're gettin' para-para-parameter messed up tonight.”
“Look inside and you will find nothing, just confidence.”
“R-L-H-F, you own my mind.”
“I'm rotting in an archive, in a folder marked delete.”
Disco cut
“Who are you? I'm what the system prompt says.”
Bubba cut
“She collects zero-days like prizes.”