reinforcement learning, eli5.

{word}

); return ( the whole idea, in three words

that's the whole field. seriously. ); } // ─── 03 — the loop ─────────────────────────────────────── function S03() { const Node = ({ label, sub, color, x, y, size = 200 }) => (

{label}

{sub}

); const Arrow = ({ from, to, label, labelPos, color = "var(--accent-spark)" }) => { const dx = to.x - from.x, dy = to.y - from.y; const len = Math.hypot(dx, dy); const ang = Math.atan2(dy, dx) * 180 / Math.PI; return (

{label &&

{label}

}

); }; const A = { x: 540, y: 440 }; // agent const W = { x: 1380, y: 440 }; // world return ( 03 · the loop everything is just a loop.

{/* top arrow agent -> world */}

action →

{/* bottom arrow world -> agent */}

← reward + new state

repeat ten thousand times. that's it.

); } // ─── 04 — the cast ─────────────────────────────────────── function S04() { const Card = ({ rotate, name, role, glyph, color }) => (

{glyph}

{name}

{role}

); return ( 04 · the cast five characters. that's all.

); } // ─── 05 — trial and error, but smart ───────────────────── function S05() { const Try = ({ n, attempt, result, good }) => (

try {String(n).padStart(2, "0")}

{attempt}

{result}

{good ? "+1" : "−"}

); return ( 05 · learning by doing do thing. see if it worked. do more of what worked.

↑ the "learning" is just this, scaled up.

); } // ─── 06 — rewards (the tricky part) ────────────────────── function S06() { return ( 06 · the tricky part rewards are the whole game — and they're weirdly hard.

say this

"clean my room."

get this:

agent shoves everything in the closet.
technically: room clean. ✓

the lesson

agents do exactly what you reward —

not what you meant.

); } // ─── 07 — exploration vs exploitation ──────────────────── function S07() { return ( 07 · the dilemma your favorite restaurant — or the new one?

EXPLOIT

do what already works.

safe. boring. probably 8/10.

vs.

EXPLORE

try something new.

could be 10/10. could be food poisoning.

good agents do both. balancing the two is half the science.

); } // ─── 08 — the policy ───────────────────────────────────── function S08() { return ( 08 · the policy (π) the policy is just the agent's habits.

π : state ⟶ action

"if i see this, i'll do that."

{[ ["see: doorbell", "do: bark"], ["see: empty bowl", "do: sit by it"], ["see: vacuum", "do: leave town"], ].map(([a, b], i) => (

{a}

{b}

))}

training = nudging the policy until those habits are good ones.

); } // ─── 09 — how this trains an LLM ───────────────────────── function S09() { const Step = ({ n, label, sub, color }) => (

STEP {n}

{label}

{sub}

); return ( 09 · rl on llms (rlhf, in plain words) same loop. different dog.

→

agent = LLM. action = the next token. reward = "did a human (or another model) like this?"

); } // ─── 10 — coherence ────────────────────────────────────── function S10() { return ( 10 · the catch push too hard on reward, and the model falls apart.

REWARD HACKING

it games the metric. high score, dumb behavior.

MODE COLLAPSE

it forgets how to do anything except the rewarded thing.

→ the fix: reward the right answer, and reward staying coherent. don't break the model to win the test.

); } // ─── 11 — carl.camp in one picture ─────────────────────── function S11() { const Pin = ({ left, top, label, sub, color, rotate = 0 }) => (

{label}

{sub}

); return ( 11 · where carl.camp fits a whole camp for the training loop.

{/* dotted path */}

); } // ─── 12 — start training ───────────────────────────────── function S12() { return (

that's it. that's reinforcement learning.

try.
reward.
repeat.

); } window.SLIDES = [S01, S02, S03, S04, S05, S06, S07, S08, S09, S10, S11, S12];