ALL INTELLIGENCE
Hacker News AIT2·Models

A Visual Explainer of Reinforcement Learning and Weight Perturbation

Original title: A visual explanation of reinforcement learning
42AI Score

This is an interactive popular-science explainer that visualizes the fundamentals of reinforcement learning, using Braitenberg vehicles to show how neural networks can learn via weight perturbation instead of backpropagation. It notes that OpenAI used a similar evolution-strategies approach in 2017 to train AI to play Atari games. The piece is educational and contains no new research results or model releases.

Full Report

Contents3 sections

Learn more

Thanks for reading chapter two of Myriad, a series exploring how minds emerge from matter. Myriad is a collaboration between Arkarup Banerjee, a neuroscientist who studies behavior, and Aatish Bhatia, a physicist turned science journalist.

How the spark moves

The sparks in this article are inspired by the work of the neuroscientist Valentino Braitenberg. We implemented their neural circuitry as follows:

vL=σ(ws bL+wc bR+d)vR=σ(wc bL+ws bR+d)\begin{aligned}v_L &= \sigma(\textcolor{#7ee081}{w_s}, b_L + \textcolor{#c49bff}{w_c}, b_R + \textcolor{#ff8fb8}{d})\ v_R &= \sigma(\textcolor{#c49bff}{w_c}, b_L + \textcolor{#7ee081}{w_s}, b_R + \textcolor{#ff8fb8}{d})\end{aligned}vLvR=σ(wsbL+wcbR+d)=σ(wcbL+wsbR+d)

where vLv_LvL and vRv_RvR are the left and right flipper speeds, bLb_LbL and bRb_RbR are the brightness levels at the left and right eyes, ws\textcolor{#7ee081}{w_s}ws and wc\textcolor{#c49bff}{w_c}wc are the straight and crossed connection strengths between the eyes and the flippers, and d\textcolor{#ff8fb8}{d}d is a drive that sets the spark’s speed when there is no light. σ(x)\sigma(x)σ(x) is the ReLU function. The spark’s forward speed is the average of the two flipper speeds, and its turning speed is proportional to their difference.

How the spark learns

There are three parameters in our simplified setup: ws,\textcolor{#7ee081}{w_s}\textrm{,}ws, wc\textcolor{#c49bff}{w_c}wc and d.\textcolor{#ff8fb8}{d}\textrm{.}d. The spark learns values for these parameters using a version of the REINFORCE algorithm published by Ronald Williams in 1992.

Each trial, the parameters were wiggled by a small random amount and the spark moved for 500 time steps, starting from a random point on the edge of the arena. Its reward was the average brightness along its path.

This reward was then compared to a running average of recent rewards. If the spark did better than usual, its parameters were nudged in the direction of this random change. If it did worse, its parameters were nudged in the opposite direction. The size of the nudge was proportional to how much better (or worse) it did:

ΔW(R‾)change=α(R‾)step sizeε(R‾)wiggle(R−R‾)(R‾)improvement\small \underset{\mathclap{\htmlClass{sans}{\text{change}}}}{\Delta W\vphantom{(\overline{R})}}\hspace{0.3em}=\hspace{0.5em}\underset{\mathclap{\htmlClass{sans}{\color{#66b3ff}\text{step size}}}}{\textcolor{#66b3ff}{\alpha}\vphantom{(\overline{R})}}\hspace{2.1em}\underset{\mathclap{\htmlClass{sans}{\color{#ff9b32}\text{wiggle}}}}{\textcolor{#ff9b32}{\varepsilon}\vphantom{(\overline{R})}}\hspace{1.2em}\underset{\mathclap{\htmlClass{sans}{\color{#ffc832}\text{improvement}}}}{\textcolor{#ffc832}{(R - \overline{R})}\vphantom{(\overline{R})}}changeΔW(R)=step sizeα(R)wiggleε(R)improvement(R−R)(R)

Here ΔW\Delta WΔW is the change in the three parameters, α\textcolor{#66b3ff}{\alpha}α is the learning rate, ε\textcolor{#ff9b32}{\varepsilon}ε is the random wiggle, and R−R‾\textcolor{#ffc832}{R - \overline{R}}R−R is how much better or worse than usual the spark did. We repeated this thousands of times.

Williams’ original approach adds randomness to each neuron’s output. We add it to the weights and biases instead, an approach known as weight perturbation. A similar idea, called evolution strategies, was used by OpenAI in 2017 to teach AI models to play Atari video games.