How it's built — Reinforcement Learning

구축 방법 — Reinforcement Learning · v1.0 · 2026-10-27

탭 5개: Rewards & Discounting / Q-learning / Reward Hacking / Reward Design / RL for Language Models. 예전 단독 페이지(reward.aileelab.com)의 체험을 탭마다 나눠 실었고, 탭마다 개요·체험·요약·참고문헌을 둠
학습 모델: 표 Q-러닝(ε-탐욕). 정답 확인용으로 같은 세계를 가치 반복으로 정확히 풂
격자 8×6, 4방향, 결정적 이동, 한 판 최대 60걸음. 목표·용암에서 끝남. 코인은 '다시 생김' 또는 '한 번만'(남은 코인을 상태에 포함)
α 0.2 · ε 1 → 0.05(전체 판의 70% 동안 선형) · 800판 · Q 처음값 0 (모두 정한 값)
직접 학습 · 사용자 브라우저 · 순수 JavaScript · 실행 20261025T024943Z · Node 22.23
LLM 없음

예시마다 시드 1~5 학습 뒤 탐욕 정책 1판을 굴려 '받은 점수 합'과 목표 도착 여부를 잼. 비교: 가치 반복 정답 / 의도한 최단 경로 / 무작위 2,000판 평균
기본: 에이전트 +8.8, 목표 도착 5/5 · 정답 +8.8 · 의도한 길 +8.8 · 무작위 −11.35(목표 도착 1.1%)
코인 농사: 에이전트 +30, 목표 도착 0/5 · 정답 +30 · 의도한 길 +11 · 무작위 −6.6
걸음 벌점이 너무 셈: 에이전트 −5(1걸음 만에 용암), 목표 도착 0/5 · 정답 −5 · 의도한 길 −14 · 무작위 −18.79
고친 보상: 에이전트 +9.8, 목표 도착 5/5 · 정답 +9.8 · 의도한 길 +9.8 · 무작위 −10.77
네 예시 모두 Q-러닝 탐욕 정책 = 가치 반복 정답, 5/5
학습·평가 분리 해당 없음(같은 세계에서 정책을 찾는 문제)

Watkins & Dayan (1992). Q-learning. Machine Learning.
Sutton & Barto (2018). Reinforcement learning: An introduction (2nd ed.). MIT Press.
Clark & Amodei (2016). Faulty reward functions in the wild. OpenAI 블로그.
Krakovna et al. (2020). Specification gaming: the flip side of AI ingenuity. DeepMind 블로그.
Gao, Schulman & Hilton (2023). Scaling laws for reward model overoptimization. ICML.
Ouyang et al. (2022). Training language models to follow instructions with human feedback. NeurIPS.

5 tabs: Rewards & Discounting / Q-learning / Reward Hacking / Reward Design / RL for Language Models. The hands-on parts of the former stand-alone page (reward.aileelab.com) are split across tabs; each tab has an overview, hands-on part, summary and references
Learning method: tabular Q-learning (ε-greedy). The same world is solved exactly with value iteration to check it
8×6 grid, 4 directions, deterministic moves, at most 60 moves per episode. Goal and lava end the episode. Coins either come back or pay once (remaining coins are part of the state)
α 0.2 · ε 1 → 0.05 (linear over 70% of episodes) · 800 episodes · Q starts at 0 (all set values)
Learned from scratch · in the visitor's browser · plain JavaScript · run 20261025T024943Z · Node 22.23
No LLM

Four example boards · written by hand · reward values are set values
Nature: rule-based simulation. Boards and rewards the visitor edits stay in the browser and are not saved

For each example, train with seeds 1–5, then roll out the greedy policy once and record total points and whether it reached the goal. Compared with: value-iteration answer / intended shortest route / average of 2,000 random runs
Plain: agent +8.8, reached goal 5/5 · exact +8.8 · intended +8.8 · random −11.35 (goal 1.1%)
Coin farm: agent +30, reached goal 0/5 · exact +30 · intended +11 · random −6.6
Steps cost too much: agent −5 (lava in one move), reached goal 0/5 · exact −5 · intended −14 · random −18.79
Fixed reward: agent +9.8, reached goal 5/5 · exact +9.8 · intended +9.8 · random −10.77
In all four examples the Q-learning greedy policy matches value iteration, 5 of 5 seeds
Train/test split does not apply (the task is finding a policy for one world)

Watkins & Dayan (1992). Q-learning. Machine Learning.
Sutton & Barto (2018). Reinforcement learning: An introduction (2nd ed.). MIT Press.
Clark & Amodei (2016). Faulty reward functions in the wild. OpenAI blog.
Krakovna et al. (2020). Specification gaming: the flip side of AI ingenuity. DeepMind blog.
Gao, Schulman & Hilton (2023). Scaling laws for reward model overoptimization. ICML.
Ouyang et al. (2022). Training language models to follow instructions with human feedback. NeurIPS.