How it's built — Language Model

구축 방법 — Language Model · v1.0 · 2026-10-27

탭 6개: Tokenization / Bigram Baseline / Training / Attention / Sampling / From Tiny to Large. 예전 단독 페이지(gpt.aileelab.com)의 체험을 탭마다 나눠 실었고, 탭마다 개요·체험·요약·참고문헌을 둠
학습 모델: 글자(자모) 단위 디코더 트랜스포머. 다음 토큰 맞히기, 교차 엔트로피 손실
2층 · 머리 4개 · 임베딩 48 · 피드포워드 192 · 문맥 64토큰 · pre-LN · GELU · 파라미터 66,987개(헌법) ~ 70,479개(영화평)
직접 학습 · 사용자 브라우저의 Web Worker(CPU) · 순수 JavaScript, 라이브러리 없음, 역전파 직접 구현
AdamW · 학습률 0.01 → 0.0005 코사인 · 워밍업 30단계 · 가중치 감쇠 0.1 · 기울기 자르기 1 · 묶음 8 × 64토큰 · 1,000단계 · 시드 1 (모두 정한 값)
기준 수치 실행 20261025T023903Z · Node 22.23 · Intel Xeon 2.1GHz 2코어
LLM 없음. 외부 AI 호출 없음

대한민국헌법 18,509자 · KoNLPy 0.6.0 kolaw 말뭉치 · 법령은 저작권법 제7조에 따라 보호받지 않는 저작물
네이버 영화평 NSMC v1.0 학습 세트에서 무작위 4,570개(약 12만 자, 자주 쓰는 글자 500개로만 된 평) · CC0 1.0 · github.com/e9t/nsmc
Tiny Shakespeare 앞 159,902자 · 퍼블릭 도메인 · github.com/karpathy/char-rnn
성격: 공개 데이터. 한국어는 유니코드 NFD로 자모 분해, 출력은 NFC로 다시 붙임
사용자 입력: 붙여 넣은 글은 브라우저 안에서만 쓰고 저장·전송하지 않음

글의 앞 90% 학습 / 뒤 10% 검증. 검증 손실은 검증 글의 고정 창 48개, 바이그램(평활 α 0.05, 정한 값)도 같은 자리에서 잼
헌법: 작은 트랜스포머 1.583(시드 1~3: 1.546~1.583) / 바이그램 2.087 / 글자 빈도만 3.462 / 고르게 찍기 4.317
영화평: 2.060(2.022~2.060) / 2.281 / 3.532 / 4.710
셰익스피어: 2.155(2.086~2.155) / 2.492 / 3.296 / 4.111
음절 단위로 자른 경우(헌법, 1,000단계, 문맥 32, 묶음 12): 3.285 / 바이그램 3.157. 자모 단위를 고른 근거
미분 검사: 해석적 기울기 vs 중앙 차분 180개, 최대 상대오차 0.17%
단계당 시간: Node에서 약 170~210 ms. 헤드리스 Chromium 40초 학습 시 194단계, 검증 손실 2.115
학습·검증 분리함(글의 다른 부분)
2026-10-26 ai/로 옮긴 뒤 tests/language-model/train.mjs constitution 1000단계 시드 1 다시 실행: 검증 손실 1.583, 기록과 같음 · 기울기 검사 최대 상대오차 0.0017

Vaswani et al. (2017). Attention is all you need. NeurIPS.
Radford et al. (2019). Language models are unsupervised multitask learners. OpenAI.
Brown et al. (2020). Language models are few-shot learners. NeurIPS.
Ouyang et al. (2022). Training language models to follow instructions with human feedback. NeurIPS.
Loshchilov & Hutter (2019). Decoupled weight decay regularization. ICLR.
데이터: NSMC v1.0 · CC0 1.0 · https://github.com/e9t/nsmc
데이터: Tiny Shakespeare · 퍼블릭 도메인 · https://github.com/karpathy/char-rnn
데이터: KoNLPy 0.6.0 kolaw(대한민국헌법) · https://konlpy.org

6 tabs: Tokenization / Bigram Baseline / Training / Attention / Sampling / From Tiny to Large. The hands-on parts of the former stand-alone page (gpt.aileelab.com) are split across tabs; each tab has an overview, hands-on part, summary and references
Trained model: character-level (jamo) decoder-only transformer. Next-token prediction with cross-entropy loss
2 layers · 4 heads · embedding 48 · feed-forward 192 · context 64 tokens · pre-LN · GELU · 66,987 (constitution) to 70,479 (reviews) parameters
Trained from scratch · in the visitor's browser, Web Worker (CPU) · plain JavaScript, no libraries, hand-written backpropagation
AdamW · learning rate 0.01 → 0.0005 cosine · 30 warm-up steps · weight decay 0.1 · gradient clip 1 · batch 8 × 64 tokens · 1,000 steps · seed 1 (all set values)
Reference run 20261025T023903Z · Node 22.23 · Intel Xeon 2.1 GHz, 2 cores
No LLM. No external AI calls

Constitution of the Republic of Korea, 18,509 characters · KoNLPy 0.6.0 kolaw corpus · statutes are not protected works under Article 7 of the Korean Copyright Act
4,570 random reviews from the NSMC v1.0 training set (about 120,000 characters, reviews using only the 500 most common characters) · CC0 1.0 · github.com/e9t/nsmc
First 159,902 characters of Tiny Shakespeare · public domain · github.com/karpathy/char-rnn
Nature: public data. Korean is split into jamo with Unicode NFD and rejoined with NFC for output
User input: pasted text is used only in the browser and is not saved or sent

First 90% of the text for training, last 10% for validation. Validation loss uses 48 fixed windows; the bigram (smoothing α 0.05, set value) is scored on the same positions
Constitution: tiny transformer 1.583 (seeds 1–3: 1.546–1.583) / bigram 2.087 / letter frequency 3.462 / uniform 4.317
Movie reviews: 2.060 (2.022–2.060) / 2.281 / 3.532 / 4.710
Shakespeare: 2.155 (2.086–2.155) / 2.492 / 3.296 / 4.111
Cut by syllable instead (constitution, 1,000 steps, context 32, batch 12): 3.285 / bigram 3.157. This is why jamo was chosen
Gradient check: analytic vs central difference on 180 entries, max relative error 0.17%
Time per step: about 170–210 ms in Node. Headless Chromium, 40 s of training: 194 steps, validation loss 2.115
Training and validation are separate parts of the text
2026-10-26, after the move to ai/: re-ran tests/language-model/train.mjs constitution, 1,000 steps, seed 1: validation loss 1.583, same as recorded · gradient check worst relative error 0.0017

Designed for texts of 500+ characters, up to a few hundred token types, fixed 1,000 steps

Vaswani et al. (2017). Attention is all you need. NeurIPS.
Radford et al. (2019). Language models are unsupervised multitask learners. OpenAI.
Brown et al. (2020). Language models are few-shot learners. NeurIPS.
Ouyang et al. (2022). Training language models to follow instructions with human feedback. NeurIPS.
Loshchilov & Hutter (2019). Decoupled weight decay regularization. ICLR.
Data: NSMC v1.0 · CC0 1.0 · https://github.com/e9t/nsmc
Data: Tiny Shakespeare · public domain · https://github.com/karpathy/char-rnn
Data: KoNLPy 0.6.0 kolaw (Constitution of Korea) · https://konlpy.org