2/14/2026

text2token: From Plain Text to Byte Tokens

Why I built a tiny text-to-token converter and what it teaches about how text is represented internally.

I built text2token as a small browser-based utility to convert plain text into byte-level tokens and decode it back.

Repository: https://github.com/wayiam/text2token

Why I Built It

When working around AI and language tooling, token representation comes up often.
This project was a simple way to make that concept visible and interactive.

What It Does

• Takes user text input
• Encodes it into byte-level token values
• Decodes token values back to original text

The app is intentionally minimal so the transformation is easy to understand.

Tech and Approach

This project is a lightweight frontend app with no backend dependency.

• HTML for structure
• CSS for UI
• JavaScript for encode/decode logic

const encoded = Array.from(new TextEncoder().encode(inputText));
const decoded = new TextDecoder().decode(new Uint8Array(encoded));

Key Takeaway

Not every useful project has to be large.
Sometimes a focused micro-tool is enough to explain an important concept clearly.