GPT-OSS Tokenizer Explorer

Enter a dialogue and see how the GPT-OSS tokenizer processes it. Use the format speaker: message in the dialogue component.

Input Dialogue

Tokenization Results

Total Tokens: 0

Instructions:

  1. Enter dialogue: Use the dialogue component to enter conversations
  2. Speaker format: Type system:, user:, or assistant: followed by your message
  3. Submit: Click 'Tokenize Dialogue' to process the conversation
  4. View results: See the tokenization details in the output area

Example:

system: You are a helpful assistant.
user: Hello! How are you today?
assistant: I'm doing well, thank you for asking!

What you'll see:

  • Total tokens: Number of tokens in the conversation
  • Tokenized output: How the tokenizer formats the conversation