GPT-OSS Tokenizer Explorer
Enter a dialogue and see how the GPT-OSS tokenizer processes it. Use the format speaker: message
in the dialogue component.
Input Dialogue
Tokenization Results
Total Tokens: 0
Instructions:
- Enter dialogue: Use the dialogue component to enter conversations
- Speaker format: Type
system:
,user:
, orassistant:
followed by your message - Submit: Click 'Tokenize Dialogue' to process the conversation
- View results: See the tokenization details in the output area
Example:
system: You are a helpful assistant.
user: Hello! How are you today?
assistant: I'm doing well, thank you for asking!
What you'll see:
- Total tokens: Number of tokens in the conversation
- Tokenized output: How the tokenizer formats the conversation