tokenize splits Python source into tokens with type, text, and position. Unlike an AST, it preserves comments and formatting details useful to formatters and checkers.

Practical example

from io import BytesIO
from tokenize import NAME, generate_tokens

source = "total = price * quantity"
names = [token.string for token in generate_tokens(iter(source.splitlines(True)).__next__) if token.type == NAME]
print(names)

Tokens and trees serve different jobs

Use tokens when whitespace, comments, or spelling matter; use an AST when expressions and statements matter. Prefer tokenize.open() for files because it honors encoding declarations.

Transformations need tests

untokenize() helps rebuild source, but a naive edit may alter spacing or semantics. Test valid and invalid code, multiline strings, and different line endings.

Keep learning

Continue with Python AST: analyze code and literals safely and PEP 8 Python: Complete Style Guide and Best Practices. A official Python documentation, accessed July 22, 2026, documents the API and behavior across versions.