pytype/tools/ subdirectory consists of a number of independent tools which
build upon the idea of “pytype as a library”. They are included as part of
pytype both for convenience, and as examples of projects that depend on pytype.
NOTE: These tools typically have both a binary and one or more library modules; the latter can be considered public, and are safe for third party code to use as dependencies.
The main pytype binary,
pytype-single, runs pytype over a single file (and
expects all imports to have a corresponding
pyi file available).
analyze_project extracts a dependency graph from a directory of python code,
pytype-single over each of its files in the correct order, so that a
pyi file is generated for all a file’s imports before the file itself is
NOTE: Since this is what users expect by default, the
analyze_project binary is
pytype and is exposed as the main user-facing entry point, though from a
code perspective the “main” project is
merge_pyi takes a python source file and the corresponding
file (either generated by pytype or hand-written), and attempts to insert type
annotations from the
pyi file as inline type annotations in the python source.
merge_pyi is written as a lib2to3 fixer, and does not depend on
pytype, though running pytype first is a convenient way to generate the input
traces is a library of utilities for type-aware python source code analysis.
During pytype’s execution, we collect “opcode traces”, snapshots of the type
information pytype infers for each opcode’s arguments. The
provides tools for working with these opcode traces, as well as joining them to
the source text and the parsed syntax tree to perform various code analysis and
Some of the main functionality the library provides:
source.Code: A class that holds a set of raw traces and a source file, and has various options to query the joined data by line number ranges.
visitor.BaseVisitor: A base class for visitors on any tree that conforms to the python
traces.trace: A function that runs pytype over a source file, generates traces, and returns a
source.Codeobject with the traces and the source file.
traces.MatchAstVisitor: An AST visitor that matches opcode traces with AST nodes, using position and symbol information from both sets of data.
annotate_ast is a library that adds type information to the nodes of a python
AST. Internally, it uses
traces.MatchAstVisitor to find opcode traces for each
node, and copies the type information from the trace to
annotate_ast is a useful starting point to build type-aware linting and
refactoring tools; these tools typically work via AST analysis and
transformation, and can be made much more precise if the AST nodes include type
xref is an indexer and cross-reference generator for python projects. The
indexer is built around a
ScopedVisitor, an AST visitor that keeps track of
nested scopes, and an
Env, an analogue of a python symbol table. It replicates
python’s symbol lookup mechanism, and uses the
traces library to associate
type information with symbols.
The indexer matches opcode traces from pytype with every AST node. Joining in this data allows us to determine:
Note that some of this information is very hard to determine purely from a lexical analysis of the source code, without adding in type inference.
xref emits cross-reference data in kythe format; however,
the kythe graph generation is cleanly separated from the indexing code, and
other output formats should be easy to add if desired.
tools/ directory also contains some utilities that are generally useful
when writing pytype-based tools:
arg_parser.py: Argument parsing that forwards arguments to
config.py: Config file reader
environment.py: Set up and validate an environment to run pytype
runner.py: Running subprocesses (typically used to help run