pytype

A static type analyzer for Python code

Home
Developer guide
Workflow
Development process
Python version upgrades
Supporting new features
Program analysis
Bytecode
Directives
Main loop
Stack frames
Typegraph
Data representation
Abstract values
Attributes
Overlays
Special builtins
Type annotations
Type stubs
Configuration
Tools

View the Project on GitHub google/pytype

Hosted on GitHub Pages — Theme by orderedlist

Pytype Tools

Introduction

The pytype/tools/ subdirectory consists of a number of independent tools which build upon the idea of “pytype as a library”. They are included as part of pytype both for convenience, and as examples of projects that depend on pytype.

NOTE: These tools typically have both a binary and one or more library modules; the latter can be considered public, and are safe for third party code to use as dependencies.

analyze_project

The main pytype binary, pytype-single, runs pytype over a single file (and expects all imports to have a corresponding pyi file available). analyze_project extracts a dependency graph from a directory of python code, and runs pytype-single over each of its files in the correct order, so that a pyi file is generated for all a file’s imports before the file itself is processed.

NOTE: Since this is what users expect by default, the analyze_project binary is named pytype and is exposed as the main user-facing entry point, though from a code perspective the “main” project is pytype-single

merge_pyi

merge_pyi takes a python source file and the corresponding pyi signature file (either generated by pytype or hand-written), and attempts to insert type annotations from the pyi file as inline type annotations in the python source.

merge_pyi is written as a lib2to3 fixer, and does not depend on pytype, though running pytype first is a convenient way to generate the input pyi file.

traces

traces is a library of utilities for type-aware python source code analysis. During pytype’s execution, we collect “opcode traces”, snapshots of the type information pytype infers for each opcode’s arguments. The traces library provides tools for working with these opcode traces, as well as joining them to the source text and the parsed syntax tree to perform various code analysis and transformation tasks.

Some of the main functionality the library provides:

annotate_ast

annotate_ast is a library that adds type information to the nodes of a python AST. Internally, it uses traces.MatchAstVisitor to find opcode traces for each node, and copies the type information from the trace to node.resolved_type and node.resolved_annotation.

annotate_ast is a useful starting point to build type-aware linting and refactoring tools; these tools typically work via AST analysis and transformation, and can be made much more precise if the AST nodes include type information.

xref

xref is an indexer and cross-reference generator for python projects. The indexer is built around a ScopedVisitor, an AST visitor that keeps track of nested scopes, and an Env, an analogue of a python symbol table. It replicates python’s symbol lookup mechanism, and uses the traces library to associate type information with symbols.

The indexer matches opcode traces from pytype with every AST node. Joining in this data allows us to determine:

Note that some of this information is very hard to determine purely from a lexical analysis of the source code, without adding in type inference.

xref emits cross-reference data in kythe format; however, the kythe graph generation is cleanly separated from the indexing code, and other output formats should be easy to add if desired.

Utilities

The tools/ directory also contains some utilities that are generally useful when writing pytype-based tools: