pytype

A static type analyzer for Python code

Home
Developer guide
Workflow
Development process
Python version upgrades
Supporting new features
Program analysis
Bytecode
Directives
Main loop
Stack frames
Typegraph
Data representation
Abstract values
Attributes
Overlays
Special builtins
Type annotations
Type stubs
TypeVars
Configuration
Style guide
Tools
Documentation debugging

View the Project on GitHub google/pytype

Hosted on GitHub Pages — Theme by orderedlist

Supporting new typing features

Introduction

The standard library typing module regularly gains new features that we then want to support in pytype. Support needs to be added in the loading, analysis, and output phases of pytype’s execution.

Loading

To allow the feature to be imported in source files and referenced in type stubs, we need to declare it in the stub for typing, located in pytype/stubs/builtins. This declaration should describe how the feature is typed.

For sufficiently simple features, adding this declaration is the only thing you need to do! For example, all pytype does for typing.runtime_checkable is pass the input through unchanged, so it can just be declared as an identity function.

Type stub parsing

Some features require a new node type in the AST representation of type stubs. For example, typing.Tuple required the addition of a pytd.TupleType node to distinguish between heterogeneous and homogeneous tuples. When you add a new node type, you also need to teach the stub parser to construct instances of it.

Conversion from a PyTD node to an abstract value

If you’ve defined a new PyTD node type or a new abstract class (see below) for your feature, you should specify how nodes are to be converted to abstract values.

Analysis

typing_overlay

Complicated features often need an entry in pytype/overlays/typing_overlay.py to control how the object behaves when interacted with.

Common techniques that you’ll see in typing_overlay:

Parameterized classes

Many of the parameterizable classes in typing_overlay.py have a corresponding parameterized class in abstract.py. For example, parameterizing typing_overlay.Tuple produces an abstract.TupleClass. If you need a parameterized object to have special behavior - e.g., instantiating a TupleClass will produce a heterogeneous abstract.Tuple, rather than a plain homogeneous Instance(tuple) - you will need to add a parameterized class, subclassing abstract.ParameterizedClass.

typing_extensions_overlay

If you implement a feature that is not available in all Python versions that pytype supports, it can be backported to earlier versions by adding it to the third-party typing_extensions module. If the name and behavior of the construct in typing_extensions are the same as in typing, you don’t have to do anything; the backport will be done automatically by our typing_extensions overlay. If you need to customize the behavior of the typing_extensions backport, add an entry to the typing_extensions member map here.

Matching

Arguably the most critical piece of supporting a new feature is defining its matching behavior, as a value being matched against an annotation and vice versa. For example, for typing.Tuple, we had to define rules for both this case:

def f(x: X): ...
f((a1, ..., a_n))  # what types X should this tuple match?

and this one:

def f(x: Tuple[A1, ..., A_n]): ...
f(x)  # what values x should match this tuple?

If your feature is a class, you should modify the matcher method _match_type_against_type; otherwise, you likely need to go one level up and change _match_value_against_type. Roughly speaking, these methods are structured as a series of isinstance checks on the value and the annotation. A good starting point is to figure out which isinstance check(s) the feature can satisfy, then determine whether the value and annotation types described by the check can match each other. If so, you’ll need to update the subst dictionary with substitutions for any type parameters involved in the match and return subst; else, return None.

Important: if you directly modify the substitution dictionary, make a copy of it first! The same dictionary may be the input to multiple matches using the same type parameters.

Output

Conversion from an abstract value to a PyTD node

If you’ve defined a new PyTD node type or a new abstract class for your feature, you’ll likely need to modify the PyTD converter to ensure that the right AST is produced for a module’s inferred types. Similar to the matcher, the converter operates via a series of isinstance checks on abstract classes, so you will need to add an isinstance check for your new class and/or modify the body of a check to return your new node.

PrintVisitor

Lastly, you may need to modify how pytd_visitors.PrintVisitor stringifies nodes of the new feature.

Partial support

It is often desirable to check in partial support for a complicated feature and finish it later. Some tips for doing this gracefully: