Supporting new typing features

Supporting new typing features

Introduction

The standard library typing module regularly gains new features that we then want to support in pytype. Support needs to be added in the loading, analysis, and output phases of pytype’s execution.

Loading

To allow the feature to be imported in source files and referenced in type stubs, we need to declare it in the stub for typing, located in pytype/stubs/builtins. This declaration should describe how the feature is typed.

For sufficiently simple features, adding this declaration is the only thing you need to do! For example, all pytype does for typing.runtime_checkable is pass the input through unchanged, so it can just be declared as an identity function.

Type stub parsing

Some features require a new node type in the AST representation of type stubs. For example, typing.Tuple required the addition of a pytd.TupleType node to distinguish between heterogeneous and homogeneous tuples. When you add a new node type, you also need to teach the stub parser to construct instances of it.

Conversion from a PyTD node to an abstract value

If you’ve defined a new PyTD node type or a new abstract class (see below) for your feature, you should specify how nodes are to be converted to abstract values.

Analysis

typing_overlay

Complicated features often need an entry in pytype/overlays/typing_overlay.py to control how the object behaves when interacted with.

Common techniques that you’ll see in typing_overlay:

For methods: subclassing abstract.PyTDFunction and overriding call(). When a PyTDFunction is invoked, the arguments are passed to the call method, which should return the result of the invocation.
For parameterizable classes: subclassing typing_overlay.TypingContainer and overriding some of the behavior of getitem_slot. The getitem_slot method implements __getitem__; i.e., it controls what happens when the class is parameterized.
- The easiest approach is to override the _get_value_info helper method. This method is passed the parameter values and returns info on what the parameterized object should look like (see the docstring) that another helper, _build_value, will use to construct the object.
- If you need more control, you can override the _build_inner and _build_value methods that are directly called by _getitem_slot. _build_inner takes the raw parameter variables and extracts the values, which are passed to _build_value. _build_value constructs the parameterized object.
- For complete control, override getitem_slot. This method takes a slice variable containing the parameter variables and returns the parameterized object.

Parameterized classes

Many of the parameterizable classes in typing_overlay.py have a corresponding parameterized class in abstract.py. For example, parameterizing typing_overlay.Tuple produces an abstract.TupleClass. If you need a parameterized object to have special behavior - e.g., instantiating a TupleClass will produce a heterogeneous abstract.Tuple, rather than a plain homogeneous Instance(tuple) - you will need to add a parameterized class, subclassing abstract.ParameterizedClass.

typing_extensions_overlay

If you implement a feature that is not available in all Python versions that pytype supports, it can be backported to earlier versions by adding it to the third-party typing_extensions module. If the name and behavior of the construct in typing_extensions are the same as in typing, you don’t have to do anything; the backport will be done automatically by our typing_extensions overlay. If you need to customize the behavior of the typing_extensions backport, add an entry to the typing_extensions member map here.

Matching

Arguably the most critical piece of supporting a new feature is defining its matching behavior, as a value being matched against an annotation and vice versa. For example, for typing.Tuple, we had to define rules for both this case:

def f(x: X): ...
f((a1, ..., a_n))  # what types X should this tuple match?

and this one:

def f(x: Tuple[A1, ..., A_n]): ...
f(x)  # what values x should match this tuple?

If your feature is a class, you should modify the matcher method _match_type_against_type; otherwise, you likely need to go one level up and change _match_value_against_type. Roughly speaking, these methods are structured as a series of isinstance checks on the value and the annotation. A good starting point is to figure out which isinstance check(s) the feature can satisfy, then determine whether the value and annotation types described by the check can match each other. If so, you’ll need to update the subst dictionary with substitutions for any type parameters involved in the match and return subst; else, return None.

Important: if you directly modify the substitution dictionary, make a copy of it first! The same dictionary may be the input to multiple matches using the same type parameters.

Output

Conversion from an abstract value to a PyTD node

If you’ve defined a new PyTD node type or a new abstract class for your feature, you’ll likely need to modify the PyTD converter to ensure that the right AST is produced for a module’s inferred types. Similar to the matcher, the converter operates via a series of isinstance checks on abstract classes, so you will need to add an isinstance check for your new class and/or modify the body of a check to return your new node.

PrintVisitor

Lastly, you may need to modify how pytd_visitors.PrintVisitor stringifies nodes of the new feature.

Partial support

It is often desirable to check in partial support for a complicated feature and finish it later. Some tips for doing this gracefully:

To prevent premature use, map a feature to overlay_utils.not_supported_yet to generate an error when it is imported in source files.
If you’ve implemented only the PyTD logic for a feature, convert the pytd nodes to ctx.convert.unsolvable to treat them as Any in abstract analysis.
If you’ve implemented only the abstract logic for a feature, convert the abstract values to pytd.AnythingType to produce Any in the inferred stub.