The standard library
typing module regularly gains new features that we then
want to support in pytype. Support needs to be added in the loading, analysis,
and output phases of pytype’s execution.
To allow the feature to be imported in source files and referenced in type
stubs, we need to declare it in the stub for
typing, located in
pytype/stubs/builtins. This declaration should describe
how the feature is typed.
For sufficiently simple features, adding this declaration is the only thing you
need to do! For example, all pytype does for
typing.runtime_checkable is pass
the input through unchanged, so it can just be declared as an identity function.
Some features require a new node type in the AST representation of type
stubs. For example,
typing.Tuple required the addition of a
node to distinguish between heterogeneous and homogeneous tuples. When you add a
new node type, you also need to teach the stub parser to construct
instances of it.
Complicated features often need an entry in
to control how the object behaves when interacted with.
Common techniques that you’ll see in
call(). When a
PyTDFunctionis invoked, the arguments are passed to the
callmethod, which should return the result of the invocation.
typing_overlay.TypingContainerand overriding some of the behavior of
__getitem__; i.e., it controls what happens when the class is parameterized.
_get_value_infohelper method. This method is passed the parameter values and returns info on what the parameterized object should look like (see the docstring) that another helper,
_build_value, will use to construct the object.
_build_valuemethods that are directly called by
_build_innertakes the raw parameter variables and extracts the values, which are passed to
_build_valueconstructs the parameterized object.
getitem_slot. This method takes a slice variable containing the parameter variables and returns the parameterized object.
Many of the parameterizable classes in
typing_overlay.py have a
corresponding parameterized class in
abstract.py. For example,
typing_overlay.Tuple produces an
abstract.TupleClass. If you
need a parameterized object to have special behavior - e.g., instantiating a
TupleClass will produce a heterogeneous
abstract.Tuple, rather than a plain
Instance(tuple) - you will need to add a parameterized class,
If you implement a feature that is not available in all Python versions that
pytype supports, it can be backported to earlier versions by adding it to the
typing_extensions module. If the name and behavior of the
typing_extensions are the same as in
typing, you don’t have to
do anything; the backport will be done automatically by our
overlay. If you need to customize the behavior of the
backport, add an entry to the
typing_extensions member map
Arguably the most critical piece of supporting a new feature is defining its
matching behavior, as a value being matched against an annotation and
vice versa. For example, for
typing.Tuple, we had to define rules for both
def f(x: X): ... f((a1, ..., a_n)) # what types X should this tuple match?
and this one:
def f(x: Tuple[A1, ..., A_n]): ... f(x) # what values x should match this tuple?
If your feature is a class, you should modify the matcher method
_match_type_against_type; otherwise, you likely need to go one level up and
_match_value_against_type. Roughly speaking, these methods are
structured as a series of
isinstance checks on the value and the annotation. A
good starting point is to figure out which
isinstance check(s) the feature can
satisfy, then determine whether the value and annotation types described by the
check can match each other. If so, you’ll need to update the
with substitutions for any type parameters involved in the match and return
subst; else, return
Important: if you directly modify the substitution dictionary, make a copy of it first! The same dictionary may be the input to multiple matches using the same type parameters.
If you’ve defined a new PyTD node type or a new abstract class for your feature,
you’ll likely need to modify the PyTD converter to ensure that
the right AST is produced for a module’s inferred types. Similar to the matcher,
the converter operates via a series of
isinstance checks on abstract classes,
so you will need to add an
isinstance check for your new class and/or modify
the body of a check to return your new node.
Lastly, you may need to modify how pytd_visitors.PrintVisitor stringifies nodes of the new feature.
It is often desirable to check in partial support for a complicated feature and finish it later. Some tips for doing this gracefully:
overlay_utils.not_supported_yetto generate an error when it is imported in source files.
ctx.convert.unsolvableto treat them as
Anyin abstract analysis.
Anyin the inferred stub.