pytype

A static type analyzer for Python code

Home
Developer guide
Workflow
Development process
Python version upgrades
Supporting new features
Program analysis
Bytecode
Directives
Main loop
Stack frames
Typegraph
Data representation
Abstract values
Attributes
Overlays
Special builtins
Type annotations
Type stubs
Configuration
Tools

View the Project on GitHub google/pytype

Hosted on GitHub Pages — Theme by orderedlist

Overlays

Introduction

pytype sometimes needs extra type information or needs to perform complex operations that cannot be expressed in a pyi file. (For more details on motivation, see Special Builtins). In such cases, we write an overlay to directly manipulate abstract values.

Example

A common pattern in Python code is to check the runtime version using sys.version_info. In order to handle version checks properly, pytype needs to load the value of sys.version_info, not just its type. To achieve this, we write an overlay that maps version_info to a method that will directly build the version tuple:

class SysOverlay(overlay.Overlay):
  """A custom overlay for the 'sys' module."""

  def __init__(self, vm):
    member_map = {
        "version_info": build_version_info
    }
    [...]

And the build method has access to the VM’s python_version attribute:

def build_version_info(vm):
  [...]
  version = []
  # major, minor
  for i in vm.python_version:
    version.append(vm.convert.constant_to_var(i))
  [...]

Mechanics

When pytype goes to resolve an import, it will first check for an overlay and, if one is present, use it in preference to loading the pyi. Overlays inherit from the overlay.Overlay class, a subclass of abstract.Module that overrides member lookup so that when a member name is present in the overlay map, the representation of that member is constructed by calling the constructor specified in the map. The constructor can be any callable that accepts a VM instance and returns an abstract.BaseValue instance.

Adding an overlay

  1. If a file named overlays/{module}_overlay.py does not yet exist for the module in question, create one and add the following boilerplate (replace foo with the module name):

    from pytype import overlay
    
    class FooOverlay(overlay.Overlay):
    
      def __init__(self, vm):
        member_map = {}
        ast = vm.loader.import_name("foo")
        super().__init__(vm, "foo", member_map, ast)
    

    Then add the new overlay to overlay_dict, and create a new target for the file in overlays/CMakeLists.txt.

  2. In the {Module}Overlay initializer, add an entry to member_map for each new member. The key should be the member name and the value the constructor.
  3. Implement the new module members! The existing overlays contain plenty of examples of how to do this.