Overlays

• Overlays
  • Introduction
  • Example
  • Mechanics
  • Adding an overlay

Introduction

pytype sometimes needs extra type information or needs to perform complex operations that cannot be expressed in a pyi file. (For more details on motivation, see Special Builtins). In such cases, we write an overlay to directly manipulate abstract values.

Example

A common pattern in Python code is to check the runtime version using sys.version_info. In order to handle version checks properly, pytype needs to load the value of sys.version_info, not just its type. To achieve this, we write an overlay that maps version_info to a method that will directly build the version tuple:

class SysOverlay(overlay.Overlay):
  """A custom overlay for the 'sys' module."""

  def __init__(self, ctx):
    member_map = {
        "version_info": build_version_info
    }
    [...]

And the build method has access to the context’s python_version attribute:

def build_version_info(ctx, module):
  [...]
  version = []
  # major, minor
  for i in ctx.python_version:
    version.append(ctx.convert.constant_to_var(i))
  [...]

Mechanics

When pytype goes to resolve an import, it will first check for an overlay and, if one is present, use it in preference to loading the pyi. Overlays inherit from the overlay.Overlay class, a subclass of abstract.Module that overrides member lookup so that when a member name is present in the overlay map, the representation of that member is constructed by calling the constructor specified in the map. The constructor can be any callable that accepts a context instance and a module name and returns an abstract.BaseValue instance.

Adding an overlay

1. If a file named overlays/{module}_overlay.py does not yet exist for the module in question, create one and add the following boilerplate (replace foo with the module name):

   from pytype.overlays import overlay
   
   class FooOverlay(overlay.Overlay):
   
     def __init__(self, ctx):
       member_map = {}
       ast = ctx.loader.import_name("foo")
       super().__init__(ctx, "foo", member_map, ast)
   

   Then add the new overlay to overlay_dict, and create a new target for the file in overlays/CMakeLists.txt.

2. In the {Module}Overlay initializer, add an entry to member_map for each new member. The key should be the member name and the value the constructor.

3. Implement the new module members! The existing overlays contain plenty of examples of how to do this.