The Import System

Welcome back to the Expert level!

To beginners, the import statement in Python seems like magic. You type import math, and suddenly a .py file from somewhere on your hard drive is evaluated and bound to the name math in your local scope.

But what if you want to import a module that doesn’t exist on disk? What if you want to import a module directly from a database, or a network request, or an encrypted ZIP file?

The Import Lifecycle

When you execute import foo, Python goes through a highly structured pipeline:

1. The Cache Check (sys.modules): Python first checks if foo is already in the sys.modules dictionary. If it is, the import system halts and returns the cached object instantly.
2. The Finders (sys.meta_path): If it’s a cache miss, Python iterates through the finders (classes) listed in sys.meta_path. Each finder is asked to locate the module by returning a ModuleSpec.
3. The Loaders: The ModuleSpec contains a Loader object. The loader is responsible for taking the spec and actually executing the code into a new module namespace.

Hijacking sys.meta_path

By writing a custom MetaPathFinder and Loader, you can completely hijack this process.

Let’s look at a concrete example that imports a module directly from a Python dictionary string in memory, completely bypassing the filesystem!

import sys
import importlib.abc
import importlib.machinery
import types

# 1. The Loader executes the code string
class StringLoader(importlib.abc.Loader):
    def __init__(self, code_str):
        self.code_str = code_str

    def exec_module(self, module):
        code = compile(self.code_str, '<string>', 'exec')
        types.FunctionType(code, module.__dict__)()

# 2. The Finder checks if the requested module is in our dictionary
class StringFinder(importlib.abc.MetaPathFinder):
    def __init__(self, modules):
        self.modules = modules

    def find_spec(self, fullname, path, target=None):
        if fullname in self.modules:
            return importlib.machinery.ModuleSpec(fullname, StringLoader(self.modules[fullname]))
        return None

# 3. Register our hook at the front of the meta path!
my_modules = {
    'fake_module': 'def hello(): return "Hello from memory!"'
}
sys.meta_path.insert(0, StringFinder(my_modules))

# 4. Trigger the import machinery!
import fake_module
print(fake_module.hello())

Notice that fake_module.py never existed on disk!

Mastering this API allows you to build highly sophisticated plug-in architectures for your applications.