`PYTHON_LAZY_IMPORTS=none` ignored for `__lazy_modules__`

[hugovk]

Bug report

Bug description:

Keyword

Given lazy keyword imports:

lazy import asyncio
lazy import doctest
lazy import imaplib
lazy import json
lazy import smtplib
lazy import unittest
lazy import urllib.request
lazy from argparse import ArgumentParser
lazy from collections import OrderedDict
lazy from pathlib import Path

Then running with -X lazy_imports=none and PYTHON_LAZY_IMPORTS=none will disable lazy imports, as expected:

❯ hyperfine --warmup 10 --runs 20 \
   "./python.exe lazy-eager.py" \
   "PYTHON_LAZY_IMPORTS=normal ./python.exe lazy-eager.py" \
   "PYTHON_LAZY_IMPORTS=none   ./python.exe lazy-eager.py" \
   "PYTHON_LAZY_IMPORTS=all    ./python.exe lazy-eager.py" \
   "./python.exe -X lazy_imports=normal lazy-eager.py" \
   "./python.exe -X lazy_imports=none   lazy-eager.py" \
   "./python.exe -X lazy_imports=all    lazy-eager.py"
Benchmark 1: ./python.exe lazy-eager.py
  Time (mean ± σ):      17.1 ms ±   0.5 ms    [User: 13.2 ms, System: 3.0 ms]
  Range (min … max):    16.2 ms …  17.9 ms    20 runs

Benchmark 2: PYTHON_LAZY_IMPORTS=normal ./python.exe lazy-eager.py
  Time (mean ± σ):      17.2 ms ±   0.4 ms    [User: 13.2 ms, System: 3.1 ms]
  Range (min … max):    16.5 ms …  17.9 ms    20 runs

Benchmark 3: PYTHON_LAZY_IMPORTS=none   ./python.exe lazy-eager.py
  Time (mean ± σ):     141.5 ms ±  11.5 ms    [User: 122.0 ms, System: 16.2 ms]
  Range (min … max):   134.8 ms … 184.1 ms    20 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 4: PYTHON_LAZY_IMPORTS=all    ./python.exe lazy-eager.py
  Time (mean ± σ):      17.1 ms ±   0.6 ms    [User: 13.2 ms, System: 3.0 ms]
  Range (min … max):    16.1 ms …  18.3 ms    20 runs

Benchmark 5: ./python.exe -X lazy_imports=normal lazy-eager.py
  Time (mean ± σ):      17.2 ms ±   0.5 ms    [User: 13.2 ms, System: 3.1 ms]
  Range (min … max):    16.6 ms …  18.2 ms    20 runs

Benchmark 6: ./python.exe -X lazy_imports=none   lazy-eager.py
  Time (mean ± σ):     136.4 ms ±   1.5 ms    [User: 119.3 ms, System: 15.5 ms]
  Range (min … max):   133.4 ms … 138.9 ms    20 runs

Benchmark 7: ./python.exe -X lazy_imports=all    lazy-eager.py
  Time (mean ± σ):      17.3 ms ±   0.5 ms    [User: 13.3 ms, System: 3.1 ms]
  Range (min … max):    16.5 ms …  18.2 ms    20 runs

Summary
  PYTHON_LAZY_IMPORTS=all    ./python.exe lazy-eager.py ran
    1.00 ± 0.05 times faster than ./python.exe lazy-eager.py
    1.01 ± 0.04 times faster than PYTHON_LAZY_IMPORTS=normal ./python.exe lazy-eager.py
    1.01 ± 0.05 times faster than ./python.exe -X lazy_imports=normal lazy-eager.py
    1.01 ± 0.05 times faster than ./python.exe -X lazy_imports=all    lazy-eager.py
    7.99 ± 0.30 times faster than ./python.exe -X lazy_imports=none   lazy-eager.py
    8.29 ± 0.74 times faster than PYTHON_LAZY_IMPORTS=none   ./python.exe lazy-eager.py

Dunder

But when using __lazy_modules__ instead:

__lazy_modules__ = [
    "argparse",
    "asyncio",
    "collections",
    "doctest",
    "imaplib",
    "json",
    "pathlib",
    "smtplib",
    "unittest",
    "urllib.request",
]
import asyncio
import doctest
import imaplib
import json
import smtplib
import unittest
import urllib.request
from argparse import ArgumentParser
from collections import OrderedDict
from pathlib import Path

-X lazy_imports=none and PYTHON_LAZY_IMPORTS=none are ignored and they're lazily imported:

❯ hyperfine --warmup 10 --runs 20 \
   "./python.exe lazy-dunder.py" \
   "PYTHON_LAZY_IMPORTS=normal ./python.exe lazy-dunder.py" \
   "PYTHON_LAZY_IMPORTS=none   ./python.exe lazy-dunder.py" \
   "PYTHON_LAZY_IMPORTS=all    ./python.exe lazy-dunder.py" \
   "./python.exe -X lazy_imports=normal lazy-dunder.py" \
   "./python.exe -X lazy_imports=none   lazy-dunder.py" \
   "./python.exe -X lazy_imports=all    lazy-dunder.py"
Benchmark 1: ./python.exe lazy-dunder.py
  Time (mean ± σ):      17.0 ms ±   0.5 ms    [User: 13.1 ms, System: 3.1 ms]
  Range (min … max):    16.3 ms …  18.4 ms    20 runs

Benchmark 2: PYTHON_LAZY_IMPORTS=normal ./python.exe lazy-dunder.py
  Time (mean ± σ):      16.6 ms ±   0.5 ms    [User: 13.1 ms, System: 2.8 ms]
  Range (min … max):    15.9 ms …  17.6 ms    20 runs

Benchmark 3: PYTHON_LAZY_IMPORTS=none   ./python.exe lazy-dunder.py
  Time (mean ± σ):      16.9 ms ±   0.5 ms    [User: 13.1 ms, System: 3.0 ms]
  Range (min … max):    15.7 ms …  17.6 ms    20 runs

Benchmark 4: PYTHON_LAZY_IMPORTS=all    ./python.exe lazy-dunder.py
  Time (mean ± σ):      16.9 ms ±   0.4 ms    [User: 13.1 ms, System: 3.0 ms]
  Range (min … max):    16.3 ms …  17.5 ms    20 runs

Benchmark 5: ./python.exe -X lazy_imports=normal lazy-dunder.py
  Time (mean ± σ):      17.0 ms ±   0.4 ms    [User: 13.1 ms, System: 3.0 ms]
  Range (min … max):    15.9 ms …  17.7 ms    20 runs

Benchmark 6: ./python.exe -X lazy_imports=none   lazy-dunder.py
  Time (mean ± σ):      16.8 ms ±   0.5 ms    [User: 13.1 ms, System: 2.9 ms]
  Range (min … max):    16.1 ms …  17.8 ms    20 runs

Benchmark 7: ./python.exe -X lazy_imports=all    lazy-dunder.py
  Time (mean ± σ):      16.9 ms ±   0.4 ms    [User: 13.1 ms, System: 3.0 ms]
  Range (min … max):    16.5 ms …  17.9 ms    20 runs

Summary
  PYTHON_LAZY_IMPORTS=normal ./python.exe lazy-dunder.py ran
    1.01 ± 0.04 times faster than ./python.exe -X lazy_imports=none   lazy-dunder.py
    1.01 ± 0.04 times faster than PYTHON_LAZY_IMPORTS=none   ./python.exe lazy-dunder.py
    1.02 ± 0.04 times faster than PYTHON_LAZY_IMPORTS=all    ./python.exe lazy-dunder.py
    1.02 ± 0.04 times faster than ./python.exe -X lazy_imports=all    lazy-dunder.py
    1.02 ± 0.04 times faster than ./python.exe -X lazy_imports=normal lazy-dunder.py
    1.02 ± 0.04 times faster than ./python.exe lazy-dunder.py

Thanks

To @henryiii for discovering this.

CPython versions tested on:

CPython main branch, 3.15

Operating systems tested on:

macOS

Linked PRs

• gh-146369: Ensure PYTHON_LAZY_IMPORTS=none overrides __lazy_modules__ #146371