Python import 问题排查
今天在某个容器环境里执行 python manage.py shell
时遇到如下错误:
Traceback (most recent call last):
File "manage.py", line 22, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 383, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 377, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 226, in fetch_command
klass = load_command_class(app_name, subcommand)
File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 38, in load_command_class
module = import_module('%s.management.commands.%s' % (app_name, name))
File "/usr/local/lib/python3.6/importlib/__init__.py", line 128, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/usr/local/lib/python3.6/importlib/_bootstrap.py", line 987, in _gcd_import
return _find_and_load(name, _gcd_import)
File "/usr/local/lib/python3.6/importlib/_bootstrap.py", line 970, in _find_and_load
return _find_and_load_unlocked(name, import_)
File "/usr/local/lib/python3.6/importlib/_bootstrap.py", line 942, in _find_and_load_unlocked
_call_with_frames_removed(import_, parent)
File "/usr/local/lib/python3.6/importlib/_bootstrap.py", line 205, in _call_with_frames_removed
return f(*args, **kwds)
File "/usr/local/lib/python3.6/importlib/_bootstrap.py", line 987, in _gcd_import
return _find_and_load(name, _gcd_import)
File "/usr/local/lib/python3.6/importlib/_bootstrap.py", line 970, in _find_and_load
return _find_and_load_unlocked(name, import_)
File "/usr/local/lib/python3.6/importlib/_bootstrap.py", line 959, in _find_and_load_unlocked
module = _load_unlocked(spec)
File "/usr/local/lib/python3.6/importlib/_bootstrap.py", line 651, in _load_unlocked
module = module_from_spec(spec)
File "/usr/local/lib/python3.6/importlib/_bootstrap.py", line 569, in module_from_spec
_init_module_attrs(spec, module)
File "/usr/local/lib/python3.6/importlib/_bootstrap.py", line 512, in _init_module_attrs
raise NotImplementedError
NotImplementedError
开始以为是项目里某些地方弄乱了pythonpath
, 后来在新的目录里执行 django-admin startproject
还是一样的错误。
看了下 load_command_class
这个函数,里面调用了 python 的import_module
函数。
vi /usr/local/lib/python3.6/site-packages/django/core/management/__init__.py +38
def load_command_class(app_name, name):
"""
Given a command name and an application name, return the Command
class instance. Allow all errors raised by the import process
(ImportError, AttributeError) to propagate.
"""
module = import_module('%s.management.commands.%s' % (app_name, name))
return module.Command()
起个 python 终端,手工调用 import_module
,错误重现:
root@13a7ab176589:~# python3
Python 3.6.2 (default, Dec 7 2018, 02:38:42)
[GCC 4.8.4] on linux
>>> from importlib import import_module
>>> m = import_module('django.core.management.commands.startproject')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
...
...
File "/usr/local/lib/python3.6/importlib/_bootstrap.py", line 512, in _init_module_attrs
raise NotImplementedError
NotImplementedError
通过 print 调试后,发现 python 会到这个目录 /usr/local/lib/python3.6/site-packages/django/core/management/commands
搜索模块。
看了下这个目录,没有 __init__.py
文件。
root@13a7ab176589:~# ll /usr/local/lib/python3.6/site-packages/django/core/management/commands
total 212
drwxr-xr-x 3 root root 4096 Apr 21 11:19 ./
drwxr-xr-x 4 root root 4096 Apr 21 15:55 ../
drwxr-xr-x 2 root root 4096 Apr 21 11:43 __pycache__/
-rw-r--r-- 1 root root 2248 Apr 20 17:39 check.py
-rw-r--r-- 1 root root 5729 Apr 20 17:39 compilemessages.py
-rw-r--r-- 1 root root 4322 Apr 20 17:39 createcachetable.py
-rw-r--r-- 1 root root 1207 Apr 20 17:39 dbshell.py
-rw-r--r-- 1 root root 3373 Apr 20 17:39 diffsettings.py
-rw-r--r-- 1 root root 8479 Apr 20 17:39 dumpdata.py
-rw-r--r-- 1 root root 3557 Apr 20 17:39 flush.py
-rw-r--r-- 1 root root 13763 Apr 20 17:39 inspectdb.py
...
手工添加 __init__.py
后,django-admin startproject
可以正常工作。
那么问题来了,为什么会没有 __init__.py
,这个文件在 python 里用来标识一个目录为 package,从而可以被 import
。
翻了几页 Django 项目在 github 上的 commit 历史,发现在这个 commit 被移除了。通过 commit 信息,继而找到这个 ticket,里面主要记录了移除 python2 相关的条目,里面提到了 namespace package
。
Namespace Package
后来有在 SO 上找个这个帖子,从 python3.3 开始,引入了 Implicit Namespace Package,如果目录没有__init__.py
,那么这个目录会被当作 namespace package
被加载,与之对应的是 regular package
。
通过测试发现确实如此,
root@13a7ab176589:~# ls ba*
bar:
__init__.py __init__.pyc __pycache__ a.py b
baz:
__pycache__ shell.py
root@13a7ab176589:~# python3
Python 3.6.2 (default, Dec 7 2018, 02:38:42)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import bar
>>> import baz
>>>
然而,import_module
却无法工作,
root@13a7ab176589:~# python3
Python 3.6.2 (default, Dec 7 2018, 02:38:42)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from importlib import import_module
>>> m = import_module('bar')
>>> m = import_module('baz')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.6/importlib/__init__.py", line 128, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/usr/local/lib/python3.6/importlib/_bootstrap.py", line 987, in _gcd_import
return _find_and_load(name, _gcd_import)
File "/usr/local/lib/python3.6/importlib/_bootstrap.py", line 970, in _find_and_load
return _find_and_load_unlocked(name, import_)
File "/usr/local/lib/python3.6/importlib/_bootstrap.py", line 959, in _find_and_load_unlocked
module = _load_unlocked(spec)
File "/usr/local/lib/python3.6/importlib/_bootstrap.py", line 651, in _load_unlocked
module = module_from_spec(spec)
File "/usr/local/lib/python3.6/importlib/_bootstrap.py", line 569, in module_from_spec
_init_module_attrs(spec, module)
File "/usr/local/lib/python3.6/importlib/_bootstrap.py", line 512, in _init_module_attrs
raise NotImplementedError
NotImplementedError
初步怀疑是 python 3.6.2 的bug,于是找 3.6.8 测试,发现可以工作。
➜ Desktop python3
Python 3.6.8 (v3.6.8:3c6b436a57, Dec 24 2018, 02:04:31)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from importlib import import_module
>>> m = import_module('bar')
>>> m = import_module('baz')
>>>
到此问题解决~
经验教训
保持工具库的版本更新(至少补丁版本需要最新),不仅可以消除安全问题,还可以规避各种诡异的bug引起的时间投入。
思考:Implicit Namespace Package
Implicit Namespace Package 解决的问题背景在这篇PEP以及 Guido 的邮件里有介绍。
正如这篇帖子所讲,99%的情况下,我们需要的是 regular package,为了解决极少数的情况,引入了 Implicit Namespace Package 从而增加语言本身的复杂度,也许不是个明智的选择。毕竟 Zen of Python 第二条就写了,
Explicit is better than implicit.