Develop Reference

Python moving modules into sub directory (without breaking existing import structure)

Supposing I am writing library code with a directory structure as follows:
- mylibrary/
|
|-----foo.py
|-----bar.py
|-----baz.py
|-----__init__.py
And to better organise I create a sub directory:
- mylibrary/
|
|-----foobar/
| |-----foo.py
| |-----bar.py
|-----baz.py
|-----__init__.py
I want all client code to keep working without updates so I want to update init.py so that imports don't break.
I've tried adding this to init.py:
from foobar import foo
Now if I open a shell I can do:
from mylibrary import foo
print(foo.Foo)
However if I do this:
from mylibrary.foo import Foo
I get No module named mylibrary.foo error.
Here is the traceback from my actual example:
Type "help", "copyright", "credits" or "license" for more information.
>>> from global_toolkit import section
>>> section.Section
<class 'global_toolkit.serialization.section.Section'>
>>> from global_toolkit.section import Section
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'global_toolkit.section'
>>>
Can anyone explain this behaviour?

Add this in your __init__.py :
from .foobar import foo, bar
import sys
for i in ['foo','bar']:
sys.modules['mylib.'+i] = sys.modules['mylib.foobar.'+i]
Now, from mylib.foo import Foo should work.

Python Pickle failing on Windows but works on Mac [duplicate]

I have a class MyClass defined in my_module. MyClass has a method pickle_myself which pickles the instance of the class in question:
def pickle_myself(self, pkl_file_path):
with open(pkl_file_path, 'w+') as f:
pkl.dump(self, f, protocol=2)
I have made sure that my_module is in PYTHONPATH. In the interpreter, executing __import__('my_module') works fine:
>>> __import__('my_module')
<module 'my_module' from 'A:\my_stuff\my_module.pyc'>
However, when eventually loading the file, I get:
File "A:\Anaconda\lib\pickle.py", line 1128, in find_class
__import__(module)
ImportError: No module named my_module
Some things I have made sure of:
I have not changed the location of my_module.py (Python pickling after changing a module's directory)
I have tried to use dill instead, but still get the same error (More on python ImportError No module named)
EDIT -- A toy example that reproduces the error:
The example itself is spread over a bunch of files.
First, we have the module ball (stored in a file called ball.py):
class Ball():
def __init__(self, ball_radius):
self.ball_radius = ball_radius
def say_hello(self):
print "Hi, I'm a ball with radius {}!".format(self.ball_radius)
Then, we have the module test_environment:
import os
import ball
#import dill as pkl
import pickle as pkl
class Environment():
def __init__(self, store_dir, num_balls, default_ball_radius):
self.store_dir = store_dir
self.balls_in_environment = [ball.Ball(default_ball_radius) for x in range(num_balls)]
def persist(self):
pkl_file_path = os.path.join(self.store_dir, "test_stored_env.pkl")
with open(pkl_file_path, 'w+') as f:
pkl.dump(self, f, protocol=2)
Then, we have a module that has functions to make environments, persist them, and load them, called make_persist_load:
import os
import test_environment
#import pickle as pkl
import dill as pkl
def make_env_and_persist():
cwd = os.getcwd()
my_env = test_environment.Environment(cwd, 5, 5)
my_env.persist()
def load_env(store_path):
stored_env = None
with open(store_path, 'rb') as pkl_f:
stored_env = pkl.load(pkl_f)
return stored_env
Then we have a script to put it all together, in test_serialization.py:
import os
import make_persist_load
MAKE_AND_PERSIST = True
LOAD = (not MAKE_AND_PERSIST)
cwd = os.getcwd()
store_path = os.path.join(cwd, "test_stored_env.pkl")
if MAKE_AND_PERSIST == True:
make_persist_load.make_env_and_persist()
if LOAD == True:
loaded_env = make_persist_load.load_env(store_path)
In order to make it easy to use this toy example, I have put it all up on in a Github repository that simply needs to be cloned into your directory of choice.. Please see the README containing instructions, which I also reproduce here:
Instructions:
1) Clone repository into a directory.
2) Add repository directory to PYTHONPATH.
3) Open up test_serialization.py, and set the variable MAKE_AND_PERSIST to True. Run the script in an interpreter.
4) Close the previous interpreter instance, and start up a new one. In test_serialization.py, change MAKE_AND_PERSIST to False, and this will programmatically set LOAD to True. Run the script in an interpreter, causing ImportError: No module named test_environment.
5) By default, the test is set to use dill, instead of pickle. In order to change this, go into test_environment.py and make_persist_load.py, to change imports as required.
EDIT: after switching to dill '0.2.5.dev0', dill.detect.trace(True) output
C2: test_environment.Environment
# C2
D2: <dict object at 0x000000000A9BDAE8>
C2: ball.Ball
# C2
D2: <dict object at 0x000000000AA25048>
# D2
D2: <dict object at 0x000000000AA25268>
# D2
D2: <dict object at 0x000000000A9BD598>
# D2
D2: <dict object at 0x000000000A9BD9D8>
# D2
D2: <dict object at 0x000000000A9B0BF8>
# D2
# D2
EDIT: the toy example works perfectly well when run on Mac/Ubuntu (i.e. Unix-like systems?). It only fails on Windows.

I can tell from your question that you are probably doing something like this, with a class method that is attempting to pickle the instance of the class. It's ill-advised to do that, if you are doing that… it's much more sane to use pkl.dump external to the class instead (where pkl is pickle or dill etc). However, it can still work with this design, see below:
>>> class Thing(object):
... def pickle_myself(self, pkl_file_path):
... with open(pkl_file_path, 'w+') as f:
... pkl.dump(self, f, protocol=2)
...
>>> import dill as pkl
>>>
>>> t = Thing()
>>> t.pickle_myself('foo.pkl')
Then restarting...
Python 2.7.10 (default, Sep 2 2015, 17:36:25)
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> f = open('foo.pkl', 'r')
>>> t = dill.load(f)
>>> t
<__main__.Thing object at 0x1060ff410>
If you have a much more complicated class, which I'm sure you do, then you are likely to run into trouble, especially if that class uses another file that is sitting in the same directory.
>>> import dill
>>> from bar import Zap
>>> print dill.source.getsource(Zap)
class Zap(object):
x = 1
def __init__(self, y):
self.y = y
>>>
>>> class Thing2(Zap):
... def pickle_myself(self, pkl_file_path):
... with open(pkl_file_path, 'w+') as f:
... dill.dump(self, f, protocol=2)
...
>>> t = Thing2(2)
>>> t.pickle_myself('foo2.pkl')
Then restarting…
Python 2.7.10 (default, Sep 2 2015, 17:36:25)
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> f = open('foo2.pkl', 'r')
>>> t = dill.load(f)
>>> t
<__main__.Thing2 object at 0x10eca8090>
>>> t.y
2
>>>
Well… shoot, that works too. You'll have to post your code, so we can see what pattern you are using that dill (and pickle) fails for. I know having one module import another that is not "installed" (i.e. in some local directory) and expecting the serialization to "just work" doesn't for all cases.
See dill issues:
https://github.com/uqfoundation/dill/issues/128
https://github.com/uqfoundation/dill/issues/129
and this SO question:
Why dill dumps external classes by reference, no matter what?
for some examples of failure and potential workarounds.
EDIT with regard to updated question:
I don't see your issue. Running from the command line, importing from the interpreter (import test_serialization), and running the script in the interpreter (as below, and indicated in your steps 3-5) all work. That leads me to think you might be using an older version of dill?
>>> import os
>>> import make_persist_load
>>>
>>> MAKE_AND_PERSIST = False #True
>>> LOAD = (not MAKE_AND_PERSIST)
>>>
>>> cwd = os.getcwd()
>>> store_path = os.path.join(cwd, "test_stored_env.pkl")
>>>
>>> if MAKE_AND_PERSIST == True:
... make_persist_load.make_env_and_persist()
...
>>> if LOAD == True:
... loaded_env = make_persist_load.load_env(store_path)
...
>>>
EDIT based on discussion in comments:
Looks like it's probably an issue with Windows, as that seems to be the only OS the error appears.
EDIT after some work (see: https://github.com/uqfoundation/dill/issues/140):
Using this minimal example, I can reproduce the same error on Windows, while on MacOSX it still works…
# test.py
class Environment():
def __init__(self):
pass
and
# doit.py
import test
import dill
env = test.Environment()
path = "test.pkl"
with open(path, 'w+') as f:
dill.dump(env, f)
with open(path, 'rb') as _f:
_env = dill.load(_f)
print _env
However, if you use open(path, 'r') as _f, it works on both Windows and MacOSX. So it looks like the __import__ on Windows is more sensitive to file type than on non-Windows systems. Still, throwing an ImportError is weird… but this one small change should make it work.

In case someone is having same problem, I had the same problem running Python 2.7 and the problem was the pickle file created on windows while I am running Linux, what I had to do is running dos2unix which has to be downloaded first using
sudo yum install dos2unix
And then you need to convert the pickle file example
dos2unix data.p

Python 3.5 dill pickling/unpickling on different servers: "KeyError: 'ClassType'"

See updates at the bottom
--
A similar question was asked here, but never resolved:
pickling and unpickling user-defined class
I'm working on a project which necessitates pickling user defined classes, and sending them to a remote server where they are unpickled and called. We use the Dill library to accomplish this, and have had a lot of success.
Unfortunately, I've run into an issue I'm having a hard time debugging. I create and pickle a class as follows:
import dill, base64
import time, random
class periodicSource(object):
def __call__(self):
while True:
time.sleep(0.1)
yield random.uniform(20,100)
periodic_src = periodicSource()
a = base64.b64encode(dill.dumps(periodic_src)).decode("ascii")
print(a)
It creates an ascii representation of the dilled class.
gANjZGlsbC5kaWxsCl9jcmVhdGVfdHlwZQpxAChjZGlsbC5kaWxsCl9sb2FkX3R5cGUKcQFYCQAAAENsYXNzVHlwZXEChXEDUnEEWA4AAABwZXJpb2RpY1NvdXJjZXEFaAFYBgAAAG9iamVjdHEGhXEHUnEIhXEJfXEKKFgIAAAAX19jYWxsX19xC2NkaWxsLmRpbGwKX2NyZWF0ZV9mdW5jdGlvbgpxDChoAVgIAAAAQ29kZVR5cGVxDYVxDlJxDyhLAUsASwFLA0tjQyl4IgB0AABqAQBkAQCDAQABdAIAagMAZAIAZAMAgwIAVgFxAwBXZAAAU3EQKE5HP7mZmZmZmZpLFEtkdHERKFgEAAAAdGltZXESWAUAAABzbGVlcHETWAYAAAByYW5kb21xFFgHAAAAdW5pZm9ybXEVdHEWWAQAAABzZWxmcReFcRhYHwAAADxpcHl0aG9uLWlucHV0LTIwLTdhNGU5MDIwYWM2Yz5xGWgLSwdDBgABAwENAXEaKSl0cRtScRx9cR0oWAYAAAByYW5kb21xHmNkaWxsLmRpbGwKX2ltcG9ydF9tb2R1bGUKcR9oFIVxIFJxIVgEAAAAdGltZXEiaB9YBAAAAHRpbWVxI4VxJFJxJXVoC05OfXEmdHEnUnEoWAoAAABfX21vZHVsZV9fcSlYCAAAAF9fbWFpbl9fcSpYBwAAAF9fZG9jX19xK05YDQAAAF9fc2xvdG5hbWVzX19xLF1xLXV0cS5ScS8pgXEwLg==
When I go to deserialize it on the other server:
a = 'gANjZGlsbC5kaWxsCl9jcmVhdGVfdHlwZQpxAChjZGlsbC5kaWxsCl9sb2FkX3R5cGUKcQFYCQAAAENsYXNzVHlwZXEChXEDUnEEWA4AAABwZXJpb2RpY1NvdXJjZXEFaAFYBgAAAG9iamVjdHEGhXEHUnEIhXEJfXEKKFgIAAAAX19jYWxsX19xC2NkaWxsLmRpbGwKX2NyZWF0ZV9mdW5jdGlvbgpxDChoAVgIAAAAQ29kZVR5cGVxDYVxDlJxDyhLAUsASwFLA0tjQyl4IgB0AABqAQBkAQCDAQABdAIAagMAZAIAZAMAgwIAVgFxAwBXZAAAU3EQKE5HP7mZmZmZmZpLFEtkdHERKFgEAAAAdGltZXESWAUAAABzbGVlcHETWAYAAAByYW5kb21xFFgHAAAAdW5pZm9ybXEVdHEWWAQAAABzZWxmcReFcRhYHwAAADxpcHl0aG9uLWlucHV0LTIwLTdhNGU5MDIwYWM2Yz5xGWgLSwdDBgABAwENAXEaKSl0cRtScRx9cR0oWAYAAAByYW5kb21xHmNkaWxsLmRpbGwKX2ltcG9ydF9tb2R1bGUKcR9oFIVxIFJxIVgEAAAAdGltZXEiaB9YBAAAAHRpbWVxI4VxJFJxJXVoC05OfXEmdHEnUnEoWAoAAABfX21vZHVsZV9fcSlYCAAAAF9fbWFpbl9fcSpYBwAAAF9fZG9jX19xK05YDQAAAF9fc2xvdG5hbWVzX19xLF1xLXV0cS5ScS8pgXEwLg=='
a = dill.loads(base64.b64decode(a.encode()))
print(a)
I get the following error:
/home/streamsadmin/anaconda3/bin/python /home/streamsadmin/git/streamsx.topology/test/python/topology/deleteme2.py
Traceback (most recent call last):
File "/home/streamsadmin/git/streamsx.topology/test/python/topology/deleteme2.py", line 40, in <module>
a = dill.loads(base64.b64decode(a.encode()))
File "/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py", line 277, in loads
return load(file)
File "/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py", line 266, in load
obj = pik.load()
File "/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py", line 524, in _load_type
return _reverse_typemap[name]
KeyError: 'ClassType'
I would expect this if I were using different version of Python on the remote system, but they're the same:
Server 1:
>>> import sys
>>> sys.version
'3.5.2 |Anaconda 4.1.1 (64-bit)| (default, Jul 2 2016, 17:53:06) \n[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]'
Server 2:
>>> import sys
>>> sys.version
'3.5.2 |Anaconda 4.2.0 (64-bit)| (default, Jul 2 2016, 17:53:06) \n[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]'
Additionally, both versions of Dill are 0.2.6. Any ideas how I could debug this?
EDIT: I've think it might be something with my environment. I'm using Python 3.5, but list the builtin types:
>>> import types
>>> dir(types)
['BuiltinFunctionType',
'BuiltinMethodType',
'ClassType',
'CodeType',
...
]
It seems ClassType is in the output which should NOT be the case since ClassType was removed in Python 3.5. This is exceedingly strange.
I'm running on a system that has both Python 2.7 and Python 3.5 installed. Could the 2.7 installation somehow be polluting the 3.5 installation?

The culprit is cloudpickle. By default in Python 3.5, types.ClassType is left unset.
>>> import types
>>> dir(types)
['BuiltinFunctionType', 'BuiltinMethodType', 'CodeType', ...]
When cloudpickle is imported, suddenly, types.ClassType becomes defined.
>>> import cloudpickle
>>> dir(types)
['BuiltinFunctionType', 'BuiltinMethodType', 'ClassType', 'CodeType', ...]
Server A uses dill to serialize objects, and also imports cloudpickle. Therefore it includes a reference to ClassType during serialization.
Server B does NOT import cloudpickle, and then tries to find a reference to ClassType during deserialization and fails. Raising the error:
Traceback (most recent call last):
File "/home/streamsadmin/git/streamsx.topology/test/python/topology/deleteme2.py", line 40, in <module>
a = dill.loads(base64.b64decode(a.encode()))
File "/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py", line 277, in loads
return load(file)
File "/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py", line 266, in load
obj = pik.load()
File "/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py", line 524, in _load_type
return _reverse_typemap[name]
KeyError: 'ClassType'
On our system, we can't remove cloudpickle from our environment, so we had to do the following workaround.
On server B, right after we import dill and sometime before the first call to dill.loads, we invoke the following line of code:
dill._dill._reverse_typemap['ClassType'] = type
This defines ClassType appropriately. And causes dill deserialization to work as expected.

I am sure cloudpickle is causing the problem. You can debug it step by step.
First Check if classType Exists in your builtin types
import types
dir(types)
if it exist than it should have worked for you, if not than move to next steps.
import cloudpickle and now check again. You will have classType in buildin types
excute below code
dill.dill._reverse_typemap['ClassType'] = type
it should work for you :)
But if you are still getting error AttributeError: module 'dill' has no attribute 'dill'
than use this one dill._dill._reverse_typemap['ClassType'] = type because dill.dill is moved to dill._dill

How to check if a module/library/package is part of the python standard library?

I have installed sooo many libraries/modules/packages with pip and now I cannot differentiate which is native to the python standard library and which is not. This causes problem when my code works on my machine but it doesn't work anywhere else.
How can I check if a module/library/package that I import in my code is from the python stdlib?
Assume that the checking is done on the machine with all the external libraries/modules/packages, otherwise I could simply do a try-except import on the other machine that doesn't have them.
For example, I am sure these imports work on my machine, but when it's on a machine with only a plain Python install, it breaks:
from bs4 import BeautifulSoup
import nltk
import PIL
import gensim

You'd have to check all modules that have been imported to see if any of these are located outside of the standard library.
The following script is not bulletproof but should give you a starting point:
import sys
import os
external = set()
exempt = set()
paths = (os.path.abspath(p) for p in sys.path)
stdlib = {p for p in paths
if p.startswith((sys.prefix, sys.real_prefix))
and 'site-packages' not in p}
for name, module in sorted(sys.modules.items()):
if not module or name in sys.builtin_module_names or not hasattr(module, '__file__'):
# an import sentinel, built-in module or not a real module, really
exempt.add(name)
continue
fname = module.__file__
if fname.endswith(('__init__.py', '__init__.pyc', '__init__.pyo')):
fname = os.path.dirname(fname)
if os.path.dirname(fname) in stdlib:
# stdlib path, skip
exempt.add(name)
continue
parts = name.split('.')
for i, part in enumerate(parts):
partial = '.'.join(parts[:i] + [part])
if partial in external or partial in exempt:
# already listed or exempted
break
if partial in sys.modules and sys.modules[partial]:
# just list the parent name and be done with it
external.add(partial)
break
for name in external:
print name, sys.modules[name].__file__
Put this is a new module, import it after all imports in your script, and it'll print all modules that it thinks are not part of the standard library.

The standard library is defined in the documentation of python. You can just search there, or put the module names into a list and check programmatically with that.
Alternatively, in python3.4 there's a new isolated mode that allows to ignore a certain number of user-defined library paths. In previous versions of python you can use -s to ignore the per-user environment and -E to ignore the system defined variables.
In python2 a very simple way to check if a module is part of the standard library is to clear the sys.path:
>>> import sys
>>> sys.path = []
>>> import numpy
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named numpy
>>> import traceback
>>> import os
>>> import re
However this doesn't work in python3.3+:
>>> import sys
>>> sys.path = []
>>> import traceback
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'traceback'
[...]
This is because starting with python3.3 the import machinery was changed, and importing the standard library uses the same mechanism as importing any other module (see the documentation).
In python3.3 the only way to make sure that only stdlib's imports succeed is to add only the standard library path to sys.path, for example:
>>> import os, sys, traceback
>>> lib_path = os.path.dirname(traceback.__file__)
>>> sys.path = [lib_path]
>>> import traceback
>>> import re
>>> import numpy
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'numpy'
I used the traceback module to get the library path, since this should work on any system.
For the built-in modules, which are a subset of the stdlib modules, you can check sys.builtin_module_names

#Bakuriu's answer was very useful to me. The only problem I experienced is if you want to check if a particular module is stdlib however is has been imported already. In that case, sys.modules will only have an entry for it so even if sys.path is stripped, the import will succeed:
In [1]: import sys
In [2]: import virtualenv
In [3]: sys.path = []
In [4]: try:
__import__('virtualenv')
except ImportError:
print(False)
else:
print(True)
...:
True
vs
In [1]: import sys
In [2]: sys.path = []
In [3]: try:
__import__('virtualenv')
except ImportError:
print(False)
else:
print(True)
...:
False
I whipped out the following solution which seems to work in both Python2 and Python3:
from __future__ import unicode_literals, print_function
import sys
from contextlib import contextmanager
from importlib import import_module
#contextmanager
def ignore_site_packages_paths():
paths = sys.path
# remove all third-party paths
# so that only stdlib imports will succeed
sys.path = list(filter(
None,
filter(lambda i: 'site-packages' not in i, sys.path)
))
yield
sys.path = paths
def is_std_lib(module):
if module in sys.builtin_module_names:
return True
with ignore_site_packages_paths():
imported_module = sys.modules.pop(module, None)
try:
import_module(module)
except ImportError:
return False
else:
return True
finally:
if imported_module:
sys.modules[module] = imported_module
You can keep track of the source code here

Dynamically importing Python module

I have a trusted remote server that stores many custom Python modules. I can fetch them via HTTP (e.g. using urllib2.urlopen) as text/plain, but I cannot save the fetched module code to the local hard disk. How can I import the code as a fully operable Python module, including its global variables and imports?
I suppose I have to use some combination of exec and imp module's functions, but I've been unable to make it work yet.

It looks like this should do the trick: importing a dynamically generated module
>>> import imp
>>> foo = imp.new_module("foo")
>>> foo_code = """
... class Foo:
... pass
... """
>>> exec foo_code in foo.__dict__
>>> foo.Foo.__module__
'foo'
>>>
Also, as suggested in the ActiveState article, you might want to add your new module to sys.modules:
>>> import sys
>>> sys.modules["foo"] = foo
>>> from foo import Foo
<class 'Foo' …>
>>>

Here's something I bookmarked a while back that covers something similar:
Customizing the Python Import System
It's a bit beyond what you want, but the basic idea is there.

Python3 version
(attempted to edit other answer but the edit que is full)
import imp
my_dynamic_module = imp.new_module("my_dynamic_module")
exec("""
class Foo:
pass
""", my_dynamic_module.__dict__)
Foo = my_dynamic_module.Foo
foo_object = Foo()
# register it on sys
import sys
sys.modules[my_dynamic_module.__name__] = my_dynamic_module

I recently encountered trying to do this while trying to write unit tests for source code examples I put into a project's readme (I wanted to avoid just linking to small files or duplicating the text in a way that could get out of sync).
I came up with the following
import sys
import types
from importlib import import_module
def compile_and_install_module(module_name: str, source_code: str) -> types.ModuleType:
"""Compile source code and install it as a module.
End result is that `import <module_name>` and `from <module_name> import ...` should work.
"""
module = types.ModuleType(module_name, "Module created from source code")
# Execute source in context of empty/fake module
exec(source_code, module.__dict__)
# Insert fake module into sys.modules. It's now a real module
sys.modules[module_name] = module
# Imports should work now
return import_module(module_name)
And a quick example of how you can use it
$ cat hello.py
def foo():
print("Hello world")
bar = 42
$ python
Python 3.9.5 (tags/v3.9.5:0a7dcbd, May 3 2021, 17:27:52) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from compile import compile_and_install_module
>>> compile_and_install_module("hello", open("hello.py").read())
<module 'hello'>
>>> import hello
>>> hello.foo()
Hello world
>>> from hello import bar
>>> bar
42
You can remove the return value and import_lib import if you