I'm learning how to use the python package Ray to parallel my code. I'm facing a problem with a class whose methods are decorated with #ray.remote.
It is okay for me to import the class in the same folder and execute the method in the class. However, after importing the class to another directory, it raises ModuleNotFoundError when the method is called.
According to the trackback of the error, it seems like the issue is about the deserialization process when the program can not find the module it requires.
A minimal working example of my issue is shown below.
Structure of the directory.
folder/
main/
a_class.py
process_in_main.py
processing/
process_not_in_main.py
Content of three scripts
# a_class.py
import ray
class Foo:
#ray.remote
def single_function(self):
return None
def combined_function(self):
ray.init()
results_id = [Foo.single_function.remote(self) for i in range(5)]
results = ray.get(results_id)
ray.shutdown()
return None
# process_in_main.py, it works fine
import a_class
instance = a_class.Foo()
instance.combined_function()
# process_not_in_main.py
import sys
sys.path.append('../')
from main import a_class
instance = a_class.Foo()
instance.combined_function() # it will raise ModuleNotFoundError: No module named 'main'
Any help would be appreciated. Thanks in advance.
Please check that the imports are really resolvable. Both scripts work for me, when I set the Pythonpath to the parent of folder and use absolute imports that start with "folder".
Dockerised example here (since I´m at windows at the moment) :
https://github.com/FelixKleineBoesing/stackoverflowSnippets/tree/master/question59809169
To preface, I think I may have figured out how to get this code working (based on Changing module variables after import), but my question is really about why the following behavior occurs so I can understand what to not do in the future.
I have three files. The first is mod1.py:
# mod1.py
import mod2
var1A = None
def func1A():
global var1
var1 = 'A'
mod2.func2()
def func1B():
global var1
print var1
if __name__ == '__main__':
func1A()
Next I have mod2.py:
# mod2.py
import mod1
def func2():
mod1.func1B()
Finally I have driver.py:
# driver.py
import mod1
if __name__ == '__main__':
mod1.func1A()
If I execute the command python mod1.py then the output is None. Based on the link I referenced above, it seems that there is some distinction between mod1.py being imported as __main__ and mod1.py being imported from mod2.py. Therefore, I created driver.py. If I execute the command python driver.py then I get the expected output: A. I sort of see the difference, but I don't really see the mechanism or the reason for it. How and why does this happen? It seems counterintuitive that the same module would exist twice. If I execute python mod1.py, would it be possible to access the variables in the __main__ version of mod1.py instead of the variables in the version imported by mod2.py?
The __name__ variable always contains the name of the module, except when the file has been loaded into the interpreter as a script instead. Then that variable is set to the string '__main__' instead.
After all, the script is then run as the main file of the whole program, everything else are modules imported directly or indirectly by that main file. By testing the __name__ variable, you can thus detect if a file has been imported as a module, or was run directly.
Internally, modules are given a namespace dictionary, which is stored as part of the metadata for each module, in sys.modules. The main file, the executed script, is stored in that same structure as '__main__'.
But when you import a file as a module, python first looks in sys.modules to see if that module has already been imported before. So, import mod1 means that we first look in sys.modules for the mod1 module. It'll create a new module structure with a namespace if mod1 isn't there yet.
So, if you both run mod1.py as the main file, and later import it as a python module, it'll get two namespace entries in sys.modules. One as '__main__', then later as 'mod1'. These two namespaces are completely separate. Your global var1 is stored in sys.modules['__main__'], but func1B is looking in sys.modules['mod1'] for var1, where it is None.
But when you use python driver.py, driver.py becomes the '__main__' main file of the program, and mod1 will be imported just once into the sys.modules['mod1'] structure. This time round, func1A stores var1 in the sys.modules['mod1'] structure, and that's what func1B will find.
Regarding a practical solution for using a module optionally as main script - supporting consistent cross-imports:
Solution 1:
See e.g. in Python's pdb module, how it is run as a script by importing itself when executing as __main__ (at the end) :
#! /usr/bin/env python
"""A Python debugger."""
# (See pdb.doc for documentation.)
import sys
import linecache
...
# When invoked as main program, invoke the debugger on a script
if __name__ == '__main__':
import pdb
pdb.main()
Just I would recommend to reorganize the __main__ startup to the beginning of the script like this:
#! /usr/bin/env python
"""A Python debugger."""
# When invoked as main program, invoke the debugger on a script
import sys
if __name__ == '__main__':
##assert os.path.splitext(os.path.basename(__file__))[0] == 'pdb'
import pdb
pdb.main()
sys.exit(0)
import linecache
...
This way the module body is not executed twice - which is "costly", undesirable and sometimes critical.
Solution 2:
In rarer cases it is desirable to expose the actual script module __main__ even directly as the actual module alias (mod1):
# mod1.py
import mod2
...
if __name__ == '__main__':
# use main script directly as cross-importable module
_mod = sys.modules['mod1'] = sys.modules[__name__]
##_modname = os.path.splitext(os.path.basename(os.path.realpath(__file__)))[0]
##_mod = sys.modules[_modname] = sys.modules[__name__]
func1A()
Known drawbacks:
reload(_mod) fails
pickle'ed classes would need extra mappings for unpickling (find_global ..)
I am developping a program which runs on two different plattforms. Depending on which plattform I want to run it, the import directories and the names of the libraries change. For this reason i set a variable called RUN_ON_PC to True or False.
I want to implement a helper which sets the paths correctly and imports the libraries with the correct name depending of the platform and gives an interface with the same name of the libraries to the main program. The module myimporthelper is either in the "/mylib" or in the "/sd/mylib" directory. The other module names in these directories differ.
I try to do the following which is not working, since the imported modules from myimporthelper.py are not visible to main.py:
main.py:
RUN_ON_PC = True
import sys
if RUN_ON_PC:
sys.path.append("/mylib1")
else:
sys.path.append("/sd/mylib1")
import myimporthelper
myimporthelper.importall(RUN_ON_PC)
a = moduleA.ClassA() -> produces NameError: name not defined
myimporthelper.py:
import sys
def importall(run_on_pc):
if (run_on_pc == True):
sys.path.append("C:\\Users\\.....\\mylib")
import module1 as moduleA
else:
sys.path.append("/sd/mylib")
import module_a as moduleA
I want to keep the main.py short and want to outsource the platform dependent importing stuff to other module. I was not able to find a solution for this and would aprecciate any help.
Thanks a lot in advance.
You just have to qualify the name with the helper module name
a = myimporthelper.moduleA.ClassA()
But the moduleA name has to be accessible. If you import it inside a function in the helper it won't be, because of scope, unless you assign it to a name you previously declared as global in the helper module function.
I faced this problem while i was expirementing with my project.My project is designed so it has multipule submoudles.
Now each submodule needs a position of an the same item on the screen,which is random each time the program ran.I thought it was a good idea to check once where is the position of the item (It does not change in runtime) , and let all modules access the position.(Since checking the position takes a long time)
This is what i did:
main.py
import Config
import sub_module
main()
def main():
position = get_pos()
Config.pos = position
sub_module.click_on_item()
Config.py
pos = None
I tried using this code , so when i ran the program , it sets the config.py module pos variable , to the position of the item on screen.
This code works fine but when i try to use it in the submodules like this:
sub_module.py
import Config
def click_on_item():
click(Config.pos)
It resets the value to None , since it reruns the Config module code.
Is there an elegant solution for this problem ? I can't let the position as an argument for the , since my case is much more complex.
One solution that i can think of is writing to the disk , and rereading but it's not fast.
Thanks for the help.
2 EDIT:
My project is multi-package , and there is the problem
this is the directory tree:
mainPackage:
__init__.py
Config.py
main.py
b.py
Packages:
Package_A:
__init__.py
a.py
main.py
import Config
from Packages.Package_A import a
import b
def main():
Config.Position = (124, 586)
a.print_val()
b.print_val()
if __name__ == '__main__':
main()
Config
Position = None
b.py
import Config
def print_val():
print Config.Position
a.py
from mainPackage import Config
def print_val():
print Config.Position
output
None -> from Package_A
(124, 586) -> from b.py
Sorry for the inconvenience since i didn't know what was causing the problem , my interest is in multi package global variable.
The problem is in a.py: from mainPackage import Config. You did an absolute import from a package and ended up importing Config twice. Python only imports a module once but in your case you used two different names (Config in some files, mainPackage.Config in others) which confused python and it imported the module twice - once by a package relative import and once by an absolute import. Add print "Importing Config" to Config.py and you will see it printed twice.
Just change a.py to import Config and all will be well.
Config.py will only be initialized on import once in your application and pos will be initially None.
main.py then sets it, and all other modules can access pos with Config.pos.
Config.py
# Initial value
pos = None
main.py
import Config
import sub_module
def main():
position = get_pos()
Config.pos = position
sub_module.click_on_item()
The rest of your files can be kept as is.
UPDATE
As #tdelaney mentioned in his answer.
Change
from mainPackage import Config
to
import Config
to avoid another Config.
For each of your module, define your functions/class with a 'pos' argument.
Module A.py
def foo(pos):
click(pos)
Module main.py
def main():
position = get_pos()
A.foo(position)
In general, do not use global var.
I have some code spread across multiple files that try to import from each other, as follows:
main.py:
from entity import Ent
entity.py:
from physics import Physics
class Ent:
...
physics.py:
from entity import Ent
class Physics:
...
I then run from main.py and I get the following error:
Traceback (most recent call last):
File "main.py", line 2, in <module>
from entity import Ent
File ".../entity.py", line 5, in <module>
from physics import Physics
File ".../physics.py", line 2, in <module>
from entity import Ent
ImportError: cannot import name Ent
I'm assume the error is due to importing entity twice - once in main.py and later in physics.py - but how can I work around the problem?
See also What happens when using mutual or circular (cyclic) imports in Python? for a general overview of what is allowed and what causes a problem WRT circular imports. See Why do circular imports seemingly work further up in the call stack but then raise an ImportError further down? for technical details on why and how the problem occurs.
You have circular dependent imports. physics.py is imported from entity before class Ent is defined and physics tries to import entity that is already initializing. Remove the dependency to physics from entity module.
While you should definitely avoid circular dependencies, you can defer imports in python.
for example:
import SomeModule
def someFunction(arg):
from some.dependency import DependentClass
this ( at least in some instances ) will circumvent the error.
This is a circular dependency. It can be solved without any structural modifications to the code. The problem occurs because in vector you demand that entity be made available for use immediately, and vice versa. The reason for this problem is that you asking to access the contents of the module before it is ready -- by using from x import y. This is essentially the same as
import x
y = x.y
del x
Python is able to detect circular dependencies and prevent the infinite loop of imports. Essentially all that happens is that an empty placeholder is created for the module (ie. it has no content). Once the circularly dependent modules are compiled it updates the imported module. This is works something like this.
a = module() # import a
# rest of module
a.update_contents(real_a)
For python to be able to work with circular dependencies you must use import x style only.
import x
class cls:
def __init__(self):
self.y = x.y
Since you are no longer referring to the contents of the module at the top level, python can compile the module without actually having to access the contents of the circular dependency. By top level I mean lines that will be executed during compilation as opposed to the contents of functions (eg. y = x.y). Static or class variables accessing the module contents will also cause problems.
In my case, I was working in a Jupyter notebook and this was happening due the import already being cached from when I had defined the class/function inside my working file.
I restarted my Jupyter kernel and the error disappeared.
To make logic clear is very important. This problem appear, because the reference become a dead loop.
If you don't want to change the logic, you can put the some import statement which caused ImportError to the other position of file, for example the end.
a.py
from test.b import b2
def a1():
print('a1')
b2()
b.py
from test.a import a1
def b1():
print('b1')
a1()
def b2():
print('b2')
if __name__ == '__main__':
b1()
You will get Import Error: ImportError: cannot import name 'a1'
But if we change the position of from test.b import b2 in A like below:
a.py
def a1():
print('a1')
b2()
from test.b import b2
And the we can get what we want:
b1
a1
b2
This is a circular dependency.
we can solve this problem by using import module or class or function where we needed.
if we use this approach, we can fix circular dependency
A.py
from B import b2
def a1():
print('a1')
b2()
B.py
def b1():
from A import a1
print('b1')
a1()
def b2():
print('b2')
if __name__ == '__main__':
b1()
I just got this error too, for a different reason...
from my_sub_module import my_function
The main script had Windows line endings. my_sub_module had UNIX line endings. Changing them to be the same fixed the problem. They also need to have the same character encoding.
As already mentioned, this is caused by a circular dependency. What has not been mentioned is that when you're using Python typing module and you import a class only to be used to annotate Types, you can use Forward references:
When a type hint contains names that have not been defined yet, that
definition may be expressed as a string literal, to be resolved later.
and remove the dependency (the import), e.g. instead of
from my_module import Tree
def func(arg: Tree):
# code
do:
def func(arg: 'Tree'):
# code
(note the removed import statement)
The problem is clear: circular dependency between names in entity and physics modules.
Regardless of importing the whole module or just a class, the names must be loaded .
Watch this example:
# a.py
import b
def foo():
pass
b.bar()
# b.py
import a
def bar():
pass
a.foo()
This will be compiled into:
# a.py
# import b
# b.py
# import a # ignored, already importing
def bar():
pass
a.foo()
# name a.foo is not defined!!!
# import b done!
def foo():
pass
b.bar()
# done!
With one slight change we can solve this:
# a.py
def foo():
pass
import b
b.bar()
# b.py
def bar():
pass
import a
a.foo()
This will be compiled into:
# a.py
def foo():
pass
# import b
# b.py
def bar():
pass
# import a # ignored, already importing
a.foo()
# import b done!
b.bar()
# done!
Try this solution: rename your working python script
You should not name your current python script with the name of some other module you import, since you will get that error.
Example:
you are working in medicaltorch.py
in that script, you have: from medicaltorch import X where medicaltorch is supposed to be a separate installed module
This will fail with the ImportError since 2 things refer to medicaltorch
So, just rename your working python script in 1.
If you are importing file1.py from file2.py and used this:
if __name__ == '__main__':
# etc
Variables below that in file1.py cannot be imported to file2.py because __name__ does not equal __main__!
If you want to import something from file1.py to file2.py, you need to use this in file1.py:
if __name__ == 'file1':
# etc
In case of doubt, make an assert statement to determine if __name__=='__main__'
Don't see this one here yet - this is incredibly stupid, but make sure you're importing the correct variable/function.
I was getting this error
ImportError: cannot import name IMPLICIT_WAIT
because my variable was actually IMPLICIT_TIMEOUT.
when I changed my import to use the correct name, I no longer got the error 🤦♂️
One way to track import error is step by step trying to run python on each of imported files to track down bad one.
you get something like:
python ./main.py
ImportError: cannot import name A
then you launch:
python ./modules/a.py
ImportError: cannot import name B
then you launch:
python ./modules/b.py
ImportError: cannot import name C (some NON-Existing module or some other error)
Also not directly relevant to the OP, but failing to restart a PyCharm Python console, after adding a new object to a module, is also a great way to get a very confusing ImportError: Cannot import name ...
The confusing part is that PyCharm will autocomplete the import in the console, but the import then fails.
Not specifically for this asker, but this same error will show if the class name in your import doesn't match the definition in the file you're importing from.
In my case, simply missed filename:
from A.B.C import func_a (x)
from A.B.C.D import func_a (O)
where D is file.
I met this error too, but my case is less common, and it does throw this error too.
My case is that I encounter this error in jupyter notebook; I write from M import c where M is a python file and c is a class in M.py, the reason for the error is because c is just created a few minutes ago, but my jupyter notebook has been running for a long time, so I just need to restart the jupyter notebook and let it reload M.py.