Develop Reference

Using a class whose methods are decorated by #ray.remote in another directory

I'm learning how to use the python package Ray to parallel my code. I'm facing a problem with a class whose methods are decorated with #ray.remote.
It is okay for me to import the class in the same folder and execute the method in the class. However, after importing the class to another directory, it raises ModuleNotFoundError when the method is called.
According to the trackback of the error, it seems like the issue is about the deserialization process when the program can not find the module it requires.
A minimal working example of my issue is shown below.
Structure of the directory.
folder/
main/
a_class.py
process_in_main.py
processing/
process_not_in_main.py
Content of three scripts
# a_class.py
import ray
class Foo:
#ray.remote
def single_function(self):
return None
def combined_function(self):
ray.init()
results_id = [Foo.single_function.remote(self) for i in range(5)]
results = ray.get(results_id)
ray.shutdown()
return None
# process_in_main.py, it works fine
import a_class
instance = a_class.Foo()
instance.combined_function()
# process_not_in_main.py
import sys
sys.path.append('../')
from main import a_class
instance = a_class.Foo()
instance.combined_function() # it will raise ModuleNotFoundError: No module named 'main'
Any help would be appreciated. Thanks in advance.

Please check that the imports are really resolvable. Both scripts work for me, when I set the Pythonpath to the parent of folder and use absolute imports that start with "folder".
Dockerised example here (since I´m at windows at the moment) :
https://github.com/FelixKleineBoesing/stackoverflowSnippets/tree/master/question59809169

Python + PyCharm File Structure issue: AttributeError: 'module' object has no attribute 'X'

I'm having the following file structure:
main.py
Crypto
GetGenerators.py
Utils
RecHash.py
ToInteger.py
Utils.py
GetGenerators.py looks like this:
import unittest
import os, sys
import gmpy2
from gmpy2 import mpz
sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from Utils.Utils import AssertInt, AssertClass
from Utils.ToInteger import ToInteger
from Utils.RecHash import RecHash
def GetGenerators(n):
AssertInt(n)
assert n >= 0, "n must be greater than or equal 0"
generators = []
# ... irrelevant code...
return generators
class GetGeneratorsTest(unittest.TestCase):
def testGetGenerators(self):
self.assertEqual(len(GetGenerators(50)), 50)
if __name__ == '__main__':
unittest.main()
When I'm using the function GetGenerators from inside main.py, it works fine.
However, when I'm running the GetGenerators.py UnitTests by rightclicking the file, "Run Unittests in GetGenerators.py", I'm getting the following error:
File "C:\Program Files (x86)\JetBrains\PyCharm 2016.3.2\helpers\pycharm\nose_helper\util.py", line 70, in resolve_name
obj = getattr(obj, part)
AttributeError: 'module' object has no attribute 'GetGenerators'
I suppose it has something to do with the structure of my files, but I don't see the problem.

I haven't had your exact problem before, but I think I've had one like it. When I use PyCharm, I find that if open and use files that I've created in a project in PyCharm, then everything works fine. I can import them, can run them; no problems. The problems I run into (which are similar to yours) are when I open a file that was not created within a PyCharm project. I can't import them, and sometimes can't even run them correctly. Maybe it's just me being stupid or maybe a real bug with PyCharm, but whatever the case is. It might be worth (if you haven't already), create a project in PyCharm and copy and paste the file contents into files you create within PyCharm. For some reason, that has worked for me in the past.

So I've ran into a similar problem with PyCharm 2022.2.2 and this solution didn't help me. Instead, what worked was checking my code to make sure I didn't have any object named 'module' defined anywhere, plus I changed some of the documents like "face_landmarks.py" and "face_recognition.py" into "landmarks.py" to avoid confusion when calling a similar line with face_recognition package in python.
I've also tried marking the project folder as a Namespace package. However, as I've done several things at once, I'm not sure if this had any impact. The problem was resolved, but the issue with file structure is there for PyCharm even 6 years later.

Test if code is executed from within a py.test session

I'd like to connect to a different database if my code is running under py.test. Is there a function to call or an environment variable that I can test that will tell me if I'm running under a py.test session? What's the best way to handle this?

A simpler solution I came to:
import sys
if "pytest" in sys.modules:
...
Pytest runner will always load the pytest module, making it available in sys.modules.
Of course, this solution only works if the code you're trying to test does not use pytest itself.

There's also another way documented in the manual:
https://docs.pytest.org/en/latest/example/simple.html#pytest-current-test-environment-variable
Pytest will set the following environment variable PYTEST_CURRENT_TEST.
Checking the existence of said variable should reliably allow one to detect if code is being executed from within the umbrella of pytest.
import os
if "PYTEST_CURRENT_TEST" in os.environ:
# We are running under pytest, act accordingly...
Note
This method works only when an actual test is being run.
This detection will not work when modules are imported during pytest collection.

A solution came from RTFM, although not in an obvious place. The manual also had an error in code, corrected below.
Detect if running from within a pytest run
Usually it is a bad idea to make application code behave differently
if called from a test. But if you absolutely must find out if your
application code is running from a test you can do something like
this:
# content of conftest.py
def pytest_configure(config):
import sys
sys._called_from_test = True
def pytest_unconfigure(config):
import sys # This was missing from the manual
del sys._called_from_test
and then check for the sys._called_from_test flag:
if hasattr(sys, '_called_from_test'):
# called from within a test run
else:
# called "normally"
accordingly in your application. It’s also a good idea to use your own
application module rather than sys for handling flag.

Working with pytest==4.3.1 the methods above failed, so I just went old school and checked with:
script_name = os.path.basename(sys.argv[0])
if script_name in ['pytest', 'py.test']:
print('Running with pytest!')

While the hack explained in the other answer (http://pytest.org/latest/example/simple.html#detect-if-running-from-within-a-pytest-run) does indeed work, you could probably design the code in such a way you would not need to do this.
If you design the code to take the database to connect to as an argument somehow, via a connection or something else, then you can simply inject a different argument when you're running the tests then when the application drives this. Your code will end up with less global state and more modulare and reusable. So to me it sounds like an example where testing drives you to design the code better.

This could be done by setting an environment variable inside the testing code. For example, given a project
conftest.py
mypkg/
__init__.py
app.py
tests/
test_app.py
In test_app.py you can add
import os
os.environ['PYTEST_RUNNING'] = 'true'
And then you can check inside app.py:
import os
if os.environ.get('PYTEST_RUNNING', '') == 'true':
print('pytest is running')

Python: intercept a class loading action

Summary: when a certain python module is imported, I want to be able to intercept this action, and instead of loading the required class, I want to load another class of my choice.
Reason: I am working on some legacy code. I need to write some unit test code before I start some enhancement/refactoring. The code imports a certain module which will fail in a unit test setting, however. (Because of database server dependency)
Pseduo Code:
from LegacyDataLoader import load_me_data
...
def do_something():
data = load_me_data()
So, ideally, when python excutes the import line above in a unit test, an alternative class, says MockDataLoader, is loaded instead.
I am still using 2.4.3. I suppose there is an import hook I can manipulate
Edit
Thanks a lot for the answers so far. They are all very helpful.
One particular type of suggestion is about manipulation of PYTHONPATH. It does not work in my case. So I will elaborate my particular situation here.
The original codebase is organised in this way
./dir1/myapp/database/LegacyDataLoader.py
./dir1/myapp/database/Other.py
./dir1/myapp/database/__init__.py
./dir1/myapp/__init__.py
My goal is to enhance the Other class in the Other module. But since it is legacy code, I do not feel comfortable working on it without strapping a test suite around it first.
Now I introduce this unit test code
./unit_test/test.py
The content is simply:
from myapp.database.Other import Other
def test1():
o = Other()
o.do_something()
if __name__ == "__main__":
test1()
When the CI server runs the above test, the test fails. It is because class Other uses LegacyDataLoader, and LegacydataLoader cannot establish database connection to the db server from the CI box.
Now let's add a fake class as suggested:
./unit_test_fake/myapp/database/LegacyDataLoader.py
./unit_test_fake/myapp/database/__init__.py
./unit_test_fake/myapp/__init__.py
Modify the PYTHONPATH to
export PYTHONPATH=unit_test_fake:dir1:unit_test
Now the test fails for another reason
File "unit_test/test.py", line 1, in <module>
from myapp.database.Other import Other
ImportError: No module named Other
It has something to do with the way python resolves classes/attributes in a module

You can intercept import and from ... import statements by defining your own __import__ function and assigning it to __builtin__.__import__ (make sure to save the previous value, since your override will no doubt want to delegate to it; and you'll need to import __builtin__ to get the builtin-objects module).
For example (Py2.4 specific, since that's what you're asking about), save in aim.py the following:
import __builtin__
realimp = __builtin__.__import__
def my_import(name, globals={}, locals={}, fromlist=[]):
print 'importing', name, fromlist
return realimp(name, globals, locals, fromlist)
__builtin__.__import__ = my_import
from os import path
and now:
$ python2.4 aim.py
importing os ('path',)
So this lets you intercept any specific import request you want, and alter the imported module[s] as you wish before you return them -- see the specs here. This is the kind of "hook" you're looking for, right?

There are cleaner ways to do this, but I'll assume that you can't modify the file containing from LegacyDataLoader import load_me_data.
The simplest thing to do is probably to create a new directory called testing_shims, and create LegacyDataLoader.py file in it. In that file, define whatever fake load_me_data you like. When running the unit tests, put testing_shims into your PYTHONPATH environment variable as the first directory. Alternately, you can modify your test runner to insert testing_shims as the first value in sys.path.
This way, your file will be found when importing LegacyDataLoader, and your code will be loaded instead of the real code.

The import statement just grabs stuff from sys.modules if a matching name is found there, so the simplest thing is to make sure you insert your own module into sys.modules under the target name before anything else tries to import the real thing.
# in test code
import sys
import MockDataLoader
sys.modules['LegacyDataLoader'] = MockDataLoader
import module_under_test
There are a handful of variations on the theme, but that basic approach should work fine to do what you describe in the question. A slightly simpler approach would be this, using just a mock function to replace the one in question:
# in test code
import module_under_test
def mock_load_me_data():
# do mock stuff here
module_under_test.load_me_data = mock_load_me_data
That simply replaces the appropriate name right in the module itself, so when you invoke the code under test, presumably do_something() in your question, it calls your mock routine.

Well, if the import fails by raising an exception, you could put it in a try...except loop:
try:
from LegacyDataLoader import load_me_data
except: # put error that occurs here, so as not to mask actual problems
from MockDataLoader import load_me_data
Is that what you're looking for? If it fails, but doesn't raise an exception, you could have it run the unit test with a special command line tag, like --unittest, like this:
import sys
if "--unittest" in sys.argv:
from MockDataLoader import load_me_data
else:
from LegacyDataLoader import load_me_data

Python App Engine import issues after app is cached

I'm using a modified version on juno (http://github.com/breily/juno/) in Google App Engine. The problem I'm having is I have code like this:
import juno
import pprint
#get('/')
def home(web):
pprint.pprint("test")
def main():
run()
if __name__ == '__main__':
main()
The first time I start the app up in the dev environment it works fine. The second time and every time after that it can't find pprint. I get this error:
AttributeError: 'NoneType' object has no attribute 'pprint'
If I set the import inside the function it works every time:
#get('/')
def home(web):
import pprint
pprint.pprint("test")
So it seems like it is caching the function but for some reason the imports are not being included when it uses that cache. I tried removing the main() function at the bottom to see if that would remove the caching of this script but I get the same problem.
Earlier tonight this code was working fine, I'm not sure what could have changed to cause this. Any insight is appreciated.

I would leave it that way. I saw a slideshare that Google put out about App Engine optimization that said you can get better performance by keeping imports inside of the methods, so they are not imported unless necessary.

Is it possible you are reassigning the name pprint somewhere? The only two ways I know of for a module-level name (like what you get from the import statement) to become None is if you either assign it yourself pprint = None or upon interpreter shutdown, when Python's cleanup assigns all module-level names to None as it shuts things down.