I've been playing around with IPython.parallel and I wanted to use some custom modules of my own, but haven't been able to do it as explained on the cookbook using dview.sync_imports(). The only thing that has worked for me was something like
def my_parallel_func(args):
import sys
sys.path.append('/path/to/my/module')
import my_module
#and all the rest
and then in the main just to
if __name__=='__main__':
#set up dview...
dview.map( my_parallel_func, my_args )
The correct way to do this would in my opinion be something like
with dview.sync_imports():
import sys
sys.path.append('/path/to/my/module')
import my_module
but this throws an error saying there is no module named my_module.
So, what is the right way of doing it using dview.sync_imports()??
The problem is that you're changing the PYTHONPATH just in the local process running the Client, and not in the remote processes running in the ipcluster.
You can observe this behaviour if you run the next piece of code:
from IPython.parallel import Client
rc = Client()
dview = rc[:]
with dview.sync_imports():
import sys
sys.path[:] = ['something']
def parallel(x):
import sys
return sys.path
print 'Local: ', sys.path
print 'Remote: ', dview.map_sync(parallel, range(1))
Basically all the modules that you want to use with sync_imports must already be in the PYTHONPATH.
If it's not in the PYTHONPATH then you must add it to the path in the function that you execute remotely, and then import the module in the function.
Related
below the folder structure of my software:
below the code of all the .py files:
run.py:
import modules.module_01.aa as a
a.test()
# test:
if __name__=="__main__":
pass
aa.py (module 1):
import libraries.qq as q
import libraries.zz as z
def test():
q.qq_fun()
z.zz_fun()
print("ciao")
qq.py (library used by aa.py):
def qq_fun():
pass
zz.py (library used by aa.py):
def zz_fun():
pass
my question is really simple, why when I run "run.py" Python say to me:
why "aa.py" can't import the module "qq.py" and "zz.py"? how can I fix this issue?
run.py
In run.py, the Python interpreter thinks you're trying to import module_01.aa from a module named module. To import aa.py, you'll need to add this code to the top of your file, which adds the directory aa.py is in to the system path, and change your import statement to import aa as a.
import sys
sys.path.insert(0, "./modules/module_01/")
aa.py
The same problem occurs in aa.py. To fix the problem in this file, you'll need to add this code to the top of aa.py, which adds the directory qq.py and zz.py are in, and remove the libraries. from both of your import statements.
import sys
sys.path.insert(0, "./modules/module_01/libraries")
I am building a python function I and I need to add my custom module in the code. My azure function looks like this:
import logging
import sys
from sys import path
import os
import time
sys.path.insert(0, './myFunctionFolderName') // Try to insert it as a standard python library.
import azure.functions as func
from myCustomModule.models import someFunctionName
def main(req: func.HttpRequest) -> func.HttpResponse:
name = "My new function"
testing_variable = someFunctionName()
return func.HttpResponse(f"Function executed {name}")]
What I have done is inserted my function folder as a standard path so python looks for the libraries in that folder as well. This function works perfectly in the local environment using Visual Studio Code. However, when I deploy and run it, it throws myCustomModule not found. Another piece of information is that I cannot access Kudu (terminal) since it is not suppored on my consumption plan for Linux. I am not sure what am I missing. Please not since its not a standard library I cannot add it in the requirements.txt. Also note, that my function folder has my function script file and my custom module so they are in the same folder as it should be in the azure python template.
Use an absolute path rather than the relative paths.
Following worked for me
import logging
import sys
from sys import path
import os
import time
dir_path = os.path.dirname(os.path.realpath(__file__))
sys.path.insert(0, dir_path)
import azure.functions as func
from myCustomModule.models import someFunctionName
def main(req: func.HttpRequest) -> func.HttpResponse:
name = "My new function"
testing_variable = someFunctionName()
return func.HttpResponse(f"Function executed {name}")]
A simpler solution than this response is to explicitly tell python to use local import adding a dot before the custom module import, like this:
import logging
import sys
from sys import path
import os
import time
import azure.functions as func
from .myCustomModule.models import someFunctionName
It works with Azure Functions
See also the offical documentation regarding Import Behavior in Functions for Python.
I'm new to Python. What I want to do is after setup esptool package (pip install esptool) call its main method with a bunch of arguments in my application. Something like:
esptool.py -p /dev/ttyUSB0 write_flash -fm qio 0x0000
There is an issue I faced. esptool is not on the packages list in python to import (it is installed with pip already). How am I gonna user import and call the main method?
Resolving import issues
You can't simply invoke import esptool because esptool.py is an executable script, thus not meant to be imported like a plain module. However, there are workarounds for importing code from executable scripts; here are two I know of:
extending sys.path
You can extend the sys.path to include the bindir containing the esptool.py script. Simple check from command line:
$ PYTHONPATH=$(which esptool.py)/.. python -c "import esptool; esptool.main()"
should print you the usage help text.
Extending sys.path in code:
import os
import sys
try:
from shutil import which
except ImportError:
from distutils.spawn import find_executable as which
bindir = os.path.dirname(which('esptool.py'))
sys.path.append(bindir) # after this line, esptool becomes importable
import esptool
if __name__ == '__main__':
esptool.main()
using import machinery
You can avoid extending sys.path by using the mechanisms of importing Python code from arbitrary files. I like this solution more than fiddling with sys.path, but unfortunately, it is not portable between Python 2 and 3.
Python 3.5+
import importlib.machinery
import importlib.util
from shutil import which
if __name__ == '__main__':
loader = importlib.machinery.SourceFileLoader('esptool', which('esptool.py'))
spec = importlib.util.spec_from_loader(loader.name, loader)
esptool = importlib.util.module_from_spec(spec)
loader.exec_module(esptool) # after this line, esptool is imported
esptool.main()
Python 2.7
import imp
from distutils.spawn import find_executable as which
if __name__ == '__main__':
esptool = imp.load_source('esptool', which('esptool.py'))
esptool.main()
Passing command line arguments
The command line arguments are stored in sys.argv list, so you will have to temporarily overwrite it in order to pass the arguments to the main function:
# assuming esptool is imported
import sys
if __name__ == '__main__':
# save the current arguments
argv_original = sys.argv[:]
# overwrite arguments to be passed to esptool argparser
sys.argv[:] = ['', '-p', '/dev/ttyUSB0', 'write_flash', '-fm', 'qio', '0x0000']
try:
esptool.main()
except Exception:
# TODO deal with errors here
pass
finally: # restore original arguments
sys.argv[:] = argv_original
I am debugging code, which has this line:
run('python /home/some_user/some_repo/pyflights/usertools/import.py /home/some_user/some_repo/pyflights/config/index_import.conf flights.map --import')
run - is some analog of os.system
So, I want to run this code without using run function. I need to import my import.py file and run it with sys.args. But how can I do this?
from some_repo.pyflights.usertools import import
There is no way to import import because import is a keyword. Moreover, importing a python file is different from running a script because most scripts have a section
if __name__ == '__main__':
....
When the program is running as a script, the variable __name__ has value __main__.
If you are ready to call a subprocess, you can use
`subprocess.call(...)`
Edit: actually, you can import import like so
from importlib import import_module
mod = import_module('import')
however it won't have the same effect as calling the script. Notice that the script probably uses sys.argv, and this must be addressed too.
Edit: here is an ersatz that you can try if you really don't want a subprocess. I don't guarantee it will work
import shlex
import sys
import types
def run(args):
"""Runs a python program with arguments within the current process.
Arguments:
#args: a sequence of arguments, the first one must be the file path to the python program
This is not guaranteed to work because the current process and the
executed script could modify the python running environment in incompatible ways.
"""
old_main, sys.modules['__main__'] = sys.modules['__main__'], types.ModuleType('__main__')
old_argv, sys.argv = sys.argv, list(args)
try:
with open(sys.argv[0]) as infile:
source = infile.read()
exec(source, sys.modules['__main__'].__dict__)
except SystemExit as exc:
if exc.code:
raise RuntimeError('run() failed with code %d' % exc.code)
finally:
sys.argv, sys.modules['__main__'] = old_argv, old_main
command = '/home/some_user/some_repo/pyflights/usertools/import.py /home/some_user/some_repo/pyflights/config/index_import.conf flights.map --import'
run(shlex.split(command))
I'm trying to use a function called start to set up my enviroment in python. The function imports os.
After I run the function and do the following
os.listdir(simdir+"main")
I get a error that says os not defined
code
>>> def setup ():
import os.path
import shutil
simdir="e:\\"
maindir="c:\\backup\\bitcois\\test exit\\"
>>> setup()
>>> os.listdir(simdir+"main")
Traceback (most recent call last):
File "<pyshell#10>", line 1, in <module>
os.listdir(simdir+"main")
NameError: name 'os' is not defined
The import statement is scoped. When importing modules they are defined for the local namespace.
From the documentation:
Import statements are executed in two steps: (1) find a module, and initialize it if necessary; (2) define a name or names in the local namespace (of the scope where the import statement occurs). [...]
So in your case the os package is only defined within function setup.
You are getting this error because you are NOT importing the whole os library but just the os.path module. In this way, the other resources at the os library are not made available for your use.
In order to be able to use the os.listdir method, you need to either import it alongside the os.path like this:
>>> def setup ():
import os.path, os.listdir
import shutil
simdir="e:\\"
maindir="c:\\backup\\bitcois\\test exit\\"
or import the full library:
>>> def setup ():
import os
import shutil
simdir="e:\\"
maindir="c:\\backup\\bitcois\\test exit\\"
You can read more here:
https://docs.python.org/2/tutorial/modules.html
try:
import os.path
import shutil
import glob
def setup ():
global simdir
simdir="e:\\"
maindir="c:\\backup\\bitcois\\test exit\\"
setup()
os.listdir(simdir+"main")
You need to return the paths and assign the returned values in the global scope. Also, import os too:
import os
def setup():
# retain existing code
return simdir, maindir
simdir, maindir = setup()
When you import os or do any sort of command within a function, the command's effect only last while that function itself is running. What you need to do is
import os
...Do your function and other code
This way, your import lasts for the whole program :).