I'm new to Python. What I want to do is after setup esptool package (pip install esptool) call its main method with a bunch of arguments in my application. Something like:
esptool.py -p /dev/ttyUSB0 write_flash -fm qio 0x0000
There is an issue I faced. esptool is not on the packages list in python to import (it is installed with pip already). How am I gonna user import and call the main method?
Resolving import issues
You can't simply invoke import esptool because esptool.py is an executable script, thus not meant to be imported like a plain module. However, there are workarounds for importing code from executable scripts; here are two I know of:
extending sys.path
You can extend the sys.path to include the bindir containing the esptool.py script. Simple check from command line:
$ PYTHONPATH=$(which esptool.py)/.. python -c "import esptool; esptool.main()"
should print you the usage help text.
Extending sys.path in code:
import os
import sys
try:
from shutil import which
except ImportError:
from distutils.spawn import find_executable as which
bindir = os.path.dirname(which('esptool.py'))
sys.path.append(bindir) # after this line, esptool becomes importable
import esptool
if __name__ == '__main__':
esptool.main()
using import machinery
You can avoid extending sys.path by using the mechanisms of importing Python code from arbitrary files. I like this solution more than fiddling with sys.path, but unfortunately, it is not portable between Python 2 and 3.
Python 3.5+
import importlib.machinery
import importlib.util
from shutil import which
if __name__ == '__main__':
loader = importlib.machinery.SourceFileLoader('esptool', which('esptool.py'))
spec = importlib.util.spec_from_loader(loader.name, loader)
esptool = importlib.util.module_from_spec(spec)
loader.exec_module(esptool) # after this line, esptool is imported
esptool.main()
Python 2.7
import imp
from distutils.spawn import find_executable as which
if __name__ == '__main__':
esptool = imp.load_source('esptool', which('esptool.py'))
esptool.main()
Passing command line arguments
The command line arguments are stored in sys.argv list, so you will have to temporarily overwrite it in order to pass the arguments to the main function:
# assuming esptool is imported
import sys
if __name__ == '__main__':
# save the current arguments
argv_original = sys.argv[:]
# overwrite arguments to be passed to esptool argparser
sys.argv[:] = ['', '-p', '/dev/ttyUSB0', 'write_flash', '-fm', 'qio', '0x0000']
try:
esptool.main()
except Exception:
# TODO deal with errors here
pass
finally: # restore original arguments
sys.argv[:] = argv_original
Related
I am setting up a package using distutils.
I need to allow access to a module that is built during the set-up process and is located in ./build/temp.linux-x86_64-3.6. I do this by including the
include_dirs=["./build/temp.linux-x86_64-3.6"]
when adding the extension to the distutils Configuration.
My question is there a way of setting this using a wildcard such as:
include_dirs=["./build/temp.linux*"]
as when I try this it fails, citing error:
Nonexistent include directory ‘build/temp.linux*’ [-Wmissing-include-dirs]
The reason I want this is that the build folder will be named differently depending on the system. Alternatively if anyone knows a way of figuring out what this temp build folder will be called that would also work.
The way I have got around this problem is as follows:
def return_major_minor_python():
import sys
return str(sys.version_info[0])+"."+str(sys.version_info[1])
def return_include_dir():
from distutils.util import get_platform
return get_platform()+'-'+return_major_minor_python()
Then when calling config.add_extension() using:
include_dirs=['build/temp.' + return_include_dir()]
So the whole process for adding a f90wrapped, f2py extension to a python package is:
def setup_fort_ext(args,parent_package='',top_path=''):
from numpy.distutils.misc_util import Configuration
from os.path import join
import sys
config = Configuration('',parent_package,top_path)
fort_src = [join('PackageName/','fortran_source.f90')]
config.add_library('_fortran_source', sources=fort_src,
extra_f90_compile_args = [ args["compile_args"]],
extra_link_args=[args["link_args"]])
sources = [join('PackageName','f90wrap_fortran_source.f90')]
config.add_extension(name='_fortran_source',
sources=sources,
extra_f90_compile_args = [ args["compile_args"]],
extra_link_args=[args["link_args"]],
libraries=['_tort'],
include_dirs=['build/temp.' + return_include_dir()])
return config
if __name__ == '__main__':
import sys
import subprocess
import os
install_numpy() #installs numpy
install_dependencies() #calls to pip to install any requirements
from numpy.distutils.core import setup
config = {'name':'PackageName',
'version':__version__,
'project_description':'Some Package description',
'description':'Some package Description',
'long_description': open('README.txt').read(),
'long_description_content_type':'text/markdown',
'author':'Your name here',
'author_email':'your email here',
'url':'link to git repo here',
'python_requires':'>=3.3',
'packages':['PackageName'],
'package_dir':{'PackageName':'PackageName'},
'package_data':{'PackageName':['*so*']},
'name': 'PackageName'
}
config_fort = setup_fort_ext(args,parent_package='PackageName',top_path='')
config2 = dict(config,**config_fort.todict())
setup(**config2)
where the source fortran_source.f90 is wrapped beforehand and the resulting wrapped source file (f90wrap_fortran_source.f90) is included as a library, and subsequently compiled by f2py.
args in the above is just a dict with the any linking or compile args you wish to pass through.
I am debugging code, which has this line:
run('python /home/some_user/some_repo/pyflights/usertools/import.py /home/some_user/some_repo/pyflights/config/index_import.conf flights.map --import')
run - is some analog of os.system
So, I want to run this code without using run function. I need to import my import.py file and run it with sys.args. But how can I do this?
from some_repo.pyflights.usertools import import
There is no way to import import because import is a keyword. Moreover, importing a python file is different from running a script because most scripts have a section
if __name__ == '__main__':
....
When the program is running as a script, the variable __name__ has value __main__.
If you are ready to call a subprocess, you can use
`subprocess.call(...)`
Edit: actually, you can import import like so
from importlib import import_module
mod = import_module('import')
however it won't have the same effect as calling the script. Notice that the script probably uses sys.argv, and this must be addressed too.
Edit: here is an ersatz that you can try if you really don't want a subprocess. I don't guarantee it will work
import shlex
import sys
import types
def run(args):
"""Runs a python program with arguments within the current process.
Arguments:
#args: a sequence of arguments, the first one must be the file path to the python program
This is not guaranteed to work because the current process and the
executed script could modify the python running environment in incompatible ways.
"""
old_main, sys.modules['__main__'] = sys.modules['__main__'], types.ModuleType('__main__')
old_argv, sys.argv = sys.argv, list(args)
try:
with open(sys.argv[0]) as infile:
source = infile.read()
exec(source, sys.modules['__main__'].__dict__)
except SystemExit as exc:
if exc.code:
raise RuntimeError('run() failed with code %d' % exc.code)
finally:
sys.argv, sys.modules['__main__'] = old_argv, old_main
command = '/home/some_user/some_repo/pyflights/usertools/import.py /home/some_user/some_repo/pyflights/config/index_import.conf flights.map --import'
run(shlex.split(command))
I have my Python unittest script like below. It takes an argument '-a' to determine whether the testcases should load the base module from foo_PC_A.py or foo_PC_B.py. I use shutil.move() to rename either .py file to foo.py, so all the testcase modules (e.g. tm1.py, tm2.py) can simply import foo. Though this looks like a workaround and not Pythonic. Is there any better way to do this? Or a better design to fundamentally resolve this issue.
(run_all_unittest.py)
if sys.argv[1] = '-a':
shutil.move('foo_PC_A.py', 'foo.py')
else:
shutil.move('foo_PC_B.py', 'foo.py')
test_module_list = ['tm1', 'tm2', ...]
for test_module_name in test_module_list:
test_module = __import__(test_module_name)
test_suites.append(unittest.TestLoader().loadTestsFromModule(test_module))
alltests = unittest.TestSuite(test_suites)
unittest.TextTestRunner().run(alltests)
if sys.argv[1] = '-a':
shutil.move('foo.py', 'foo_PC_A.py')
else:
shutil.move('foo.py', 'foo_PC_B.py')
(tm1.py)
from foo import MyTestCase
...
(foo_PC_A.py)
import <some module only available on PC A>
class MyTestCase(unittest.TestCase):
...
(foo_PC_B.py)
# since the PC A modules are not available on PC B,
# just call the pre-built executable via subprocess
import subprocess
class MyTestCase(unittest.TestCase):
...
def test_run(self):
subprocess.call(...)
You can fool Python into thinking the module has already been loaded. Just import the module dynamically and use sys.modules:
import sys
import importlib
if sys.argv[1] = '-a':
sys.modules['foo'] = importlib.import_module('foo_PC_A')
else:
sys.modules['foo'] = importlib.import_module('foo_PC_A')
When any module runs import foo or from foo import ..., Python will use that path.
Note that if foo is moved to a package, the full Python path must be specified, as in:
sys.modules['path.to.foo'] = ...
I've been playing around with IPython.parallel and I wanted to use some custom modules of my own, but haven't been able to do it as explained on the cookbook using dview.sync_imports(). The only thing that has worked for me was something like
def my_parallel_func(args):
import sys
sys.path.append('/path/to/my/module')
import my_module
#and all the rest
and then in the main just to
if __name__=='__main__':
#set up dview...
dview.map( my_parallel_func, my_args )
The correct way to do this would in my opinion be something like
with dview.sync_imports():
import sys
sys.path.append('/path/to/my/module')
import my_module
but this throws an error saying there is no module named my_module.
So, what is the right way of doing it using dview.sync_imports()??
The problem is that you're changing the PYTHONPATH just in the local process running the Client, and not in the remote processes running in the ipcluster.
You can observe this behaviour if you run the next piece of code:
from IPython.parallel import Client
rc = Client()
dview = rc[:]
with dview.sync_imports():
import sys
sys.path[:] = ['something']
def parallel(x):
import sys
return sys.path
print 'Local: ', sys.path
print 'Remote: ', dview.map_sync(parallel, range(1))
Basically all the modules that you want to use with sync_imports must already be in the PYTHONPATH.
If it's not in the PYTHONPATH then you must add it to the path in the function that you execute remotely, and then import the module in the function.
Suppose I have two modules:
a.py:
import b
print __name__, __file__
b.py:
print __name__, __file__
I run the "a.py" file. This prints:
b C:\path\to\code\b.py
__main__ C:\path\to\code\a.py
Question: how do I obtain the path to the __main__ module ("a.py" in this case) from within the "b.py" library?
import __main__
print(__main__.__file__)
Perhaps this will do the trick:
import sys
from os import path
print(path.abspath(str(sys.modules['__main__'].__file__)))
Note that, for safety, you should check whether the __main__ module has a __file__ attribute. If it's dynamically created, or is just being run in the interactive python console, it won't have a __file__:
python
>>> import sys
>>> print(str(sys.modules['__main__']))
<module '__main__' (built-in)>
>>> print(str(sys.modules['__main__'].__file__))
AttributeError: 'module' object has no attribute '__file__'
A simple hasattr() check will do the trick to guard against scenario 2 if that's a possibility in your app.
The python code below provides additional functionality, including that it works seamlessly with py2exe executables.
I use similar code to like this to find paths relative to the running script, aka __main__. as an added benefit, it works cross-platform including Windows.
import imp
import os
import sys
def main_is_frozen():
return (hasattr(sys, "frozen") or # new py2exe
hasattr(sys, "importers") # old py2exe
or imp.is_frozen("__main__")) # tools/freeze
def get_main_dir():
if main_is_frozen():
# print 'Running from path', os.path.dirname(sys.executable)
return os.path.dirname(sys.executable)
return os.path.dirname(sys.argv[0])
# find path to where we are running
path_to_script=get_main_dir()
# OPTIONAL:
# add the sibling 'lib' dir to our module search path
lib_path = os.path.join(get_main_dir(), os.path.pardir, 'lib')
sys.path.insert(0, lib_path)
# OPTIONAL:
# use info to find relative data files in 'data' subdir
datafile1 = os.path.join(get_main_dir(), 'data', 'file1')
Hopefully the above example code can provide additional insight into how to determine the path to the running script...
Another method would be to use sys.argv[0].
import os
import sys
main_file = os.path.realpath(sys.argv[0]) if sys.argv[0] else None
sys.argv[0] will be an empty string if Python gets start with -c or if checked from the Python console.
import sys, os
def getExecPath():
try:
sFile = os.path.abspath(sys.modules['__main__'].__file__)
except:
sFile = sys.executable
return os.path.dirname(sFile)
This function will work for Python and Cython compiled programs.
I use this:
import __main__
name = __main__.__file__ # this gets file path
lastBar =int(name[::-1].find('/')) # change '/' for linux or '\' for windows
extension = 3 #3 for python file .py
main_file_name=name[::-1][extension:lastBar][::-1] #result