Develop Reference

Call another installed package in Python

I'm new to Python. What I want to do is after setup esptool package (pip install esptool) call its main method with a bunch of arguments in my application. Something like:
esptool.py -p /dev/ttyUSB0 write_flash -fm qio 0x0000
There is an issue I faced. esptool is not on the packages list in python to import (it is installed with pip already). How am I gonna user import and call the main method?

Resolving import issues
You can't simply invoke import esptool because esptool.py is an executable script, thus not meant to be imported like a plain module. However, there are workarounds for importing code from executable scripts; here are two I know of:
extending sys.path
You can extend the sys.path to include the bindir containing the esptool.py script. Simple check from command line:
$ PYTHONPATH=$(which esptool.py)/.. python -c "import esptool; esptool.main()"
should print you the usage help text.
Extending sys.path in code:
import os
import sys
try:
from shutil import which
except ImportError:
from distutils.spawn import find_executable as which
bindir = os.path.dirname(which('esptool.py'))
sys.path.append(bindir) # after this line, esptool becomes importable
import esptool
if __name__ == '__main__':
esptool.main()
using import machinery
You can avoid extending sys.path by using the mechanisms of importing Python code from arbitrary files. I like this solution more than fiddling with sys.path, but unfortunately, it is not portable between Python 2 and 3.
Python 3.5+
import importlib.machinery
import importlib.util
from shutil import which
if __name__ == '__main__':
loader = importlib.machinery.SourceFileLoader('esptool', which('esptool.py'))
spec = importlib.util.spec_from_loader(loader.name, loader)
esptool = importlib.util.module_from_spec(spec)
loader.exec_module(esptool) # after this line, esptool is imported
esptool.main()
Python 2.7
import imp
from distutils.spawn import find_executable as which
if __name__ == '__main__':
esptool = imp.load_source('esptool', which('esptool.py'))
esptool.main()
Passing command line arguments
The command line arguments are stored in sys.argv list, so you will have to temporarily overwrite it in order to pass the arguments to the main function:
# assuming esptool is imported
import sys
if __name__ == '__main__':
# save the current arguments
argv_original = sys.argv[:]
# overwrite arguments to be passed to esptool argparser
sys.argv[:] = ['', '-p', '/dev/ttyUSB0', 'write_flash', '-fm', 'qio', '0x0000']
try:
esptool.main()
except Exception:
# TODO deal with errors here
pass
finally: # restore original arguments
sys.argv[:] = argv_original

run import.py with sys parameters

I am debugging code, which has this line:
run('python /home/some_user/some_repo/pyflights/usertools/import.py /home/some_user/some_repo/pyflights/config/index_import.conf flights.map --import')
run - is some analog of os.system
So, I want to run this code without using run function. I need to import my import.py file and run it with sys.args. But how can I do this?
from some_repo.pyflights.usertools import import

There is no way to import import because import is a keyword. Moreover, importing a python file is different from running a script because most scripts have a section
if __name__ == '__main__':
....
When the program is running as a script, the variable __name__ has value __main__.
If you are ready to call a subprocess, you can use
`subprocess.call(...)`
Edit: actually, you can import import like so
from importlib import import_module
mod = import_module('import')
however it won't have the same effect as calling the script. Notice that the script probably uses sys.argv, and this must be addressed too.
Edit: here is an ersatz that you can try if you really don't want a subprocess. I don't guarantee it will work
import shlex
import sys
import types
def run(args):
"""Runs a python program with arguments within the current process.
Arguments:
#args: a sequence of arguments, the first one must be the file path to the python program
This is not guaranteed to work because the current process and the
executed script could modify the python running environment in incompatible ways.
"""
old_main, sys.modules['__main__'] = sys.modules['__main__'], types.ModuleType('__main__')
old_argv, sys.argv = sys.argv, list(args)
try:
with open(sys.argv[0]) as infile:
source = infile.read()
exec(source, sys.modules['__main__'].__dict__)
except SystemExit as exc:
if exc.code:
raise RuntimeError('run() failed with code %d' % exc.code)
finally:
sys.argv, sys.modules['__main__'] = old_argv, old_main
command = '/home/some_user/some_repo/pyflights/usertools/import.py /home/some_user/some_repo/pyflights/config/index_import.conf flights.map --import'
run(shlex.split(command))

Python Gooey Run Directory

I'm using the latest Gooey library on Python 2.7 on Windows as a GUI for a simple argparse, but for some reason, the script keeps giving me the [Errno2] No File Exists.
I think it is because there is a space in the path of the Anaconda installation (i.e. C:\Users\FirstName LastName\Etc.) but I'm stumped.
I have tried str.replace all the \ with \\, but I keep getting the same error message. Any ideas of what to do?
Code:
from __future__ import print_function
import pandas as pd
import numpy as np
import glob
import sys
import os
import json
from argparse import ArgumentParser
from gooey import Gooey, GooeyParser
#Gooey(program_name="CPT Lookup")
def parse_args():
stored_args = {}
parser = GooeyParser(description='CPT Lookup')
#Eventually make into checkboxes
parser.add_argument('year',
action='store',
default=stored_args.get('year'),
widget='FileChooser',
help="CSV file with extracted year")
parser.add_argument('CPT',
action='store',
default=stored_args.get('CPT'),
widget='TextField',
help='CPT Code')
args = parser.parse_args()
return args
def loadCSV(year):
#DO I DO SOMETHING LIKE YEAR.REPLACE('\','\\')?
df = pd.read_csv(year)
return df
if __name__ == '__main__':
conf = parse_args()
print("Opening CSV file")
sales_df = loadCSV(conf.year)

This was an issue with the Gooey library itself (I'm the author). It wasn't quoting incoming file paths correctly.
If you pull down the latest release from PyPi (pip install -U gooey), your example script should run without issue.

argparse in python3 - cannot add argument with - in name?

This is my sample program:
#!/usr/bin/python
import os
import string
import argparse
parser = argparse.ArgumentParser(description="get all versions")
parser.add_argument(test-version)
version = parser.parse_args()
print(version.test-version)
When I run the python script, it throws the error:
AttributeError: 'Namespace' object has no attribute 'test'
If I try the same example with argument "test" instead of "test-version" it works fine.
It is important for me to have the - in my argument name.
How can I fix this issue?
Thanks

The issue is that you can't have a - in a variable name, because it's the subtraction operator. So, in your code, the interpreter is trying to do version.test - version, which doesn't work because version doesn't have a .test attribute and you can't subtract a namespace from something. If you want to use - in the argument, you need to set the dest= parameter in .add_argument().
For example:
import argparse
parser = argparse.ArgumentParser(description="get all versions")
parser.add_argument('--test-version', dest='test_version')
version = parser.parse_args(['--test-version', '1'])
print(version.test_version)
or if you want it as a status flag:
import argparse
parser = argparse.ArgumentParser(description="get all versions")
parser.add_argument('--test-version', action='store_true', dest='test_version')
version = parser.parse_args(['--test-version'])
print(version.test_version)

Try this:
print(getattr(version, 'test-version'))

Import custom modules on IPython.parallel engines with sync_imports()

I've been playing around with IPython.parallel and I wanted to use some custom modules of my own, but haven't been able to do it as explained on the cookbook using dview.sync_imports(). The only thing that has worked for me was something like
def my_parallel_func(args):
import sys
sys.path.append('/path/to/my/module')
import my_module
#and all the rest
and then in the main just to
if __name__=='__main__':
#set up dview...
dview.map( my_parallel_func, my_args )
The correct way to do this would in my opinion be something like
with dview.sync_imports():
import sys
sys.path.append('/path/to/my/module')
import my_module
but this throws an error saying there is no module named my_module.
So, what is the right way of doing it using dview.sync_imports()??

The problem is that you're changing the PYTHONPATH just in the local process running the Client, and not in the remote processes running in the ipcluster.
You can observe this behaviour if you run the next piece of code:
from IPython.parallel import Client
rc = Client()
dview = rc[:]
with dview.sync_imports():
import sys
sys.path[:] = ['something']
def parallel(x):
import sys
return sys.path
print 'Local: ', sys.path
print 'Remote: ', dview.map_sync(parallel, range(1))
Basically all the modules that you want to use with sync_imports must already be in the PYTHONPATH.
If it's not in the PYTHONPATH then you must add it to the path in the function that you execute remotely, and then import the module in the function.