Develop Reference

Is there a generic way to execute different python files, depending on sys.args?

I would like to create a python file that can be run from the terminal - this file will be in charge of running various other python files depending on the functionality required along with their required arguments, respectively. For example, this is the main file:
import sys
from midi_to_audio import arguments, run
files = ["midi_to_audio.py"]
def main(file, args):
if file == "midi_to_audio.py":
if len(args) != arguments:
print("Incorrect argument length")
else:
run("test","t")
if __name__ == '__main__':
sys.argv.pop(0)
file = sys.argv[0]
sys.argv.pop(0)
if file not in files:
print("File does not exist")
else:
main(file, sys.argv)
And this is the first file used in the example (midi_to_audio.py):
arguments = 2
def run(file, output_file):
print("Ran this method")
So depending on which file I've specified when running the cmd via the terminal, it will go into a different file and call its run method. If the arguments are not as required in each file, it will not run
For example: >python main.py midi_to_audio.py file_name_here output_name_here
My problem is that, as I add more files with their own "arguments" and "run" functions, I wonder if python is going to get confused with which arguments or which run function to execute. Is there a more safer/generic way of doing this?
Also, is there a way of getting the names of the python files depending on which files I've imported? Because for now I have to import the file and manually add their file name to the files list in main.py

Your runner could look like this, to load a module by name and check it has run, and check the arguments given on the command line, and finally dispatch to the module's run function.
import sys
import importlib
def main():
args = sys.argv[1:]
if len(args) < 1:
raise Exception("No module name given")
module_name = args.pop(0).removesuffix(".py") # grab the first argument and remove the .py suffix
module = importlib.import_module(module_name) # import a module by name
if not hasattr(module, 'run'): # check if the module has a run function
raise Exception(f"Module {module_name} does not have a run function")
arg_count = getattr(module, 'arguments', 0) # get the number of arguments the module needs
if len(args) != arg_count:
raise Exception(f"Module {module_name} requires {arg_count} arguments, got {len(args)}")
module.run(*args)
if __name__ == '__main__':
main()
This works with the midi_to_audio.py module in your post.

make Python3 script as a CLI

I have a single python3 script which has the following structure. I wish to make this code available as a CLI utility (and not a python3 module) via pip. The reason it not being a python3 module is because the logic is very straight forward and I see no benefit refactoring the code into smaller python files to make it a module.
Code deflection.py
def func1():
"""some useful function here"""
def main(args):
""" My MAIN Logic here!!"""
def parse_args():
"""Parse Arguments if Passed else use configuration file"""
parser = argparse.ArgumentParser(description='what the CLI should do.')
parser.add_argument('--ip', type=str, required=False, help='descp#1')
# Add more arguments (trimmed for code brevity)
return parser.parse_args()
if __name__ == '__main__':
args = parse_args()
CONF = dict() # create a dict for reading a `conf.json` file from `/etc/` folder
with open(CONF_PATH) as cFile:
_conf = json.load(cFile)
CONF = _conf['Key_Of_Interest']
# Check Argument conditions
if condition_1:
print('Starting Script in Default Mode. Reading Conf File conf.json')
try:
main(...) # pass all the default args here
except KeyboardInterrupt as e:
# if CTRL C pressed safe exit
sys.exit(0)
elif condition_2:
# if one particular argument wasn't mentioned, stop script
sys.exit(1)
else:
print('Starting Script with Custom Arguments.')
try:
main(..) # custom args to main function
except KeyboardInterrupt as e:
# safe exit if CTRL C pressed
sys.exit(0)
I am following the Python-Packaging Tutorial which does mention CLI for python modules.
Current Directory Structure
.
|-- bin
| `-- deflection
|-- deflection
| |-- deflection.py
| `-- __init__.py
|-- MANIFEST.in
|-- README.rst
`-- setup.py
setup.py
from setuptools import setup
def readme():
with open('README.rst') as f:
return f.read()
setup(name='deflection',
version='0.1',
description='Extract Micro-Epsilon OptoNCDT values and store into InfluxDB',
long_description=readme(),
url='https://mypersonalgitlabLink.com/awesomeCLIProject',
author='Monty Python',
author_email='Monty#python.org',
license='GPLv3',
packages=['deflection'],
scripts=['bin/deflection']
install_requires=[
'influxdb-python'
],
zip_safe=False)
At this point I am not sure what should be written in bin/deflection file?
#!/usr/bin/env python3
from .deflection import main # NOT SURE Here! because main() requires arguments
I can decide simply upon chmod +x deflection.py but I have a dependency of influxdb-python which I wish to ship via pip i.e. when one does
`pip3 install deflection`
the users can directly do $ deflection --arg1='test' and use the script.
How do I achieve this without using click or any other helper modules and stick to core pip?

At this point I am not sure what should be written in bin/deflection file?
Nothing. You shouldn't ship the executable inside the bin folder of your source tree. The executable will be created upon installation.
I suggest you to use flit and pyproject.toml. It will greatly simplifies your project. First add a pyproject.toml file (instead of setup.py):
[build-system]
requires = ['flit']
build-backend = 'flit.buildapi'
[tool.flit.metadata]
module = 'deflection'
requires-python = '>=3'
description-file = 'README.rst'
requires = ['influxdb-python']
[tool.flit.scripts]
deflection = 'deflection.deflection:main'
Then upload your code to PyPI with flit publish.

as mentioned by #schlamar in the comment section:
I added all the code within the __name__=='__main__' block into a standalone function called main() and renamed the main(args) function to send_data(args).
def send_data(args):
""" refactor the function name"""
def main():
args = parse_args()
CONF = dict() # create a dict for reading a `conf.json` file from `/etc/` folder
with open(CONF_PATH) as cFile:
_conf = json.load(cFile)
CONF = _conf['Key_Of_Interest']
# Check Argument conditions
if condition_1:
print('Starting Script in Default Mode. Reading Conf File conf.json')
try:
send_data(...) # pass all the default args here
except KeyboardInterrupt as e:
# if CTRL C pressed safe exit
sys.exit(0)
elif condition_2:
# if one particular argument wasn't mentioned, stop script
sys.exit(1)
else:
print('Starting Script with Custom Arguments.')
try:
send_data(..) # custom args to main function
except KeyboardInterrupt as e:
# safe exit if CTRL C pressed
sys.exit(0)
in my bin/deflection I added
#!/usr/bin/env python3
import deflection
if __name__ == '__main__':
deflection.main()
Works all well now when I checked it in virtualenv using pip install . in the repo and $ deflection to check if it runs

How to compile multiple subprocess python files into single .exe file using pyinstaller

I have a similar question to this one:Similar Question.
I have a GUI and where the user can input information and the other scripts use some of that information to run.I have 4 different scripts for each button. I run them as a subprocess so that the main gui doesn’t act up or say that it’s not responding. This is an example of what I have since the code is really long since I used PAGE to generate the gui.
###Main.py#####
import subprocess
def resource_path(relative_path):
#I got this from another post to include images but I'm also using it to include the scripts"
try:
# PyInstaller creates a temp folder and stores path in _MEIPASS
base_path = sys._MEIPASS
except Exception:
base_path = os.path.abspath(".")
return os.path.join(base_path, relative_path)
Class aclass:
def get_info(self):
global ModelNumber, Serial,SpecFile,dateprint,Oper,outputfolder
ModelNumber=self.Model.get()
Serial=self.SerialNumber.get()
outputfolder=self.TEntry2.get()
SpecFile= self.Spec_File.get()
return ModelNumber,Serial,SpecFile,outputfolder
def First(self):
aclass.get_info(self) #Where I use the resource path function
First_proc = subprocess.Popen([sys.executable, resource_path('first.py'),str(ModelNumber),str(Serial),str(path),str(outputfolder)])
First_proc.wait()
#####First.py#####
import numpy as np
import scipy
from main import aclass
ModelNumber = sys.argv[1]
Serial = sys.argv[2]
path = sys.argv[3]
path_save = sys.argv[4]
and this goes on for my second,third, and fourth scripts.
In my spec file, I added:
a.datas +=[('first.py','C\\path\\to\\script\\first.py','DATA')]
a.datas +=[('main.py','C\\path\\to\\script\\main.py','DATA')]
this compiles and it works, but when I try to convert it to an .exe, it crashes because it can't import first.py properly and its own libraries (numpy,scipy....etc). I've tried adding it to the a.datas, and runtime_hooks=['first.py'] in the spec file...and I can't get it to work. Any ideas? I'm not sure if it's giving me this error because it is a subprocess.

Assuming you can't restructure your app so this isn't necessary (e.g., by using multiprocessing instead of subprocess), there are three solutions:
Ensure that the .exe contains the scripts as an (executable) zipfile—or just use pkg_resources—and copy the script out to a temporary directory so you can run it from there.
Write a multi-entrypoint wrapper script that can be run as your main program, and also run as each script—because, while you can't run a script out of the packed exe, you can import a module out of it.
Using pkg_resources again, write a wrapper that runs the script by loading it as a string and running it with exec instead.
The second one is probably the cleanest, but it is a bit of work. And, while we could rely on setuptools entrypoints to some of the work, trying to explain how to do this is much harder than explaining how to do it manually,1 so I'm going to do the latter.
Let's say your code looked like this:
# main.py
import subprocess
import sys
spam, eggs = sys.argv[1], sys.argv[2]
subprocess.run([sys.executable, 'vikings.py', spam])
subprocess.run([sys.executable, 'waitress.py', spam, eggs])
# vikings.py
import sys
print(' '.join(['spam'] * int(sys.argv[1])))
# waitress.py
import sys
import time
spam, eggs = int(sys.argv[1]), int(sys.argv[2]))
if eggs > spam:
print("You can't have more eggs than spam!")
sys.exit(2)
print("Frying...")
time.sleep(2)
raise Exception("This sketch is getting too silly!")
So, you run it like this:
$ python3 main.py 3 4
spam spam spam
You can't have more eggs than spam!
We want to reorganize it so there's a script that looks at the command-line arguments to decide what to import. Here's the smallest change to do that:
# main.py
import subprocess
import sys
if sys.argv[1][:2] == '--':
script = sys.argv[1][2:]
if script == 'vikings':
import vikings
vikings.run(*sys.argv[2:])
elif script == 'waitress':
import waitress
waitress.run(*sys.argv[2:])
else:
raise Exception(f'Unknown script {script}')
else:
spam, eggs = sys.argv[1], sys.argv[2]
subprocess.run([sys.executable, __file__, '--vikings', spam])
subprocess.run([sys.executable, __file__, '--waitress', spam, eggs])
# vikings.py
def run(spam):
print(' '.join(['spam'] * int(spam)))
# waitress.py
import sys
import time
def run(spam, eggs):
spam, eggs = int(spam), int(eggs)
if eggs > spam:
print("You can't have more eggs than spam!")
sys.exit(2)
print("Frying...")
time.sleep(2)
raise Exception("This sketch is getting too silly!")
And now:
$ python3 main.py 3 4
spam spam spam
You can't have more eggs than spam!
A few changes you might want to consider in real life:
DRY: We have the same three lines of code copied and pasted for each script, and we have to type each script name three times. You can just use something like __import__(sys.argv[1][2:]).run(sys.argv[2:]) with appropriate error handling.
Use argparse instead of this hacky special casing for the first argument. If you're already sending non-trivial arguments to the scripts, you're probably already using argparse or an alternative anyway.
Add an if __name__ == '__main__': block to each script that just calls run(sys.argv[1:]), so that during development you can still run the scripts directly to test them.
I didn't do any of these because they'd obscure the idea for this trivial example.
1 The documentation is great as a refresher if you've already done it, but as a tutorial and explanatory rationale, not so much. And trying to write the tutorial that the brilliant PyPA guys haven't been able to come up with for years… that's probably beyond the scope of an SO answer.

How do I make PyCharm show code coverage of programs that use multiple processes?

Let's say I create this simple module and call it MyModule.py:
import threading
import multiprocessing
import time
def workerThreaded():
print 'thread working...'
time.sleep(2)
print 'thread complete'
def workerProcessed():
print 'process working...'
time.sleep(2)
print 'process complete'
def main():
workerThread = threading.Thread(target=workerThreaded)
workerThread.start()
workerProcess = multiprocessing.Process(target=workerProcessed)
workerProcess.start()
workerThread.join()
workerProcess.join()
if __name__ == '__main__':
main()
And then I throw this together to unit test it:
import unittest
import MyModule
class MyModuleTester(unittest.TestCase):
def testMyModule(self):
MyModule.main()
unittest.main()
(I know this isn't a good unit test because it doesn't actually TEST it, it just runs it, but that's not relevant to my question)
If I run this unit test in PyCharm with code coverage, then it only shows the code inside the workerThreaded() and main() functions as being covered, even though it clearly covers the workerProcessed() function as well.
How do I get PyCharm to include code that was started in a new process process in its code coverage? Also, how can I get it to include the if __name__ == '__main__': block as well?
I'm running PyCharm 2.7.3, as well as Python 2.7.3.

Coverage.py can measure code run in subprocesses, details are at http://nedbatchelder.com/code/coverage/subprocess.html

I managed to make it work with subprocesses, not sure if this will work with threads or with python 2.
Create .covergerc file in your project root
[run]
concurrency=multiprocessing
Create sitecustomize.py file in your project root
import atexit
from glob import glob
import os
from functools import partial
from shutil import copyfile
from tempfile import mktemp
def combine_coverage(coverage_pattern, xml_pattern, old_coverage, old_xml):
from coverage.cmdline import main
# Find newly created coverage files
coverage_files = [file for file in glob(coverage_pattern) if file not in old_coverage]
xml_files = [file for file in glob(xml_pattern) if file not in old_xml]
if not coverage_files:
raise Exception("No coverage files generated!")
if not xml_files:
raise Exception("No coverage xml file generated!")
# Combine all coverage files
main(["combine", *coverage_files])
# Convert them to xml
main(["xml"])
# Copy combined xml file over PyCharm generated one
copyfile('coverage.xml', xml_files[0])
os.remove('coverage.xml')
def enable_coverage():
import coverage
# Enable subprocess monitoring by providing rc file and enable coverage collecting
os.environ['COVERAGE_PROCESS_START'] = os.path.join(os.path.dirname(__file__), '.coveragerc')
coverage.process_startup()
# Get current coverage files so we can process only newly created ones
temp_root = os.path.dirname(mktemp())
coverage_pattern = '%s/pycharm-coverage*.coverage*' % temp_root
xml_pattern = '%s/pycharm-coverage*.xml' % temp_root
old_coverage = glob(coverage_pattern)
old_xml = glob(xml_pattern)
# Register atexit handler to collect coverage files when python is shutting down
atexit.register(partial(combine_coverage, coverage_pattern, xml_pattern, old_coverage, old_xml))
if os.getenv('PYCHARM_RUN_COVERAGE'):
enable_coverage()
This basically detects if the code is running in PyCharm Coverage and collects newly generated coverage files. There are multiple files, one for the main process and one for each subprocess. So we need to combine them with "coverage combine" then convert them to xml with "coverage xml" and copy the resulted file over PyCharm's generated xml file.
Note that if you kill the child process in you tests coverage.py will not write the data file.
It does not require anything else just hit "Run unittests with Coverage" button in PyCharm.
That's it.

Executing code in __init__

I'm trying to untar xz/bx2/gz files in the init section of my class. I'm using the following code :
class myClass(object):
def __init__(self, *args):
for i in args:
try:
f = tarfile.open(i)
print("Extracting ", i)
f.extractall()
f.close()
except tarfile.ReadError:
print("File not a tarball, or any of .xz/.bz2/.gz archives.")
exit()
if __name__ == "__main__":
<???>
The only problem here is, I'm not sure what to call after "main", in order to initialize and run the init method. I've just started out, and am a bit unclear.
If I'm to write a function named unpack() which does the untarring rather than putting it under init, i know i can do something like :
if __name__ == "__main__":
start = myClass()
start.unpack()
Since I want to do the unpacking in init itself, how would I do it in this case ?
Edit:
Sorry in case I'm not clear, I'm trying to run this script from the command line as :
# python script.py file1.tar.bz2 file2.tar.bz2 file3.tar.bz2
So the *args should be populated with the file names, and hence the code to extract it should run, atleast from what I know.
Thank you,

You just call myClass() :
if __name__ == "__main__":
start = myClass(sys.argv)