I need to run a script foo.py, but I need to also insert some debugging lines to run before the code in foo.py. Currently I just put those lines in foo.py and I'm careful not to commit that to Git, but I don't like this solution.
What I want is a separate file bar.py that I don't commit to Git. Then I want to run:
python /somewhere/bar.py /somewhere_else/foo.py
What I want this to do is first run some lines of code in bar.py, and then run foo.py as __main__. It should be in the same process that the bar.py lines ran in, otherwise the debugging lines won't help.
Is there a way to make bar.py do this?
Someone suggested this:
import imp
import sys
# Debugging code here
fp, pathname, description = imp.find_module(sys.argv[1])
imp.load_module('__main__', fp, pathname, description)
The problem with that is that because it uses import machinery, I need to be on the same folder as foo.py to run that. I don't want that. I want to simply put in the full path to foo.py.
Also: The solution needs to work with .pyc files as well.
Python has a mechanism for running code at startup; the site module.
"This module is automatically imported during initialization."
The site module will attempt to import a module named sitecustomize before __main__ is imported.
It will also attempt to import a module named usercustomize if your environment instructs it to.
For example, you could put a sitecustomize.py file in your site-packages folder that contains this:
import imp
import os
if 'MY_STARTUP_FILE' in os.environ:
try:
file_path = os.environ['MY_STARTUP_FILE']
folder, file_name = os.path.split(file_path)
module_name, _ = os.path.splitext(file_name)
fp, pathname, description = imp.find_module(module_name, [folder])
except Exception as e:
# Broad exception handling since sitecustomize exceptions are ignored
print "There was a problem finding startup file", file_path
print repr(e)
exit()
try:
imp.load_module(module_name, fp, pathname, description)
except Exception as e:
print "There was a problem loading startup file: ", file_path
print repr(e)
exit()
finally:
# "the caller is responsible for closing the file argument" from imp docs
if fp:
fp.close()
Then you could run your script like this:
MY_STARTUP_FILE=/somewhere/bar.py python /somewhere_else/foo.py
You could run any script before foo.py without needing to add code to reimport __main__.
Run export MY_STARTUP_FILE=/somewhere/bar.py and not need to reference it every time
You can use execfile() if the file is .py and uncompyle2 if the file is .pyc.
Let's say you have your file structure like:
test|-- foo.py
|-- bar
|--bar.py
foo.py
import sys
a = 1
print ('debugging...')
# run the other file
if sys.argv[1].endswith('.py'): # if .py run right away
execfile(sys.argv[1], globals(), locals())
elif sys.argv[1].endswith('.pyc'): # if .pyc, first uncompyle, then run
import uncompyle2
from StringIO import StringIO
f = StringIO()
uncompyle2.uncompyle_file(sys.argv[1], f)
f.seek(0)
exec(f.read(), globals(), locals())
bar.py
print a
print 'real job'
And in test/, if you do:
$ python foo.py bar/bar.py
$ python foo.py bar/bar.pyc
Both, outputs the same:
debugging...
1
real job
Please also see this answer.
You probably have something along the lines of:
if __name__ == '__main__':
# some code
Instead, write your code in a function main() in foo and then do:
if __name__ == '__main__':
main()
Then, in bar, you can import foo and call foo.main().
Additionaly, if you need to change the working directory, you can use the os.chdir(path) method, e.g. os.chdir('path/of/bar') .
bar.py would have to behave like the Python interpreter itself and run foo.py (or foo.pyc, as you asked for that) as if it was the main script. This is surprisingly hard. My 90% solution to do that looks like so:
def run_script_as_main(cmdline):
# It is crucial to import locally what we need as we
# later remove everything from __main__
import sys, imp, traceback, os
# Patch sys.argv
sys.argv = cmdline
# Clear the __main__ namespace and set it up to run
# the secondary program
maindict = sys.modules["__main__"].__dict__
builtins = maindict['__builtins__']
maindict.clear()
maindict['__file__'] = cmdline[0]
maindict['__builtins__'] = builtins
maindict['__name__'] = "__main__"
maindict['__doc__'] = None
# Python prepends a script's location to sys.path
sys.path[0] = os.path.dirname(os.path.abspath(cmdline[0]))
# Treat everything as a Python source file, except it
# ends in '.pyc'
loader = imp.load_source
if maindict["__file__"].endswith(".pyc"):
loader = imp.load_compiled
with open(cmdline[0], 'rb') as f:
try:
loader('__main__', maindict["__file__"], f)
except Exception:
# In case of an exception, remove this script from the
# stack trace; if you don't care seeing some bar.py in
# traceback, you can leave that out.
ex_type, ex_value, ex_traceback = sys.exc_info()
tb = traceback.extract_tb(ex_traceback, None)[1:]
sys.stderr.write("Traceback (most recent call last):\n")
for line in traceback.format_list(tb):
sys.stderr.write(line)
for line in traceback.format_exception_only(ex_type, ex_value):
sys.stderr.write(line)
This mimics the default behavior of the Python interpreter
relatively closely. It sets system globals __name__, __file__,
sys.argv, inserts the script location into sys.path, clears the global namespace etc.
You would copy that code to bar.py or import it from somewhere and use it like so:
if __name__ == "__main__":
# You debugging setup goes here
...
# Run the Python program given as argv[1]
run_script_as_main(sys.argv[1:])
Then, given this:
$ find so-foo-bar
so-foo-bar
so-foo-bar/debugaid
so-foo-bar/debugaid/bar.py
so-foo-bar/foo.py
and foo.py looking like this:
import sys
if __name__ == "__main__":
print "My name is", __name__
print "My file is", __file__
print "My command line is", sys.argv
print "First 2 path items", sys.path[:2]
print "Globals", globals().keys()
raise Exception("foo")
... running foo.py via bar.py gives you that:
$ python so-foo-bar/debugaid/bar.py so-foo-bar/foo.py
My name is __main__
My file is so-foo-bar/foo.py
My command line is ['so-foo-bar/foo.py']
First 2 path items ['~/so-foo-bar', '/usr/local/...']
Globals ['__builtins__', '__name__', '__file__', 'sys', '__package__', '__doc__']
Traceback (most recent call last):
File "so-foo-bar/foo.py", line 9, in
raise Exception("foo")
Exception: foo
While I'd guess that run_script_as_main is lacking in some interesting details, it's pretty close to the way Python would run foo.py.
The foo.py need not be on the same folder as the main folder. Use
export PYTHONPATH=$HOME/dirWithFoo/:$PYTHONPATH
To add the path that foo resides to Python path. Now you can
from foo import myfunc
myfunc()
And make myfunc do whatever u want
Have you tried this?
bar.py
imp.load_source('__main__', '../../foo.py')
for me it runs whatever is in foo.py (before and inside main)
After, maybe what you do in bar.py might be more tricky that my basic test.
The good thing with load_source is that it imports the .pyc if it already exists or initiates a "compile" of the .py and then imports the .pyc.
This solution is what ended up working well for me:
#!/usr/bin/env python
import imp
import sys
import os.path
file_path = sys.argv.pop(1)
assert os.path.exists(file_path)
folder, file_name = os.path.split(file_path)
###############################################################################
# #
# Debugging cruft here
# #
###############################################################################
module_name, _ = os.path.splitext(file_name)
sys.path.append(folder)
imp.load_module('__main__', *imp.find_module(module_name))
Related
I have a Python project in which I have the following folder structure:
> root
> download_module
> __init__.py
> downloadProcess.py
> sharedFunctions.py
> someHelper.py
> useSharedFunction.py
The download_module/__init__.py has the following code:
from .sharedFunctions import stringArgumentToDate
from .downloadProcess import downloadProcessMethod
The sharedFunctions.py file contains the following function:
def stringArgumentToDate(arg):
dateformat = "%m/%d/%Y"
date = None
if arg.isnumeric():
date = datetime.fromtimestamp(int(arg))
if date == None:
date = datetime.strptime(arg, dateformat)
return date
Then on the useSharedFunction.py I try to import the shared function and use it like this.
from download_module import stringArgumentToDate
from download_module import downloadProcessMethod
def main():
arg = '03/14/2022'
dateArg = stringArgumentToDate(arg)
if __name__ == '__main__':
main()
When I try to run this by using python3 useSharedFunction.py I got the following error:
Traceback (most recent call last):
File "useSharedFunction.py", line 4, in <module>
from download_module import stringArgumentToDate
File "/Users/jacobo/Documents/project/download_module/__init__.py", line 2, in <module>
from .download_module import downloadAndProcessMethod
File "/Users/jacobo/Documents/project/download_module/downloadProcess.py", line 10, in <module>
from sharedFunctions import stringArgumentToDate, otherFunction
ModuleNotFoundError: No module named 'sharedFunctions'
I do believe the error is in downloadProcess since at the beggining of the file we got this import:
from sharedFunctions import stringArgumentToDate, otherFunction
from someHelper import Helper
Which refers to sibling files.
However I'm unsure what will be a proper fix to allow to run the downloadProcess.py main independently but also, being able to call it one of its method from a root or any other file out of the module.
Consider this structure:
┬ module
| ├ __init__.py
| ├ importing_submodule.py
| └ some_submodule.py
├ __main__.py
├ some_submodule.py
└ module_in_parent_dir.py
with content:
__main__.py
import module
/module/__init__.py
from . import importing_submodule
/module/importing_submodule.py
from some_submodule import SomeClass
/module/some_submodule.py
print("you imported from module")
class SomeClass:
pass
/some_submodule.py
print("you imported from root")
class SomeClass:
pass
/module_in_parent_dir.py
class SomeOtherClass:
pass
How sibling import works
(skip this section if you know already)
Now lets run __main__.py and it will say "you imported from root".
But if we change code a bit..
/module/importing_submodule.py
from module.some_submodule import SomeClass
It now says "You imported from module" as we wanted, probably with scary red line in IDE saying "Unresolved reference" if you didn't config working directory in IDE.
How this happen is simple: script root(Current working directory) is decided by main script(first script that's running), and python uses namespaces.
Python's import system uses 2 import method, and for convenience let's call it absolute import and relative import.
Absolute import: Import from dir listed in sys.path and current working directory
Relative import: Import relative to the very script that called import
And what decide the behavior is whether we use . at start of module name or not.
Since we imported by from some_submodule without preceeding dot, python take it as 'Absolute import'(the term we decided earlier).
And then when we also specified module name like from module.some_submodule python looks for module in path list or in current working directory.
Of course, this is never a good idea; script root can change via calls like os.chdir() then submodules inside module folder may get lost.
Therefore, the best practices for sibling import is using relative import inside module folder.
/module/importing_submodule.py
from .some_submodule import SomeClass
Making script that work in both way
To make submodule import it's siblings when running as main script, yet still work as submodule when imported by other script, then use try - except and look for ImportError.
For importing_submodule.py as an example:
/module/importing_submodule.py
try:
from .some_submodule import SomeClass
except ImportError:
# attempted relative import with no known parent package
# because this is running as main script, there's no parent package.
from some_submodule import SomeClass
Importing modules from parent directory is a bit more tricky.
Since submodule is now main script, relative import to parent level directory doesn't work.
So we need to add the parent directory to sys.path, when the script is running as main script.
/module/importing_submodule.py
try:
from .some_submodule import SomeClass
except ImportError:
# attempted relative import with no known parent package
# because this is running as main script, there's no parent package.
from some_submodule import SomeClass
# now since we don't have parent package, we just append the path.
from sys import path
import pathlib
path.append(pathlib.Path(__file__).parent.parent.as_posix())
print("Importing module_in_parent_dir from sys.path")
else:
print("Importing module_in_parent_dir from working directory")
# Now either case we have parent directory of `module_in_parent_dir`
# in working dir or path, we can import it
# might need to suppress false IDE warning this case.
# noinspection PyUnresolvedReferences
from module_in_parent_dir import SomeOtherClass
Output:
"C:\Program Files\Python310\python.exe" .../module/importing_module.py
you imported from module
Importing module_in_parent_dir from sys.path
Process finished with exit code 0
"C:\Program Files\Python310\python.exe" .../__main__.py
you imported from module
Importing module_in_parent_dir from working directory
Process finished with exit code 0
I have a very simple script that i wish to test out.
Master script (caller.py):
#!/usr/bin/env python2.7
import test2
test2.printMe()
Function/Module Script (test2.py):
def printMe():
print 'this actually works'
When I run caller.py, it works. Because the test2.py script exists in the same directory from which I'm calling caller.py.
What happens if i need to make the function(s) defined in test2.py available to the caller.py script, through a variable? There are times when the content of the test2.py function script wont always be in a file. Sometimes, it's inside a variable.
So basically, im looking for a way to access the functions that are in a variable just as import would access functions that are in a file.
I wish to be able to do something like this:
from commands import *
def run_command(cmd):
status, text = getstatusoutput(cmd)
print text
run_command('python /tmp/test2.py')
test2.printMe()
Please advise.
Try this:
from sys import path as sys_path
from os import path
def import_module(path_to_py_file):
dir_name, module_name = path.split(path.splitext(path_to_py_file)[0])
sys_path.append(dir_name)
return __import__(module_name)
Now you can do this:
test2 = import_module(r'/tmp/test2.py')
test2.printMe()
I am debugging code, which has this line:
run('python /home/some_user/some_repo/pyflights/usertools/import.py /home/some_user/some_repo/pyflights/config/index_import.conf flights.map --import')
run - is some analog of os.system
So, I want to run this code without using run function. I need to import my import.py file and run it with sys.args. But how can I do this?
from some_repo.pyflights.usertools import import
There is no way to import import because import is a keyword. Moreover, importing a python file is different from running a script because most scripts have a section
if __name__ == '__main__':
....
When the program is running as a script, the variable __name__ has value __main__.
If you are ready to call a subprocess, you can use
`subprocess.call(...)`
Edit: actually, you can import import like so
from importlib import import_module
mod = import_module('import')
however it won't have the same effect as calling the script. Notice that the script probably uses sys.argv, and this must be addressed too.
Edit: here is an ersatz that you can try if you really don't want a subprocess. I don't guarantee it will work
import shlex
import sys
import types
def run(args):
"""Runs a python program with arguments within the current process.
Arguments:
#args: a sequence of arguments, the first one must be the file path to the python program
This is not guaranteed to work because the current process and the
executed script could modify the python running environment in incompatible ways.
"""
old_main, sys.modules['__main__'] = sys.modules['__main__'], types.ModuleType('__main__')
old_argv, sys.argv = sys.argv, list(args)
try:
with open(sys.argv[0]) as infile:
source = infile.read()
exec(source, sys.modules['__main__'].__dict__)
except SystemExit as exc:
if exc.code:
raise RuntimeError('run() failed with code %d' % exc.code)
finally:
sys.argv, sys.modules['__main__'] = old_argv, old_main
command = '/home/some_user/some_repo/pyflights/usertools/import.py /home/some_user/some_repo/pyflights/config/index_import.conf flights.map --import'
run(shlex.split(command))
I have 2 files in a directory: loader.py & mod1.py. Loader.py dynamically instanciates a class in mod1.py and calls a method on it. Here is mod1.py
class MyClass1:
def run(self):
print "Class1 running"
Here is loader:
def run():
mod = __import__('mod1')
cls = getattr(mod, 'MyClass1')
inst = cls()
inst.run()
run()
If I run this just straight python: "python loader.py" I see:
Class1 running
Which is what you expect. If I then run it under fabric: "fab -f loader.py run" I see
Class1 running
Class1 running
Done.
Which makes sense, run() is called by fabric AND loader.py when it's loaded by fabric.
However, if I remove the explicit call to run in loader.py so it's only called once under fabric, I get
ImportError: No module named mod1
Why does running under fabric make a difference? Is there a way to make this work under fabric?
Here's my insight.
This weird behavior is explained in docs, quote:
Fabric does a normal import (actually an __import__) of your fabfile
in order to access its contents – it does not do any eval-ing or
similar. In order for this to work, Fabric temporarily adds the found
fabfile’s containing folder to the Python load path (and removes it
immediately afterwards.)
Look at the modified loader.py:
import os
import sys
print sys.path # current dir is in sys.path
def run():
print sys.path # current dir is NOT in sys.path
# no problem - we'll just add it
sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)))
mod = __import__('mod1')
cls = getattr(mod, 'MyClass1')
inst = cls()
inst.run()
Looks ugly, but it fixes the problem.
Here's how fabric does manipulations with imports while loading fab file. Also this issue is relevant.
Hope that helps.
Suppose I have two modules:
a.py:
import b
print __name__, __file__
b.py:
print __name__, __file__
I run the "a.py" file. This prints:
b C:\path\to\code\b.py
__main__ C:\path\to\code\a.py
Question: how do I obtain the path to the __main__ module ("a.py" in this case) from within the "b.py" library?
import __main__
print(__main__.__file__)
Perhaps this will do the trick:
import sys
from os import path
print(path.abspath(str(sys.modules['__main__'].__file__)))
Note that, for safety, you should check whether the __main__ module has a __file__ attribute. If it's dynamically created, or is just being run in the interactive python console, it won't have a __file__:
python
>>> import sys
>>> print(str(sys.modules['__main__']))
<module '__main__' (built-in)>
>>> print(str(sys.modules['__main__'].__file__))
AttributeError: 'module' object has no attribute '__file__'
A simple hasattr() check will do the trick to guard against scenario 2 if that's a possibility in your app.
The python code below provides additional functionality, including that it works seamlessly with py2exe executables.
I use similar code to like this to find paths relative to the running script, aka __main__. as an added benefit, it works cross-platform including Windows.
import imp
import os
import sys
def main_is_frozen():
return (hasattr(sys, "frozen") or # new py2exe
hasattr(sys, "importers") # old py2exe
or imp.is_frozen("__main__")) # tools/freeze
def get_main_dir():
if main_is_frozen():
# print 'Running from path', os.path.dirname(sys.executable)
return os.path.dirname(sys.executable)
return os.path.dirname(sys.argv[0])
# find path to where we are running
path_to_script=get_main_dir()
# OPTIONAL:
# add the sibling 'lib' dir to our module search path
lib_path = os.path.join(get_main_dir(), os.path.pardir, 'lib')
sys.path.insert(0, lib_path)
# OPTIONAL:
# use info to find relative data files in 'data' subdir
datafile1 = os.path.join(get_main_dir(), 'data', 'file1')
Hopefully the above example code can provide additional insight into how to determine the path to the running script...
Another method would be to use sys.argv[0].
import os
import sys
main_file = os.path.realpath(sys.argv[0]) if sys.argv[0] else None
sys.argv[0] will be an empty string if Python gets start with -c or if checked from the Python console.
import sys, os
def getExecPath():
try:
sFile = os.path.abspath(sys.modules['__main__'].__file__)
except:
sFile = sys.executable
return os.path.dirname(sFile)
This function will work for Python and Cython compiled programs.
I use this:
import __main__
name = __main__.__file__ # this gets file path
lastBar =int(name[::-1].find('/')) # change '/' for linux or '\' for windows
extension = 3 #3 for python file .py
main_file_name=name[::-1][extension:lastBar][::-1] #result