I am using this approach to get the current directory of a file:
import pathlib
pathlib.Path(__file__).parent.resolve()
Is it possible to extract this string into a function, that will use the path of the caller? My attempt:
# a.py
import bb.b
print(bb.b.get_current_path(__file__)) # OK: path to a.py directory (.)
print(bb.b.get_current_path()) # WRONG: path to b.py directory (bb)
# bb/b.py
import pathlib
def get_current_path(file=__file__):
return pathlib.Path(file).parent.resolve()
Is it possible to avoid using __file__ in a.py?
You can use inspect module to inspect the stack. Though it satisfies the given condition
# bb/b.py
import inspect
import pathlib
stack = inspect.stack()
file_importer = stack[6][1]
def get_current_path(file=file_importer):
return pathlib.Path(file).parent.resolve()
I'm not sure if import will be always at the frame with the index 6. Search through code_context can be used instead:
file_importer = next(frameinfo[1] for frameinfo in stack
if frameinfo[4] and frameinfo[4][0].startswith('import'))
But this approach breaks the possibility to run b.py, so that exception handling of StopIteration is required with desired behaviour.
Both approaches return the direct importer of b.py, e.g., having a file c.py and calling it from a.py returns the path of c.py
# c.py
import bb.b
get_current_path = bb.b.get_current_path
# a.py
import bb.c
print(bb.c.get_current_path(__file__)) # OK: path to a.py directory (.)
print(bb.c.get_current_path()) # WRONG: path to c.py directory (bb)
Hence, depending on further conditions desired behaviour could be reached by processing inspect.stack().
Related
I have a Python project in which I have the following folder structure:
> root
> download_module
> __init__.py
> downloadProcess.py
> sharedFunctions.py
> someHelper.py
> useSharedFunction.py
The download_module/__init__.py has the following code:
from .sharedFunctions import stringArgumentToDate
from .downloadProcess import downloadProcessMethod
The sharedFunctions.py file contains the following function:
def stringArgumentToDate(arg):
dateformat = "%m/%d/%Y"
date = None
if arg.isnumeric():
date = datetime.fromtimestamp(int(arg))
if date == None:
date = datetime.strptime(arg, dateformat)
return date
Then on the useSharedFunction.py I try to import the shared function and use it like this.
from download_module import stringArgumentToDate
from download_module import downloadProcessMethod
def main():
arg = '03/14/2022'
dateArg = stringArgumentToDate(arg)
if __name__ == '__main__':
main()
When I try to run this by using python3 useSharedFunction.py I got the following error:
Traceback (most recent call last):
File "useSharedFunction.py", line 4, in <module>
from download_module import stringArgumentToDate
File "/Users/jacobo/Documents/project/download_module/__init__.py", line 2, in <module>
from .download_module import downloadAndProcessMethod
File "/Users/jacobo/Documents/project/download_module/downloadProcess.py", line 10, in <module>
from sharedFunctions import stringArgumentToDate, otherFunction
ModuleNotFoundError: No module named 'sharedFunctions'
I do believe the error is in downloadProcess since at the beggining of the file we got this import:
from sharedFunctions import stringArgumentToDate, otherFunction
from someHelper import Helper
Which refers to sibling files.
However I'm unsure what will be a proper fix to allow to run the downloadProcess.py main independently but also, being able to call it one of its method from a root or any other file out of the module.
Consider this structure:
┬ module
| ├ __init__.py
| ├ importing_submodule.py
| └ some_submodule.py
├ __main__.py
├ some_submodule.py
└ module_in_parent_dir.py
with content:
__main__.py
import module
/module/__init__.py
from . import importing_submodule
/module/importing_submodule.py
from some_submodule import SomeClass
/module/some_submodule.py
print("you imported from module")
class SomeClass:
pass
/some_submodule.py
print("you imported from root")
class SomeClass:
pass
/module_in_parent_dir.py
class SomeOtherClass:
pass
How sibling import works
(skip this section if you know already)
Now lets run __main__.py and it will say "you imported from root".
But if we change code a bit..
/module/importing_submodule.py
from module.some_submodule import SomeClass
It now says "You imported from module" as we wanted, probably with scary red line in IDE saying "Unresolved reference" if you didn't config working directory in IDE.
How this happen is simple: script root(Current working directory) is decided by main script(first script that's running), and python uses namespaces.
Python's import system uses 2 import method, and for convenience let's call it absolute import and relative import.
Absolute import: Import from dir listed in sys.path and current working directory
Relative import: Import relative to the very script that called import
And what decide the behavior is whether we use . at start of module name or not.
Since we imported by from some_submodule without preceeding dot, python take it as 'Absolute import'(the term we decided earlier).
And then when we also specified module name like from module.some_submodule python looks for module in path list or in current working directory.
Of course, this is never a good idea; script root can change via calls like os.chdir() then submodules inside module folder may get lost.
Therefore, the best practices for sibling import is using relative import inside module folder.
/module/importing_submodule.py
from .some_submodule import SomeClass
Making script that work in both way
To make submodule import it's siblings when running as main script, yet still work as submodule when imported by other script, then use try - except and look for ImportError.
For importing_submodule.py as an example:
/module/importing_submodule.py
try:
from .some_submodule import SomeClass
except ImportError:
# attempted relative import with no known parent package
# because this is running as main script, there's no parent package.
from some_submodule import SomeClass
Importing modules from parent directory is a bit more tricky.
Since submodule is now main script, relative import to parent level directory doesn't work.
So we need to add the parent directory to sys.path, when the script is running as main script.
/module/importing_submodule.py
try:
from .some_submodule import SomeClass
except ImportError:
# attempted relative import with no known parent package
# because this is running as main script, there's no parent package.
from some_submodule import SomeClass
# now since we don't have parent package, we just append the path.
from sys import path
import pathlib
path.append(pathlib.Path(__file__).parent.parent.as_posix())
print("Importing module_in_parent_dir from sys.path")
else:
print("Importing module_in_parent_dir from working directory")
# Now either case we have parent directory of `module_in_parent_dir`
# in working dir or path, we can import it
# might need to suppress false IDE warning this case.
# noinspection PyUnresolvedReferences
from module_in_parent_dir import SomeOtherClass
Output:
"C:\Program Files\Python310\python.exe" .../module/importing_module.py
you imported from module
Importing module_in_parent_dir from sys.path
Process finished with exit code 0
"C:\Program Files\Python310\python.exe" .../__main__.py
you imported from module
Importing module_in_parent_dir from working directory
Process finished with exit code 0
My directory looks like this :
- HttpExample:
- __init__.py
- DBConnection.py
- getLatlong.py
I want to import DBConnection and import getLatlong in __init__.py. There is no error in my __init__.py until I run it, I received :
System.Private.CoreLib: Exception while executing function: Functions.HttpExample. System.Private.CoreLib: Result: Failure
Exception: ModuleNotFoundError: No module named 'getLatlong'
I'm trying to use function in getLatlong to use the information input by user from __init__.py to getLatlong. Below is the code:
__init__.py :
from getLatlong import result
from DBConnection import blobService, container_name, account_key, file_path
def main(req: func.HttpRequest) -> func.HttpResponse:
logging.info('Python HTTP trigger function processed a request.')
section = req.params.get('section')
bound = req.params.get('bound')
km_location = req.params.get('km_location')
location = req.params.get('location')
if not section:
try:
req_body = req.get_json()
except ValueError:
pass
else:
section = req_body.get('section')
if section and bound and km_location:
result(section, km_location, bound, location).getResult() #HERE
return func.HttpResponse(f"Hello {section}, {bound}!")
#elif section is None or bound is None or km_location is None:
# return func.HttpResponse("All values are mandatory!")
I am also receiving compile error at getLatlong to import DBConnection to this class. The following values will pass to getLatlong.py. The code :
from DBConnection import blobService, container_name, account_key, file_path #Another import error here says : Unable to import DBConnection
class result:
def __init__(self, section, bound, km_location, location):
self.section = section
self.bound = bound
self.km_location = km_location
self.location = location
def getResult(self):
print(self.section)
print(self.bound)
print(self.km_location)
print(self.location)
I've tried every way to import these files before I lost my mind..
You get these errors, because Python does not know where to look for the files you want to import. Depending on which Python version you are using, I see three ways to solve this:
You could add HttpExample to your PYTHONPATH and than your imports should work as you have them currently.
Another way would be to use the sys module and append the path to HttpExample, e.g.
import sys
sys.path.append('PATH/TO/HttpExample')
But you would have to do this in all files, where you want to import something from the parent folder.
Or you use relative imports, which have been available since Python 2.5 (See PEP238). Those are only available in modules, but as you have your __init__.py file, it should work. For relative imports you are using dots . to tell Python where to look for the import. One dot . tells Python to look for the desired import in the parent folder. You could also use .. to go up two levels. But one level should be enough in your case.
So in your case changing your code to this, should solve your problem.
In __init.py__:
from .getLatlong import result
from .DBConnection import blobService, container_name, account_key, file_path
In getLangLong.py:
from .DBConnection import blobService, container_name, account_key, file_path
You could try from __app__.HttpExample import getLatlong.
There is a document about how to import module in the Shared Code folder. Check this doc:Folder structure.
It says Shared code should be kept in a separate folder in __app__. And in my test this could work for me.
I have just started a python project. The directory structure is as follows:
/algorithms
----/__init__.py
----/linkedlist
--------/__init__.py
--------/file1.py
--------/file2.py
/tests
----/test_linkedlist
You can also check the Github repository.
In each of the sub folders under algorithms, in the __init__ file I am including the following for all the files one by one:
from .file1 import *
from .file2 import *
And so on.
The task that I am trying to achieve is running all tests together using the query:
python3 -m unittest discover tests
Each file in the tests directory starts as follows:
from algorithms.linkedlist import *
import unittest
Right now if I want to add a new file to the linkedlist directory, I create the file and then add another from .filename import * in the __init__ file.
How do I write a script in the __init__ file so that each time I create a new file, I do not have to manually insert the import command?
So the __init__ is in the same folder? As the docs say The import statement is syntactic sugar for the __import__ function.
So we can use:
import importlib
import glob
for file in glob.iglob('*.py'):
importlib.__import__(file)
Some reasons why this does not work:
You want to load the functions in the module - the import * from syntax. With this code you can only run file1.test.
You run the script loading from another directory, which confuses glob. We have to specify the actual working directory.
__import__ prefers to know the module name.
To find the solution I combine the import * from function from this answer with pkgutil.walk_packages from this blog.
import importlib
import pkgutil
def custom_import_all(module_name):
""" Use to dynamically execute from module_name import * """
# get a handle on the module
mdl = importlib.import_module(module_name)
# is there an __all__? if so respect it
if "__all__" in mdl.__dict__:
names = mdl.__dict__["__all__"]
else:
# otherwise we import all names that don't begin with _
names = [x for x in mdl.__dict__ if not x.startswith("_")]
# now drag them in
globals().update({k: getattr(mdl, k) for k in names})
__path__ = pkgutil.extend_path(__path__, __name__)
for importer, modname, ispkg in pkgutil.walk_packages(path=__path__, prefix=__name__+'.'):
custom_import_all(modname)
I need to run a script foo.py, but I need to also insert some debugging lines to run before the code in foo.py. Currently I just put those lines in foo.py and I'm careful not to commit that to Git, but I don't like this solution.
What I want is a separate file bar.py that I don't commit to Git. Then I want to run:
python /somewhere/bar.py /somewhere_else/foo.py
What I want this to do is first run some lines of code in bar.py, and then run foo.py as __main__. It should be in the same process that the bar.py lines ran in, otherwise the debugging lines won't help.
Is there a way to make bar.py do this?
Someone suggested this:
import imp
import sys
# Debugging code here
fp, pathname, description = imp.find_module(sys.argv[1])
imp.load_module('__main__', fp, pathname, description)
The problem with that is that because it uses import machinery, I need to be on the same folder as foo.py to run that. I don't want that. I want to simply put in the full path to foo.py.
Also: The solution needs to work with .pyc files as well.
Python has a mechanism for running code at startup; the site module.
"This module is automatically imported during initialization."
The site module will attempt to import a module named sitecustomize before __main__ is imported.
It will also attempt to import a module named usercustomize if your environment instructs it to.
For example, you could put a sitecustomize.py file in your site-packages folder that contains this:
import imp
import os
if 'MY_STARTUP_FILE' in os.environ:
try:
file_path = os.environ['MY_STARTUP_FILE']
folder, file_name = os.path.split(file_path)
module_name, _ = os.path.splitext(file_name)
fp, pathname, description = imp.find_module(module_name, [folder])
except Exception as e:
# Broad exception handling since sitecustomize exceptions are ignored
print "There was a problem finding startup file", file_path
print repr(e)
exit()
try:
imp.load_module(module_name, fp, pathname, description)
except Exception as e:
print "There was a problem loading startup file: ", file_path
print repr(e)
exit()
finally:
# "the caller is responsible for closing the file argument" from imp docs
if fp:
fp.close()
Then you could run your script like this:
MY_STARTUP_FILE=/somewhere/bar.py python /somewhere_else/foo.py
You could run any script before foo.py without needing to add code to reimport __main__.
Run export MY_STARTUP_FILE=/somewhere/bar.py and not need to reference it every time
You can use execfile() if the file is .py and uncompyle2 if the file is .pyc.
Let's say you have your file structure like:
test|-- foo.py
|-- bar
|--bar.py
foo.py
import sys
a = 1
print ('debugging...')
# run the other file
if sys.argv[1].endswith('.py'): # if .py run right away
execfile(sys.argv[1], globals(), locals())
elif sys.argv[1].endswith('.pyc'): # if .pyc, first uncompyle, then run
import uncompyle2
from StringIO import StringIO
f = StringIO()
uncompyle2.uncompyle_file(sys.argv[1], f)
f.seek(0)
exec(f.read(), globals(), locals())
bar.py
print a
print 'real job'
And in test/, if you do:
$ python foo.py bar/bar.py
$ python foo.py bar/bar.pyc
Both, outputs the same:
debugging...
1
real job
Please also see this answer.
You probably have something along the lines of:
if __name__ == '__main__':
# some code
Instead, write your code in a function main() in foo and then do:
if __name__ == '__main__':
main()
Then, in bar, you can import foo and call foo.main().
Additionaly, if you need to change the working directory, you can use the os.chdir(path) method, e.g. os.chdir('path/of/bar') .
bar.py would have to behave like the Python interpreter itself and run foo.py (or foo.pyc, as you asked for that) as if it was the main script. This is surprisingly hard. My 90% solution to do that looks like so:
def run_script_as_main(cmdline):
# It is crucial to import locally what we need as we
# later remove everything from __main__
import sys, imp, traceback, os
# Patch sys.argv
sys.argv = cmdline
# Clear the __main__ namespace and set it up to run
# the secondary program
maindict = sys.modules["__main__"].__dict__
builtins = maindict['__builtins__']
maindict.clear()
maindict['__file__'] = cmdline[0]
maindict['__builtins__'] = builtins
maindict['__name__'] = "__main__"
maindict['__doc__'] = None
# Python prepends a script's location to sys.path
sys.path[0] = os.path.dirname(os.path.abspath(cmdline[0]))
# Treat everything as a Python source file, except it
# ends in '.pyc'
loader = imp.load_source
if maindict["__file__"].endswith(".pyc"):
loader = imp.load_compiled
with open(cmdline[0], 'rb') as f:
try:
loader('__main__', maindict["__file__"], f)
except Exception:
# In case of an exception, remove this script from the
# stack trace; if you don't care seeing some bar.py in
# traceback, you can leave that out.
ex_type, ex_value, ex_traceback = sys.exc_info()
tb = traceback.extract_tb(ex_traceback, None)[1:]
sys.stderr.write("Traceback (most recent call last):\n")
for line in traceback.format_list(tb):
sys.stderr.write(line)
for line in traceback.format_exception_only(ex_type, ex_value):
sys.stderr.write(line)
This mimics the default behavior of the Python interpreter
relatively closely. It sets system globals __name__, __file__,
sys.argv, inserts the script location into sys.path, clears the global namespace etc.
You would copy that code to bar.py or import it from somewhere and use it like so:
if __name__ == "__main__":
# You debugging setup goes here
...
# Run the Python program given as argv[1]
run_script_as_main(sys.argv[1:])
Then, given this:
$ find so-foo-bar
so-foo-bar
so-foo-bar/debugaid
so-foo-bar/debugaid/bar.py
so-foo-bar/foo.py
and foo.py looking like this:
import sys
if __name__ == "__main__":
print "My name is", __name__
print "My file is", __file__
print "My command line is", sys.argv
print "First 2 path items", sys.path[:2]
print "Globals", globals().keys()
raise Exception("foo")
... running foo.py via bar.py gives you that:
$ python so-foo-bar/debugaid/bar.py so-foo-bar/foo.py
My name is __main__
My file is so-foo-bar/foo.py
My command line is ['so-foo-bar/foo.py']
First 2 path items ['~/so-foo-bar', '/usr/local/...']
Globals ['__builtins__', '__name__', '__file__', 'sys', '__package__', '__doc__']
Traceback (most recent call last):
File "so-foo-bar/foo.py", line 9, in
raise Exception("foo")
Exception: foo
While I'd guess that run_script_as_main is lacking in some interesting details, it's pretty close to the way Python would run foo.py.
The foo.py need not be on the same folder as the main folder. Use
export PYTHONPATH=$HOME/dirWithFoo/:$PYTHONPATH
To add the path that foo resides to Python path. Now you can
from foo import myfunc
myfunc()
And make myfunc do whatever u want
Have you tried this?
bar.py
imp.load_source('__main__', '../../foo.py')
for me it runs whatever is in foo.py (before and inside main)
After, maybe what you do in bar.py might be more tricky that my basic test.
The good thing with load_source is that it imports the .pyc if it already exists or initiates a "compile" of the .py and then imports the .pyc.
This solution is what ended up working well for me:
#!/usr/bin/env python
import imp
import sys
import os.path
file_path = sys.argv.pop(1)
assert os.path.exists(file_path)
folder, file_name = os.path.split(file_path)
###############################################################################
# #
# Debugging cruft here
# #
###############################################################################
module_name, _ = os.path.splitext(file_name)
sys.path.append(folder)
imp.load_module('__main__', *imp.find_module(module_name))
i have a 2.6 python script and library in the following directory structure:
+ bin
\- foo.py
+ lib
\+ foo
\- bar.py
i would like users to run bin/foo.py to instantiate the classes within lib/foo.py. to achieve this, in my bin/foo.py script i have the following code:
from __future__ import absolute_import
import foo
klass = foo.bar.Klass()
however, this results in:
AttributeError: 'module' object has no attribute 'bar'
ie it thinks that foo is itself rather than the library foo - renaming bin/foo.py to bin/foo-script.py works as expected.
is there a way i can keep the bin/foo.py script and import lib/foo.py?
The current directory is on the path by default, so you need to remove that before you import the other foo module:
import sys
sys.path = [dir for dir in sys.path if dir != '']
Alternatively, prepend the lib directory so that it takes precedence:
import sys
sys.path = ['../lib'] + sys.path
If you just write import foo, it will definitely load the foo module in the current scope. Assuming lib and foo as packages, won't you need to write something like this in order to make it work?
import lib.foo.bar as foobar
klass = foobar.Klass()