I want to know abspath of python script followed by steps below.
built it to byte code by py_compile.
execute it to check abspath.
But I got 2 results when I execute it.I found the results based on the path of script followed by py_compile.
Here is my script test.py :
import os
import inspect
print os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
Build it with py_compile, then got 2 results when I enter different path of test.py:
1.enter the folder and compile with only script name.Then chdir to execute
[~]cd /usr/local/bin/
[/usr/local/bin/]python -m py_compile test.py
[/usr/local/bin/]cd ~
[~]python /usr/local/bin/test.pyc
/home/UserXX
2.In other folder and compile with absolute script name.
[~]python -m py_compile /usr/local/bin/test.py
[~]python /usr/local/bin/test.pyc
/usr/local/bin
how come got 2 different results?
When we want to get the path of one python file, typically we can use any of following methods:
1.
a)
import os
import inspect
print os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe()))).replace('\\', '/')
b)
import os
import inspect
print os.path.dirname(os.path.abspath(inspect.stack()[0][1])).replace('\\', '/')
2.
a)
import os
import sys
print os.path.dirname(os.path.abspath(__file__)).replace('\\', '/')
b)
import os
import sys
print os.path.dirname(os.path.abspath(sys.argv[0])).replace('\\', '/')
For most of scenarios, we use 1 is enough, we seldom to use inspect like 2, as inspect maybe slower.
When will we use 2? Say use inspect to get file path?
One scenario I can remember is execfile, when fileA.py execfile fileB.py in its program, and fileA & fileB not in the same folder. (Maybe more scenario)
Then if we use __file__ in fileB.py, you will find its directory is just same as fileA.py, because here the directory will be the caller's directory.
Then we had to use inspect to get fileB.py's directory.
Anyway, for your situation, if your test.py just at the top of callgraph, just suggest you to use __file__, it's quicker, no need to use inspect.
With this if you use -m py_compile, it works for you.
Finally, why inspect not work with -m py_compile?
Unfortunately, I did not find official document to explain this.
But suppose your test.py is in the folder tt, then let's do cd ..; python -m py_compile tt/test.py, you will get a test.pyc in tt folder.
Let open this pyc file, although you will see something not suitable for man to read, you still can find some clue:
One line is something like:
currentframe(^#^#^#^#(^#^#^#^#(^#^#^#^#s^G^#^#^#tt/a.pyt
Do you see the folder name tt already in pyc file?
If you use inspect.stack() to test, it will more clear, print inspect.stack()[0][1] will always take your current compile folder in pyc file if you use -m py_compile.
This directly means during the process of py_compile, something was fixed to pyc file. This something I call fix makes you can just run your program in the same folder you do -m py_compile.
Hope this can give you some clue & helps you.
Related
I have a very simple scenario like this:
example/
common.py
folder1/
script.py
where script.py file should import common.py module.
If I cd into folder1/ and run the script (i.e. by calling python3 script.py on the command line), the import breaks with the usual error ModuleNotFoundError: No module named 'common'.
I know I could turn the whole parent folder into a package by adding __init__.py files in each subdirectory, however this solution still prevents me to run the script directly from inside folder1/.
How can I fix this?
If you turn both directories into python packages, and the top directory is in your PYTHONPATH, you can run script.py like so:
python3 -m example.folder1.script
If you need to add a parent directory to the search path, you can always do:
import os
import sys
DIR = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
sys.path.insert(0, DIR)
os.path.realpath(__file__) gets the realpath to the file, and os.path.dirname gets the directory name of this file. Doing it twice gets the example directory here. Now, since this is added to the search path, you can import common.
However, you should really consider having a dedicated scripts or bin directory, rather than try to run scripts from subdirectories. Think about writing a library, and import this library rather than individual files.
I am using py.test to test my python code. The relevant structure of my project is
-myproject
file.py
file2.py
-test/
input.txt
input2.dat
test_part.py
-regress/
file2_old.py
__init__.py
-test/
test_file2_regression.py
test_part.py imports file and file2 and test_file2_regression.py imports file2 and regress.file2_old. If I run pytest in the console, I get the import errors, that the packages don't exist. Running python -m pytest on the other hand works perfectly fine, but only if I run it from the myproject/ directory.
What is the correct way to do this, to get it working from anywhere in my project? I already tried modifying the PYTHONPATH but I honestly have no idea how to properly do it.
Further information:
I don't have any setup files, and my __init__s are just empty files. If manipulating the PYTHONPATH is necessary it needs to be done relative to myproject as I use it on several machines. I am running python 2.7.
I already check out:
Importing correctly with pytest
Good Integration Practices
but it didn't really help me.
The solution which works with the pytest command from any directory in the project is to include before the imports in the test*.py files:
import os
from sys import path
PATH = os.path.abspath(os.path.dirname(__file__))
path.append(os.path.join(PATH, os.pardir, os.pardir))
where the proper number of os.pardir is used to navigate to the project directory, from there the __init__.py files allow to import the modules.
Both argv[0] and inspect.getsourcefile don't provide the necessary information. argv[0] contains the location of the used py.test module and the getsourcefile method simply returns None.
Edit: since Python 3.4 we can use instead of os.path the modern pathlib:
from pathlib import Path
from sys import path
PATH = Path(__file__).resolve()
path.append(PATH.parents[2])
Having the same issue and similar success in searching for "the best way of doing this", I concluded for myself to avoid the situation whenever possible (by running the actual scripts consequently from top level), but to answer your question, my current approach (for instance for unit testing from a parallel folder) is
from sys import argv, path
from os.path import dirname, join
path.append(join(dirname(argv[0]), ".."))
This makes the interpreter also search in the folder above where the script is started. Another approach (instead of using argv) is to use the introspect module to obtain the filename. These work better for me than using __file__, as the latter is not always defined.
Edit 29.10.:
Alternative to argv[0] is to use
from inspect import getsourcefile
from sys import path
from os.path import dirname, join
this_file = getsourcefile(lambda _: None)
path.append(join(dirname(this_file), ".."))
I hope this will work at least for the requested purpose, see also How do I get the path of the current executed file in Python?.
The simplest - if it works in your case - is of course:
from os.path import dirname, join
path.append(join(dirname(__file__), ".."))
Here's my directory structure:
my_package
|
+--__init__.py
|
+--setup.py
|
+--module.py
|
+--sub_package
|
+--__init__.py
|
+--script.py
The script script.py needs to import a function from module.py, and I need to be able to run script.py using the Python interpreter.
I know that Guido calls this an "anti-pattern". And I know you have 10 reasons why I shouldn't do this. I probably agree with most of them - really, I do. I wouldn't be asking this question if I could avoid this. So can we just skip the part where we go over why this is bad and move on to the solution? Thanks!
I also know that there are about a 1000 other SO questions about this. I've probably read them all by now. Most of them were made obsolete by changes to Python's import system, and the rest just aren't right.
What I have tried:
Using from .module import my_function in script.py and then either running python script.py inside the directory sub_package or python sub_package/script.py inside the directory my_package. Either way, I get the error:
SystemError: Parent module '' not loaded, cannot perform relative import
Using from module import my_function or from my_package.module import my_function and running script.py as above (either from sub_package or my_package). I get:
ImportError: No module named 'module'
(or similarly with my_package instead of module)
Running python -m sub_package/script from the directory my_package or python -m my_package/sub_package/script from the parent directory of my_package. I get:
No module named sub_package/script
(or similarly with my_package/sub_package/script)
Is there anything else I should be trying? I would really rather avoid messing with sys.path or PYTHONPATH for a whole host of reasons.
I believe you can just say
import module
from within script.py if you do the following:
from your terminal in the mypackage directory run:
$ python setup.py install
This will allow you to make the import statement that you want. The one issue that I have found with this method is that you will have to call
$ python setup.py install
every time you make a change to module.py but if you are ok with that this method should work
In fact, python interpreter will find module in sys.path.It means if you add your module's directory in this example "somepath/my_package" to sys.path, python will find it by "import module".
So how to achieve this goal?
If your directory absolute path never change, you can do this below:
sys.path.append(yourAbsPath)
But sometimes the path of directory might change, but the relative path will not change.So i recommend to follow this, try to add it to the beginning of your script.py:
import sys
import os
my_packagePath = os.path.dirname(os.path.dirname(__file__))
sys.path.append(my_packagePath)
import module
#from module import yourFunction #replace yourFunction with your real function
In the cmd tool, just type "python yourScript.py's abspath".
I have the following directory structure
project/
bin/
script
stuff/
__init__.py
mainfile.py
Inside of script, I have the following to enable command line execution:
#!/usr/bin/env python
from stuff import mainfile
This works, but I would have expected needing to jump up one level...
from ..stuff import mainfile
or
from project.stuff import mainfile
What am I missing here?
Actually none of your examples should work out of the box.
Let's modify bin/script.py somewhat:
#! /usr/bin/env python
import sys
print sys.path
from stuff import mainfile
This should yield something like
['.../project/bin', ...]
Traceback (most recent call last):
File "bin/script.py", line 6, in <module>
from stuff import mainfile
ImportError: No module named stuff
Only the directory of the script (not the current directory) is added to sys.path automatically:
As initialized upon program startup, the first item of this list, path[0], is the directory containing the script that was used to invoke the Python interpreter.* If the script directory is not available (e.g. if the interpreter is invoked interactively or if the script is read from standard input), path[0] is the empty string, which directs Python to search modules in the current directory first. Notice that the script directory is inserted before the entries inserted as a result of PYTHONPATH.
Hence there's no stuff module on sys.path. I'm not sure about your environment setup, but this is the canonical result when no additional parameters are set up (e.g. PYTHONPATH).
Similarily,
from ..stuff import mainfile
will result in the classic ValueError: Attempted relative import in non-package. You are informed of the fact that you can only do relative imports relative to actual modules. Since inside script .. does not refer to a module (because the script itself is the top level module so to say), a relative import does not work here. In a manner of speaking from Python's perspective, there is no module above the script and thus .. does not refer to something tangible when used in the context of the script.
Note that this also means that it does not help to only make project and project/bin into modules themselves by dropping __init__.py marker files. Relative imports to a parent of the script are only possible if the parent of the script is actually something python has a concept of.
This is one of the reasons, why the -m command line switch exists making it possible to run a module from the command line. For example, given the above relative import
python -m project.bin.script
does the trick, but only if executed from the proper directory (projectdir/..).
That problem is even worse with
from project.stuff import mainfile
because the project directory is only automatically on sys.path when you start a script from the directory above project and do not specify a main script to run:
cd <projectdir>/..
python -m project.bin.script
# works
cd <projectdir>
python -m bin.script
# does not work, because `sys.path` starts with <projectdir>
# and it's `stuff.mainfile` now, not `project.stuff.mainfile`.
If you want to import modules from project in your script, fix up sys.path to your needs:
import sys
import os
sys.path.insert(0, os.path.dirname(sys.path[0]))
from stuff import mainfile
You need to first add the parent directory into sys.path. Try the following:
# This is a file in bin/ directory.
this_file_path = os.path.dirname(__file__)
sys.path.append(os.path.join(this_file_path, '..'))
import stuff.mainfile
I have a Python script that uses built-in modules but also imports a number of custom modules that exist in the same directory as the main script itself.
For example, I would call
python agent.py
and agent.py has a number of imports, including:
import checks
where checks is in a file in the same directory as agent.py
agent/agent.py
agent/checks.py
When the current working directory is agent/ then everything is fine. However, if I call agent.py from any other directory, it is obviously unable to import checks.py and so errors.
How can I ensure that the custom modules can be imported regardless of where the agent.py is called from e.g.
python /home/bob/scripts/agent/agent.py
Actually your example works because checks.py is in the same directory as agent.py, but say checks.py was in the preceeding directory, eg;
agent/agent.py
checks.py
Then you could do the following:
path = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
if not path in sys.path:
sys.path.insert(1, path)
del path
Note the use of __file__.
You should NOT need to fiddle with sys.path. To quote from the Python 2.6 docs for sys.path:
As initialized upon program startup, the first item of this list,
path[0], is the directory containing the script that was used to
invoke the Python interpreter. If the script directory is not
available (e.g. if the interpreter is invoked interactively or if the
script is read from standard input), path[0] is the empty string,
which directs Python to search modules in the current directory first.
Notice that the script directory is inserted before the entries
inserted as a result of PYTHONPATH.
=== amod.py ===
def whoami():
return __file__
=== ascript.py ===
import sys
print "sys.argv", sys.argv
print "sys.path", sys.path
import amod
print "amod __file__", amod.whoami()
=== result of running ascript.py from afar ===
C:\somewhere_else>\python26\python \junk\timport\ascript.py
sys.argv ['\\junk\\timport\\ascript.py']
sys.path ['C:\\junk\\timport', 'C:\\WINDOWS\\system32\\python26.zip', SNIP]
amod __file__ C:\junk\timport\amod.py
and if it's re-run, the last line changes of course to ...amod.pyc. This appears not to be a novelty, it works with Python 2.1 and 1.5.2.
Debug hints for you: Try two simple files like I have. Try running Python with -v and -vv. Show us the results of your failing tests, including full traceback and error message, and your two files. Tell us what platform you are running on, and what version of Python. Check the permissions on the checks.py file. Is there a checks.something_else that's causing interference?
You need to add the path to the currently executing module to the sys.path variable. Since you called it on the command line, the path to the script will always be in sys.argv[0].
import sys
import os
sys.path.append(os.path.split(sys.argv[0])[0])
Now when import searches for the module, it will also look in the folder that hosts the agent.py file.
There are several ways to add things to the PYTHONPATH.
Read http://docs.python.org/library/site.html
Set the PYTHONPATH environment variable prior to running your script.
You can do this python -m agent to run agent.py from your PYTHONPATH.
Create .pth files in your lib/site-packages directory.
Install your modules in lib/site-packages.
I think you should consider making the agent directory into a proper Python package. Then you place this package anywhere on the python path, and you can import checks as
from agent import checks
See http://docs.python.org/tutorial/modules.html
If you know full path to check.py use this recipe (http://code.activestate.com/recipes/159571/)
If you want to add directory to system path -- this recipe (http://code.activestate.com/recipes/52662/). In this case I have to determine application directory (sys.argv[0]) an pass this value to AddSysPath function. If you want to look at production sample please leave a comment on this thread so I post it later.
Regards.
To generalize my understanding of your goal, you want to be able to import custom packages using import custom_package_name no matter where you are calling python from and no matter where your python script is located.
A number of answers mention what I'm about to describe, but I feel like most of the answers assume a lot of previous knowledge. I'll try to be as explicit as I can.
To achieve the goal of allowing custom packages to be imported via the import statement, they have to be discoverable somewhere through the path that python uses to search for packages. Python actually uses multiple paths, but we'll only focus on the one that can be found by combining the output of sys.prefix (in your python interpreter) with /lib/pythonX.Y/site-packages (or lib/site-packages if you're using windows) where X.Y is your python version.
Concretely, find the path that your python uses by running:
import sys
your_path = sys.prefix + '/lib/pythonX.Y/site-packages'
print(your_path)
This path should look something like /usr/local/lib/python3.5/site-packages if you're using python 3.5, but it could be much different depending on your setup.
Python uses this path (and a few others) to find the packages that you want to import. So what you can do is put your custom packages in the /usr/local/lib/python3.5/site-packages folder. Don't forget to add an init.py file to the folder.
To be concrete again, in terminal type:
cd your_path
cp path_to_custom_package/custom_package ./
Now you should be able to import everything your custom package like you would if the package was located in the same directory (i.e. import package.subpackage for each subpackage file in your package should work).