In a program, and obviously being influenced by the way Java does things, I want to read a static file (a log configuration file, actually) from a directory within the interpreter's PYTHONPATH. I know I could do something like:
import foo
a = foo.__path__
conf = open(a[0] + "/logging.conf")
but I don't know if this is the "Pythonic" way of doing things. How could I distribute the logging configuration file in a way that my application does not need to be externally configured to read it?
In general, that's fine, though I'm not sure you want a[0] above (that will just give you the first character of the path), and you should use os.path.join instead of just appending / to be cross-platform compatible. You might consider making the path canonical, i.e. os.path.abspath(os.path.dirname(foo.__path__)). Note that it won't work if __path__ is in a zip file or other import trickery is in use, but I wouldn't worry about that (it's not normal to do so for the main program in Python, unlike Java).
If you do want to support zipped files, there's pkg_resources, but that's somewhat deprecated at this point (there's no corresponding API I could see in the new packaging module).
Here's a snippet based on the link Nix posted upthread but written in a more functional style:
def search_path(pathname_suffix):
cands = [os.path.join(d,pathname_suffix) for d in sys.path]
try:
return filter(os.path.exists, cands)[0]
except IndexError:
return None
Related
I would like to find out where the source code for the mindstorms module is for the Mindstorms Robot Inventor.
At the start of each file there is a starting header of
from Mindstorms import ...
Etc..
That is what I want to find.
I have tryed multiple python methods to file the file path, but they all return ./projects/8472.py
Thanks,
henos
You've likely been using __file__ and similar, correct? Yes, that will give you the currently running file, but since you can't edit the code for the actual Mindstorms code, it's not helpful. Instead, you want to inspect the directory itself, like you would if your regular Python code were accessing a data file elsewhere.
Most of the internal systems are .mpy files (see http://docs.micropython.org/en/v1.12/reference/mpyfiles.html), so directly reading the code from the device is less than optimal. Additionally, this means that many "standard library" packages are missing or incomplete; you can't import pathlib but you can import os, but you can't use os.walk(). Those restrictions make any sort of directory traversal a little more frustrating, but not impossible.
For instance, the file runtime/extensions.music.mpy looks like the following (note: not copy-pasted since the application doesn't let you):
M☐☐☐ ☐☐☐
☐6runtime/extensions/music.py ☐☐"AbstractExtensions*☐☐$abstract_extension☐☐☐☐YT2 ☐☐MusicExtension☐☐4☐☐☐Qc ☐|
☐☐ ☐ ☐☐ ☐☐☐☐ ☐2 ☐☐play_drum2☐☐☐play_noteQc ☐d:
☐☐ *☐☐_call_sync#☐,☐+/-☐☐drumb6☐YQc☐ ☐☐drum_nos☐musicExtension.playDrum☐☐E☐
☐ *☐ #☐,☐+/-☐☐instrumentb*☐☐noteb*☐☐durationb6☐YQc☐ ☐☐☐☐s☐musicExtension.playNote
Sure, you can kind of see what's going on, but it isn't that helpful.
You'll want to use combinations of os.listdir and print here, since the MicroPython implementation doesn't give access to os.walk. Example code to get you started:
import os
import sys
print(os.uname()) # note: doesn't reflect actual OS version, see https://stackoverflow.com/questions/64449448/how-to-import-from-custom-python-modules-on-new-lego-mindstorms-robot-inventor#comment115866177_64508469
print(os.listdir(".")) # ['util', 'projects', 'runtime', ...]
print(os.listdir("runtime/extenstions")) # ['__init__.mpy', 'abstract_extension.mpy', ...]
with open("runtime/extensions/music/mpy", "r") as f:
for line in f:
print(line)
sys.exit()
Again, the lack of copy-paste from the console is rough, so even when you do get to the "show contents of the file on screen" part, it's not that helpful.
One cool thing to note though is that if you load up a scratch program, you can read the code in regular .py. It's about as intelligible as you'd expect, since it uses very low-level calls and abstractions, not the hub.light_matrix.show_image("CLOCK6") that you'd normally write.
I have a Python script that needs some data that's stored in a file that will always be in the same location as the script. I have a setup.py for the script, and I want to make sure it's pip installable in a wide variety of environments, and can be turned into a standalone executable if necessary.
Currently the script runs with Python 2.7 and Python 3.3 or higher (though I don't have a test environment for 3.3 so I can't be sure about that).
I came up with this method to get the data. This script isn't part of a module directory with __init__.py or anything, it's just a standalone file that will work if just run with python directly, but also has an entry point defined in the setup.py file. It's all one file. Is this the correct way?
def fetch_wordlist():
wordlist = 'wordlist.txt'
try:
import importlib.resources as res
return res.read_binary(__file__, wordlist)
except ImportError:
pass
try:
import pkg_resources as resources
req = resources.Requirement.parse('makepw')
wordlist = resources.resource_filename(req, wordlist)
except ImportError:
import os.path
wordlist = os.path.join(os.path.dirname(__file__), wordlist)
with open(wordlist, 'rb') as f:
return f.read()
This seems ridiculously complex. Also, it seems to rely on the package management system in ways I'm uncomfortable with. The script no longer works unless it's been pip-installed, and that also doesn't seem desirable.
Resources living on the filesystem
The standard way to read a file adjacent to your python script would be:
a) If you've got python>=3.4 I'd suggest you use the pathlib module, like this:
from pathlib import Path
def fetch_wordlist(filename="wordlist.txt"):
return (Path(__file__).parent / filename).read_text()
if __name__ == '__main__':
print(fetch_wordlist())
b) And if you're still using a python version <3.4 or you still want to use the good old os.path module you should do something like this:
import os
def fetch_wordlist(filename="wordlist.txt"):
with open(os.path.join(os.path.dirname(__file__), filename)) as f:
return f.read()
if __name__ == '__main__':
print(fetch_wordlist())
Also, I'd suggest you capture exceptions in the outer callers, the above methods are standard way to read files in python so you don't need wrap them in a function like fetch_wordlist, said otherwise, reading files in python is an "atomic" operation.
Now, it may happen that you've frozen your program using some freezer such as cx_freeze, pyinstaller or similars... in that case you'd need to detect that, here's a simple way to check it out:
a) using os.path:
if getattr(sys, 'frozen', False):
app_path = os.path.dirname(sys.executable)
elif __file__:
app_path = os.path.dirname(__file__)
b) using pathlib:
if getattr(sys, 'frozen', False):
app_path = Path(sys.executable).parent
elif __file__:
app_path = Path(__file__).parent
Resources living inside a zip file
The above solutions would work if the code lives on the file system but it wouldn't work if the package is living inside a zip file, when that happens you could use either importlib.resources (new in version 3.7) or pkg_resources combo as you've already shown in the question (or you could wrap up in some helpers) or you could use a nice 3rd party library called importlib_resources that should work with the old&modern python versions:
pypi: https://pypi.org/project/importlib_resources/
documentation: https://importlib-resources.readthedocs.io/en/latest/
Specifically for your particular problem I'd suggest you take a look to this https://importlib-resources.readthedocs.io/en/latest/using.html#file-system-or-zip-file.
If you want to know what that library is doing behind the curtains because you're not willing to install any 3rd party library you can find the code for py2 here and py3 here in case you wanted to get the relevant bits for your particular problem
I'm going to go out on a limb and make an assumption because it may drastically simplify your problem. The only way I can imagine that you can claim that this data is "stored in a file that will always be in the same location as the script" is because you created this data, once, and put it in a file in the source code directory. Even though this data is binary, have you considered making the data a literal byte-string in a python file, and then simply importing it as you would anything else?
You're right about the fact that your method of reading a file is a bit unnecessarily complex. Unless you have got a really specific reason to use the importlib and pkg_resources modules, it's rather simple.
import os
def fetch_wordlist():
if not os.path.exists('wordlist.txt'):
raise FileNotFoundError
with open('wordlist.txt', 'rb') as wordlist:
return wordlist.read()
You haven't given much information regarding your script, so I cannot comment on why it doesn't work unless it's installed using pip. My best guess: your script is probably packed into a python package.
I've looked around the docs on the pytest website, but haven't found a clear example of working with 'test resources', such as reading in fixed files during unit tests. Something similar to what http://jlorenzen.blogspot.com/2007/06/proper-way-to-access-file-resources-in.html describes for Java.
For example, if I have a yaml file checked in to source control, what is the right way to write a test which loads from that file? I think this boils down to understanding the right way to access a 'resource file' on the python equivalent of the classpath (PYTHONPATH?).
This seems like it should be simple. Is there an easy solution?
Perhaps what you are looking for is pkg_resources or pkgutil. For example, if you have a module within your python source called "resources", you could read your "resourcefile" using:
with open(pkg_resources.resource_filename("resources", "resourcefile")) as infile:
for line in infile:
print(line)
or:
with tempfile.TemporaryFile() as outfile:
outfile.write(pkgutil.get_data("resources", "resourcefile"))
The latter even works when your "script" is an executable zip file. The former works without needing to unpack your resources from an egg.
Note that creating a subdirectory of your source does not make it a module. You need to add a file named __init__.py within the directory for it to be visible as a module for the purposes of pkg_resources and pkgutil. __init__.py can be empty.
I think "resource file" is whatever definition you give to it in python (in Java, resource files can be bundled into jar files with ordinary Java classes, and Java provides library functions to access this information).
An equivalent solution might be to access the PYTHONPATH environment variable, define your "resource file" as a relative path, and then troll the PYTHONPATH looking for it. Here's an example:
pythonpath = os.env['PYTHONPATH']
file_relative_path = os.path.join('subdir', 'resourcefile') // e.g. subdir/resourcefile
for dir in pythonpath.split(os.pathsep):
resource_path = os.path.join(dir, file_relative_path)
if os.path.exists(resource_path):
return resource_path
This code snippet returns a full path for the first file that exists on the PYTHONPATH.
I'm working on a mod_wsgi script.. at the beginning is:
sys.path.extend(map(os.path.abspath, ['/media/server/www/webroot/']))
But I've noticed, that every time I update the script the sys.path var keeps growing with duplicates of this extension:
['/usr/lib64/python25.zip'
'/usr/lib64/python2.5'
'/usr/lib64/python2.5/plat-linux2'
'/usr/lib64/python2.5/lib-tk'
'/usr/lib64/python2.5/lib-dynload'
'/usr/lib64/python2.5/site-packages'
'/usr/lib64/python2.5/site-packages/Numeric'
'/usr/lib64/python2.5/site-packages/gtk-2.0'
'/usr/lib64/python2.5/site-packages/scim-0.1'
'/usr/lib/python2.5/site-packages'
'/media/server/www/webroot'
'/media/server/www/webroot'
'/media/server/www/webroot'
'/media/server/www/webroot']
It resets every time I restart apache.. is there any way to make sure this doesn't happen? I want the module path to be loaded only once..
No need to worry about checking or using abspath yourself. Use the ‘site’ module's built-in addsitedir function. It will take care of these issues and others (eg. pth files) automatically:
import site
site.addsitedir('/media/server/www/webroot/')
(This function is only documented in Python 2.6, but it has pretty much always existed.)
One fairly simple way to do this is to check to see if the path has already been extended before extending it::
path_extension = map(os.path.abspath,['/media/server/www/webroot/'])
if path_extension[0] not in sys.path:
sys.path.extend(path_extension)
This has the disadvantage, however, of always scanning through most of sys.path when checking to see if it's been extended. A faster, though more complex, version is below::
path_extension = map(os.path.abspath,['/media/server/www/webroot/'])
if path_extension[-1] not in reversed(sys.path):
sys.path.extend(path_extension)
A better solution, however, is probably to either add the path extensions to your PYTHONPATH environment variable or put a .pth file into your site-packages directory:
http://docs.python.org/install/index.html
The mod_wsgi documentation on code reloading covers this.
I have a fair number of Python scripts that contain reusable code that are used and referenced by other Python scripts. However, these scripts tend to be scattered across different directories and I find it to be somewhat tedious to have to include (most often multiple) calls to sys.path.append on my top-level scripts. I just want to provide the 'import' statements without the additional file references in the same script.
Currently, I have this:
import sys
sys.path.append('..//shared1//reusable_foo')
import Foo
sys.path.append('..//shared2//reusable_bar')
import Bar
My preference would be the following:
import Foo
import Bar
My background is primarily in the .NET platform so I am accustomed to having meta files such as *.csproj, *.vbproj, *.sln, etc. to manage and contain the actual file path references outside of the source files. This allows me to just provide 'using' directives (equivalent to Python's import) without exposing all of the references and allowing for reuse of the path references themselves across multiple scripts.
Does Python have equivalent support for this and, if not, what are some techniques and approaches?
The simple answer is to put your reusable code in your site-packages directory, which is in your sys.path.
You can also extend the search path by adding .pth files somewhere in your path.
See https://docs.python.org/2/install/#modifying-python-s-search-path for more details
Oh, and python 2.6/3.0 adds support for PEP370, Per-user site-packages Directory
If your reusable files are packaged (that is, they include an __init__.py file) and the path to that package is part of your PYTHONPATH or sys.path then you should be able to do just
import Foo
This question provides a few more details.
(Note: As Jim said, you could also drop your reusable code into your site-packages directory.)
You can put the reusable stuff in site-packages. That's completely transparent, since it's in sys.path by default.
You can put someName.pth files in site-packages. These files have the directory in which your actual reusable stuff lives. This is also completely transparent. And doesn't involve the extra step of installing a change in site-packages.
You can put the directory of the reusable stuff on PYTHONPATH. That's a little less transparent, because you have to make sure it's set. Not rocket science, but not completely transparent.
In one project, I wanted to make sure that the user could put python scripts (that could basically be used as plugins) anywhere. My solution was to put the following in the config file for that project:
[server]
PYPATH_APPEND: /home/jason:/usr/share/some_directory
That way, this would add /home/jason and /usr/share/some_directory to the python path at program launch.
Then, it's just a simple matter of splitting the string by the colons and adding those directories to the end of the sys.path. You may want to consider putting a module in the site-packages directory that contains a function to read in that config file and add those directories to the sys.path (unfortunately, I don't have time at the moment to write an example).
As others have mentioned, it's a good idea to put as much in site-packages as possible and also using .pth files. But this can be a good idea if you have a script that needs to import a bunch of stuff that's not in site-packages that you wouldn't want to import from other scripts.
(there may also be a way to do this using .pth files, but I like being able to manipulate the python path in the same place as I put the rest of my configuration info)
The simplest way is to set (or add to) PYTHONPATH, and put (or symlink) your modules and packages into a path contained in PYTHONPATH.
My solution was to package up one utility that would import the module:
my_util is in site packages
import my_util
foo = myutil.import_script('..//shared1//reusable_foo')
if foo == None:
sys.exit(1)
def import_script(script_path, log_status = True):
"""
imports a module and returns the handle
"""
lpath = os.path.split(script_path)
if lpath[1] == '':
log('Error in script "%s" in import_script' % (script_path))
return None
#check if path is already in sys.path so we don't repeat
npath = None
if lpath[0] == '':
npath = '.'
else:
if lpath[0] not in sys.path:
npath = lpath[0]
if npath != None:
try:
sys.path.append(npath)
except:
if log_status == True:
log('Error adding path "%s" in import_script' % npath)
return None
try:
mod = __import__(lpath[1])
except:
error_trace,error_reason = FormatExceptionInfo()
if log_status == True:
log('Error importing "%s" module in import_script: %s' % (script_path, error_trace + error_reason))
sys.path.remove(npath)
return None
return mod