Develop Reference

Writing Python pip-able package: How to access files in the package?

I'm trying to develop a visualization package based on CEFpython technology, so the function has to access an html file. So the structure of the package is basically:
viz_tool
--> init.py
--> main.py
--> index.html
and in the main.py file, I defined a function:
def view():
index_filepath = os.path.realpath("index.html")
print(index_filepath)
browser = cef.CreateBrowserSync(url='file://' + index_filepath,
window_title="viz tool")
....
The goal is, after installing this package, I can do:
from viz_tool import view
view()
but of course currently the "index_filepath" would be my current working directory, so it won't be able to find the "index.html" file.
How should I change the code so I could access the "index.html" file, which presumably would be in the directory where the package is installed?
Thank you very much! and please let me know if I need to clarify anything

Look at the documentation of pkg_resources.
It has a method called resource_filename that you can use to get absolute path to any internal module.
from pkg_resources import resource_filename
from foo.bar import bazz # say bazz has template folder
pth_to_template = resource_filename(bazz.__name__, "template")
This will return the absolute path to <path to foo>/foo/bar/bazz/template.
I would highly recommend not putting html file in root folder and also avoid using hardcoded paths in any form, including doing os.path.join recursively to traverse multiple levels. It makes moving files in future a mess.
Edit:
For accessing resources from the same module by relative path, you can do
pth = resource_filename(__name__, "templates/something.html")
Edit 2:.
You will also have to add all the non-python files in package_data as here in order for setup.py to add them to the distribution package.

How to make python config file, in which relative paths are defined, but when scripts in other directories import config, paths are correct?

I have the following directory structure for a program I'm writing in python:
\code\
main.py
config.py
\module_folder1\
script1.1.py
\data\
data_file1
data_file2
My config.py is a set of global variables that are set by the user, or generally fixed all the time. In particular config.py defines path variables to the 2 data files, something like path1 = os.path.abspath("../data/data_file1"). The primary use is to run main.py which imports config (and the other modules I wrote) and all is good.
But sometimes I need to run script1.1.py by itself. Ok, no problem. I can add to script1.1 the usual if __name__ == '__main__': and I can import config. But then I get path1 = "../code/data/data_file1" which doesn't exist. I thought that since the path is created in config.py the path would be relative to where config.py lives, but it's not.
So the question is, how can I have a central config file which defines relative paths, so I can import the config file to scripts in different directories and have the paths still be correct?
I should mention that the code repo will be shared among multiple machines, so hardcoding an absolute path is not an option.

You know the correct relative path to the file from the directory where config.py is located
You know the correct relative path to the directory where config.py is located (in your case, ..)
Both of this things are system-independent and do not change unless you change the structure of you project. Just add them together using os.path.join('..', config.path_repative_to_config)

(Not sure who posted this as a comment, then deleted it, but it seems to work so I'm posting as an answer.) The trick is to use os.path.dirname(__file__) in the config file, which gives the directory of the config file (/code/) regardless of where the script that imports config is.
Specifically to answer the question, in the config file define
path1 = os.path.abspath(os.path.join(os.path.join(os.path.join( os.path.dirname(__file__) , '..'), 'data' ), 'data_file1' ) )

Flask configuration paths

So I have this code in my init:
app = Flask(__name__)
app.config.from_object('config')
app.config.from_pyfile('app.cfg')
My folder structure goes like this:
/config.py
/app.cfg
/app/__init__.py
However, this code seems to find my config.py in the root, but it's searching inside my app folder for my app.cfg. How do I make it look for my configuration file in the correct place, without having to put it in my app folder.
Should I be using from object and rename my app.cfg to app.py

According to the official docs (http://flask.pocoo.org/docs/0.10/api/#flask.Config):
from_pyfile(filename, silent=False)
Updates the values in the config from a Python file. This function behaves as if the file was imported as module with thefrom_object() function.
Parameters: filename – the filename of the config. This can either be an absolute filename or a filename relative to the root path.silent – set to True if you want silent failure for missing files.
So, if you don't want to put your cfg file in your app folder one of the things you can do is to set relative route to root folder (that seems to be "./app" directory) in your "from_pyfile" call.
from_pyfile('../app.cfg')
should work..

Django can't find template - Polls App Part 3

I am following the polls app tutorial part 3 and I cannot get the templates to be found
This is the exact error
polls/index.html
Request Method: GET
Request URL: http://localhost:8000/polls/
Django Version: 1.4.3
Exception Type: TemplateDoesNotExist
Exception Value:
polls/index.html
Exception Location: c:\Python27\lib\site-packages\django\template\loader.py in find_template, line 138
Python Executable: c:\Python27\python.exe
Python Version: 2.7.2
So in my settings.py I have put the directory there. "C:/Scripts/mysite/template" and created /polls/index.html
TEMPLATE_DIRS = (
# Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
# Always use forward slashes, even on Windows.
# Don't forget to use absolute paths, not relative paths.
"C:/Scripts/template"
"C:/Scripts/mysite/template"
)
It for some reason cannot find it though. Any ideas?
This is the exact instruction from the tutorial.
create a directory, somewhere on your filesystem, whose contents Django can access. (Django runs as whatever user your server runs.)

You're missing a comma at the end of the first line:
"C:/Scripts/template", # <--- HERE
"C:/Scripts/mysite/template"
without that, Python automatically concatenates the two strings (because they're inside parens) so that it just becomes "C:/Scripts/templateC:/Scripts/mysite/template".

Debug
from your applications main folder, run the following command: python manage.py shell this is going to bootstrap your app with the settings file.
type from django.conf import settings and hit enter
type settings.TEMPLATE_DIRS and check the output. Do you see the template directories that you specified?
Absolute paths relative to the settings.py file
I generally use absolute paths relative to the settings.py file. That way collaborators can share a main settings file, and no matter what system/environment you deploy to your paths will be correct.
to do this:
# settings.py
import os
# ... other settings
TEMPLATE_DIRS = (
os.path.join(os.path.normpath(os.path.dirname(__file__)), 'templates'),
)
let me know if the debug step didn't help, and i'll try and supply some more help.

Accessing resource files in Python unit tests & main code

I have a Python project with the following directory structure:
project/
project/src/
project/src/somecode.py
project/src/mypackage/mymodule.py
project/src/resources/
project/src/resources/datafile1.txt
In mymodule.py, I have a class (lets call it "MyClass") which needs to load datafile1.txt. This sort of works when I do:
open ("../resources/datafile1.txt")
Assuming the code that creates the MyClass instance created is run from somecode.py.
The gotcha however is that I have unit tests for mymodule.py which are defined in that file, and if I leave the relative pathname as described above, the unittest code blows up as now the code is being run from project/src/mypackage instead of project/src and the relative filepath doesn't resolve correctly.
Any suggestions for a best practice type approach to resolve this problem? If I move my testcases into project/src that clutters the main source folder with testcases.

I usually use this to get a relative path from my module. Never tried in a unittest tho.
import os
print(os.path.join(os.path.dirname(__file__),
'..',
'resources'
'datafile1.txt'))
Note: The .. tricks works pretty well, but if you change your directory structure you would need to update that part.

On top of the above answers, I'd like to add some Python 3 tricks to make your tests cleaner.
With the help of the pathlib library, you can explicit your ressources import in your tests. It even handles the separators difference between Unix (/) and Windows ().
Let's say we have a folder structure like this :
`-- tests
|-- test_1.py <-- You are here !
|-- test_2.py
`-- images
|-- fernando1.jpg <-- You want to import this image !
`-- fernando2.jpg
You are in the test_1.py file, and you want to import fernando1.jpg. With the help to the pathlib library, you can read your test resource with an object oriented logic as follows :
from pathlib import Path
current_path = Path(os.path.dirname(os.path.realpath(__file__)))
image_path = current_path / "images" / "fernando1.jpg"
with image_path.open(mode='rb') as image :
# do what you want with your image object
But there's actually convenience methods to make your code more explicit than mode='rb', as :
image_path.read_bytes() # Which reads bytes of an object
text_file_path.read_text() # Which returns you text file content as a string
And there you go !

in each directory that contains Python scripts, put a Python module that knows the path to the root of the hierarchy. It can define a single global variable with the relative path. Import this module in each script. Python searches the current directory first so it will always use the version of the module in the current directory, which will have the relative path to the root of the current directory. Then use this to find your other files. For example:
# rootpath.py
rootpath = "../../../"
# in your scripts
from rootpath import rootpath
datapath = os.path.join(rootpath, "src/resources/datafile1.txt")
If you don't want to put additional modules in each directory, you could use this approach:
Put a sentinel file in the top level of the directory structure, e.g. thisisthetop.txt. Have your Python script move up the directory hierarchy until it finds this file. Write all your pathnames relative to that directory.
Possibly some file you already have in the project directory can be used for this purpose (e.g. keep moving up until you find a src directory), or you can name the project directory in such a way to make it apparent.

You can access files in a package using importlib.resources (mind Python version compatibility of the individual functions, there are backports available as importlib_resources), as described here. Thus, if you put your resources folder into your mypackage, like
project/src/mypackage/__init__.py
project/src/mypackage/mymodule.py
project/src/mypackage/resources/
project/src/mypackage/resources/datafile1.txt
you can access your resource file in code without having to rely on inferring file locations of your scripts:
import importlib.resources
file_path = importlib.resources.files('mypackage').joinpath('resources/datafile1.txt')
with open(file_path) as f:
do_something_with(f)
Note, if you distribute your package, don't forget to include the resources/ folder when creating the package.

The filepath will be relative to the script that you initially invoked. I would suggest that you pass the relative path in as an argument to MyClass. This way, you can have different paths depending on which script is invoking MyClass.