I have a question regarding the import//from statement in python.
In my views.py file (Project/App/views.py) I have this line:
from django.views.generic import TemplateView, ListView
Why do I have to include 'django' in that line? Why is it not enough to specify which directory (views) that the generic file is located in? This is what I have done in many of my previous python-only scripts - an example being:
from random import foo
as well as in my current django url.py file. There, I have:
from app.views import view
Why don't I have to specify that further, like with the first example where 'django' is included in the path-specification? How come I don't have to write it like this:
from project.app.views import view
Thank you!
Welcome to the wild world of Python's import system!
To expand on freude's answer a little bit, you've ran into one of the most consistently confusing portions of the Python language: relative vs absolute imports. While the import examples that you gave are syntactically fine, they hide some of the complexity what's going on behind the scenes. When you run:
from django.views.generic import TemplateView, ListView
Python searches through the PYTHONPATH (which you can see with something like print(sys.path)) for a package or module named django. It end up finding one somewhere among your installed libraries. Similarly, when you run:
from project.app.views import view
It searches those same paths, but instead finds the project package in the directory that the Python interpreter is currently executing in. However, if you had installed a library named project, how would it know which one you actually meant? This is generally solved by using absolute imports and by being explicit if you intend to use relative imports like this. If you wanted to be more precise in your example, you would specify that you wanted to import it relative to the current module by using a . - like so:
from .project.app.views import view
You can even see this in action in an example in the django tutorial.
See this classic answer on the subject for more detailed information.
A python script sees certain paths pointing to the the global default location like site-packages or dist-packages (you may find those directories in the python directory tree and random and django are located in one of them) or specified by the environment variable PYTHONPATH. Usually, PYTHONPATH includes your project directory (actually you may add there whatever directory you want). Your example suggests that the packages django as well as apps and random are located in those paths while project is not there. In python a package is represented by a directory containing __init__.py file as well as some other files representing modules. Now you may import modules from packages in those locations using relative paths like you have shown in your examples.
Related
Hey I'm working on a project that has a set hierarchical modules with a folder structure set up like so:
module_name/
__init__.py
constants.py
utils.py
class_that_all_submodules_need_access_to.py
sub_module_a/
__init__.py
sub_module_a_class_a.py
sub_module_a_class_b.py
useful_for_sub_module_a/
__init__.py
useful_class_a.py
useful_class_b.py
sub_module_b/
__init__.py
sub_module_b_class_a.py
sub_module_b_class_b.py
etc etc etc...
The problem is, I can't figure out how to set up the imports in the init.py's so that I can access class_that_all_submodules_need_access_to.py from sub_module_a/useful_for_sub_module_a/useful_class_a.py.
I've tried looking this up on Google/StackOverflow to exhaustion and I've come up short. The peculiar thing is that PyCharm has the paths set up in such a way that I don't encounter this bug when working on the project in PyCharm, but only from other environments.
So here's one particularly inelegant solution that I've come up with. My sub_module_a/useful_for_sub_module_a/init.py looks like:
import sys
import os
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..', '..')))
import module_name
This is similar in sub_module_*/, where instead of 3 '..'s it's just two (i.e. '..', '..' instead of '..', '..', '..' for the sys.path.insert line above). And then in sub_module_a/useful_for_sub_module_a/useful_class_a.py, I have to import module_name/constants.py (and others) like this:
import module_name.constants
from module_name.class_that_all_submodules_need_access_to import ImportantClass
While this solution works, I was wondering if there is a better/more elegant way to set up the imports and/or folder hierarchy? I'm concerned about messing with the python system path for users of this module. Is that even a valid concern?
There are two kinds of Python import: absolute and relative, see Python import. But the first import thing is that you need to understand how python finds your package? If you just put your package under your home folder, then Python knows nothing about it. You can check this blog [Python search] (https://leemendelowitz.github.io/blog/how-does-python-find-packages.html).
Thus to import modules, the first thing is that let Python know your package. After knowing the locations where Python would search packages, there are often two ways to accomplish this goal:
(1) PYTHONPATH environment variable. Set this variable in your environment configuration file, e.g., .bash_profile. This is also the simplest way.
(2) Use setuptools, which could help you distribute your package. This is a long story. I would not suggest you choose it, unless you would like to distribute your package in the future. However, it worth to know it.
If you set your path correctly, the Python import is just straight forward. If you would like use the absolute import, try
from module_name import class_that_all_submodules_need_access_to
If you would like to use the relative import, it depends on which module you are now. Suppose you are writing the module module_name.sub_module_a.sub_moduel_a_class_a, then try
from .class_that_all_submodules_need_access_to import XX_you_want
Note that relative import supports from .xx import xx format only.
Thanks.
I have two Python packages where one needs to be imported by the other. The directory structure is like follows:
workspace/
management/
__init__.py
handle_management.py
other_management.py
utils/
__init__.py
utils_dict.py
I'm trying to import functionality from the utils project in the handle_management.py file:
import utils.utils_dict
Error I'm getting when trying to run handle_management.py:
ImportError: No module named utils.utils_dict
I've read a lot about how to resolve this problem and I can't seem to find a solution that works.
I started with Import a module from a relative path - I tried the applicable solutions but none worked.
Is the only solution to make workspace/ available via site_packages? If so, what is the best way to do this?
EDIT:
I've tried to add the /home/rico/workspace/ to the PYTHONPATH - no luck.
EDIT 2:
I was able to successfully use klobucar's solution but I don't think it will work as a general solution since this utility is going to be used by several other developers. I know I can use some Python generalizations to determine the relative path for each user. I just feel like there is a more elegant solution.
Ultimately this script will run via cron to execute unit testing on several Python projects. This is also going to be available to each developer to ensure integration testing during their development.
I need to be able to make this more general.
EDIT 3:
I'm sorry, but I don't really like any of these solutions for what I'm trying to accomplish. I appreciate the input - and I'm using them as a temporary fix. As a complete fix I'm going to look into adding a script available in the site_packages directory that will add to the PYTHONPATH. This is something that is needed across several machines and several developers.
Once I build my complete solution I'll post back here with what I did and mark it as a solution.
EDIT 4:
I feel as though I didn't do a good job expressing my needs with this question. The answers below addressed this question well - but not my actual needs. As a result I have restructured my question in another post. I considered editing this one, but then the answers below (which would be very helpful for others) wouldn't be meaningful to the change and would seem out of place and irrelevant.
For the revised version of this question please see Unable to import Python package universally
You have 2 solutions:
Either put workspace in your PYTHONPATH:
import sys
sys.path.append('/path/to/workspace')
from utils import utils_dict
(Note that if you're running a script inside workspace, that is importing handle_management, most probably workspace is already in your PYTHONPATH, and you wouldn't need to do that, but it seems it's not the case HERE).
Or, make "workspace" a package by adding an empty (or not) __init__.py file in the workspace directory. Then:
from ..utils import utils_dict
I would prefer the second, because you would have a problem if there's another module called "utils" in you PYTHONPATH
Apart from that, you are importing wrong here: import utils.utils_dict.py. You don't need to include the extension of the file ".py". You are not importing the file, you are importing the module (or package if it's a folder), so you don't want the path to that file, you need its name.
What you need to do is add workspace to your import path. I would make a wrapper that does this for you in workspace or just put workspace in you PYTHONPATH as an environment variable.
import sys
# Add the workspace folder path to the sys.path list
sys.path.append('/path/to/workspace/')
from workspace.utils import utils_dict
Put "workspace/" in your PYTHONPATH to make the packages underneath available to you when it searches.
This can be done from your shell profile (if you are on *nix, or environment variables on windows.
For instance, on OSX you might add to your ~/.profile (if workspace is in your home directory):
export PYTHONPATH=$HOME/workspace:$PYTHONPATH
Another option is to use virtualenv to make your project area its own contained environment.
I do a lot of work on different projects (I'm a scientist) in a fairly standardised directory structure. e.g.:
project
/analyses/
/lib
/doc
/results
/bin
I put all my various utility scripts in /bin/ because cleanliness is next to godliness. However, I have to hard code paths (e.g. ../../x/y/z) and then I have to run things within ./bin/ or they break.
I've used Django and that has /manage.py which runs various django-things and automatically handles the path. I've also used fabric to run various user defined functions.
Question: How do I do something similar? and what's the best way? I can easily write something in /manage.py to inject the root dir into sys.path etc, but then I'd like to be able to do "./manage.py foo" which would run /bin/foo.py. Or is it possible to get fabric to call executables from a certain directory?
Basically - I want something easy and low maintenance. I want to be able to drop an executable script/file/whatever into ./bin/ and not have to deal with path issues or import issues.
What is the best way to do this?
Keep Execution at TLD
In general, try to keep your runtime at top-level. This will straighten out your imports tremendously.
If you have to do a lot of import addressing with relative imports, there's probably a
better way.
Modifying The Path
Other poster's have mentioned the PYTHONPATH. That's a great way to do it permanently in your shell.
If you don't want to/aren't able to manipulate the PYTHONPATH project path directly you can use sys.path to get yourself out of relative import hell.
Using sys.path.append
sys.path is just a list internally. You can append to it to add stuff to into your path.
Say I'm in /bin and there's a library markdown in lib/. You can append a relative paths with sys.path to import what you want.
import sys
sys.path.append('../lib')
import markdown
print markdown.markdown("""
Hello world!
------------
""")
Word to the wise: Don't get too crazy with your sys.path additions. Keep your schema simple to avoid yourself a lot confusion.
Overly eager imports can sometimes lead to cases where a python module needs to import itself, at which point execution will halt!
Using Packages and __init__.py
Another great trick is creating python packages by adding __init__.py files. __init__.py gets loaded before any other modules in the directory, so it's a great way to add imports across the entire directory. This makes it an ideal spot to add sys.path hackery.
You don't even need to necessarily add anything to the file. It's sufficient to just do touch __init__.py at the console to make a directory a package.
See this SO post for a more concrete example.
In a shell script that you source (not run) in your current shell you set the following environment variables:
PATH=$PATH:$PROJECTDIR/bin
PYTHONPATH=$PROJECTDIR/lib
Then you put your Python modules and package tree in your projects ./lib directory. Python automatically adds the PYTHONPATH environment variable to sys.path.
Then you can run any top-level script from the shell without specifying the path, and any imports from your library modules are looked for in the lib directory.
I recommend very simple top-level scripts, such as:
#!/usr/bin/python
import sys
import mytool
mytool.main(sys.argv)
Then you never have to change that, you just edit the module code, and also benefit from the byte-code caching.
You can easily achieve your goals by creating a mini package that hosts each one of your projects. Use paste scripts to create a simple project skeleton. And to make it executable, just install it via setup.py develop. Now your bin scripts just need to import the entry point to this package and execute it.
I'm using this code in one of my apps in Google App Engine. I encountered problems with the way that individual files are referenced. For example, in __init.py__ the files decorators.py, errors.py, etc are imported as follows:
import reddit.decorators
import reddit.errors
import reddit.helpers
import reddit.objects
Since the files are all within the same module, shouldn't they be imported like this instead:
import decorators
import errors
import helpers
import objects
The absolute reference works only if the reddit package is on the system path, which seems not to be the case in Google App Engine, for some reason.
Is this a problem with the source, or do I need to examine my application configuration within Google App Engine more closely?
If you want to use a package, you will have to install the whole directory somewhere where Python can find it, i.e., to a directory that is in sys.path. You should never attempt to use the package contents as standalone modules, since this is not how the package is designed for.
Since the working directory of your main script (.) is in sys.path, you should be able to use the reddit package simply by putting the whole package directory within the same directory as your main script. If you cannot import reddit in Google App Engine, you will have to check your setup there. Unfortunately, I do not know how GAE works or what you are allowed to install there, but I guess it should work, since they allow you to put arbitrary Python modules and packages to your webspace, don't they?
Concerning your original question, you are refering to the wrong section of the manual. For intra-package references, you should either use absolute imports:
import reddit.decorators as decorators
or relative ones:
from . import decorators
If the absolute import syntax works depends on your Python version. This is ambiguous:
import decorators
Do you mean a global module (/decorators.py)? Or a module within the package (/reddit/decorators.py)? Python 2.x will look for a relative import first, then try an absolute import if the relative one fails. Starting with version 2.6, using absolute-style imports is deprecated and should not be used any more. Since 3.0, the statement above will only be interpreted as absolute import and not look for a relative one. Explicit absolute imports will work as expected in both versions.
I have created a folder with all my modules for my GAE application and with external libraries like Jinja2 to keep everything sorted in one place. I have folders structure like this:
lib\
\utils\
\__init__.py
\firepython
\jinja2
\jsonpickle
__init__.py
sessions.py
When I try to load Jinja from utils__init__.py, I get error ImportError: No module named jinja2.environment. When I look at Jinja2 imports instructions, I see them look like from jinja2.loaders. I try to change them to be like from lib.jinja2.loaders but some other errors then appear about imports. More than that I don't think it's a good practice to change these imports in external libraries source if there is a more convenient and right way to import modules properly. I also have added some paths to PYTHONPATH but it doesn't solve all problems. How can I properly import an external package that is placed in another folder, may be with a deep structure?
Indeed you should not have to change imports in external libraries - though depending on your environment, you might even have too.
PYTHONPATH
Modifying your PYTHONPATH should suffice; PYTHONPATH should contain a 'lib' path that is either absolute or relative to your home, eg.
Then you could simply do
from jinja2 import WHATEVER
sys.path.append
Another way to go without PYTHONPATH is to use sys.path.append() and add your paths from your python code. I actually favor that, as it also allows to have per-application paths.
use virtualenv
Details would be a bit long to be put here, but please follow the official doc
These options applies to general python development rather than GAE specificities; if it does not work on your development machine you should post more details (exact imports, absolute paths, pythonpath...).
A proper project structure and use of appcfg.py should workout dependencies when uploading to google: please take a look at this good answer: How do I manage third-party Python libraries with Google App Engine? (virtualenv? pip?) and follow those guidelines.
A nice way to go with GAE is through yaml application directives. Please take a look at the doc for includes: http://code.google.com/appengine/docs/python/config/appconfig.html#Includes
Also remember that GAE officially supports python 2.5, and 2.7 support is experimental
Python 2.7 is now officially supported
To properly import a module, you need to make sure, that python knows where to find it.
To do so, for each external library append it's parent directory to the sys.path (in run-time), or setup PYTHONPATH environment (before running).
For example:
import sys
sys.path.append('/my/lib')
# now we can import from lib
import jsonpickle # will load /my/lib/jsonpickle/__init__.py
See http://docs.python.org/tutorial/modules.html#the-module-search-path . to understand what python does when you call import.