I have a specific problem which might require a general solution. I am currently learning apache thrift. I used this guide.I followed all the steps and i am getting a import error as Cannot import module UserManager. So the question being How does python import lookup take place. Which directory is checked first. How does it move upwards?
How does sys.path.append('') work?
I found out the answer for this here. I followed the same steps. But i am still facing the same issue. Any ideas why? Anything more i should put up that could help debug you guys. ?
Help is appreciated.
On windows, Python looks up modules from the Lib folder in the default python path, for example from "C:\Python34\Lib\". You can add your Python libaries in a custom folder ("my-lib" or sth.) in there, but you need a file in order to tell Python that you can import from there. This file is called __init__.py , and is totally empty. That data structure should look like this:
my-lib
__init__.py
/myfolder
mymodule.py
(This is how every Python module works. For example urllib.request, it's at "%PYTHONPATH%\Lib\urllib\request.py")
You can import from the "mymodule.py" file by typing
import my-lib
and then using
mylib.mymodule.myfunction
or you can use
from my-lib import mymodule
And then just using the name of you function.
You can now use sys.path.append to append the path you pass into the function to the folders Python looks for the modules (Please note that thats not permanent). If the path of your modules should be static, you should consider putting these in the Lib folder. If that path is relative to your file you could look for the path of the file you execute from, and then append the sys.path relative to your file, but i reccomend using relative imports.
If you consider doing that, i recommend reading the docs, you can do that here: https://docs.python.org/3/reference/import.html#submodules
If I got you right, you're using Python 3.3 from Blender but try to include the 3.2 standard library. This is bound to give you a flurry of issues, you should not do that. Find another way. It's likely that Blender offers a way to use the 3.3 standard library (and that's 99% compatible with 3.2). Pure-Python third party library can, of course, be included by fiddling with sys.path.
The specific issue you're seeing now is likely caused by the version difference. As people have pointed out in the comments, Python 3.3 doesn't find the _tkinter extension module. Although it is present (as it works from Python 3.2), it is most likely in a .so file with an ABI tag that is incompatible with Blender's Python 3.3, hence it won't even look at it (much like a module.txt is not considered for import module). This is a good thing. Extension modules are highly version-specific, slight ABI mismatches (such as between 3.2 and 3.3, or two 3.3 compiled with different options) can cause pretty much any kind of error, from crashes to memory leaks to silent data corruption or even something completely different.
You can verify whether this is the case via import _tkinter; print(_tkinter.file) in the 3.2 shell. Alternatively, _tkinter may live in a different directory entirely. Adding that directory won't actually fix the real issue outlined above.
For any new readers coming along that are still having issues, try the following. This is cleaner than using sys.path.append if your app directory is structured with your .py files that contain functions for import underneath your script that imports those files. Let me illustrate.
Script that imports files: main.py
Function files named like: func1.py
main.py
/functionfolder
__init__.py
func1.py
func2.py
The import code in your main.py file should look as follows:
from functionfolder import func1
from functionfolder import func2
As Agilix correctly stated, you must have an __init__.py file in your "functionfolder" (see directory illustration above).
In addition, this solved my issue with Pylance not resolving the import, and showing me a nagging error constantly. After a rabbit-hole of sifting through GitHub issues, and trying too many comparatively complicated proposed solutions, this ever-so-simple solution worked for me.
You may try with declaring sys.path.append('/path/to/lib/python') before including any IMPORT statements.
I just created a __init__.py file inside my new folder, so the directory is initialised, and it worked (:
Related
A few hours ago I was careless enough to name my short script as code.py. Apparently, there is such a package which is used e.g. by ptvsd or pdb. This led to my code.py to be imported instead and caused a bunch of nested unhandled exceptions with missing imports upon trying to debug my code. What was making it more frustrating is that traceback didn't show any sign of importing my code.py file, so I spent quite a while to find the source of the problem.
I'd like to avoid such situations in the future, so my question is: what's the best practice to ensure that the modules you use aren't importing your code by mistake due to such a name collision?
This is a common gotcha, and actually there's no failsafe way to avoid it. At least you can make sure your modules all live in packages (at least one package if that's a small project with no reusable code) so that you'd use them as from mypackage import code instead of import code (also make sure you use either absolute imports etc), and that you always run your code from the directory containing the package(s), not from within the package directory itself (python inserts the current working directory in first position of sys.path).
This won't prevent ALL possible name masking issues but it should minimize them. Now from experience, once you've experienced this kind of issues at least once, you usually spot the symptoms very quickly - the most common and quite obvious being that some totally unrelated stlib or third-part module starts crashing with ImportErrors or AttributeErrors (with "module X has no attribute Y" messages). At this point, if you just added a new module to your own code, chances are it's the new module that breaks everything, so you can just rename it (make sure you clean up .pyo/.pyc files if any) and see if it solves the issue. Else check the traceback to find out which imports fails, most of the time you'll find you have a module or package by the same name in your current working directory.
You can't avoid completely, that somebody is able to import your module by mistake.
You can structure your code better in subpackages going from "well known" to "less known" names. E.g. if you are developing code for a certain company then you might want to structure like:
company.country.location.department.function
If your code is then getting more accepted and used by others, you can bring the code in the upper namespace, so that it is made available in company.country.location.department.function
and company.country.location.department
You can modify sys.path at the beginning of your main module, before you start importing other modules:
import sys
sys.path.append(sys.path.pop(0))
so that the main module's starting directory is placed at the last of the module search paths rather than at the front, in order for other modules of the same name to take precedence.
EDIT: To all the downvoters, this answer actually works.
For example, running code.py with the following content:
import pdb
pdb.run('print("Hello world")')
would raise:
AttributeError: module 'pdb' has no attribute 'run'
because code.py has no run defined, while running code.py with the following content instead:
import sys
sys.path.append(sys.path.pop(0))
import pdb
pdb.run('print("Hello world")')
would execute pdb.run properly:
> <string>(1)<module>()
(Pdb)
I am running python 3.4 on the main.py file in the same directory.
/root directory is not in python path. It is simply the current directory that the script is executing in. All pycache folders were deleted after each test
So why exactly is __init__.py important? I thought it was necessary as stated in this post:
What is __init__.py for?
If you remove the init.py file, Python will no longer look for submodules inside that directory, so attempts to import the module will fail.
Right now, it seems to me that __init__.py is nothing more than an optional constructor where we do housekeeping and other optional things like specifying the "all" variable, etc. But not a critical item to have.
Image showing the results of the test:
Can someone explain the discrepancy or what is the cause of this issue?
As confusing as it may be, although the basics will work without __init__.py files, you should probably still use them. Many external tools, as well as package-related functions in the standard library, will not work as expected without them. More words of wisdom here (as well as a misleading accepted answer): Is __init__.py not required for packages in Python 3.3+.
Found Answer
In essence, init.py is not needed, and its purpose is for legacy and optional housekeeping tasks that you may or may not want or need in Python versions 2.7 vs 3.0+. However, it is important to take into account that they have slightly different behavior during more complex parsing if you are building something more complex.
Please refer to the following links for additional reading material:
https://www.python.org/dev/peps/pep-0420/#namespace-packages-today
How do I create a namespace package in Python?
What's the difference between a Python module and a Python package?
https://softwareengineering.stackexchange.com/questions/276888/python-namespace-vs-module-with-underscores
I have two Python packages where one needs to be imported by the other. The directory structure is like follows:
workspace/
management/
__init__.py
handle_management.py
other_management.py
utils/
__init__.py
utils_dict.py
I'm trying to import functionality from the utils project in the handle_management.py file:
import utils.utils_dict
Error I'm getting when trying to run handle_management.py:
ImportError: No module named utils.utils_dict
I've read a lot about how to resolve this problem and I can't seem to find a solution that works.
I started with Import a module from a relative path - I tried the applicable solutions but none worked.
Is the only solution to make workspace/ available via site_packages? If so, what is the best way to do this?
EDIT:
I've tried to add the /home/rico/workspace/ to the PYTHONPATH - no luck.
EDIT 2:
I was able to successfully use klobucar's solution but I don't think it will work as a general solution since this utility is going to be used by several other developers. I know I can use some Python generalizations to determine the relative path for each user. I just feel like there is a more elegant solution.
Ultimately this script will run via cron to execute unit testing on several Python projects. This is also going to be available to each developer to ensure integration testing during their development.
I need to be able to make this more general.
EDIT 3:
I'm sorry, but I don't really like any of these solutions for what I'm trying to accomplish. I appreciate the input - and I'm using them as a temporary fix. As a complete fix I'm going to look into adding a script available in the site_packages directory that will add to the PYTHONPATH. This is something that is needed across several machines and several developers.
Once I build my complete solution I'll post back here with what I did and mark it as a solution.
EDIT 4:
I feel as though I didn't do a good job expressing my needs with this question. The answers below addressed this question well - but not my actual needs. As a result I have restructured my question in another post. I considered editing this one, but then the answers below (which would be very helpful for others) wouldn't be meaningful to the change and would seem out of place and irrelevant.
For the revised version of this question please see Unable to import Python package universally
You have 2 solutions:
Either put workspace in your PYTHONPATH:
import sys
sys.path.append('/path/to/workspace')
from utils import utils_dict
(Note that if you're running a script inside workspace, that is importing handle_management, most probably workspace is already in your PYTHONPATH, and you wouldn't need to do that, but it seems it's not the case HERE).
Or, make "workspace" a package by adding an empty (or not) __init__.py file in the workspace directory. Then:
from ..utils import utils_dict
I would prefer the second, because you would have a problem if there's another module called "utils" in you PYTHONPATH
Apart from that, you are importing wrong here: import utils.utils_dict.py. You don't need to include the extension of the file ".py". You are not importing the file, you are importing the module (or package if it's a folder), so you don't want the path to that file, you need its name.
What you need to do is add workspace to your import path. I would make a wrapper that does this for you in workspace or just put workspace in you PYTHONPATH as an environment variable.
import sys
# Add the workspace folder path to the sys.path list
sys.path.append('/path/to/workspace/')
from workspace.utils import utils_dict
Put "workspace/" in your PYTHONPATH to make the packages underneath available to you when it searches.
This can be done from your shell profile (if you are on *nix, or environment variables on windows.
For instance, on OSX you might add to your ~/.profile (if workspace is in your home directory):
export PYTHONPATH=$HOME/workspace:$PYTHONPATH
Another option is to use virtualenv to make your project area its own contained environment.
I have created a folder with all my modules for my GAE application and with external libraries like Jinja2 to keep everything sorted in one place. I have folders structure like this:
lib\
\utils\
\__init__.py
\firepython
\jinja2
\jsonpickle
__init__.py
sessions.py
When I try to load Jinja from utils__init__.py, I get error ImportError: No module named jinja2.environment. When I look at Jinja2 imports instructions, I see them look like from jinja2.loaders. I try to change them to be like from lib.jinja2.loaders but some other errors then appear about imports. More than that I don't think it's a good practice to change these imports in external libraries source if there is a more convenient and right way to import modules properly. I also have added some paths to PYTHONPATH but it doesn't solve all problems. How can I properly import an external package that is placed in another folder, may be with a deep structure?
Indeed you should not have to change imports in external libraries - though depending on your environment, you might even have too.
PYTHONPATH
Modifying your PYTHONPATH should suffice; PYTHONPATH should contain a 'lib' path that is either absolute or relative to your home, eg.
Then you could simply do
from jinja2 import WHATEVER
sys.path.append
Another way to go without PYTHONPATH is to use sys.path.append() and add your paths from your python code. I actually favor that, as it also allows to have per-application paths.
use virtualenv
Details would be a bit long to be put here, but please follow the official doc
These options applies to general python development rather than GAE specificities; if it does not work on your development machine you should post more details (exact imports, absolute paths, pythonpath...).
A proper project structure and use of appcfg.py should workout dependencies when uploading to google: please take a look at this good answer: How do I manage third-party Python libraries with Google App Engine? (virtualenv? pip?) and follow those guidelines.
A nice way to go with GAE is through yaml application directives. Please take a look at the doc for includes: http://code.google.com/appengine/docs/python/config/appconfig.html#Includes
Also remember that GAE officially supports python 2.5, and 2.7 support is experimental
Python 2.7 is now officially supported
To properly import a module, you need to make sure, that python knows where to find it.
To do so, for each external library append it's parent directory to the sys.path (in run-time), or setup PYTHONPATH environment (before running).
For example:
import sys
sys.path.append('/my/lib')
# now we can import from lib
import jsonpickle # will load /my/lib/jsonpickle/__init__.py
See http://docs.python.org/tutorial/modules.html#the-module-search-path . to understand what python does when you call import.
In a Python system for which I develop, we usually have this module structure.
mymodule/
mymodule/mymodule/feature.py
mymodule/test/feature.py
This allows our little testing framework to easily import test/feature.py and run unit tests. However, we now have the need for some shell scripts (which are written in Python):
mymodule/
mymodule/scripts/yetanotherfeature.py
mymodule/test/yetanotherfeature.py
yetanotherfeature.py is installed by the module Debian package into /usr/bin. But we obviously don't want the .py extension there. So, in order for the test framework to still be able to import the module I have to do this symbolic link thingie:
mymodule/
mymodule/scripts/yetanotherfeature
mymodule/scripts/yetanotherfeature.py # -> mymodule/scripts/yetanotherfeature
mymodule/test/yetanotherfeature.py
Is it possible to import a module by filename in Python, or can you think of a more elegant solution for this?
The imp module is used for this:
daniel#purplehaze:/tmp/test$ cat mymodule
print "woho!"
daniel#purplehaze:/tmp/test$ cat test.py
import imp
imp.load_source("apanapansson", "mymodule")
daniel#purplehaze:/tmp/test$ python test.py
woho!
daniel#purplehaze:/tmp/test$
You could most likely use some tricker by using import hooks, I wouldn't recommend it though. On the other hand I would also probably do it the other way around , have your .py scripts somewhere, and make '.py'less symbolic links to the .py files. So your library could be anywhere and you can run the test from within by importing it normall (since it has the py extension), and then /usr/bin/yetanotherfeature points to it, so you can run it without the py.
Edit: Nevermind this (at least the hooks part), the import imp solution looks very good to me :)
Check out imp module:
http://docs.python.org/library/imp.html
This will allow you to load a module by filename. But I think your symbolic link is a more elegant solution.
Another option would be to use setuptools:
"...there’s no easy way to have a script’s filename match local conventions on both Windows and POSIX platforms. For another, you often have to create a separate file just for the “main” script, when your actual “main” is a function in a module somewhere... setuptools fixes all of these problems by automatically generating scripts for you with the correct extension, and on Windows it will even create an .exe file..."
https://pythonhosted.org/setuptools/setuptools.html#automatic-script-creation