I have a few questions regarding python setup scripts or rather how to properly setup a module (as I am doing this for the first time and kind of struggling).
For simplicity I just post a link to the corresponding github repository rather than explaining the project in detail. I am fully aware that the project as it is will not work (e.g. the file constants.py is missing) but for starters I would like the "structure" to work.
There are two main components in this project, i.e. pymap and agb - both dependent on each other (which should not be a problem I guess). I also would like to use scripts located in the bin/ directory which of course use the modules pymap and agb. For installation I use sudo ./setup.py develop which installs the modules as I can now use them in a python3 shell. The line import pymap.pymap_gui will throw an error (since constants.py is not yet in the project) however the import can be resolved.
When - on the other hand - calling the scripts with pymap.py the same import can not even be resolved:
ModuleNotFoundError: No module named 'pymap.pymap_gui'; 'pymap' is not a package
How can this be even though from a python3 shell the import works perfectly fine?
Furthermore - where to improve my project structure? Is my setup the way to go (not minding the messy code and not yet working project itself - I am viewing this from kinda structure-ish perspective).
The problem was simply that the module had the same name as the script (pymap and pymap.py). Sorry for bothering!
Related
I have a specific problem which might require a general solution. I am currently learning apache thrift. I used this guide.I followed all the steps and i am getting a import error as Cannot import module UserManager. So the question being How does python import lookup take place. Which directory is checked first. How does it move upwards?
How does sys.path.append('') work?
I found out the answer for this here. I followed the same steps. But i am still facing the same issue. Any ideas why? Anything more i should put up that could help debug you guys. ?
Help is appreciated.
On windows, Python looks up modules from the Lib folder in the default python path, for example from "C:\Python34\Lib\". You can add your Python libaries in a custom folder ("my-lib" or sth.) in there, but you need a file in order to tell Python that you can import from there. This file is called __init__.py , and is totally empty. That data structure should look like this:
my-lib
__init__.py
/myfolder
mymodule.py
(This is how every Python module works. For example urllib.request, it's at "%PYTHONPATH%\Lib\urllib\request.py")
You can import from the "mymodule.py" file by typing
import my-lib
and then using
mylib.mymodule.myfunction
or you can use
from my-lib import mymodule
And then just using the name of you function.
You can now use sys.path.append to append the path you pass into the function to the folders Python looks for the modules (Please note that thats not permanent). If the path of your modules should be static, you should consider putting these in the Lib folder. If that path is relative to your file you could look for the path of the file you execute from, and then append the sys.path relative to your file, but i reccomend using relative imports.
If you consider doing that, i recommend reading the docs, you can do that here: https://docs.python.org/3/reference/import.html#submodules
If I got you right, you're using Python 3.3 from Blender but try to include the 3.2 standard library. This is bound to give you a flurry of issues, you should not do that. Find another way. It's likely that Blender offers a way to use the 3.3 standard library (and that's 99% compatible with 3.2). Pure-Python third party library can, of course, be included by fiddling with sys.path.
The specific issue you're seeing now is likely caused by the version difference. As people have pointed out in the comments, Python 3.3 doesn't find the _tkinter extension module. Although it is present (as it works from Python 3.2), it is most likely in a .so file with an ABI tag that is incompatible with Blender's Python 3.3, hence it won't even look at it (much like a module.txt is not considered for import module). This is a good thing. Extension modules are highly version-specific, slight ABI mismatches (such as between 3.2 and 3.3, or two 3.3 compiled with different options) can cause pretty much any kind of error, from crashes to memory leaks to silent data corruption or even something completely different.
You can verify whether this is the case via import _tkinter; print(_tkinter.file) in the 3.2 shell. Alternatively, _tkinter may live in a different directory entirely. Adding that directory won't actually fix the real issue outlined above.
For any new readers coming along that are still having issues, try the following. This is cleaner than using sys.path.append if your app directory is structured with your .py files that contain functions for import underneath your script that imports those files. Let me illustrate.
Script that imports files: main.py
Function files named like: func1.py
main.py
/functionfolder
__init__.py
func1.py
func2.py
The import code in your main.py file should look as follows:
from functionfolder import func1
from functionfolder import func2
As Agilix correctly stated, you must have an __init__.py file in your "functionfolder" (see directory illustration above).
In addition, this solved my issue with Pylance not resolving the import, and showing me a nagging error constantly. After a rabbit-hole of sifting through GitHub issues, and trying too many comparatively complicated proposed solutions, this ever-so-simple solution worked for me.
You may try with declaring sys.path.append('/path/to/lib/python') before including any IMPORT statements.
I just created a __init__.py file inside my new folder, so the directory is initialised, and it worked (:
I'm using Python to develop few company-specific applications. There is a custom shared module ("library") that describes some data and algorithms and there are dozens of Python scripts that work with this library. There's quite a lot of these files, so they are organized in subfolders
myproject
apps
main_apps
app1.py
app2.py
...
utils
util1.py
util2.py
...
library
__init__.py
submodule1
__init__.py
file1.py
...
submodule2
...
Users want to run these scripts by simply going, say, to myproject\utils and launching "py util2.py some_params". Many of these users are developers, so quite often they want to edit a library and immediately re-run scripts with updated code. There are also some 3rd party libraries used by this project and I want to make sure that everyone is using the same versions of these libs.
Now, there are two key problems I encountered:
how to reference (library) from (apps)?
how to manage 3rd party dependencies?
The first problem is well-familiar to many Python developers and was asked on SO for many times: it's quite difficult to instruct Python to import package from "....\library". I tested several different approaches, but it seems that python is reluctant to search for packages anywhere, but in standard libraries locations or the folder of the script itself.
Relative import doesn't work since script is not a part of a library (and even if it was, this still doesn't work when script is executed directly unless it's placed in the "root" project folder which I'd like to avoid)
Placing .pth file (as one might think from reading this document) to script folder apparently doesn't have any effect
Of course direct meddling with sys.path work, but boilerplate code like this one in each and every one of the script files looks quite terrible
import sys, os.path
here = os.path.dirname(os.path.realpath(__file__))
module_root = os.path.abspath(os.path.join(here, '../..'))
sys.path.append(python_root)
import my_library
I realize that this happens because Python wants my library to be properly "installed" and that's indeed would be the only right way to go had this library was developed separately from the scripts that use it. But unfortunately it's not the case and I think that re-doing "installation" of library each time it's changed is going to be quite inconvenient and prone to errors.
The second problem is straightforward. Someone adds a new 3rd party module to our app/lib and everyone else start seeing import problems once they update their apps. Several branches of development, different moments when user does pip install, few rollbacks - and everyone eventually ends using different versions of 3rd party modules. In my case things are additionally complicated by the fact that many devs work a lot with older Python 2.x code while I'd like to move on to Python 3.x
While looking for a possible solution for my problems, I found a truly excellent virtual environments feature in Python. Things looked quite bright:
Create a venv for myproject
Distribute a Requirements.txt file as part of app and provide a script that populates venv accordingly
Symlink my own library to venv site_packages folder so it'll be always detected by Python
This solution looked quite natural & robust. I'm explicitly setting my own environment for my project and place whatever I need into this venv, including my own lib that I can still edit on the fly. And it indeed work. But calling activate.bat to make this python environment active and another batch file to deactivate it is a mess, especially on Windows platform. Boilerplate code that is editing sys.path looks terrible, but at least it doesn't interfere with UX like this potential fix do.
So there's a question that I want to ask.
Is there a way to bind particular python venv to particular folders so python launcher will automatically use this venv for scripts from these folders?
Is there a better alternative way to handle this situation that I'm missing?
Environment for my project is Python 3.6 running on Windows 10.
I think that I finally found a reasonable answer. It's enough to just add shebang line pointing to python interpreter in venv, e.g.
#!../../venv/Scripts/python
The full project structure will look like this
myproject
apps
main_apps
app1.py (with shebang)
app2.py (with shebang)
...
utils
util1.py (with shebang)
util2.py (with shebang)
...
library
__init__.py
submodule1
__init__.py
file1.py
...
submodule2
...
venv
(python interpreter, 3rd party modules)
(symlink to library)
requirements.txt
init_environment.bat
and things work like this:
venv is a virtual python environment with everything that project needs
init_environment.bat is a script that populates venv according to requirements.txt and places a symlink to my library into venv site-modules
all scripts start with shebang line pointing (with relative path) to venv interpreter
There's a full custom environment with all the libs including my own and scripts that use it will all have very natural imports. Python launcher will also automatically pick Python 3.6 as interpreter & load the relevant modules whenever any user-facing script in my project is launched from console or windows explorer.
Cons:
Relative shebang won't work if a script is called from other folder
User will still have to manually run init_environment.bat to update virtual environment according to requirements.txt
init_environment scrip on Windows require elevated privileges to make a symlink (but hopefully that strange MS decision will be fixed with upcoming Win10 update in April'17)
However I can live with these limitations. Hope that this will help others looking for similar problems.
Would be still nice to still hear other options (as answers) and critics (as comments) too.
So recently I have found about a NEAT algorithm and wanted to give it a try using NEAT-Python(not sure if this is even the correct source :| ). So I created my virtual environment activated it and installed the neat-python using pip in the VE. When I then tried to run one of the examples from their GitHub page it threw an error like this:
ImportError: No module named visualize
So I checked my source files, and actually the neat-python doesn't include the visualize.py script, however it is in their GitHub repository. I then tried to add it myself by downloading just the visualize.oy script dragging it inside my VE and adding it to all the textfiles the NEAT brought with it, like the installed-filex.txt etc. However it still threw the same error.
I'm still fairly new to VE and GitHub so please don't be too hard on me :] thanks in advance.
-Jorge
I think you could simply copying the visualize.py into the same directory as the script you are running.
If you wanted it in your lib/site-packages directory so you could import it with the neat module:
copy visualize.py into lib/site-packages/neat/ and modify __init__.py to add the line import neat.visualize as visualize. Delete the __pycache__ directory. Make sure you have modules installed: Numpy, GraphViz, and Matplotlib. When you've done the above, you should be able to import neat and access neat.visualize.
I don't recommend doing this though for several reasons:
Say you wanted to update your neat module. Your visualize.py file is technically not part of the module. So it wouldn't be updated along with your neat module.
the visualize.py file seems to be written in the context of the examples as opposed to being for general use with the module, so contextually, it doesn't belong there.
At some point in the future, you might also forget that this wasn't a part of the module, but your code acts as if it was part of the API. So your code will break in some other neat installation.
I'm trying to make an application in Python that will be run from a USB drive on a variety of different computers. Let's just assume for a second that the client computer has Python installed and put that to one side.
My application uses cherrypy to launch a local web server, but it's safe to assume that this won't be already installed on the computer it's run on.
What would be the best of addressing this problem as transparently as possible?
It should be usable by a non-technical user and they shouldn't have to worry about installing any dependencies themselves.
At the moment, I've packaged a copy of the cherrypy source with my app and if the program detects that it isn't already installed, it runs the setup.py but with the --user flag, avoiding any permissions issues.
This doesn't seem like great practice.
Is it possible to just include the built package with my app? I've tried doing this with something like the following:
/myapp/libs/cherrypy
I copied the cherrypy directory from the CherryPy archive. This is pre-build, and so probably won't work. I also tried an alternative where I run setup.py (which created a build directory) and copied those files instead.
/myapp/somefile.py
import myapp.libs.cherrypy as cherrypy
In theory, this should work... But it doesn't. When I try to run the server (using cherrpy.quickstart()) it runs two instances of the server, which obviously starts causing problems.
So, the question(s): First off, am I going about this completely the wrong way? How can I make a third-party package available to my app?
This is due to PYTHONPATH issues. I would recommend using virtual envs and pip as standard when working with packages you've imported or obatined externally.
Some great notes here: https://python-guide.readthedocs.org/en/latest/
If you wnat to import your own code. I'd set your PYTHONPATH (in the case below the dev_folder) to a root development directory and follow this structure...
dev_folder \
- project_name \
- main_script.py
- helper.py
- libary1 \
- __init__.py
- lib1.py
- libary2 \
- __init__.py
- lib2.py
You'd obviously come up with better names for the library folders/packages :-)
Hope this helps.
I have two Python packages where one needs to be imported by the other. The directory structure is like follows:
workspace/
management/
__init__.py
handle_management.py
other_management.py
utils/
__init__.py
utils_dict.py
I'm trying to import functionality from the utils project in the handle_management.py file:
import utils.utils_dict
Error I'm getting when trying to run handle_management.py:
ImportError: No module named utils.utils_dict
I've read a lot about how to resolve this problem and I can't seem to find a solution that works.
I started with Import a module from a relative path - I tried the applicable solutions but none worked.
Is the only solution to make workspace/ available via site_packages? If so, what is the best way to do this?
EDIT:
I've tried to add the /home/rico/workspace/ to the PYTHONPATH - no luck.
EDIT 2:
I was able to successfully use klobucar's solution but I don't think it will work as a general solution since this utility is going to be used by several other developers. I know I can use some Python generalizations to determine the relative path for each user. I just feel like there is a more elegant solution.
Ultimately this script will run via cron to execute unit testing on several Python projects. This is also going to be available to each developer to ensure integration testing during their development.
I need to be able to make this more general.
EDIT 3:
I'm sorry, but I don't really like any of these solutions for what I'm trying to accomplish. I appreciate the input - and I'm using them as a temporary fix. As a complete fix I'm going to look into adding a script available in the site_packages directory that will add to the PYTHONPATH. This is something that is needed across several machines and several developers.
Once I build my complete solution I'll post back here with what I did and mark it as a solution.
EDIT 4:
I feel as though I didn't do a good job expressing my needs with this question. The answers below addressed this question well - but not my actual needs. As a result I have restructured my question in another post. I considered editing this one, but then the answers below (which would be very helpful for others) wouldn't be meaningful to the change and would seem out of place and irrelevant.
For the revised version of this question please see Unable to import Python package universally
You have 2 solutions:
Either put workspace in your PYTHONPATH:
import sys
sys.path.append('/path/to/workspace')
from utils import utils_dict
(Note that if you're running a script inside workspace, that is importing handle_management, most probably workspace is already in your PYTHONPATH, and you wouldn't need to do that, but it seems it's not the case HERE).
Or, make "workspace" a package by adding an empty (or not) __init__.py file in the workspace directory. Then:
from ..utils import utils_dict
I would prefer the second, because you would have a problem if there's another module called "utils" in you PYTHONPATH
Apart from that, you are importing wrong here: import utils.utils_dict.py. You don't need to include the extension of the file ".py". You are not importing the file, you are importing the module (or package if it's a folder), so you don't want the path to that file, you need its name.
What you need to do is add workspace to your import path. I would make a wrapper that does this for you in workspace or just put workspace in you PYTHONPATH as an environment variable.
import sys
# Add the workspace folder path to the sys.path list
sys.path.append('/path/to/workspace/')
from workspace.utils import utils_dict
Put "workspace/" in your PYTHONPATH to make the packages underneath available to you when it searches.
This can be done from your shell profile (if you are on *nix, or environment variables on windows.
For instance, on OSX you might add to your ~/.profile (if workspace is in your home directory):
export PYTHONPATH=$HOME/workspace:$PYTHONPATH
Another option is to use virtualenv to make your project area its own contained environment.