I keep getting ImportError: No module named twitter when it's in my lib module
Details:
the twitter module is located in the lib directory.
The lib directory has an empty init.py
I'm getting the error on a page just outside the directory
When I take the twitter module out of the lib directory and just say import twitter, it works
When I put the twitter module back inside the lib and say from lib import twitter, or when I say import twitter, or saying import lib.twitter it says, go to **** and like it
I tried doing this: https://cloud.google.com/appengine/docs/python/tools/libraries27?hl=en , which suggests to:
To tell your app how to find libraries in this directory, create (or modify) a file named appengine_config.py in the root of your project, then add these lines:
from google.appengine.ext import vendor
# Add any libraries installed in the "lib" folder.
vendor.add('lib')
and then tried import twitter, and tried from lib import twitter, and it says, "haha, nice try."
Now what?
tl;dr - the comments are right. 'lib' is not a module, it's a folder containing modules. It needs to be on your Python import path before your import twitter, using the vendor.add('lib'), which should set up your import path correctly when you deploy or use dev_appserver.py. If you place the vendor.add(lib) snippet in appengine_config.py file and run dev_appserver.py or deploy, and the twitter module is inside lib, you should be able to import twitter. If you're running outside of App Engine/devappserver, you'll need to make sure lib is on the PYTHONPATH.
Here is the longer explanation of what the lib folder is; what vendoring is; and how they fit into the bigger picture of managing dependencies on App Engine.
With App Engine, you can specify some dependencies provided by the sandbox in app.yaml. You can se the list of available libraries here. For any other dependencies (which can only be pure Python libraries), the libraries need to be directly included inside your project. Directly including the code for these dependencies as part of your project (rather than just listing them in requirements.txt and expecting them to get installed at deploy time) is commonly called 'vendoring'.
Now, even though we are including dependencies directly, we'd rather just use pip and a requirements.txt, for a few reasons I will mention below. But pip is usually used to install into libraries into the system libraries or a virtualenv, not into the project itself. So a feature was added to pip and a feature was added to App Engine to fix this.
The pip feature install dependencies into a folder rather than your system libraries or a virtualenv. You use the -t flag like this:
pip install -r requirements.txt -t lib
Where 'lib' is the folder to install into. This way, you still specify your dependencies in requirements.txt, and use pip to install them, they just get installed right into the directory specified. As the comments noted, the lib folder is itself not a module - it just contains them. So your lib folder should not have an __init__.py, it should just contain a folder like 'twitter' that does have an __init__.py. Since lib is not a module, from lib import twitter doesn't actually make sense. Also note that lib is an arbitrary name, but the one we usually pick by convention.
There are a few big advantages to vendoring using pip rather than just manually downloading dependencies and adding them to the project. One advantage is you don't need to check the dependencies into source control - just add the requirements.txt, and tell other users to use the pip -t command to vendor the dependencies too. Another advantage is it's just more organized to cleanly separate your code from third-party modules. The point of the vendoring features is to keep these advantages while still following App Engine requirements to include the dependencies in the directory at deploy time.
The App Engine vendor extension which you are using was made to recognize that a folder contains modules that have been 'vendored' and add it to the path. That is the vendor extension you are using in your snippet. It makes sure the lib folder is on your import path so you can import the modules you installed into it.
Once you run the vendor.add command, you should be able to import your modules in the lib folder. As the comment notes, you need to make sure it runs before import twitter. A good practice is to make sure those vendor commands run before anything else. You can accomplish this by putting that code inside a file named appengine_config.py in your directory. That's a special file that runs before anything else. Take a look at an example of doing that here.
A few last notes that might be helpful:
appengine_config.py will run in the GAE environment or when you run dev_appserver, which emulates the GAE environment. However if you're ever running outside of a GAE environment, make sure your PYTHONPATH includes the lib folder you want to import from.
Since you can vendor libraries using pip, you might ask why would you ever include a dependency using app.yaml? The answer is that since only pure Python libraries can be vendored, you still should use app.yaml dependencies for any libraries which need C libraries, such as MySQL.
If you have vendored libraries and app.yaml libaries, you don't want to vendor libraries that will be in the GAE sandbox since the versions might conflict. In that case, it's a good idea to have a separate requirements.txt for dependencies you want to vendor, and dependencies you want included only when running locally but that GAE will provide in the sandbox. Here is an example of that for MySQL.
Create a folder called "libs" (in the home directory of your webapp)
If one library, then just do: pip install -t libs/ your_library_name
if multiple libraries, do:
pip freeze > requirements.txt,
pip install -r requirements.txt -t libs
Now, create a file called appengine_config.py (in the home directory, again)
In the above file, add:
from google.appengine.ext import vendor
vendor.add('libs')
These get uploaded onto your app engine environment and you'll be able to run it!
Related
I'm trying to follow the tutorial at https://www.juliensobczak.com/tell/2016/12/26/anki-scripting.html
And I'm getting the "listcards.py" basic script set up, having cloned anki and installed the virtual environment as well as the requirements from the anki/requirements.txt file.
However, when I run the script from the tutorial entitled "listcards.py", I get a notice that the module 'anki.sched' is not found. ("ModuleNotFoundError: No module named 'anki.sched')
While I could pip install each package, I have a feeling that there must be an underlying reason that these packages are missing- is there a way to have python automatically pull in the named module even if it isn't pre-installed in the manner of how node.js installed referenced dependencies automatically, so that I won't have to manually install every missing package?
I ran into this same problem. anki.sched is a package contained within the anki repository you cloned, so it does exist on your machine. You won't be able to install it using pip.
The solution for me was to write the absolute path of the anki repository you cloned in sys.path.append rather than a relative path. For example, if your script exists in /Users/anki/scripts and your cloned anki repository exists in /Users/anki/anki write this in your script before importing anki modules:
sys.path.append("/Users/anki/anki")
rather than this (which is what is provided in the tutorial):
sys.path.append("../anki")
I'm not 100% sure why the latter fails, but Anki must be looking for the anki.sched module in the wrong place due to the relative reference.
What I did, and I know this is probably not the correct way, is to simply wipe out the root anki folder and copy all the application scripts to it, then the imports worked.
I have made a GUI application using python and PyQt5. I want to package this app but there doesn't seems to be a straight forward way to do this. Moreover what I have found answers to is to package a python module and not an application. I have read various articles and the official docs but still don't seem to have a proper answer to this, though there are several workarounds through which I could achieve the same, I just want to know what is the standard way.
This is my directory structure :
Moodly/
Moodly/
__init__.py
controller.py
logic.py
models.py
view.py
resoure.py
style.py
sounds/
notify.wav
message.wav
setup.py
MANIFEST.in
setup.cfg
run.py
moodly.png
Moodly.desktop
What do I want to achieve: The user is given with a tar file of Moodly. The user extracts it, runs the command
python setup.py install
in the terminal, the setup places all the files in the proper place and creates a Moodly.desktop file probably in usr/local/share/applications clicking on which user can run the app.
My way of achieving this:
setup.py
from setuptools import setup
setup(
name="Moodly",
version="1.0",
author="Akshay Agarwal",
author_email="agarwal.akshay.akshay8#gmail.com",
packages=["Moodly"],
include_package_data=True ,
url="http://github.com/AkshayAgarwal007/Moodly",
entry_points = {
'gui_scripts': [
'moodly = Moodly.controller:main',
],
},
# license="LICENSE.txt",
description="Student Intimation system",
# long_description=open("README.txt").read(),
# Dependent packages (distributions)
)
MANIFEST.in
include Moodly/sounds/notify.wav
include Moodly/sounds/message.wav
Now with no setup.cfg I run the command:
python setup.py install
This succesfully installs Moodly to /usr/lib/python-3.4/site-packages
alongwith the sounds directory.And now from the terminal when I type in moodly(as specified in entry points in setup.py) my GUI application launches successfully.
Now I just need the setup to create the Moodly.desktop alongwith moodly.png in usr/local/share/applications which I am trying to achieve through this:
setup.cfg
[install]
install_data=/usr/local/share/applications
Adding this to setup.py
data_files = [("Moodly", ["moodly.png","Moodly.desktop",])],
But this somehow seems to copy the files inside python-3.4/site-packages/Moodly rather than the specified destination but it used to work well with distutils
This guy also seems to have faced the same issue
Some other links I have used:
python-packaging
starting with distutils
So the way I am trying to do it , how much of it is correct and what is the standard way to do it. How can I possibly place that Moodly.desktop in the right place or what could be a better alternative way to do the entire process.
Moreover would using Pyinstaller be a better idea. Pyinstaller would package the app with PyQt5, requests and beautifulsoup4 (external modules that I have used) which I don't want. I want to use the install_requires option provided by setuptools and not unnecessary make the user download the modules which they already might have.
The .desktop file isn't supposed to be installed using Distutils. Distutils is only concerned with installing Python packages.
To install .desktop files, icons and other files incidental to distribution level packaging, you should look at build automation systems, such as CMake.
The first step in this process is to get CMake to build a Python project. You should take a look here for how to do that: https://bloerg.net/2012/11/10/cmake-and-distutils.html
Beyond that, installing .desktop files is easy. Assuming you've written a .desktop file and put it somewhere, installing it is a matter of doing:
install(PROGRAMS com.akshay.moodly.desktop DESTINATION ${XDG_APPS_INSTALL_DIR})
in your CMakeLists.txt file.
Note that you install the .desktop file to ${XDG_APPS_INSTALL_DIR} (that's a CMake variable), not a hardcoded path like /usr/local/share/applications or something. The user (and pretty much every automated distro package builder) will always install your package to a temporary path and then copy files over into their packages. Never assume that your app will live in /usr/bin or /usr/local/bin or whatever. The user could install things into /opt/Moodly or even $HOME/Moodly.
I'm trying to install the icalendar Python module for use in my Google App Engine project. I have created a Python virtual environment in my project folder, and installed the icalendar package in it. If I run the Python interpreter from the venv, I can import icalendar and use it just fine (help(icalendar) shows it's getting the module from the correct path: venv/lib/python2.7/site-packages/icalendar). I have created a symlink called icalendar pointing to that directory in my project root (same place as app.yaml). The project can deploy and run on GAE, but anytime it tries to use icalendar, I get "Cannot import module 'icalendar'".
I used the exact same steps to install the Twilio library in my app, which worked fine. If I don't include the symlink to the Twilio module, the error message is actually 'No module named twilio', instead of "Cannot import..". Not sure if that's relevant.
I've seen this: https://groups.google.com/forum/?fromgroups=#!topic/google-appengine/FM_NXd9cbus, which is exactly my problem, but there was no solution!
What am I doing wrong?
Edit: I also tried just putting the icalendar module directory in the project root, w/o the symlink. No difference.
You need to check how to use python libraries in GAE.
Pay attention as the docs are a bit confusing, you need to pick one (and only one) solution for a library, you can't mix both (they're both chapters on the above-mentioned page):
Requesting a library - for libs supplied by GAE - for these check notes for what you may need to do for your local development server
Installing a library - for additional libraries you need to include in your app - for these check also the restrictions the library needs to meet
I have a Flask project which I've put the flask module (version 0.9) directly beside my app.py file. I've done this so that I can bundle everything into a version control repository that won't require anyone else using it to install additional Python modules.
I want to use flask-login so I've tried to manually install it by downloading the latest version and putting the flask_login.py file in my "local" flask/ext/ directory. However, while I can import flask and import flask.ext, I am unable to import flask.ext.login with Python throwing ImportError: No module named flask.ext.login. import flask.ext.flask_login throws an import error as well.
Is there something I have to do differently if Flask and it's extensions are local to the app.py?
The solution is to put the flask_login.py file in the same directory as my app.py file. No modification of the flask/ext/__init__.py file is necessary.
The flask.ext module is intended only as a an extension importer, not a repository for installed extensions. Based on the import path, I thought the flask/ext folder was where extensions were to be copied. This was incorrect. Extensions simply need to be somewhere on the python path.
Python doesn't throw the exception in your case, this Flask module is responsible for modifying the standard import hook in the ext folder:
https://github.com/mitsuhiko/flask/blob/master/flask/exthook.py
So, I don't think putting flask_login.py into flask/ext is the right way to use extensions in Flask. The documentation recommends to use pip install flask-login or python setup.py install. After that you can do:
import flask_login
If you still want to do it manually, remove the setup() call from ext/__ init__.py.
It probably has a good reason why the Flask guys did it this way though :-)
Right, but the point was to manually install everything so it could be self-contained in a version control repository that won't require additional installs. I've looked at the setup.py file but can't determine how it achieves the installation. – Soviut Oct 25 at 10:06
Why not use virtualenv to achieve this? You can push your env to source control as well, although that's not normal usage.
See: http://www.virtualenv.org/en/latest/
I have the following folder structure
lib
my_module
I have moved all the libraries I need into the lib/ folder.
In my module/__init__.py, I think I will do:
import sys
sys.path.append('../lib/')
import my_dependency
Then when I need to use this dependency, I will refer to it as
my_module.my_dependency
Is this a bad usage of Python import?
NOTE: the dependencies consists of some third-party libraries not available via pip/easy_install and some C++ stuff that I wrote.
sys.path.append('../lib/') assumes that the current working directory is the directory of your script, which may or may not be the case.
A version that doesn't depend on the working directory is:
import sys, os
sys.path.append(os.path.join(os.path.split(os.path.split(os.path.abspath(sys.argv[0]))[0])[0], "lib"))
import my_dependency
The above in plain language takes the full path to the script, chops off the last two components (script directory and script filename) and appends lib.
If the libraries you use are third-party modules—especially popular ones—namespacing them like that is going to get inconsistent pretty fast. You will end up with code that sometimes is referenced as bar and sometimes as foo.bar. Maintaining such a codebase will not be worth whatever gains you expect to get from prefixing them like that.
If you keep third-party code in your own repository, consider replacing it with a requirements.txt file that can be fed to utilities like easy_install and pip. Both these tools support a --user switch that installs to your home directory (not touching system stuff).
pip install -r requirements.txt --user