How do you install a third party library that has other dependencies? I am trying to import Urban Airship into my python based Google App Engine project. I understand that the basic premise is that I do something like
import sys
sys.path.insert(0, 'libs')
from urbanairship import *
So I cloned urbanairship from their GitHub account. As an aside, the folder urbanairship actual is under another folder which was originally python-library. Through my several attempts at getting this working I have tried imports like
from python-library.urbanairship import *
from python-library.urbanairship import core
from urbanairship import *
from urbanairship import core
and then tried renaming python-library to pythonlibrary
from pythonlibrary.urbanairship import *
from pythonlibrary.urbanairship import core
A more core issue is that urbanairship has a setup process which installs other dependency libraries. So I ran:
sudo python setup.py install
That installed the requests dependency on my local machine which wasn't available in GAE, so I cloned the Requests package from Github also which had its own setup.py file to install more dependencies. Things still didn't work and I thought - this can't be the right way to do this which is why I'm not reaching out to SO for help. What is the proper way to do this?
You only need to copy the packages to your root directory (alongside app.yaml), which will put them in the global namespace for your app, and thus make them available in your code as if they were installed in the system.
As far as I can tell, in your case, you only need to copy the urbanairship and requests folder/package, and then import them as usual. If some other requirement arises, just rinse & repeat.
Just to make it clear, the rest of the code you see on the github repo includes docs, tests, and other files to make setup work properly, but you're only interested in the folder (urbanairship, requests) itself.
Related
I'm a beginner in Python and I have no experience with GitHub at all. I want to import the module semsimlib from the following URL: https://github.com/timvdc/semsimlib
I have looked on the internet for help on how to do this but most of it is very unclear and doesn't seem to work for me. Can anyone provide a detailed explanation on how to do this in a easy way?
It looks the repo does not provide appropriate scripts to simply install the package. There is no setup.py file and there is no distribution on pypi.
What you can do is go to site-packages folder inside your python installation or inside your virtual environment. Then run git clone https://github.com/timvdc/semsimlib. You should now be able to import semsimlib. Keep in mind that you will also have to install all the other dependencies your self one by one since there is also no requirements file.
You can also clone the repo into any folder on your computer and at the top of your script put:
import sys
sys.path.append("path/to/semsimlib/folder")
semsimlib will now be importable. However, I would try to get it to work with the first method.
Well I'm trying to use Datastore in a personal project using the Google App Engine. Though, I can't import the datastore module, no matter how hard I try.
I've been using the online console during the whole time (in order to avoid to have to solve problems first on my PC and then on GCloud...)
So, I'm using
from google.cloud import datastore
Unfortunately, that's not working at all. The last error I have is
ImportError: No module named google.protobuf
But before I had things like Can't import Datastore.
What I did was removing the integrality of /lib, and reinstalling every dependancy with pip. Here is my requirements.txt:
# This requirements file lists all third-party dependencies for this project.
#
# Run 'pip install -r requirements.txt -t lib/' to install these dependencies
# in `lib/` subdirectory.
#
# Note: The `lib` directory is added to `sys.path` by `appengine_config.py`.
Flask==0.10
google.cloud==0.25.0
protobuf==3.3.0
(The last line was added to try to resolve the last error I got). Before having this error, I got
Also, a little clarification question: I've seen (while looking for answers) people using gcloud and some using google.cloud. What's the difference? What should I use?
Also, pip show google.cloud shows nothing.
What am I missing?
Thank you
Well, if anyone is wondering, here's how I solved the problem. What fixed it was changing the Flask version to 0.12 (don't know why, but that's what happened).
I deleted lib to be sure I was starting from scratch. Then, I used this requirements.txt file:
Flask==0.12
google-cloud==0.25.0
click==5.1
(click is needed by Flask 0.12).
I keep getting ImportError: No module named twitter when it's in my lib module
Details:
the twitter module is located in the lib directory.
The lib directory has an empty init.py
I'm getting the error on a page just outside the directory
When I take the twitter module out of the lib directory and just say import twitter, it works
When I put the twitter module back inside the lib and say from lib import twitter, or when I say import twitter, or saying import lib.twitter it says, go to **** and like it
I tried doing this: https://cloud.google.com/appengine/docs/python/tools/libraries27?hl=en , which suggests to:
To tell your app how to find libraries in this directory, create (or modify) a file named appengine_config.py in the root of your project, then add these lines:
from google.appengine.ext import vendor
# Add any libraries installed in the "lib" folder.
vendor.add('lib')
and then tried import twitter, and tried from lib import twitter, and it says, "haha, nice try."
Now what?
tl;dr - the comments are right. 'lib' is not a module, it's a folder containing modules. It needs to be on your Python import path before your import twitter, using the vendor.add('lib'), which should set up your import path correctly when you deploy or use dev_appserver.py. If you place the vendor.add(lib) snippet in appengine_config.py file and run dev_appserver.py or deploy, and the twitter module is inside lib, you should be able to import twitter. If you're running outside of App Engine/devappserver, you'll need to make sure lib is on the PYTHONPATH.
Here is the longer explanation of what the lib folder is; what vendoring is; and how they fit into the bigger picture of managing dependencies on App Engine.
With App Engine, you can specify some dependencies provided by the sandbox in app.yaml. You can se the list of available libraries here. For any other dependencies (which can only be pure Python libraries), the libraries need to be directly included inside your project. Directly including the code for these dependencies as part of your project (rather than just listing them in requirements.txt and expecting them to get installed at deploy time) is commonly called 'vendoring'.
Now, even though we are including dependencies directly, we'd rather just use pip and a requirements.txt, for a few reasons I will mention below. But pip is usually used to install into libraries into the system libraries or a virtualenv, not into the project itself. So a feature was added to pip and a feature was added to App Engine to fix this.
The pip feature install dependencies into a folder rather than your system libraries or a virtualenv. You use the -t flag like this:
pip install -r requirements.txt -t lib
Where 'lib' is the folder to install into. This way, you still specify your dependencies in requirements.txt, and use pip to install them, they just get installed right into the directory specified. As the comments noted, the lib folder is itself not a module - it just contains them. So your lib folder should not have an __init__.py, it should just contain a folder like 'twitter' that does have an __init__.py. Since lib is not a module, from lib import twitter doesn't actually make sense. Also note that lib is an arbitrary name, but the one we usually pick by convention.
There are a few big advantages to vendoring using pip rather than just manually downloading dependencies and adding them to the project. One advantage is you don't need to check the dependencies into source control - just add the requirements.txt, and tell other users to use the pip -t command to vendor the dependencies too. Another advantage is it's just more organized to cleanly separate your code from third-party modules. The point of the vendoring features is to keep these advantages while still following App Engine requirements to include the dependencies in the directory at deploy time.
The App Engine vendor extension which you are using was made to recognize that a folder contains modules that have been 'vendored' and add it to the path. That is the vendor extension you are using in your snippet. It makes sure the lib folder is on your import path so you can import the modules you installed into it.
Once you run the vendor.add command, you should be able to import your modules in the lib folder. As the comment notes, you need to make sure it runs before import twitter. A good practice is to make sure those vendor commands run before anything else. You can accomplish this by putting that code inside a file named appengine_config.py in your directory. That's a special file that runs before anything else. Take a look at an example of doing that here.
A few last notes that might be helpful:
appengine_config.py will run in the GAE environment or when you run dev_appserver, which emulates the GAE environment. However if you're ever running outside of a GAE environment, make sure your PYTHONPATH includes the lib folder you want to import from.
Since you can vendor libraries using pip, you might ask why would you ever include a dependency using app.yaml? The answer is that since only pure Python libraries can be vendored, you still should use app.yaml dependencies for any libraries which need C libraries, such as MySQL.
If you have vendored libraries and app.yaml libaries, you don't want to vendor libraries that will be in the GAE sandbox since the versions might conflict. In that case, it's a good idea to have a separate requirements.txt for dependencies you want to vendor, and dependencies you want included only when running locally but that GAE will provide in the sandbox. Here is an example of that for MySQL.
Create a folder called "libs" (in the home directory of your webapp)
If one library, then just do: pip install -t libs/ your_library_name
if multiple libraries, do:
pip freeze > requirements.txt,
pip install -r requirements.txt -t libs
Now, create a file called appengine_config.py (in the home directory, again)
In the above file, add:
from google.appengine.ext import vendor
vendor.add('libs')
These get uploaded onto your app engine environment and you'll be able to run it!
I have a Flask project which I've put the flask module (version 0.9) directly beside my app.py file. I've done this so that I can bundle everything into a version control repository that won't require anyone else using it to install additional Python modules.
I want to use flask-login so I've tried to manually install it by downloading the latest version and putting the flask_login.py file in my "local" flask/ext/ directory. However, while I can import flask and import flask.ext, I am unable to import flask.ext.login with Python throwing ImportError: No module named flask.ext.login. import flask.ext.flask_login throws an import error as well.
Is there something I have to do differently if Flask and it's extensions are local to the app.py?
The solution is to put the flask_login.py file in the same directory as my app.py file. No modification of the flask/ext/__init__.py file is necessary.
The flask.ext module is intended only as a an extension importer, not a repository for installed extensions. Based on the import path, I thought the flask/ext folder was where extensions were to be copied. This was incorrect. Extensions simply need to be somewhere on the python path.
Python doesn't throw the exception in your case, this Flask module is responsible for modifying the standard import hook in the ext folder:
https://github.com/mitsuhiko/flask/blob/master/flask/exthook.py
So, I don't think putting flask_login.py into flask/ext is the right way to use extensions in Flask. The documentation recommends to use pip install flask-login or python setup.py install. After that you can do:
import flask_login
If you still want to do it manually, remove the setup() call from ext/__ init__.py.
It probably has a good reason why the Flask guys did it this way though :-)
Right, but the point was to manually install everything so it could be self-contained in a version control repository that won't require additional installs. I've looked at the setup.py file but can't determine how it achieves the installation. – Soviut Oct 25 at 10:06
Why not use virtualenv to achieve this? You can push your env to source control as well, although that's not normal usage.
See: http://www.virtualenv.org/en/latest/
I have the following folder structure
lib
my_module
I have moved all the libraries I need into the lib/ folder.
In my module/__init__.py, I think I will do:
import sys
sys.path.append('../lib/')
import my_dependency
Then when I need to use this dependency, I will refer to it as
my_module.my_dependency
Is this a bad usage of Python import?
NOTE: the dependencies consists of some third-party libraries not available via pip/easy_install and some C++ stuff that I wrote.
sys.path.append('../lib/') assumes that the current working directory is the directory of your script, which may or may not be the case.
A version that doesn't depend on the working directory is:
import sys, os
sys.path.append(os.path.join(os.path.split(os.path.split(os.path.abspath(sys.argv[0]))[0])[0], "lib"))
import my_dependency
The above in plain language takes the full path to the script, chops off the last two components (script directory and script filename) and appends lib.
If the libraries you use are third-party modules—especially popular ones—namespacing them like that is going to get inconsistent pretty fast. You will end up with code that sometimes is referenced as bar and sometimes as foo.bar. Maintaining such a codebase will not be worth whatever gains you expect to get from prefixing them like that.
If you keep third-party code in your own repository, consider replacing it with a requirements.txt file that can be fed to utilities like easy_install and pip. Both these tools support a --user switch that installs to your home directory (not touching system stuff).
pip install -r requirements.txt --user