I am trying to add BigQuery to a Django-AppEngine project and I am finding a lot of problems in doing that since the google library for BigQuery use a package named google and this folder is already in use for AppEngine purposes. What I find now is that, if I install BigQuery, it will overwrite this package and then nothing works!
Is there maybe someone who faced this problem before and has any idea how to solve this?
It is some way to combine existing folders using pip or something else?
Thanks!!
Possible Solution:
For those who are facing the same problem, I was able to find a solution which was not too bad. Just creating an appengine_config.py file and adding them there:
from google.appengine.ext import vendor
vendor.add('sitepackages/prod')
...but still looking for a better solution.
*note: all my third party libraries are placed there instead of in a lib folder like Google says.
I could solved the problem I was facing by adding an extra appengine_config.py file containing this two lines:
from google.appengine.ext import vendor
vendor.add('sitepackages/prod')
This file is going to be automatically called by google appengine adding the libs on sitepackages/prod(in this case) to the libs on our virtualenv.
Thanks to #snakecharmerb for showing me the way to do this.
Related
I am quite new to Python and Django. I have a problem with integrating a python package (openpyxl) to my django app. I'd like to use the methods of these files into my views.py file.
My problem is first that I don't know where's the best place to put the openpyxl folder containing all the files in my file hierarchy.
My hierarchy looks like this:
http://imgur.com/t4iOX98
Is it well placed? Should I put it outside the international folder? inside the carte_interactive folder?
And my biggest problem is inside the __init__.py of openpyxl. I get errors lines like this one:
from openpyxl.xml import LXML
Where there is no resolved reference to LXML, but is actually defined in the xml file of openpyxl.
Is it my bad file placement that caused this? or is it Django?, or is it openpyxl's fault? Do anyone have an idea?
You can see openpyxl's source files here, where I downloaded them:
https://bitbucket.org/openpyxl/openpyxl/src
If you need any more details, please ask!
Thanks in advance!
I applaud your enthusiasm for wanting to learn Django while being new to Python. That said, the way you have things set up right now will make your life unnecessarily difficult to manage.
I would first recommend reading up on best practices for setting up a Django project. Just doing a quick google search for "Django project layout best practices" will give you a lot of resources, but they'll all essentially tell you to do what's in the SO answer above.
The second very basic thing is using pip to install and use other python packages. This is especially important for a django project, where you often have a lot of dependencies outside of Django. Pip is a program to install additionaly python packages. They get installed in your PYTHONPATH, which is just a list of filepaths on disk where python will look for additional packages. If you're on a *NIX system, this is usually in something like /usr/lib/python2.7/. Once you have something in your python path, you can from any piece of code, use other libraries you've installed via the python import system. Essentially, all this more or less does is look through each location in your PYTHONPATHs for the library you're trying to import.
Finally, in regards specifically to lxml, you will want to install it via apt or some other package installer. (e.g. on ubuntu, apt install python-lxml
In order to keep track of all your external python-dependencies, stuff them in a file named "requirements.txt" in the top level directory. This is a pretty standard thing to do for Django projects, so don't worry about shipping code with ALL dependencies inside the project.
Thanks to all of you! I'm using Jetbrains Pycharm and when I wrote import openpyxl, it gave me the choice to install the package. I suppose it does it with pip, which would certainly have worked the same. And I put the package in requirements.txt, so that other users would only have to install this requirement!
It works now! And thanks for the link on the best practices. I'll read that!
I am thinking about doing some test on app engine. And I have read about the library google has originally provided. I wonder what should I do if I want to use some external libraries that were installed before as part of Python2.7 such as Pyaudio and wx..
I was aware of that I should claim the library in the yaml file but what else do I need to do? Will naming and assigning a new environment variable help? If it does help, what paths should I include here? Thanks.
You should read up on the python sandbox and runtime restrictions, https://developers.google.com/appengine/docs/python/#Python_The_sandbox
This tells you how to use other libraries.
The libraries you list can not be used in appengine.
Also read up on 3rd party libraries that are directly supported https://developers.google.com/appengine/docs/python/tools/libraries27
I want to use the soundcloud python library in a web app I am developing in Google App Engine. However, I can't find any file called "soundcloud.py" in the soundcloud library files I downloaded. When using pip install it works fine on my local computer.
What files exactly do I need to move - or what exact steps do I need to take - in order to be able to "import soundcloud" within Google App Engine.
I already tried moving all the *.py files into my main app directory, but still got this error:
import soundcloud
ImportError: No module named soundcloud
You need to include your soundcloud external library into your App Package. Where exactly to place them within your application is fairly dependant on frameworks your application uses, but the soundcloud must exist on your PYTHONPATH somewhere.
Also, soundcloud isn't a solitary library. You'll also need fudge and simplejson (Included in appengine since 1.4.2). Just grab both package folders from your local pip install, and stick em somewhere in your application that's already loaded in your path.
To see where your local installed pip folders are, just fake that you're uninstalling one of the packages. On my machine, this looks like this:
sparker% sudo pip uninstall fudge
Password:
Uninstalling fudge:
/Library/Python/2.7/site-packages/fudge
/Library/Python/2.7/site-packages/fudge-1.0.3-py2.7.egg-info
Proceed (y/n)? n
You only need to grab the fudge folder to include in your application, but I'd take both. It's always nice later to know which version of applications you're bundling with your app, and that's the easiest way to know.
Ok, I think I figured out. What I needed to do what copy the soundcloud folder, along with the fudge, simplejson and requests folders into the root folder of my webapp. Thank you VooDooNOFX -- although you didn't directly answer the precise question, you sparked the thinking that helped me figure it out.
I would like to build an admin section into my App Engine app that will allow me to download and install python modules dynamically.
Any ideas on the most efficient way to accomplish this, without the ability to write to file system?
Are there any examples of this being done?
Does python27 support of setuptools make this easier?
Edit:
My initial thought is that this could be accomplished by downloading an egg or zip file dynamically. Saving it to the blobstore, and loading if from there.
Is this posible?
What kind of performance issues would this create?
On GAE you has no access to the file system, that's why you can't install any third-party packages on your instance, you can only distribute they with your own code.
In your app directory create a folder called 'lib'. For any module you want to add, just unzip it and add it to lib. Then redeploy your application using the console or Google App Launcher. Repeat every time you want to add a new module.
I'm not 100% sure about this. But it seems to me that if don't want a manual process involved then I would suggest dynamically adding the module content as a blob store entry and loading the module at runtime.
But the trick as the previous answer states is that to use a package, its code needs to be present in your app.
What's the best way to install two python modules with the same name? I currently depend on two different facebook libraries: pyfacebook and Facebook's new python-sdk. Both of these libraries install themselves as the module 'facebook'. I can think of a bunch of hacky solutions but before I go an hack away I was curious if there was a pythonic way of dealing with this situation.
I'm using virtualenv and pip.
(Yes, I will eventually deprecate one of them, but I had two different engineers working on two different problems and they didn't realize that they were using a different module until integration)
First, I'd suggest you guys go over what other libraries you're all using so you can get a concesus on how you're building your application.
To support this type of thing place each module within it's own folder, put in an __init__.py file, then you can do this:
import Folder1.facebook as pyfacebook
import Folder2.facebook as facebooksdk
The easiest solution would be to include one (or both) of the modules in your project instead of installing it. Then, you can have more control over the module name and importing.