On their Built-in Third-party Libraries page, it says that lxml 3.7.3 is supported but I can't seem to find a way to use that.
If I write in app.yaml:
- name: lxml
version: latest
When I log etree.LXML_VERSION inside the app, I get (2, 3, 0, 0).
Setting the version specifically to "3.7.3" (instead of latest) doesn't seem to work. When I try to deploy, gcloud says:
ERROR: (gcloud.app.deploy) An error occurred while parsing file: [... /app.yaml]
lxml version "3.7.3" is not supported, use one of: "2.3", "2.3.5"
in "... app.yaml", line 17, column 1
I have installed lxml locally inside the ./lib folder, but the folder is ignored on deploy.
Am I doing something wrong?
The problems seemed to happen because of an old google cloud sdk.
The sdk was installed using apt-get and somehow it was not getting updated. This also included an older Python SDK
At this moment the latest Python SDK is 1.9.62, I was using 1.9.52.
Updating the sdk seemed to fix the problem
sudo apt-get update && sudo apt-get install google-cloud-sdk
It wouldn't be the 1st time that inconsistencies exist between the documentation, what's included in the cloud SDK, what's included in the GAE language specific SDK and what's actually available on GAE (in production). See, for example, PyCharm - Can't create App Engine Application using DJango.
The deployment error message you got suggests that the 3.7.3 version doesn't actually exist on GAE, despite being marked as available in the documentation.
The lxml library is on the list of a few special libraries which need extra attention. Most likely because they aren't "pure Python code with no C extensions" and thus they can't be included in the SDKs with the other libraries, so they need to be installed separately on your local system. From Using built-in bundled libraries with the local development server (check that entire section for related info):
Many of the built-in libraries provided by the runtime are
automatically available to the local development server. However, the
following built-in libraries must be installed locally before you can
use them with the local development server:
lxml
matplotlib
mysqldb
numpy
PIL
crcmod
pycrypto
Now, if you really want the 3.7.3 version, you may be out of luck: if indeed it's not "pure Python code with no C extensions" then you also won't be able to vendor it into your app either. Still, it's worth a try. A new try, note that you also need to:
take it out of the app.yaml file's libraries: section - you're not requesting a GAE-provided library anymore
complete the entire vendoring in procedure (you didn't mention creating the appengine_config.py file, for example).
If that doesn't work then you'll have to settle for one of the available versions mentioned in the deployment error messages. You'll need to:
add that version to the app.yaml file's libraries: section
install that version on your local system
drop the vendoring in attempt, if you went for it
Related
I would like to deploy an application on a cloudfoundry instance, consisting of an Rmarkdown file which should contain both R and python chunks. I'd like to render the document on cloudfoundry, rather than doing so locally and just pushing up the HTML file.
For knitr to work with python, it uses the reticulate package which states in the docs:
Note that for reticulate to bind to a version of Python it must be compiled with shared library support (i.e. with the --enable-shared flag).
I tried to use the cloudfoundry python buildpack to provide python for reticulate, but got the error Python shared library not found, Python bindings not loaded, and reticulate::py_discover_config() on the cloudfoundry app gives:
> reticulate::py_discover_config()
python: /home/vcap/deps/2/bin/python/ # or something very like this
libpython: [NOT FOUND]
which I think suggests that python hasn't been compiled with --enable-shared.
Tantalisingly, the changelog has this line * Add --enable-shared to Python runtime options from 6 years ago, but it looks to be for a specific version that is no longer relevant.
Is there a way I can get a version of python on cloudfoundry which will work with reticulate? Ideally without having to build it from scratch myself...
I have found that I can install a version of python that reticulate is happy to work with, using reticulate itself. Ahead of rendering the Rmarkdown document, I can use:
reticulate::install_miniconda()
reticulate::py_install(readLines("requirements.txt"))
This creates a conda environment with all my requirements and allows my code to run.
The disadvantage is that it shifts the python installation from the app staging (which has 15 minutes to complete by default on cloudfoundry), to the app start command (which has 60s by default). I had to increase the default timeout to 120s to give enough time to install the requirements and render the document.
Following #DanielMikusa's comment, I have also raised a feature request on the python buildpack, to hopefully enable shared library support in future, but this works for now for me.
I am trying to push a flask app (Python 3.5) to Cloud Foundry (CF). The application takes a POST request (text file) and returns a message. It works locally (tested via Postman). However, when attempting to push it to CF, it gives the error -
ImportError: No module named queue
Here is my code which contains queue.
import queue as Queue
self._batch_queue = Queue.Queue(self.BATCH_QUEUE_MAX)
self._example_queue = Queue.Queue(self.BATCH_QUEUE_MAX * self._hps.batch_size)
I've tried the solutions suggested here, but neither of these solve my problem. I think the issue is with the Python in CF not having queue package. (I could be wrong).
Anyone ideas on how to go about solving this will be very appreciated. Thanks in advance!
As mentioned in the comments by #KlausD, it seems like you have the wrong version of Python installed. On Cloud Foundry, you would set the version by including a file called runtime.txt in the root of your project (i.e. the directory from which you're running cf push).
https://docs.cloudfoundry.org/buildpacks/python/index.html#runtime
That file is used to tell the Python buildpack which version of Python to install for you. Suggestions would be python-3.5.x or python-3.6.x which would install the latest 3.5 or 3.6 release. You can specify an exact version like python-3.5.5 but it's not recommended as it's easy to forget to update that file when new versions of Python come out.
You can see which versions of Python are supported by the buildpack here.
https://buildpacks.cloudfoundry.org/#/buildpacks/python/v1.6.17
(Note that link goes to the latest version of the buildpack at the time I wrote this, it will get out of date. In the future, just click the most recent version of the buildpack to see what ships with it).
I've installed mesos 0.26 successfully on a vm machine.
The installation has been performed on an ubuntu trusty thar system
by following this manual:
https://open.mesosphere.com/getting-started/install/
So far so good. I wanted to write a tiny python framwork.
For this I need to install the eggs via easy_install:
(I've downloaded the eggs accordingly for the trusty thar ubuntu and the 0.26 mesos):
wget http://downloads.mesosphere.io/master/ubuntu/14.04/mesos-0.26.0-py2.7-linux-x86_64.egg
easy_install mesos-0.26.0-py2.7-linux-x86_64.egg
all went fine, however, if I start python in the shell
and type in
import mesos.interface
I get the message: ImportError: No module named interface
As someone suggested, it may be that there is no longer a binding for python, or that they have renamed the API calls. Well, I looked in the version updates here:
http://mesos.apache.org/documentation/latest/upgrades/
Since the transition from 0.19.x to 0.20.x there hasn't been any changes regarding the mesos.interface part, or at least they are not mentioning it here.
To increase the confusion I also get the following error message when I'm typing in python: import mesos.native
There I receive: ImportError: No module named interface.mesos_pb2. To put it into a nutshell: what is going wrong here, and how can it be fixed? And yes, I've googled various web pages, with terms such as "mesos python bindings", mesos +api +python, etc. And yes, I have also consulted the official mesos webpage. There are nice refences for Java and C++ but not for python, or at least they are very well hidden.
Thanks in advance for any hints.
Solved. For what reasons ever:
export PYTHONPATH=${PYTHONPATH}:/usr/lib/python2.7/site-packages/
is required to set the PYTHONPATH. After that step it works like a charm.
What python version does app engine use?
This is related to problems I'm having on mac osx with getting pycrypto to work. Because the python version now in mac is 2.7.1. When Google says the python version for app engine is '2.7' do they mean precisely 2.7.0? Or is python 2.7.1 okay too?
Can this cause problems for my local install of pycrypto? ( which I installed with pip: sudo pip install pycrypto==2.6 )The pycrypto library works correctly, but it somehow messes up app engine running a development server. The logging doesn't show all errors or logs which I expect to appear there.
When Google says the python version for app engine is '2.7' do they mean precisely 2.7.0? Or is python 2.7.1 okay too?
Any subversion of python 2.7 works. 2.7.0 works, but I have used 2.7.6 and 2.7.10 extensively as well. So I'm sure that 2.7.1 will work too.
Can this cause problems for my local install of pycrypto? ( which I installed with pip using: 'sudo pip install pycrypto==2.6' )The pycrypto library works correctly, but it somehow messes up app engine running a development server. The logging doesn't show all errors or logs which I expect to appear there.
I know you have asked about this before but I will give you an other (better) way to include pycrypto in your appengine project.
In your app.yaml, specify the following:
libraries:
- name: pycrypto
version: "2.6" # or "latest", or "2.3"
See https://cloud.google.com/appengine/docs/python/tools/libraries27 for more info about which libraries are supported by appengine and how you can import which versions.
It seems as though you're asking the question which inspired your title, about python verson 2.7.x, as a way to get insight into your actual problem, which is related to pycrypto. Tim's answer dealt with the question of which sub-versions of 2.7.x are supported.
It appears you're using both the app.yaml libraries stanza and pip to get an instance of pycrypto. So:
Is the pip-installed pycrypto residing on your dev-box somewhere like /usr/local/lib/python-2.7, or is it in the app's folder, specifically?
When the dev-server is running, is it possible your python path has it using the pycrypto installed by pip on your system, with that pycrypto being a different version than the one in production? This might cause different behaviour on the dev-server vs. production.
...All this said, it still seems odd, as Tim has pointed out, that you should see a disruption to logging based on this. What is it you're logging, and are you using any exception handling which might be dropping the responsibility to log?
Finally, is there a setting when running the dev-server which you can use that would enable more verbosity? Try that to see if you can raise the logging level enough to make the exceptions you're looking for visible.
These are my thoughts so far on how to get out of your jam.
I'm trying to install the icalendar Python module for use in my Google App Engine project. I have created a Python virtual environment in my project folder, and installed the icalendar package in it. If I run the Python interpreter from the venv, I can import icalendar and use it just fine (help(icalendar) shows it's getting the module from the correct path: venv/lib/python2.7/site-packages/icalendar). I have created a symlink called icalendar pointing to that directory in my project root (same place as app.yaml). The project can deploy and run on GAE, but anytime it tries to use icalendar, I get "Cannot import module 'icalendar'".
I used the exact same steps to install the Twilio library in my app, which worked fine. If I don't include the symlink to the Twilio module, the error message is actually 'No module named twilio', instead of "Cannot import..". Not sure if that's relevant.
I've seen this: https://groups.google.com/forum/?fromgroups=#!topic/google-appengine/FM_NXd9cbus, which is exactly my problem, but there was no solution!
What am I doing wrong?
Edit: I also tried just putting the icalendar module directory in the project root, w/o the symlink. No difference.
You need to check how to use python libraries in GAE.
Pay attention as the docs are a bit confusing, you need to pick one (and only one) solution for a library, you can't mix both (they're both chapters on the above-mentioned page):
Requesting a library - for libs supplied by GAE - for these check notes for what you may need to do for your local development server
Installing a library - for additional libraries you need to include in your app - for these check also the restrictions the library needs to meet