I would like to deploy an application on a cloudfoundry instance, consisting of an Rmarkdown file which should contain both R and python chunks. I'd like to render the document on cloudfoundry, rather than doing so locally and just pushing up the HTML file.
For knitr to work with python, it uses the reticulate package which states in the docs:
Note that for reticulate to bind to a version of Python it must be compiled with shared library support (i.e. with the --enable-shared flag).
I tried to use the cloudfoundry python buildpack to provide python for reticulate, but got the error Python shared library not found, Python bindings not loaded, and reticulate::py_discover_config() on the cloudfoundry app gives:
> reticulate::py_discover_config()
python: /home/vcap/deps/2/bin/python/ # or something very like this
libpython: [NOT FOUND]
which I think suggests that python hasn't been compiled with --enable-shared.
Tantalisingly, the changelog has this line * Add --enable-shared to Python runtime options from 6 years ago, but it looks to be for a specific version that is no longer relevant.
Is there a way I can get a version of python on cloudfoundry which will work with reticulate? Ideally without having to build it from scratch myself...
I have found that I can install a version of python that reticulate is happy to work with, using reticulate itself. Ahead of rendering the Rmarkdown document, I can use:
reticulate::install_miniconda()
reticulate::py_install(readLines("requirements.txt"))
This creates a conda environment with all my requirements and allows my code to run.
The disadvantage is that it shifts the python installation from the app staging (which has 15 minutes to complete by default on cloudfoundry), to the app start command (which has 60s by default). I had to increase the default timeout to 120s to give enough time to install the requirements and render the document.
Following #DanielMikusa's comment, I have also raised a feature request on the python buildpack, to hopefully enable shared library support in future, but this works for now for me.
Related
I have a VERY LONG script including the function "ox.distance.get_nearest_edge".
When I run it under my local computer with osmnx version 1.1.1, I get a warning that the function is deprecated and may be removed in a future release. I now need to let the script run on a server, I installed osmnx, but the script does'nt work anymore even though I installed the same version (1.1.1). (I also took care that python, geopandas, networkx etc were the same).
The error message I get is "AttributeError: 'osmnx.distance' has no attribute 'get_nearest_edge'"
(I tried to use the new function from osmnx instead (ox.distance.nearest_edges), but it seems to work very differently, and I run into errors trying to adapt my (long) script correcly).
Make old functions work would be a good way to keep old scripts working. But why is the funktion not becoming available again when I install the old osmnx-package with all of its dependencies? Do I have to install even older package versions of osmnx? I use anaconda package controlling, and JupyterLab to run my script (just in case that is a relevant detail).
Re-verify the version you're running in your script. The get_nearest_nodes and get_nearest_edges functions were deprecated in v1.1 but not removed until the release of v1.2. If you do want to use a newer version, you can take advantage of the much faster and more flexible nearest_nodes and nearest_edges functions that replaced them. See the docs and usage examples for details and demonstrations.
On a Linux server, I have some Python scripts using the built-in sqlite3 module (+ some Sqlite extensions built from source, as detailed in Upgrade Python's sqlite3 on Debian).
For another Python script, I need a newer version of the Sqlite shared library than the one I already have on the system. Reason: I need Sqlite higher than 3.25.0 for Window Functions.
If I install it from source here and do make and make install, it will probably overwrite previous versions of this library on the server, and could potentially break other OS tools using it.
How do you handle the general problem of having multiple versions of the Sqlite shared library?
I don't think Python virtual environments can be used for this context, or would it be possible?
Note: pip3 install --upgrade sqlite3 does not exist: we cannot upgrade Python's built-in sqlite3 package like this. And by the way we probably should not, since it could break some OS tools using Python + sqlite3.
This is very tricky and will need a little code change in your scripts.
What to do:
First, check the sqlite3 library version included with python just in case:
python -c "import sqlite3; print(sqlite3.connect(':memory:').execute('SELECT sqlite_version();').fetchall())
In my computer (python 3.8, windows) the output is [('3.35.5',)] which means python has the sqlite 3.35.5 library. I have no sqlite installed in my system: this is the library that comes with python3.8.
IF your python sqlite3 library is not the one you need :-( you have an alternative: you can use the pysqlite3 instead of the sqlite3 standard library. In this case:
You'll need to build the pysqlite3 library by yourself using the Sqlite3 'amalgamation' that matches the version you want to use (more on later).
You'll need to install the library, and...
You will need to change your python script imports
import pysqlite3 as sqlite3 # instead of sqlite3
Ok, what is the 'amalgamation` and how to build pysqlite3?
The amalgamation is the whole sqlite3 library in just one .c file (with the sqlite3.h file). You can get it from the sqlite3 download page: sqlite3.36 amalgamation.
Once you have the amalgamation, follow the instructions to build statically pysqlite3, and install the package.
Now you can use pysqlite3 in your code.
If you want 2 different version of sqlite3 (python3) on 2 different environments, you can do that.
Since you mentioned that sqlite3 is part of the std library, it seems like you can try the pysqlite3 package instead.
If you can't run pip, run the following command first.
sudo apt install python3-pip
Then,
pip install virtualenv
python3 -m venv sqlitev1 #(whatever name you want)
source sqlitev1/bin/activate
pip install pysqlite3==0.4.4 #(this can be whatever version u want)
source deactivate
python3 -m venv sqlitev2 #(whatever name you want)
source sqlitev2/bin/activate
pip install pysqlite3==0.4.4 #(this can be whatever version u want)
source deactivate
Now you have 2 python environments, sqlitev1 and sqlitev2, with 2 different version of sqlite3.
It might be super hacky but you can make the new version of sqlite and then make sure that the path pointing to the new version is on the pythonpath environment before the built in one. Python will scan the python path from first to last to find an import, so the new version first in the python path for the processes that want the new version and then exclude that path with the old processes that need the built in one. You can accomplish this with a bash script that loads the env and then runs the python process for the new services.
Again this is super hacky so last resort.
If you want a different version of Sqlite than that installed with your distro, and a Python that uses that version, then you could
Compile sqlite to an alternative location
Compile Python to a different location, and point it to the custom Sqlite installation.
The "pointing" is covered in the accepted answer to this question. The question body itself shows how you might compile sqlite to a custom location.
The other answer to that question proposes setting the LD_LIBRARY_PATH environment variable to the directory containing the custom sqlite build to avoid having to compile Python. This might work with a virtualenv (it could be set in the preactive hook, for example).
See also
What is LD_LIBRARY_PATH and how to use it?
Set LD_LIBRARY_PATH before importing in python
Another approach would be to compile pysqlite3 in a virtualenv over the custom sqlite build. You can read about this in this blog post (I won't copy the details as it isn't clear what licence is used by the blog).
On their Built-in Third-party Libraries page, it says that lxml 3.7.3 is supported but I can't seem to find a way to use that.
If I write in app.yaml:
- name: lxml
version: latest
When I log etree.LXML_VERSION inside the app, I get (2, 3, 0, 0).
Setting the version specifically to "3.7.3" (instead of latest) doesn't seem to work. When I try to deploy, gcloud says:
ERROR: (gcloud.app.deploy) An error occurred while parsing file: [... /app.yaml]
lxml version "3.7.3" is not supported, use one of: "2.3", "2.3.5"
in "... app.yaml", line 17, column 1
I have installed lxml locally inside the ./lib folder, but the folder is ignored on deploy.
Am I doing something wrong?
The problems seemed to happen because of an old google cloud sdk.
The sdk was installed using apt-get and somehow it was not getting updated. This also included an older Python SDK
At this moment the latest Python SDK is 1.9.62, I was using 1.9.52.
Updating the sdk seemed to fix the problem
sudo apt-get update && sudo apt-get install google-cloud-sdk
It wouldn't be the 1st time that inconsistencies exist between the documentation, what's included in the cloud SDK, what's included in the GAE language specific SDK and what's actually available on GAE (in production). See, for example, PyCharm - Can't create App Engine Application using DJango.
The deployment error message you got suggests that the 3.7.3 version doesn't actually exist on GAE, despite being marked as available in the documentation.
The lxml library is on the list of a few special libraries which need extra attention. Most likely because they aren't "pure Python code with no C extensions" and thus they can't be included in the SDKs with the other libraries, so they need to be installed separately on your local system. From Using built-in bundled libraries with the local development server (check that entire section for related info):
Many of the built-in libraries provided by the runtime are
automatically available to the local development server. However, the
following built-in libraries must be installed locally before you can
use them with the local development server:
lxml
matplotlib
mysqldb
numpy
PIL
crcmod
pycrypto
Now, if you really want the 3.7.3 version, you may be out of luck: if indeed it's not "pure Python code with no C extensions" then you also won't be able to vendor it into your app either. Still, it's worth a try. A new try, note that you also need to:
take it out of the app.yaml file's libraries: section - you're not requesting a GAE-provided library anymore
complete the entire vendoring in procedure (you didn't mention creating the appengine_config.py file, for example).
If that doesn't work then you'll have to settle for one of the available versions mentioned in the deployment error messages. You'll need to:
add that version to the app.yaml file's libraries: section
install that version on your local system
drop the vendoring in attempt, if you went for it
Kcachegrind serves as a wonderful utility to visually represent the hotspot to the source line level when profiling code. I found it pretty useful when micro optimizing my C++ code base.
For my latest python project I started using Kcachegrind to process the output from profilestats. Kcachegrind is a linux only utility but various unofficial ports are available and one I am using is qcachegrind.
Generally it works to a large extent and suffices for most issues except I am having a hard time getting the source annotation work.
On the source Tab I am being greeted with the familiar source missing message
There is no source available for the following function:
'main C:\Projects\module\src\source.py:397'
This is because no debug information is present
Recompile source and redo the profile run.
The function is located in the ELF Object:
'(unknown)'
Using the option
Settings -> Configure -> Source Annotation
and adding the Source Base Directory was not useful.
I have a feeling that the utility wants an ELF Object which is not relevant for Python. Any help in this regard would be useful.
Relevant Information:
Python 2.7
profilestats (2.0)
QCachegrind 0.7.4
Windows 2012R2
I second #taleniat comment. I am an OSX user. I was having some trouble getting qcachegrind to work so I ended up using pyprof2calltree and it works perfectly, source code tab included. YMMV.
First run your script with cProfile
python -m cProfile -o report.profile report.py
Then you can use pyprof2calltree to launch qcachegrind (no need for intermediate conversion).
pyprof2calltree -k -i report.profile
by the way, Python 2.7.10 and qcachegrind 0.7.4 installed via homebrew on OSX 10.11
Python has a win32service package that seems to allow windows service creation. I have carefully checked available google examples, for example this one:
Is it possible to run a Python script as a service in Windows? If possible, how?
I have placed code into ~/Documents/test.py and executed following under elevated command prompt:
> python test.py install
> python test.py debug
Unfortunately, every example i tried fails with error:
Debugging service TestService - press Ctrl+C to stop.
Error 0xC0000005 - Python could find the service class in the module
AttributeError: 'module' object has no attribute 'AppServerSvc'
(null): (null)
I have tested it with latest version of ActivePython (2.7.2.5 32-bit) on Windows 7 64-bit and Windows 8 64-bit. Of course, if service is started maually via service manager, it will not start either.
Am i doing something wrong or Python is not intended to be used for service creation on modern operating systems? Maybe i need a specific version of windows/python/pywin32? Of course, i can manually test all combinations starting from windows XP but it will take lots of time :(. Maybe someone already has experience with Python and windows services and can hint me what i'm doing wrong?
update
Tested on Windows XP 32-bit virtual macine (same code, same activepython distribution) - same error.
It appears that there is a bug in the ActiveState build for PyWin32. Installing ActiveState python 2.7.2.5 and running the linked sample code above, I get the same error that you are reporting.
But if I download Python 2.7.3 (2.7.2.5 does not seem to be available for download) and add PyWin32 build 214 (ActiveState seems to be using the 214 version of PyWin32). Then everything seems to work just fine. I also tried the latest build of PyWin32 (218) and it also worked correctly.
So I guess you can try reporting the problem to ActiveState (I don't have a support contract with them) and unless you have a requirement for using ActiveState, you can just switch to the standard Python builds.
I have been using services with the standard Python builds for years running on everything from Windows 2000 up to Server 2008 and Windows 7 with no problems. So I have good reason to believe that it will work for you also.
If you want to work with ActiveState to get the problem fixed, then the bug appears to be in their build of PythonService.cpp in the LoadPythonServiceInstance function. I looked at the registry entries that were created and they look fine, it is the PythonService.exe that is failing at loading your class. Based on the error message it appears to have loaded the module correctly and is just having trouble finding the class.