I tried several different ways to embed the Python grpcio module into my Bazel project but unfortunately, none of them works correctly.
As far as I know, Bazel does not support injection of plugins into the Python environment (so you can directly run import grpcio). Note that Bazel does not support virtual-env and other tools (buildout...).
I found a way to include Pypi packages thanks to this proof of concept but unfortunately it does not work for grpc.io (environment variables are missing).
I am trying to debug the plugin, but I wonder if there is a better way to include grpcio module since the code is based on Bazel?
As Nathaniel mentioned in the comments, bazel support for gRPC Python is still work in progress.
However, pubref https://github.com/pubref/rules_protobuf offers rules for bazel that support gRPC for all languages (including Python). I have used them for Java and they worked very well.
There is also a gRPC blog post about those rules: http://www.grpc.io/blog/bazel_rules_protobuf
Related
I recently install Openrefine and it's great, especially enjoying the python execution option.
Within the python execution, one can import additional packages, this can be seen in this example where the random package is imported.
Example of Openrefine python execution which returns random word out of the first 50 words
Now, I want to use a special package within the Openrefine tool, which is installed on one of my Conda environments. can I activate a particular Conda env that will be executed in Openrefine tool?
TL;DR: just wrap your Python package with FastAPI and communicate via HTTP requests.
OpenRefine and Jython
OpenRefine is using Jython, a Java implementation of Python.
Therefore, you can not "just" activate a conda environment, but you have to provide a Jython compatible package.
There is a tutorial in the OpenRefine wiki describing how to extend Jython with PyPi modules.
Please note that currently 2.7 is the newest Jython implementation. Jython 3 is still in it's planing and development phase. See Jython 3 Roadmap for details. This makes it difficult to use external libraries, as Python 2 had its end of life on 01.01.2020 and accordingly (most) libraries stopped supporting Python 2.
Also, some Python packages rely on C libraries that are not compatible with Jython. Check the Appendix A of the Definitive Guide to Jython for more details on using external tools and libraries.
Alternative solution using FastAPI
Personally I find it easier to just wrap the Python packages I want to use with FastAPI and communicate from OpenRefine via HTTP requests. Depending on your data you can then add new columns by fetching URLs or use GET/POST requests in Jython.
I recently created a GitHub Gist showing how to wrap the NER component of a spaCy model with FastAPI to be then used via OpenRefine.
I have created a python library using sklearn and some other dependencies. I want other developers to be able to use it in their programs, in a non-public environment(e.g.within a organization) They will use this library for to write their own applications.
Some questions that I have are -
What is a best way to make it available to other developers?
Let's say , the developers have their own python installation, and they use a version 1.x of a package(e.g. sklearn etc) but my
package uses 2.x, will there be a problem? If yes, how can i ensure they
can use my library.
I want to make my library available for both Python 2.7 and 3.x users. Do I need two different deployments? Currently my library
works(no version specific calls for 2.7/3.x) in both 2.7 and 3.x, if
the correct dependencies are pre-installed by the user.
The best way is to publish at PyPI. That way your user have ti just run pip install $LIB and got it with all dependencies (if you properly configured dependencies). See Python Packaging User Guide.
Just recommend your users to use virtualenv. Virtual environments are the way to separate and install different versions of Python libraries and programs to coexist at one system.
Very much depends on the nature of your library. There are libraries that can be installed to both Python 2 and 3 from one source and there are libraries that require different package for every python version.
As a follow-up to How do I write a setup.py for a twistd/twisted plugin that works with setuptools, distribute, etc?, how does one make a debian package for a twisted plugin?
Assuming the setup.py is properly written, using cdbs/python-central/dh_python2 should just work, but I haven't had much luck so far.
The trick with those tools is that they basically run 'python setup.py install --root=' and then package up whatever ends up in '', so perhaps once the previous question is properly answered, then this question becomes moot?
Anyone here has successfully packaged a twisted plugin for debian?
Apparently the issue is with 'python-support', which is plain broken when it comes to twisted plugins.
This message from Ubuntu's Matthias Klose explains the issue and offers a solution:
packaging of twisted plugins with python-support is broken by design.
Even python policy mentions explicitly that you should use the same
packaging helper for packages sharing the same python namespace.
You should use dh_python2 for that, or (deprecated) build with
dh_pycentral using `include-links'.
-- Matthias Klose
Argh, I've tried to do this and failed. I think it's possible depending on which Debian/Ubuntu releases you want to target, and how much effort you want to put in.
There are two approaches:
Have your package stick the plugin file in twisted/plugins/ in the twisted tree. This is a pain because Twisted is packaged using different methods in different releases (python-support in Lucid vs dh_python2 in Natty IIRC) and (roughly speaking) your package needs to be packaged the same way as Twisted is to make this work.
Have a twisted/plugins/ directory installed alongside your code. Then, IIRC, the problem becomes having the forest of symlinks that gets created include the twisted directory (as it's not a package).
I am trying to define a process for migrating django projects from my development server to my production server using git, and it's driving me crazy that distutils installs python modules system-wide. I've read the documentation but unless I'm missing something it seems to be mostly about how to change the installation directory. I need to be able to use different versions of the same module in different projects running on the same server, and deploy projects from git without having to download and install dependencies.
tl;dr: I need to know how to install python modules, using distutils, into my project's source tree for version control without compromising other projects using different versions of the same module.
I'm new to python, so I apologize in advance if this is common knowledge.
Besides the already mentioned virtualenv which is a good option but has the potential drawback of requiring a third-party module, Distutils itself has options to install modules into arbitrary locations. In particular, there is the home scheme which allows you to "build and maintain a personal stash of Python modules". It's described in the Python documentation set here.
Perhaps you are looking for virtualenv. It will allow you to install packages into a separate virtual Python "root".
for completeness sake, virtualenvwrapper makes every day work with virtualenv a lot quicker and simpler once you are working on multiple projects and/or on multiple development platforms at the same time.
If you are looking something akin to npm or yarn of the JavaScript world or composer of the PHP world, then you may want to look at pipenv (not to be confused with pip). Here's a guide to get you started.
Alternatively there is also Poetry, which some people say is even better, but I haven't used it yet.
I'm building a Django app, which I comfortably run (test :)) on a Ubuntu Linux host. I would like to package the app without source code and distribute it to another production machine. Ideally the app could be run by ./runapp command which starts a CherryPy server that runs the python/django code.
I've discovered several ways of doing this:
Distributing the .pyc files only and building and installing all the requirements on target machine.
Using one of the many tools to package Python apps into a distributable package.
I'm really gunning for nr.2 option, I'd like to have my Django app contained, so it's possible to distribute it without needing to install or configure additional things. Searching the interwebs provided me with more questions than answers and a very sour taste that Django packing is an arcane art that everybody knows but nobody speaks about. :)
I've tried Freeze (fails), Cx_freeze (easy install version fails, repository version works, but the app output fails) and red up on dbuilder.py (which is supposed to work but doesn't work really - I guess). If I understand correctly most problems originate form the way that Django imports modules (example) but I have no idea how to solve it.
I'll be more than happy if anyone can provide any pointers or good resources online regarding packing/distributing standalone Django applications.
I suggest you base your distro on setuptools (a tool that enhances the standard Python distro mechanizm distutils).
Using setuptools, you should be able to create a Python egg containing your application. The egg's metadata can contain a list of dependencies that will be automatically installed by easy_install (can include Django + any third-party modules/packages that you use).
setuptools/distutils distros can include scripts that will be installed to /usr/bin, so that's how you can include your runapp script.
If you're not familiar with virtualenv, I suggest you take a look at that as well. It is a way to create isolated Python environments, it will be very useful for testing your distro.
Here's a blog post with some info on virtualenv, as well as a discussion about a couple of other nice to know tools: Tools of the Modern Python Hacker: Virtualenv, Fabric and Pip
The --noreload option will stop Django auto-detecting which modules have changed. I don't know if that will fix it, but it might.
Another option (and it's not ideal) is to obscure some of your core functionality by packaging it as a dll, which your plain text code will call.