I want to use neo4j for large graph querying (12M nodes, 1.5G relations). I have tested the performance, and using cypher it's unsatisfactory as for the web-server.
Since java API query is >10x faster than the cypher, I want to write necessary query functions in java and handle everything in Python (my website backend is written in Python).
Can you give me some hints how to approach the problem of running Java functions within Python?
I have managed to run JPype. But only one java instance can access the embedded database. For that, py4j using background JVM process may be better solution. Yet, I cannot make py4j working. Do you have an experience with py4j? Or Python and neo4j java API?
The best way to achieve this is writing unmanaged extensions to your Neo4j server in Java. These unmanaged extensions expose new REST endpoints which can then be called from python via http.
A while ago I've created a kind of template for unmanaged extensions using gradle as build system.
Related
I'm trying to implement the python DB-API for a small "database" that we built internally. This database does not expose an ODBC interface (or JDBC for that matter). My goal is to create a sqlalchemy for this so that I can use it with an application like Superset for example. I have created JDBC drivers in the past and that requires full Java implementation of the methods from the interfaces. In case of Python's DB-API, I couldn't find any example. Even the one I saw with psycopg2 https://github.com/psycopg/psycopg2 is fully written in C and I'm not an expert on C.
Any way to implement the DB-API only in python? Is there any examples available? (Sorry if my understanding of db-api is not correct.)
You can find plenty of DB-API drivers written in Python. The specific libraries depend on how your database communicates and packs/unpacks data.
If you're database is listening on a port, you'll probably be using the socket module.
If you're doing any kind of batch inserting or unpacking, you want to check out the struct module as well.
If you don't need support for Python 2, with Python 3.3+ you get memoryview().cast(), which may also come handy with regard to unpacking data.
Python 3.8 comes with the shared memory module, which can help you out when you start optimizing.
If your database runs on a specific platform, ctypes comes handy for pulling out OS specific tweaks (Like manually implementing shared memory if you can't use Python 3.8).
Pandas used to support DB-API connections directly. It currently only supports the SQlite DB-API officially, but you can piggyback it, which will allow you to test yourself with a known tool.
I am leaning Python programming language. I have no problems with Python. I read Python official docs, and I can write small programs in Python. I want to familiarize myself with mysql database because it is useful in learning software development concepts. I've installed mysql database and Django on my computer. I have Ubuntu 14.04 and python 3.4 installed. I've configured Django settings to use mysql database. I tested Django connection to mysql db and all things work properly.
I am a complete newbie with web development. I didn't create my own website and I didn't start developing any web application.
My purpose currently is to master creation of mysql database and tables, making changes/migrations/queries, using Django models and Python.
Is it reasonable/possible to use Django ORM for work with mysql database without simultaneous development of a web application/local application? As I've said, I don't have my own website. I want just to try using mysql and Django together on my computer in order to get deeper knowledge as to Django and mysql in this respect.
My final purpose is development in Python, including work with mysql database.
Mysql without Python and Django is of no use for me.
If you want to learn and use MySQL, then start without anything above it - no Django, no ORM, not even a Python script. Learn to configure your mysql server (the server process I mean - doesn't have to be on a distinct computer), to work with the command-line mysql client (database creation, tables creations / modifications, adding/updating/deleting rows, and, most important, doing simple and complex queries).
While you're at it, learn about proper relational data modeling (normalisations etc) so you fully understand how to design your schemas.
Once you're confortable with this, spend some time (should be quite fast at this point) learning to do the same things from python scripts with your python's version mysql connector.
Then if you want to learn web development and Django, well go for it. The ORM is quite easy to use when you already have a good understanding of what happens underneath, so by that time you shouldn't have much problems with this part. What you'll still have to learn are the HTTP protocol (trying to do web programming without understanding the HTTP protocol is like trying to win a car race without knowing how to drive a car - it's not technically impossible but it might end up being very painful experience), then front-end stuff (html/css/javascript) and finally the views / templates parts of Django (which should be easy once you know HTTP and html).
You can of course jump right in into Django, but you will probably fight with way too many concepts at once and end up spending twice more time to figure out how everything works and why it works that way.
I have a Monet database to which I need to make a query remotely. There's a plugin for python which would make easier to make the query, but I'm creating a grails application.
So I need to create a Python module to make the query, and somehow integrate it to grails.
Does somebody knows how to do this? Thanks.
You don't do that.
Grails is written in Groovy, which is a JVM language; which means it can run any Java code. Monet provides JDBC drivers which should work with no problems.
You can implement python to java with http://www.jython.org
Jython is an implementation of the Python language for the Java platform
what is the advantage of using a python virtualbox API instead of using XPCOM?
The advantage is that pyvb is lot easier to work with.
On the contrary the documentation for the python API of XPCOM doesn't exist, and the API is not pythonic at all. You can't do introspection to find methods/attributes of an object, etc. So you have to check the C++ source to find how it works or some python scripts already written (like vboxshell.py and VBoxWebSrv.py).
On the other hand pyvb is really just python wrapper that call VirtuaBoxManager on the command line. I don't know if it's a real disadvantage or not?
I would generally recommend against either one. If you need to use virtualization programmatically, take a look at libvirt, which gives you cross platform and cross hypervisor support; which lets you do kvm/xen/vz/vmware later on.
That said, the SOAP api is using two extra abstraction layers (the client and server side of the HTTP transaction), which is pretty clearly then just calling the XPCOM interface.
If you need local host only support, use XPCOM. The extra indirection of libvirt/SOAP doesn't help you.
If you need to access virtualbox on a various hosts across multiple client machines, use SOAP or libvirt
If you want cross platform support, or to run your code on Linux, use libvirt.
From sun's site on VirtualBox python APIs:
SOAP allows to control remote VMs over
HTTP, while XPCOM is much more
high-performing and exposes certain
functionality not available with SOAP.
They use very different technologies
(SOAP is procedural, while XPCOM is
OOP), but as it is ultimately API to
the same functionality of the
VirtualBox, we kept in bindings
original semantics, so other that
connection establishment, code could
be written in such a way that people
may not care what communication
channel with VirtualBox instance is
used.
From that article, I'm having trouble seeing the difference between "python virtualbox API" and "XPCOM". Could you provide a link to the API you're thinking of?
I have an application for Tomcat which needs to offer/consume web services. Since Java web services are a nightmare (xml, code generation, etc.) compared with what is possible in Python, I would like to learn from your experience using jython instead of java for offerring/consuming web services.
What I have done so far involves adapting http://pywebsvcs.sourceforge.net/ to Jython. I still get errors (namespaces, types and so), although some of it is succesful for the simplest services.
I've put together more details on how to use webservices in jython using axis. Read about it here: How To Script Webservices with Jython and Axis.
PyServlet helps you configure Tomcat to serve up Jython scripts from a URL. You could use this is a "REST-like" way to do some basic web services without much effort. (It is also described here.)
We used a similar home grown framework to provide a variety of data services in a large multiple web application very successfully.