I used to connect elasticsearch 7 self managed cluster using following code.
from elasticsearch import Elasticsearch,RequestsHttpConnection
es = Elasticsearch(['hostname'], timeout=1000,http_auth=('user_name', 'password'),use_ssl=True,verify_certs=True,connection_class=RequestsHttpConnection,scheme="https",port=9200)
After updating the Elasticsearch to 8.5 most of the parameters were invalid. Need a help to figure out the correct way to connect elastic cluster in elastic search 8.5.
In Elasticsearch 8.X, there have been significant changes in the Elasticsearch API.
Now, in the Elasticsearch 8.X, the scheme and port need to be included explicitly as part of the hostname, scheme://hostname:port e.g.(https://localhost:9200)
The http_auth should be updated to basic_auth instead. You can have a look at all the additional available options here. So the new snippet to connect would be something like -
es = Elasticsearch(['https://hostname:port'], timeout=1000 ,basic_auth=('user_name', 'password'),verify_certs=True)
There are significant changes in the requests/responses while doing querying as well, so I would suggest giving this a read.
If you are doing migration to elasticsearch 8.X from 7.X, for short-term, there is another workaround as well which would not require any code changes and just setting the ELASTIC_CLIENT_APIVERSIONING=1 env variable in your python application.
Enable compatibility mode and upgrade Elasticsearch
Upgrade your Elasticsearch client to 7.16:
$ python -m pip install --upgrade 'elasticsearch>=7.16,<8
If you have an existing application enable the compatibility mode by setting ELASTIC_CLIENT_APIVERSIONING=1 environment variable. This will instruct the Elasticsearch server to accept and respond with 7.x-compatibile requests and responses.
https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/migration.html#migration-compat-mode
Related
I am trying to execute a python code from Databricks which primarily establish a connection from Python to SQL server using JDBC.
I used 'jaydebeapi' python library and when I run the code it gives error saying "JayDeBeApi throws AttributeError: '_jpype.PyJPField' object has no attribute 'getStaticAttribute' "
I searched in the internet and found the Jpype library is used in jaydebeapi is the problem and I downgraded the same to 0.6.3 version.
But still I am getting the same error. Can anyone explain me how to make this change and run in databricks.
Or is there any alternative library which I can use.
Why not directly follow the offical documents of databricks below to install Microsoft JDBC Driver for SQL Server for Spark Connector and refer to the sample code of Python using JDBC connect SQL Server.
SQL Databases using the Apache Spark Connector
SQL Databases using JDBC and its Python example with the jdbc url of MS SQL Server
If you were using Azure, there are the same documents for Azure Databricks, as below.
SQL Databases using the Apache Spark Connector for Azure Databricks
SQL Databases using JDBC for Azure Databricks
This is a known issue with JayDeBeApi, you may check out the issue on GitHub.
Due to a bug in 0.6.3 private variables were exposed as part of the interface. Also 0.6.3 had a default class customizer which automatically created a property to get and set if the methods matched a Java bean pattern. This property customizer was loaded late after many common java.lang classes were already loaded, and was not retroactive thus only user loaded classes that happened after initializer would have the customization. The private variable bug would mask the property customizer as the property customizer was not to override fields. Some libraries were unknowingly accessing private variables assuming they were using the property customizer.
The customizer was both unnecessary and resulted in frequent errors for new programmers. The buggy behavior has been removed and the problematic property customizer has been disable by default in 0.7.
Add lines to the module to enable the old property behavior. But this will not reenable the previous buggy access to private variables. Thus code that was exploiting the previous behavior which bypassed the getter/setter of java will need to use the reflection API.
To enable the property customizer, use
try:
import jpype.beans
except ImportError:
pass
Hope this helps.
I'm completely working on python and need to connect my instance of cloud SQL to my python project(sort of Software). Now what I need is that without using cloud_sql_proxy I need to make the connection only and only using python so that client need not need to install Google Cloud SDK.
used cloud_sql_proxy need a way to execute that without google SDK
cloud_sql_proxy -instances=Instance-Name:tcp:3306
I expect that without installing google SDK by only using python client can access the database
If you really need to do this:
Expose your cloudsql instance to the ip address that the python code runs on. Do this in the console under cloudsql -> Connections -> Authorized networks.
Connect via the ip address of the instance using your chosen database connection tool. Looking at your snippet, you are using postgres, so I would suggest psycopg2
Otherwise, if your python code is also running in GCP, you can use the internal ip (provided that they are in the same network)
I need to run BigQuery on Python but the Google BigQuery module doesn't exist
from google.cloud import bigquery
client = bigquery.Client(project='PROJECT_ID')
query = "SELECT...."
dataset = client.dataset('dataset')
table = dataset.table(name='table')
job = client.run_async_query('my-job', query)
job.destination = table
job.write_disposition= 'WRITE_TRUNCATE'
job.begin()
Do you guys know how to do the connection?
Looks like you do not have bigquery module installed, you could install it with -
pip install --upgrade google-cloud-bigquery
Ref - Installing the client library
As per Document, need to install client library for bigquery.
One thing that you need to correct is to setup credentials to connect your big query with PYTHON. You will also need to setup environment variable GOOGLE_APPLICATION_CREDENTIALS pointing towards the location of your credential file .
If your problem is on the connection to BigQuery:
client = bigquery.Client() creates the connection using your default credentials. Default credentials can be set on terminal using gcloud auth login. More on that you can see here: https://cloud.google.com/sdk/gcloud/reference/auth/login
If your problem it to install the library, consider running on terminal pip install --upgrade google-cloud-bigquery -- library docs can be found here: https://googleapis.dev/python/bigquery/latest/index.html
I have some entities added on datastore - made a little web app (in Python) to read and write from the datastore through endpoints. I am able to use the webapp through endpoints from javascript.
I want to access the web app from an installed application on PC. Is there a way to access endpoints from installed applications written in Python? How?
Is there any other way to access the datastore from PC installed applications written in Python?
That's one of the beauties of AppEngine's endpoints. You should use the Python Client Library for Google's APIs to communicate with your endpoints
pip install --upgrade google-api-python-client
Then you'll construct a resource object to communicate with your api using the apiclient.discovery.build function. For eg:
from apiclient.discovery import build
api_root = 'https://<APP_ID>.appspot.com/_ah/api'
api = 'api_name'
version = 'api_version'
discovery_url = '%s/discovery/v1/apis/%s/%s/rest' % (api_root, api, version)
service = build(api, version, discoveryServiceUrl=discovery_url)
You can then perform operations at service.<endpoint_method> etc
A more complete example with authentication can be found here.
EDIT:
Or as #Zig recommends, directly use the Google Cloud API
pip install googledatastore
Trivial example
I've been playing around with and am new to Datastax Enterprise, and I was trying to use the dsetool create_core command to generate a Solr schema automatically for my table in Cassandra. I am using Datastax Enterprise 4.6.5.
I saw in the docs that there was authentication for dsetool using JMX or Kerberos. Right now, I'm using internal authentication, but dsetool asks for HTTP authentication if I try to run the create_core command.
I don't know anything about JMX or Kerberos, and I'm also new to Cassandra and Solr, so I have 3 questions.
Which protocol should I be using to be able to talk to Solr with authentication?
Can I or should I be using internal authentication at all if I use one of the two protocols above?
How does using one of these protocols change how I talk to Cassandra or Solr, for example, using the Datastax python connector or a Solr python client?
Because dsetool create_core is not supported with internal authentication, you can use the Solr HTTP API directly to create your core. Try something like the following from your command line:
curl "http://localhost:8983/solr/admin/cores?action=CREATE&name=<keyspace>.<table>&generateResources=true"