I am newbie to elasticsearch, I know there is two official client elasticsearch supplies, but when I use the python elasticsearch, i can't find how to use the transport client..
I read the whole doc which is as following:
https://elasticsearch-py.readthedocs.io/en/master/index.html
I also search some docs, i can't find the way to use elasticsearch with python.also, in one doc, it says:
Using the native protocol from anything other than Java is not
recommended, as it would entail implementing a lot of custom
serialization.
does this mean python elasticsearch can't use transport client?
The transport client is written in Java, so the only way to use it from Python is by switching to Jython.
I think the previous answer is out of date now, if this is the transport client you mean.
I've made use of this API to do things like use the _rank_eval API, which is still considered "experimental" so hasn't made it into the official client yet.
def rank_eval(self, query, ratings, metric_name):
res = self.es.transport.perform_request(
"GET",
"/%s/_rank_eval" % INDEX,
body=self.rank_request(query, ratings, metric_name),
)
return res
Related
I am new to working with APIs in general and am writing code in python that needs to consume/interact with an API someone else has set up. I was wondering if there is any package out there that would build some sort of custom client class to interact with an API given a file outlining the API in some way (like a json or something where each available endpoint and http verb could be outlined in terms of stuff like allowed payload json schema for posts, general params allowed and their types, expected response json schema, the header key/value for a business verb, etc.). It would be helpful if I could have one master file outlining the endpoints available and then some package uses that to generate a client class we can use to consume the API as described.
In my googling most API packages I have found in python are much more focused on the generation of APIs but this isn't what I want.
Basically I believe you are looking for the built in requests package.
response = requests.get(f'{base_url}{endpoint}',
params={'foo': self.bar,
'foo_2':self.bar_2},
headers={'X-Api-Key': secret}
)
And from here, you can build you own class, pass it to a dataframe or whatever.
In the requests package is basically everything you need. Status handling, exception handling everything you need.
Please check the docs.
https://pypi.org/project/requests/
I'm currently building a python gRPC server that serializes tons of different proto messages into json to store them into a no-sql db. I'd like to simplify extension of this server such that we can add new types without rewriting the gRPC server and redeploying. Ideally, we would like to define a new message, put it in a proto file and update only the client. The server should expect any type at first but knows a .proto file or folder where to look for when it comes to serializing/deserializing.
I've read about the Any type and I'm exploring whether this is my way to do this. There is some documentation on it but very few examples to work with. One thing that I don't quite get is how to store/retrieve the type of an "Any" field.
All documentation use https as protocol for the type of an Any field (e.g. type.googleapis.com/google.protobuf.Duration). This is also the default. How would it look like if I use the local file system? How would I store this in the proto message on the client side?
How can I retrieve the type on the server side?
Where can I find a similar example?
Apologies, this is only a partial answer.
I've recently begun using Any in a project and can provide some perspective. I have a similar (albeit simpler) requirement to what you outline. Enveloped message content but, in my case, clients are required to ship a descriptor to the server and identify a specific method to help it (un)marshal etc.
I've been using Google's new Golang APIv2 and only familiar with it from Golang and Rust (not Python). The documentation is lacking but the Golang documents will hopefully help:
anypb
protoregistry
I too struggled with understanding the concept (implementation) of the global registry and so I hacked the above solution. The incoming message metadata provides sufficient context to the server that it can construct the message type and marshal the bytes into it.
I am working with the AWS Transcribe streaming service that boto3 does not support yet, so to make HTTP/2 requests, I need to manually setup the authorization header with the "AWS Signature Version 4"
I've found some example implementation, but I was hoping to just call whatever function boto3/botocore have implemented using the same configuration object.
Something like
session = boto3.Session(...)
auth = session.generate_signature('POST', '/stream-transcription', ...)
Any pointers in that direction?
Contrary to the AWS SDKs for most other programming languages, boto3/botocore don't offer the functionality to sign arbitrary requests using "AWS Signature Version 4" yet. However there is at least already an open feature request for that: https://github.com/boto/botocore/issues/1784
In this feature request, existing alternatives are discussed as well. One is the third-party Python library aws-requests-auth, which provides a thin wrapper around botocore and requests to sign HTTP-requests. That looks like the following:
import requests
from aws_requests_auth.boto_utils import BotoAWSRequestsAuth
auth = BotoAWSRequestsAuth(aws_host="your-service.domain.tld",
aws_region="us-east-1",
aws_service="execute-api")
response = requests.get("https://your-service.domain.tld",
auth=auth)
Another alternative presented in the feature request is to implement the necessary glue-code on your own, as shown in the following gist: https://gist.github.com/rhboyd/1e01190a6b27ca4ba817bf272d5a5f9a.
Did you check this SDK? Seems very recent but might do what you need.
https://github.com/awslabs/amazon-transcribe-streaming-sdk/tree/master
It looks like it handles the signing: https://github.com/awslabs/amazon-transcribe-streaming-sdk/blob/master/amazon_transcribe/signer.py
I have not tested this, but you can likely accomplish this by following along with with this SigV4 unit test:
https://github.com/boto/botocore/blob/master/tests/unit/test_auth_sigv4.py
Note, this constructs a request using the botocore.awsrequest.AWSRequest helper. You'll likely need to dig around to figure out how to send the actual HTTP request (perhaps with httpsession.py)
I am trying to develop a data streaming API with the same functionality as Twitter’s streaming API (https://dev.twitter.com/streaming/reference/post/statuses/filter), namely a data stream with filtering capabilities. I am generating a lot of data and want to serve it to clients.
I understand how to make an app that serves the all clients the same data. That’s relatively easy. The difficulty I’m having comes from allowing clients to specify data filters and serving unique data to each client.
My thoughts:
First I thought to open a streaming http requests (like Twitter). I could create an endpoint that accepts GET requests with parameters (e.g. https://stream.example.com/v1/filter.json?track=twitter). According to this answer Streaming API vs Rest API?, this doesn’t scale easily and requires a lot of resources.
Then I thought to use websockets and let the client supply a filter message (e.g. locations=-122.75,36.8,-121.75,37.8). However, I can’t find a good example of a WS server dishing out unique data to each client. What would this class look like if it inherited tornado.websocket.WebSocketHandler or similar implementation?
I also considered pushing the data to a messaging service (RabbitMQ ) or database (Redis) and subscribing clients as they connect to their unique channel. (I think like this question Any ideas how to create parameterised streaming api?). I don’t know of an efficient way to create the unique channels. This also seems overly complicated.
I would prefer to do this in Python but I would also consider using Ruby and JS implementations.
Not too familiar with Python but I think that this should be possible using Websockets. Here's my take on it in Ruby, hopefully it is of any help. These are stripped down versions with much of the websocket functionality removed just to demonstrate.
However for best practices regarding the use of streaming API I can't be of much help I'm afraid.
Server
require 'em-websocket'
require 'json'
def filtered_stream(filter, ws)
loop do
# do something with filter, transform or send different kinds of data
ws.send "#{filter} - hello"
sleep 2
end
end
EM.run {
EM::WebSocket.run(:host => "127.0.0.1", :port => 9999) do |ws|
ws.onopen { |handshake|
# We can access some information in the handshake object such as headers
# Perhaps even add the client to a list / table
}
ws.onmessage { |msg|
# use ws.close to close the connection if needed
# for example if the client attempts to send an invalid message or multiple messages?
filter = JSON.parse(msg)['locations']
Thread.new { filtered_stream(filter, ws) }
}
end
}
Client
require 'websocket-eventmachine-client'
EM.run do
ws = WebSocket::EventMachine::Client.connect(
:uri => 'ws://localhost:9999',
)
ws.onopen do
# Unsure on how to supply the filter, headers is an option too
ws.send "{\"locations\":\"-122.75,36.8,-121.75,37.8\"}"
end
ws.onmessage do |msg|
p msg
end
end
Using Python, I would like to pull data from NetSuite, along with adding/updating data in NetSuite. For example, I would like to create sales orders and add line items via Python.
I'm aware that they have a WSDL that I could potentially use. (And I was hoping that they would also have an API, but apparently not...) Does anyone have examples working with this WSDL in Python? Are there better ways to integrate with NetSuite?
I have deal with Netsuite Webservice for around one week, there is not clear documentation but once you are align with the other regular webservices behaivour is very simple adapt yourself. I attach one little script to login and to get the full list of services and data types in netsuite.
Use the factory method to create the objects to interact with the netsuite webservice.
I have used suds as SOAP Python Library.
# -*- coding: utf-8 -*-
from suds.client import Client
import os, time, sys, datetime
import suds
import logging
logging.basicConfig(level=logging.INFO)
logging.getLogger('suds.client').setLevel(logging.DEBUG)
reload(sys)
sys.setdefaultencoding('utf8')
NS_HOST = 'https://webservices.netsuite.com'
email ='myemail#xxxxxx.com'
pwd ='mypwd'
account = "99999999"
NS_ENDPOINT = "2016_2"
NS_ROLE = 3
wsdl = NS_HOST + "/wsdl/v" + NS_ENDPOINT + "_0/netsuite.wsdl"
client = Client(url=wsdl)
#You can get the methods and types with this object
print client
ApplicationInfo = client.factory.create('ns16:ApplicationInfo')
ApplicationInfo.applicationId = "xxxxx-XXXX-XXXX-XXXX-XXXXXXXX"
client.set_options(location= NS_HOST + "/services/NetSuitePort_" + NS_ENDPOINT, soapheaders={'applicationInfo':ApplicationInfo})
passport = client.factory.create('ns4:Passport')
passport.email = email
passport.password = pwd
passport.account = account
recordRef = client.factory.create('ns4:RecordRef')
recordRef.name="MX - Gerencia de Contabilidad"
passport.role = recordRef
client.service.login(passport)
Netsuite has provided toolkits for Java, .Net and PHP to access their webservices. For other languages either there are third party toolkits or you have to send Raw SOAP requests.
For my Python based projects I'm using Raw SOAP requests method. I suggest that first you get familiar with Netsuite Web services using any of the available toolkits and then for Python use this knowledge to generate raw SOAP requests. SOAPUI can also be of great help.
Have you explored restlets they are a generally good alternate for webservices.
There are a number of Python libraries available for processing SOAP messages, tried using SUDS, as it is the only one capable of properly consuming the 2014.1 Netsuite WSDL. This was only possible after some tweaks. The Netsuite WSDL is large and complex and took an enormous amount of time to load, even after caching AND loading from local. The SUDS library has not been maintained for quite a while, there is a maintained fork called SUDS-jurko that I have not yet tried. Ultimately, I ended up using raw soap messages to communicate with the Netsuite webservice for specific tasks.