Develop Reference

Accessing Elasticsearch with Python 3

I want to use the Python 3 module urllib to access an Elasticsearch database at localhost:9200. My script gets a valid request (generated by Kibana) piped to STDIN in JSON format.
Here is what I did:
import json
import sys
import urllib.parse
import urllib.request
er = json.load(sys.stdin)
data = urllib.parse.urlencode(er)
data = data.encode('ascii')
uri = urllib.request.Request('http://localhost:9200/_search', data)
with urllib.request.urlopen(uri) as repsonse:
response.read()
(I understand that my repsonse.read() doesn't make much sense by itself but I just wanted to keep it simple.)
When I execute the script, I get an
HTTP Error 400: Bad request
I am very sure that the JSON data I'm piping to the script is correct, since I had it printed and fed it via curl to Elasticsearch, and got back the documents I expected to get back.
Any ideas where I went wrong? Am I using urllib correctly? Do I maybe mess up the JSON data in the urlencode line? Am I querying Elasticsearch correctly?
Thanks for your help.

With requests you can do one of two things
1) Either you create the string representation of the json object yourself and send it off like so:
payload = {'param': 'value'}
response = requests.post(url, data=json.dumps(payload))
2) Or you have requests do it for you like so:
payload = {'param': 'value'}
response = requests.post(url, json = payload)
So depending on what actually comes out of the sys.stdin call (probably - as Kibana would be sending that if the target was ElasticSearch - a string representation of a json object == equivalent of doing json.dumps on a dictionary), but you might have to adjust a bit depending on the output of sys.stdin.
My guess is that your code could work by just doing so:
import sys
import requests
payload = sys.stdin
response = requests.post('http://localhost:9200/_search', data=payload)
And if you then want to do some work with it in Python, requests has a built in support for this too. You just call this:
json_response = response.json()
Hope this helps you on the right track. For further reading om json.dumps/loads - this answer has some good stuff on it.

For anyone who doesn't want to use requests (for example if you're using IronPython where its not supported):
import urllib2
import json
req = urllib2.Request(url, json.dumps(data), headers={'Content-Type': 'application/json'})
response = urllib2.urlopen(req)
Where 'url' can be something like this (example below is search in index):
http://<elasticsearch-ip>:9200/<index-name>/_search/

HTTPS GET API call using base64 encoding

I am trying to make an HTTP GET API call to one of my server, which support HTTP basic authentication using an API key in base64 encoding. so basically I want to add my authorization header in base64 encoding to my request.
The one method of authorization I know is:
>>> import requests
>>> r = requests.get('https://test.com/test-API-Gateway/v0/deployments', auth=('user', 'password'), verify=False)).text
>>> print r
{"statusCode":401,"statusMsg":Unauthorized,"result":[]}
But my server does not return anything, since it does not take id and password for authentication, rather it needs the base64 encoding header. Can you please tell me how to achieve this?
Thanks in advance.

The Python Requests library does allow you to add custom headers. You should be able to create the appropriate header (with your base64 encoding) and pass it as a parameter, like so:
import requests
url = 'https://test.com/test-API-Gateway/v0/deployments'
myheaders = {'my-header-param': 'somedata'}
r = requests.get(url, headers=myheaders, verify=False)).text
The related documentation can be found here.

Sending JSON request with Python

I'm new to web services and am trying to send the following JSON based request using a python script:
http://myserver/emoncms2/api/post?apikey=xxxxxxxxxxxxx&json={power:290.4,temperature:19.4}
If I paste the above into a browser, it works as expected. However, I am struggling to send the request from Python. The following is what I am trying:
import json
import urllib2
data = {'temperature':'24.3'}
data_json = json.dumps(data)
host = "http://myserver/emoncms2/api/post"
req = urllib2.Request(host, 'GET', data_json, {'content-type': 'application/json'})
response_stream = urllib2.urlopen(req)
json_response = response_stream.read()
How do I add the apikey data into the request?
Thank you!

Instead of using urllib2, you can use requests. This new python lib is really well written and it's easier and more intuitive to use.
To send your json data you can use something like the following code:
import json
import requests
data = {'temperature':'24.3'}
data_json = json.dumps(data)
payload = {'json_payload': data_json, 'apikey': 'YOUR_API_KEY_HERE'}
r = requests.get('http://myserver/emoncms2/api/post', data=payload)
You can then inspect r to obtain an http status code, content, etc

Even though this doesnt exactly answer OPs question, it should be mentioned here that requests module has a json option that can be used like this:
import requests
requests.post(
'http://myserver/emoncms2/api/post?apikey=xxxxxxxxxxxxx',
json={"temperature": "24.3"}
)
which would be equivalent to the curl:
curl 'http://myserver/emoncms2/api/post?apikey=xxxxxxxxxxxxx' \
-H 'Content-Type: application/json' \
--data-binary '{"temperature":"24.3"}'

Maybe the problem is that json.dumps puts " and in the json you put in the url there are no "s.
For example:
data = {'temperature':'24.3'}
print json.dumps(data)
prints:
{"temperature": "24.3"}
and not:
{temperature: 24.3}
like you put in your url.
One way of solving this (which is trouble prone) is to do:
json.dumps(data).replace('"', '')

How to send a POST request using django?

I dont want to use html file, but only with django I have to make POST request.
Just like urllib2 sends a get request.

Here's how you'd write the accepted answer's example using python-requests:
post_data = {'name': 'Gladys'}
response = requests.post('http://example.com', data=post_data)
content = response.content
Much more intuitive. See the Quickstart for more simple examples.

In Python 2, a combination of methods from urllib2 and urllib will do the trick. Here is how I post data using the two:
post_data = [('name','Gladys'),] # a sequence of two element tuples
result = urllib2.urlopen('http://example.com', urllib.urlencode(post_data))
content = result.read()
urlopen() is a method you use for opening urls.
urlencode() converts the arguments to percent-encoded string.

The only thing you should look at now:
https://requests.readthedocs.io/en/master/

You can use urllib2 in django. After all, it's still python. To send a POST with urllib2, you can send the data parameter (taken from here):
urllib2.urlopen(url[, data][, timeout])
[..] the HTTP request will be a POST instead of a GET when the data parameter is provided

Pay attention, that when you're using 🐍 requests , and make POST request passing your dictionary in data parameter like this:
payload = {'param1':1, 'param2':2}
r = request.post('https://domain.tld', data=payload)
you are passing parameters form-encoded.
If you want to send POST request with only JSON (most popular type in server-server integration) you need to provide a str() in data parameter. In case with JSON, you need to import json lib and make like this:
payload = {'param1':1, 'param2':2}
r = request.post('https://domain.tld', data=json.dumps(payload))`
documentation is here
OR:
just use json parameter with provided data in the dict
payload = {'param1':1, 'param2':2}
r = request.post('https://domain.tld', json=payload)`

Put json data into url with python

I am trying to encode a json parameter within a url for use with the mongolab restAPI.
My url looks something like this
url = 'https://api.mongolab.com/api/1/databases/db/collections/coll?q={"q": "10024"}&apiKey=mykey
I am trying to open it using
urllib2.urlopen(url)
but I run into errors saying that my apikey is incorrect. I know this isn't true because if I copy and paste the url into my browser I get a correct response. I also know that I can access the rest api as long as I don't have the query there (so it must be a json/formatting problem).
So does anyone know how I could encode the json query
{"q": "10024"}
into the url? Thanks!

You'll have to properly URL-encode the string. Use the urllib.quote_plus() function:
url = 'https://api.mongolab.com/api/1/databases/db/collections/coll?q={q}&apiKey={key}'
query = urllib.quote_plus('{"q": "10024"}')
urllib2.urlopen(url.format(q=query, key=your_api_key))

You could also use the requests library.
Example:
import requests
payload = {'q': '10024', 'apiKey': 'mykey'}
r = requests.get("https://api.mongolab.com/api/1/databases/db/collections/coll", params=payload)
print(r.url)
Output:
https://api.mongolab.com/api/1/databases/db/collections/coll?q=10024&apiKey=mykey