Develop Reference

How to post a kafka schema using python

I am trying to post a kafka schema using python.
From the CLI I would use a syntax like:
curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"schema": "{\"type\":\"record\",\"name\":\"VisualDetections\",\"namespace\":\"com.namespace.something\",\"fields\":[{\"name\":\"vehicle_id\",\"type\":\"int\"},{\"name\":\"source\",\"type\":\"string\"},{\"name\":\"width\",\"type\":\"int\"},{\"name\":\"height\",\"type\":\"int\"},{\"name\":\"annotated_frame\",\"type\":[\"string\",\"null\"]},{\"name\":\"version\",\"type\":\"string\"},{\"name\":\"fps\",\"type\":\"int\"},{\"name\":\"mission_id\",\"type\":\"int\"},{\"name\":\"sequence\",\"type\":{\"type\":\"array\",\"items\":{\"type\":\"record\",\"name\":\"sequence_record\",\"fields\":[{\"name\":\"frame_id\",\"type\":\"int\"},{\"name\":\"timestamp\",\"type\":\"long\"},{\"name\":\"localization\",\"type\":{\"type\":\"array\",\"items\":{\"type\":\"record\",\"name\":\"localization_record\",\"fields\":[{\"name\":\"latitude\",\"type\":\"double\"},{\"name\":\"longitude\",\"type\":\"double\"},{\"name\":\"class\",\"type\":\"string\"},{\"name\":\"object_id\",\"type\":\"int\"},{\"name\":\"confidence\",\"type\":\"double\"},{\"name\":\"bbox\",\"type\":{\"type\":\"record\",\"name\":\"bbox\",\"fields\":[{\"name\":\"x_min\",\"type\":\"int\"},{\"name\":\"y_min\",\"type\":\"int\"},{\"name\":\"x_max\",\"type\":\"int\"},{\"name\":\"y_max\",\"type\":\"int\"}]}}]}}}]}}}]}"}' http://server_ip:8081/subjects/VisualDetections-value/versions/
When I tried to tranfer this function to python I tried something like:
import requests
import json
topic = 'VisualDetections'
headers = {'Content-Type': 'application/vnd.schemaregistry.v1+json'}
with open(avro_path) as fp:
data = {'schema': json.load(fp)}
data_json = json.dumps(data)
cmd = 'http://server_ip:8081/subjects/{}-value/versions/'.format(topic)
response = requests.post(cmd, headers=headers, data=data_json)
The above returns a code {"error_code":500,"message":"Internal Server Error"}. I have tried other options like:
with open(avro_path) as fp:
data = json.load(fp)
with error code:
"error_code":422,"message":"Unrecognized field: name"
In the above the avro_path just contains the avro schema in a json file (can be uploaded if useful also).
I am not sure how I could post this data exactly. Also, I did not take into consideration the -H argument of post in CLI since I couldn't find a equivalent python argument (not sure it plays any role though). Can anyone provide a solution to this issue.

For the second error, the payload needs to be {'schema': "schema string"}
For the first, I think its a matter of the encoding; json.load will read the file to a dict rather than just a string.
Notice
>>> import json
>>> schema = {"type":"record"} # example when using json.load() ... other data excluded
>>> json.dumps({'schema': schema})
'{"schema": {"type": "record"}}' # the schema value is not a string
>>> json.dumps({'schema': json.dumps(schema)})
'{"schema": "{\\"type\\": \\"record\\"}"}' # here it is
Try just reading the file
url = 'http://server_ip:8081/subjects/{}-value/versions/'.format(topic)
with open(avro_path) as fp:
data = {'schema': fp.read().strip()}
response = requests.post(cmd, headers=headers, data=json.dumps(data))
Otherwise, you would json.load then use json.dumps twice as shown above
You may also try json=data rather than data=json.dumps(data)

Interact with Jupyter Notebooks via API

The problem: I want to interact with Jupyter from another application via Jupyter API, in particular I want to run my notebooks from the app at least (Perfect variant for me is to edit some paragraphs before running it). I've read the API documentation but haven't found what I need.
I've used for that purpose Apache Zeppelin which have the same structure (Notebooks and paragraphs).
Does anybody used Jupyter for the purpose I've just described?

Ignoring if the use of Jupyter API is the best solution for the problem (not clearly described in the question), the code below does what you have asked for: it will execute remotely a Jupyter notebook over http and get some results. It is not production ready, it more an example of how it can be done. Did not test it with cells that generate lots of output - think it will need adjustments.
You can also change/edit the code programmatically by altering the code array.
You will need to change the notebook_path, base and headers according to your configuration, see code for details.
import json
import requests
import datetime
import uuid
from pprint import pprint
from websocket import create_connection
# The token is written on stdout when you start the notebook
notebook_path = '/Untitled.ipynb'
base = 'http://localhost:9999'
headers = {'Authorization': 'Token 4a72cb6f71e0f05a6aa931a5e0ec70109099ed0c35f1d840'}
url = base + '/api/kernels'
response = requests.post(url,headers=headers)
kernel = json.loads(response.text)
# Load the notebook and get the code of each cell
url = base + '/api/contents' + notebook_path
response = requests.get(url,headers=headers)
file = json.loads(response.text)
code = [ c['source'] for c in file['content']['cells'] if len(c['source'])>0 ]
# Execution request/reply is done on websockets channels
ws = create_connection("ws://localhost:9999/api/kernels/"+kernel["id"]+"/channels",
header=headers)
def send_execute_request(code):
msg_type = 'execute_request';
content = { 'code' : code, 'silent':False }
hdr = { 'msg_id' : uuid.uuid1().hex,
'username': 'test',
'session': uuid.uuid1().hex,
'data': datetime.datetime.now().isoformat(),
'msg_type': msg_type,
'version' : '5.0' }
msg = { 'header': hdr, 'parent_header': hdr,
'metadata': {},
'content': content }
return msg
for c in code:
ws.send(json.dumps(send_execute_request(c)))
# We ignore all the other messages, we just get the code execution output
# (this needs to be improved for production to take into account errors, large cell output, images, etc.)
for i in range(0, len(code)):
msg_type = '';
while msg_type != "stream":
rsp = json.loads(ws.recv())
msg_type = rsp["msg_type"]
print(rsp["content"]["text"])
ws.close()
Useful links based on which this code is made (that I recommend reading if you want more info):
https://jupyter-client.readthedocs.io/en/latest/messaging.html#python-api
https://github.com/jupyter/jupyter/wiki/Jupyter-Notebook-Server-API
Note that there is also https://jupyter-client.readthedocs.io/en/stable/index.html, but as far as I could tell it does not support HTTP as a transport.
For reference this works with notebook-5.7.4, not sure about other versions.

Extending the code by #vladmihaisima
from websocket import create_connection, WebSocketTimeoutException
while msg_type != "stream":
try:
rsp = json.loads(ws.recv())
print(rsp["msg_type"])
print(rsp)
msg_type = rsp["msg_type"]
if msg_type == "error":
raise Exception(rsp['content']['traceback'][0])
except WebSocketTimeoutException as _e:
print("No output")
return

I believe that using of remote Jupyter Notebook is over-engineering in your case.
I see good way is pass necessary parameters to python program with well logging.

requests.post with Python

I'm connecting to a login protected API with a Python script here below.
import requests
url = 'https://api.json'
header = {'Content-Type': 'application/x-www-form-urlencoded'}
login = ('kjji#snm.com', 'xxxxx')
mnem = 'inputRequests':'{'inputRequests':'[{'function':'GDSP','identifier':'ibm','mnemonic':'IQ_TOTAL_REV'}]}}
r = requests.post(url, auth=login, data=mnem, headers=header)
print(r.json())
The connection is established but I am getting an error from the API because of the format of the data request.The original format is here below. I cannot find a way to enter this in the mnem here above:
inputRequests={inputRequests:
[
{function:"xxx",identifier:"xxx",mnemonic:"xxx"},
]
}
The error given is
C:\Users\xxx\Desktop>pie.py
File "C:\Users\xxx\Desktop\pie.py", line 6
mnem={'inputRequests':'{'inputRequests':'[{'function':'xxx','identifier':'xx','mnemonic':'xxx'}]}}
^
SyntaxError: invalid syntax
I am unsure on how to proceed from here. I cannot find anything in the requests documentation that points to how to insert several variables in the data field.

The requests module in Python receive protogenic Python dict as the JSON data in post request but not a string. Therefore, you may try to define mnem like this:
mnem = {
'inputRequests':[
{'function':'GDSP',
'identifier':'ibm',
'mnemonic':'IQ_TOTAL_REV'
}
]}

the data parameter should be a dictionary.
therefore to pass the three parameters try using:
mnem = {'function':'GDSP','identifier':'ibm','mnemonic':'IQ_TOTAL_REV'}

Problems with python + json vs. curl

so when I run the python code the server (google) give me a different response than when I run curl command. Can someone tell me where I'm wrong please?
code:
import urllib2, simplejson
def MapsWIFI(card):
req = urllib2.Request("https://www.googleapis.com/geolocation/v1/geolocate?key=AI...")
jWifi = """
{
"wifiAccessPoints": [
{
"macAddress": "64:D1:A3:0A:11:65",
"channel": 6,
},
... #some AP here
]
}
"""
print jWifi
req.add_header("Content-Type", "application/json")
jWifiReport = urllib2.urlopen(req,simplejson.dumps(jWifi)).read()
print jWifiReport
APdetected = str(len(wifiCell))
mapsDict = simplejson.loads(jWifiReport)
location = str(mapsDict.get("location",{}))[1:-1]
accuracy = "Accuracy: "+str(mapsDict.get("accuracy",{}))[1:-1]
mapMe = "|---"+location.split(",")[0]+"\n|---"+location.split(",")[1][1:]+"\n|---$
return mapMe
MapsWIFI("wlp8s0")
And the command is:
curl -d #file2.json -H "Content-Type: application/json" -i "https://www.googleapis.com/geolocation/v1/geolocate?key=AI..."
where file2.json contains exactly jWifi in that format.
The problem is that, as said, the location returned by the code is different from the location returned by curl. I don't get error code so I thing that the syntax is correct.

The data is already a JSON encoded string, you don't want to encode it twice.
Pass it in without encoding it again:
jWifiReport = urllib2.urlopen(req, jWifi).read()
You only need to encode if you have a Python data structure (a dictionary in this case).

Needed advice to automate REST services test

I am kind of newbie to REST and testing dept. I needed to write automation scripts to test our REST services.We are planning to run these scripts from a Jenkins CI job regularly. I prefer writing these in python as we already have UI functionality testing scripts in python generated by selenium IDE, but I am open to any good solution.I checked httplib,simplejson and Xunit, but looking for better solutions available out there.
And also, I would prefer to write a template and generate actual script for each REST API by reading api info from xml or something. Advance thanks to all advices.

I usually use Cucumber to test my restful APIs. The following example is in Ruby, but could easily be translated to python using either the rubypy gem or lettuce.
Start with a set of RESTful base steps:
When /^I send a GET request for "([^\"]*)"$/ do |path|
get path
end
When /^I send a POST request to "([^\"]*)" with the following:$/ do |path, body|
post path, body
end
When /^I send a PUT request to "([^\"]*)" with the following:$/ do |path, body|
put path, body
end
When /^I send a DELETE request to "([^\"]*)"$/ do |path|
delete path
end
Then /^the response should be "([^\"]*)"$/ do |status|
last_response.status.should == status.to_i
end
Then /^the response JSON should be:$/ do |body|
JSON.parse(last_response.body).should == JSON.parse(body)
end
And now we can write features that test the API by actually issuing the requests.
Feature: The users endpoints
Scenario: Creating a user
When I send a POST request to "/users" with the following:
"""
{ "name": "Swift", "status": "awesome" }
"""
Then the response should be "200"
Scenario: Listing users
Given I send a POST request to "/users" with the following:
"""
{ "name": "Swift", "status": "awesome" }
"""
When I send a GET request for "/users"
Then the response should be "200"
And the response JSON should be:
"""
[{ "name": "Swift", "status": "awesome" }]
"""
... etc ...
These are easy to run on a CI system of your choice. See these links for references:
http://www.anthonyeden.com/2010/11/testing-rest-apis-with-cucumber-and-rack-test/
http://jeffkreeftmeijer.com/2011/the-pain-of-json-api-testing/
http://www.cheezyworld.com/2011/08/09/running-your-cukes-in-jenkins/

import openpyxl
import requests
import json
from requests.auth import HTTPBasicAuth
urlHead='https://IP_ADDRESS_HOST:PORT_NUMBER/'
rowStartAt=2
apiColumn=2
#payloadColumn=3
responseBodyColumn=12
statusCodeColumn=13
headerTypes = {'Content-Type':'application/json',
'Accept':'application/json',
'Authorization': '23324'
}
wb = openpyxl.load_workbook('Excel_WORKBOOK.xlsx')
# PROCESS EACH SHEET
for sheetName in (wb.get_sheet_names()):
print ('Sheet Name = ' + sheetName)
flagVar = input('Enter N To avoid APIs Sheets')
if (flagVar=='N'):
print ('Sheet got skipped')
continue
#get a sheet
sheetObj = wb.get_sheet_by_name(sheetName)
#for each sheet iterate the API's
for i in range(2, sheetObj.max_row+1):
#below is API with method type
apiFromSheet = (sheetObj.cell(row=i, column=apiColumn).value)
if apiFromSheet is None:
continue
#print (i, apiFromSheet)
#Let's split the api
apiType = apiFromSheet.split()[0]
method = apiFromSheet.split()[1]
if (apiType!='GET'):
continue
#lets process GET API's
absPath = urlHead + method
print ("REQUESTED TYPE AND PATH = ", apiType, absPath)
print('\n')
res = requests.get(absPath, auth=HTTPBasicAuth(user, pwd), verify=False, headers=headerTypes)
#LET's write res body into relevant cell
sheetObj.cell(row=i, column=responseBodyColumn).value = (res.text)
sheetObj.cell(row=i, column=statusCodeColumn).value = (res.status_code)
wb.save('Excel_WORKBOOK.xlsx')
`#exit(0)`