The problem: I want to interact with Jupyter from another application via Jupyter API, in particular I want to run my notebooks from the app at least (Perfect variant for me is to edit some paragraphs before running it). I've read the API documentation but haven't found what I need.
I've used for that purpose Apache Zeppelin which have the same structure (Notebooks and paragraphs).
Does anybody used Jupyter for the purpose I've just described?
Ignoring if the use of Jupyter API is the best solution for the problem (not clearly described in the question), the code below does what you have asked for: it will execute remotely a Jupyter notebook over http and get some results. It is not production ready, it more an example of how it can be done. Did not test it with cells that generate lots of output - think it will need adjustments.
You can also change/edit the code programmatically by altering the code array.
You will need to change the notebook_path, base and headers according to your configuration, see code for details.
import json
import requests
import datetime
import uuid
from pprint import pprint
from websocket import create_connection
# The token is written on stdout when you start the notebook
notebook_path = '/Untitled.ipynb'
base = 'http://localhost:9999'
headers = {'Authorization': 'Token 4a72cb6f71e0f05a6aa931a5e0ec70109099ed0c35f1d840'}
url = base + '/api/kernels'
response = requests.post(url,headers=headers)
kernel = json.loads(response.text)
# Load the notebook and get the code of each cell
url = base + '/api/contents' + notebook_path
response = requests.get(url,headers=headers)
file = json.loads(response.text)
code = [ c['source'] for c in file['content']['cells'] if len(c['source'])>0 ]
# Execution request/reply is done on websockets channels
ws = create_connection("ws://localhost:9999/api/kernels/"+kernel["id"]+"/channels",
header=headers)
def send_execute_request(code):
msg_type = 'execute_request';
content = { 'code' : code, 'silent':False }
hdr = { 'msg_id' : uuid.uuid1().hex,
'username': 'test',
'session': uuid.uuid1().hex,
'data': datetime.datetime.now().isoformat(),
'msg_type': msg_type,
'version' : '5.0' }
msg = { 'header': hdr, 'parent_header': hdr,
'metadata': {},
'content': content }
return msg
for c in code:
ws.send(json.dumps(send_execute_request(c)))
# We ignore all the other messages, we just get the code execution output
# (this needs to be improved for production to take into account errors, large cell output, images, etc.)
for i in range(0, len(code)):
msg_type = '';
while msg_type != "stream":
rsp = json.loads(ws.recv())
msg_type = rsp["msg_type"]
print(rsp["content"]["text"])
ws.close()
Useful links based on which this code is made (that I recommend reading if you want more info):
https://jupyter-client.readthedocs.io/en/latest/messaging.html#python-api
https://github.com/jupyter/jupyter/wiki/Jupyter-Notebook-Server-API
Note that there is also https://jupyter-client.readthedocs.io/en/stable/index.html, but as far as I could tell it does not support HTTP as a transport.
For reference this works with notebook-5.7.4, not sure about other versions.
Extending the code by #vladmihaisima
from websocket import create_connection, WebSocketTimeoutException
while msg_type != "stream":
try:
rsp = json.loads(ws.recv())
print(rsp["msg_type"])
print(rsp)
msg_type = rsp["msg_type"]
if msg_type == "error":
raise Exception(rsp['content']['traceback'][0])
except WebSocketTimeoutException as _e:
print("No output")
return
I believe that using of remote Jupyter Notebook is over-engineering in your case.
I see good way is pass necessary parameters to python program with well logging.
Related
I am working with an app that sends data to a server with a POST request,
POST https://www.somedomain.com//sendImage HTTP/2.0
looking like this:
{
"user": {
"consent": true,
"currentNumberIs": 1,
"images": {
"data": "BASE64ENCODED IMAGE",
"docType": "avatar"
},
"totalNumberOfImages": 1
}
}
I want to replace the data part of this Json, but only if the docType is avatar. Trying to use a python script for that, that I found here and edited:
def response(flow: http.HTTPFlow) -> None:
if "somedomain.com" in flow.request.pretty_url:
request_data = json.loads(flow.request.get_text())
if request_data["user"]["images"]["docType"] == "avatar":
data = json.loads(flow.response.get_text())
data["user"]["images"]["data"] = "NEWDATA"
flow.response.text = json.dumps(data)
Launched mitmproxy with -s script.py, but according to the web console, the specific request does not trigger the script at all. Which kinda limits the scope to debug.
Would glady appreciate any help.
As #nneonneo mentioned in the comments, I would first recommend to make extensive use of mitmproxy.ctx.log() to make sure that your event hook is triggered properly. Second, if I understand things correctly, you intend to modify the request and not the response? If you want to modify request contents before they are sent to the server, you need to use the request hook and not the response hook:
def request(flow: http.HTTPFlow) -> None:
# this is executed after we have received the request
# from the client, but before it is sent to the server.
def response(flow: http.HTTPFlow) -> None:
# this is executed after we have sent the request
# to the server and received the response at the proxy.
Finally, you currently read from flow.request.text and then later assign to flow.response.text. I don't know your specific use case, but usually that should be flow.request.text as well.
You're altering the flow variable in a function, but not using the edited flow. If you return the new flow you can then use it and post it.
def response(flow: http.HTTPFlow) -> http.HTTPFlow:
if "somedomain.com" in flow.request.pretty_url:
request_data = json.loads(flow.request.get_text())
if request_data["user"]["images"]["docType"] == "avatar":
data = json.loads(flow.response.get_text())
data["user"]["images"]["data"] = "NEWDATA"
flow.response.text = json.dumps(data)
return flow
I am trying to youtube videos to merge with audio(formate_id:140) and video(formate_id:313) using the youtube-dl library. But it downloads files in the local system. I want it directly download to the client side. I don't want to store files in the local system.
Client-side means downloading files directly to a user system through a browser.
Local system means it's my server.
Example
ydl_opts = {
'format': '313+140',
'keepvideo':'false',
}
with youtube_dl.YoutubeDL(ydl_opts) as ydl:
ydl.download(['https://www.youtube.com/watch?v=Wt8VRxUYDso'])
If any way like merging process and response streaming process work at a time.
r = requests.get(videoUrl, stream=True)
response = StreamingHttpResponse(streaming_content=r)
response['Content-Disposition'] = f'attachement; filename="{fileName}"'
return response
Please guide me in solving this coding problem.
After some investigation, I found an option for youtube_dl called progress_hooks that receives a downloaded_bytes that looks like a bytes buffer of some type (did not find exactly in the documentation).
This hook can maybe be used to send each buffer as a chunk in a Django StreamingHttpResponse. The problem is that it expects to receive the data using a generator passed to streaming_content argument.
So we needed a way to turn the callback hook into a generator. This is similar to FTP client downloading process. I used this also with this great answer and came up with something like:
from queue import Queue
from django.http import StreamingHttpResponse
import youtube_dl
def my_view(request):
q = Queue()
job_done = object()
def process_bytes(progress):
if progress["status"] == "downloading":
byte_chunk = progress["downloaded_bytes"]
q.put(byte_chunk)
q.join()
elif progress["status"] == "finished":
q.put(job_done)
ydl_opts = {
'format': '313+140',
'keepvideo': 'false',
'progress_hooks': [process_bytes],
}
with youtube_dl.YoutubeDL(ydl_opts) as ydl:
ydl.download(['https://www.youtube.com/watch?v=Wt8VRxUYDso'])
def chunk_consumer():
while True:
chunk = q.get(True, None) # block, no timeout
if chunk is job_done:
break
else:
yield chunk
q.task_done()
return StreamingHttpResponse(streaming_content=chunk_consumer())
Anyway, I'm not exactly sure whether the client (browser or js) will be able to piece the chunks together into a file.
I have looked through the FTX api documentation found here: https://docs.ftx.us/#overview
And I've looked at example code found in this repo: https://github.com/ftexchange/ftx/tree/master/rest
I can't 'get' or 'post' anything that requires the Authentication. I am using the api key on my account that has 'full trade permissions', and when I look at: print(request.headers) the headers look like they are in the right format.
I've tried: using google colab instead of vs code, updating all my libraries, generating a new api key, restarting kernel and computer. I can pull something like 'markets' because it doesn't need the Authentication.
Let me know if you need any more information, below is a portion of the code I have that isolates the problem and returns {'success': False, 'error': 'Not logged in'}
import time
import urllib.parse
from typing import Optional, Dict, Any, List
from requests import Request, Session, Response
import hmac
ep = 'https://ftx.us/api/wallet/balances'
ts = int(time.time() * 1000)
s = Session()
request = Request('GET', ep)
prepared = request.prepare()
signature_payload = f'{ts}{prepared.method}{prepared.path_url}'.encode()
if prepared.body:
signature_payload += prepared.body
signature = hmac.new(secret.encode(), signature_payload, 'sha256').hexdigest()
request.headers['FTX-KEY'] = key
request.headers['FTX-SIGN'] = signature
request.headers['FTX-TS'] = str(ts)
response = s.send(prepared)
data = response.json()
print(data)
I've faced with the same problem.
You need to change this part:
prepared.headers['FTX-KEY'] = key
prepared.headers['FTX-SIGN'] = signature
prepared.headers['FTX-TS'] = str(ts)
PS. I believe that the FTX needs to fix their API documentation
PSS. I've checked the a part of https://github.com/ftexchange/ftx/tree/master/rest code. I beleave FTX guys just do a copy-paste into docs this code but originally it belongs to more a sophisticated object oriented solution that will work correctly because they pass into method an already created request and use a prepared variable just to calculate path_url and method
For ftx.us, you need to use different headers:
prepared.headers['FTXUS-KEY'] = key
prepared.headers['FTXUS-TS'] = str(ts)
prepared.headers['FTXUS-SIGN'] = signature
I was playing around the rabbitmq HTTP API and came across a weird scenario. When I look at my queues through the web interface, the status of both of them shows as IDLE. .
However when I use the HTTP API, the return for both the queue shows as 'running'. The code im using is below:
import requests
import json
uri = 'http://localhost:15672/api/queues'
r = requests.get(uri, auth=("guest","guest"))
parsed = json.loads(r.content)
#print json.dumps(parsed, indent=4)
for i in parsed:
print '{:<20} : {}'.format(i.get('name'), i.get('state'))
Output:
test queue : running
test2 : running
Can someone explain this behaviour to me?
Check the Management_console source code here: https://github.com/rabbitmq/rabbitmq-management/blob/master/priv/www/js/formatters.js#L479
function fmt_object_state(obj) {
if (obj.state == undefined) return '';
var colour = 'green';
var text = obj.state;
var explanation;
if (obj.idle_since !== undefined) {
colour = 'grey';
explanation = 'Idle since ' + obj.idle_since;
text = 'idle';
}
The console shows "idle" if the field idle_since is not null.
If there is "traffic" in your queue you will have a json like that:
"policy":"",
"exclusive_consumer_tag":"",
"consumers":0,
"consumer_utilisation":"",
"memory":176456,
"recoverable_slaves":"",
"state":"running",
if the queue is in idle (without traffic) you will have a json like that:
"idle_since":"2015-06-25 10:15:07",
"consumer_utilisation":"",
"policy":"",
"exclusive_consumer_tag":"",
"consumers":0,
"recoverable_slaves":"",
"state":"running",
As you can see the field "idle_since" is not null.
In both cases the queue is always in running state.
In conclusion it is just a web-view formatting.
I am kind of newbie to REST and testing dept. I needed to write automation scripts to test our REST services.We are planning to run these scripts from a Jenkins CI job regularly. I prefer writing these in python as we already have UI functionality testing scripts in python generated by selenium IDE, but I am open to any good solution.I checked httplib,simplejson and Xunit, but looking for better solutions available out there.
And also, I would prefer to write a template and generate actual script for each REST API by reading api info from xml or something. Advance thanks to all advices.
I usually use Cucumber to test my restful APIs. The following example is in Ruby, but could easily be translated to python using either the rubypy gem or lettuce.
Start with a set of RESTful base steps:
When /^I send a GET request for "([^\"]*)"$/ do |path|
get path
end
When /^I send a POST request to "([^\"]*)" with the following:$/ do |path, body|
post path, body
end
When /^I send a PUT request to "([^\"]*)" with the following:$/ do |path, body|
put path, body
end
When /^I send a DELETE request to "([^\"]*)"$/ do |path|
delete path
end
Then /^the response should be "([^\"]*)"$/ do |status|
last_response.status.should == status.to_i
end
Then /^the response JSON should be:$/ do |body|
JSON.parse(last_response.body).should == JSON.parse(body)
end
And now we can write features that test the API by actually issuing the requests.
Feature: The users endpoints
Scenario: Creating a user
When I send a POST request to "/users" with the following:
"""
{ "name": "Swift", "status": "awesome" }
"""
Then the response should be "200"
Scenario: Listing users
Given I send a POST request to "/users" with the following:
"""
{ "name": "Swift", "status": "awesome" }
"""
When I send a GET request for "/users"
Then the response should be "200"
And the response JSON should be:
"""
[{ "name": "Swift", "status": "awesome" }]
"""
... etc ...
These are easy to run on a CI system of your choice. See these links for references:
http://www.anthonyeden.com/2010/11/testing-rest-apis-with-cucumber-and-rack-test/
http://jeffkreeftmeijer.com/2011/the-pain-of-json-api-testing/
http://www.cheezyworld.com/2011/08/09/running-your-cukes-in-jenkins/
import openpyxl
import requests
import json
from requests.auth import HTTPBasicAuth
urlHead='https://IP_ADDRESS_HOST:PORT_NUMBER/'
rowStartAt=2
apiColumn=2
#payloadColumn=3
responseBodyColumn=12
statusCodeColumn=13
headerTypes = {'Content-Type':'application/json',
'Accept':'application/json',
'Authorization': '23324'
}
wb = openpyxl.load_workbook('Excel_WORKBOOK.xlsx')
# PROCESS EACH SHEET
for sheetName in (wb.get_sheet_names()):
print ('Sheet Name = ' + sheetName)
flagVar = input('Enter N To avoid APIs Sheets')
if (flagVar=='N'):
print ('Sheet got skipped')
continue
#get a sheet
sheetObj = wb.get_sheet_by_name(sheetName)
#for each sheet iterate the API's
for i in range(2, sheetObj.max_row+1):
#below is API with method type
apiFromSheet = (sheetObj.cell(row=i, column=apiColumn).value)
if apiFromSheet is None:
continue
#print (i, apiFromSheet)
#Let's split the api
apiType = apiFromSheet.split()[0]
method = apiFromSheet.split()[1]
if (apiType!='GET'):
continue
#lets process GET API's
absPath = urlHead + method
print ("REQUESTED TYPE AND PATH = ", apiType, absPath)
print('\n')
res = requests.get(absPath, auth=HTTPBasicAuth(user, pwd), verify=False, headers=headerTypes)
#LET's write res body into relevant cell
sheetObj.cell(row=i, column=responseBodyColumn).value = (res.text)
sheetObj.cell(row=i, column=statusCodeColumn).value = (res.status_code)
wb.save('Excel_WORKBOOK.xlsx')
`#exit(0)`