Here's what I'm doing.
I'm sending a get request to a reddit oembed endpoint. I want to parse the returned json and grab the raw html to embed a reddit post onto my django page. The error I receive when I try to do this is
json.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 1)
Here's an example of that code in action. (It's inside a function)
endpoint = requests.get("https://www.reddit.com/oembed?url=https://www.reddit.com/r/nba/comments/n6l2zu/the_crew_lock_in_their_predictions_and_ernie_has/")
return endpoint.json()['html']
Here is the html it should return. I'm thinking maybe I have to reformat it? Could someone help me out here? Thanks!
'\n <blockquote class="reddit-card" >\n The crew lock in their predictions and Ernie has the Jazz going to the Finals from\n nba\n </blockquote>\n <script async src="https://embed.redditmedia.com/widgets/platform.js" charset="UTF-8"></script>\n'
EDIT:
Here is the result of printing endpoint.json()
{
"provider_url":"https://www.reddit.com/",
"version":"1.0",
"title":"The crew lock in their predictions and Ernie has the Jazz going to the Finals",
"provider_name":"reddit",
"type":"rich",
"html":"\n <blockquote class=\"reddit-card\" >\n The crew lock in their predictions and Ernie has the Jazz going to the Finals from\n nba\n </blockquote>\n <script async src=\"https://embed.redditmedia.com/widgets/platform.js\" charset=\"UTF-8\"></script>\n",
"author_name":"tanookiben"
}
import requests
import json
def get_response():
endpoint = requests.get("https://www.reddit.com/oembed?url=https://www.reddit.com/r/nba/comments/n6l2zu/the_crew_lock_in_their_predictions_and_ernie_has/")
if endpoint.status_code == 200:
return json.loads(endpoint.text)
return {}
print(get_response())
It seems like reddit responses error message as below when you make requests. So it is better to check response status code first.
<!doctype html>
<html>
<head>
<title>Too Many Requests</title>
<style>
body {
font: small verdana, arial, helvetica, sans-serif;
width: 600px;
margin: 0 auto;
}
h1 {
height: 40px;
background: transparent url(//www.redditstatic.com/reddit.com.header.png) no-repeat scroll top right;
}
</style>
</head>
<body>
<h1>whoa there, pardner!</h1>
<p>we're sorry, but you appear to be a bot and we've seen too many requests
from you lately. we enforce a hard speed limit on requests that appear to come
from bots to prevent abuse.</p>
<p>if you are not a bot but are spoofing one via your browser's user agent
string: please change your user agent string to avoid seeing this message
again.</p>
<p>please wait 6 second(s) and try again.</p>
<p>as a reminder to developers, we recommend that clients make no
more than <a href="http://github.com/reddit/reddit/wiki/API">one
request every two seconds</a> to avoid seeing this message.</p>
</body>
</html>
I wasn't sending a proper header. Kept getting a 429 error back, and because there was no json returned, I was receiving the json decoder error.
Related
I am making a info screen for a few 3D printers, I have a rest API where I get the data from. The data should be updated continuously and it will be name and time. I however want to just start getting the current name displaying on the HTML site.
I have a working Python program that loops and updating the current name and that output needs to go to a specific field, in this case (HTML file field INPUT NAME 1) I will have multiple printers so needs to be easy to direct the data to the HTML tags... Have been trying flask but I do not know if that is the best for this?
python program:
import sched, time
from time import time, sleep
import flask
from flask import Flask, redirect, url_for, render_template, request, flash
import requests
app=Flask(__name__,template_folder='templates')
#printer-S3-nr1
response_ums3_1 = requests.get("http://192.168.50.203/api/v1/print_job/name")
response_ums5_1 = requests.get("http://192.168.50.201/api/v1/print_job/name")
#app.route("/")
def profile():
while (True):
sleep(10 - time()%10)
#ULTIMAKER S3 NR 1
if response_ums3_1.status_code == 404:
return render_template("server.html", s3name1 = "No Prints Running")
if response_ums3_1.status_code == 200:
return render_template("server.html", s3name1 = response_ums3_1.text.strip('"'))
#ULTIMAKER S3 NR 1
#ULTIMAKER S5 NR 1
if response_ums5_1.status_code == 404:
return render_template("server.html", s5name1 = "No Prints Running")
if response_ums5_1.status_code == 200:
return render_template("server.html", s5name1 = response_ums5_1.text.strip('"'))
#ULTIMAKER S5 NR 1
if __name__ == "__main__":
app.run(debug=True)
```
<!DOCTYPE html>
<html>
<head>
<style>
#customers {
font-family: Arial, Helvetica, sans-serif;
border-collapse: collapse;
width: 100%;
}
#customers td,
#customers th {
border: 1px solid #ddd;
padding: 8px;
}
#customers tr:nth-child(even) {
background-color: #f2f2f2;
}
#customers tr:hover {
background-color: #ddd;
}
#customers th {
padding-top: 12px;
padding-bottom: 12px;
text-align: left;
background-color: #1b77f7;
color: white;
}
</style>
</head>
<H1 align="center"> Printer Status </H1>
<body>
<table id="customers">
<tr>
<th>Skrivare</th>
<th>Färdig</th>
<th>Namn</th>
<th>Video</th>
</tr>
<tr>
<td>Ultimaker-S5-nr1</td>
<td>INPUT TIME</td>
<td>{{s5name1}}</td>
<td><img src="http://192.168.50.201:8080/?action=stream" alt="Ultimaker-S5-nr1" style="width:202.5px;height:151.85px;" </td>
</tr>
<tr>
<td>Ultimaker S3 nr1</td>
<td>INPUT TIME</td>
<td>{{s3name1}}</td>
<td><img src="http://192.168.50.203:8080/?action=stream" alt="Ultimaker S3 nr1" style="width:202.5px;height:151.85px;" </td>
</tr>
<tr>
</table>
</body>
</html>
```
In Flask, if you want to send data back from server to the front end, then you have to use the return command. Right now, all you're doing is sending output (your print command) to your console. When you return values from your server to the front end, you'll have to use Jinja Template to display the values e.g.
#server code
if response.status_code == 202:
return render_template("index.htm", INPUT_NAME_1 = response.text)
#html page e.g index.htm
....
<td>{{INPUT_NAME_1}}</td>
...
So from the above code (#server code), you're sending a variable called INPUT_NAME_1 back to an html page called index.htm and on that page you get the value of the variable by using the Jinja syntax of double curly brackets i.e. {{}}
Flask is a web framework. It allows you to build web-based applications which use Python. Another framework is Django. If your target is a web application, then you can use any of them.
Note: Right now, your code doesn't include anything to make it web-based. You haven't even imported the Flask libraries to your python code and that brings me to my next point.
Your question shows you're still missing some basics and continuing down this path will lead to more frustration for you. I would advise that you first understand the basics of Flask e.g. read materials that introduce you to Flask. A quick Googling returned the following - https://pymbook.readthedocs.io/en/latest/flask.html,
https://flask.palletsprojects.com/en/2.0.x/
Once you get the basics of Flask, you should then figure out if you want your web application to reside on your local machine or you want to host it on the web. For web hosting, there are cheap (or free) hosts like python anywhere, heroku, Google App Engine. If you decide to go the route of Google App Engine, check out our website where we have a GUI for it
I am creating a REST service using Bottle and MongoDB. The problem is I am getting a 500 error code when trying to insert a document.
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html>
<head>
<title>Error: 500 Internal Server Error</title>
<style type="text/css">
html {background-color: #eee; font-family: sans;}
body {background-color: #fff; border: 1px solid #ddd;
padding: 15px; margin: 15px;}
pre {background-color: #eee; border: 1px solid #ddd; padding: 5px;}
</style>
</head>
<body>
<h1>Error: 500 Internal Server Error</h1>
<p>Sorry, the requested URL <tt>'http://localhost:8080/create'</tt>
caused an error:</p>
<pre>Unhandled exception</pre>
</body>
</html>
cURL
curl -H "Content-Type: application/json" -X POST -d '{"id" : "10011-2017-TEST","certificate_number" : 9278833,"business_name" : "ACME TEST INC.","date" : "Feb 20 2017","result" : "No Violation Issued","sector" : "Test Retail Dealer - 101"}' http://localhost:8080/create
The thing is though, I am able to successfully insert a document even though I get the 500 error code. Why am I getting the 500 error code? I have been trying to implement exception handling for both MongoDB and Bottle. Not sure if that is why I am getting the code. If not, how can I properly implement exception handling? I have read up on abort but I have had trouble with that. Also, when I comment out return result , I get a 200 response code, but I want to return the id of the document inserted. Thanks
Source Code:
#!/usr/bin/python
# -*- coding: utf-8 -*-
from bson import json_util
import json
import bottle
from bottle import route, run, request, abort, post, error
import pymongo
from pymongo import MongoClient
import datetime as datetime
connection = MongoClient('localhost', 27017)
db = connection['city']
collection = db['inspections']
#route('/create', method='POST')
def insert_document():
try:
data = request.json
result = collection.insert_one(data).inserted_id
except Exception, e:
print ('EXCEPTION: ', e)
return result
if __name__ == '__main__':
run(host='localhost', port=8080)
Change
return result
to
return str(result)
or simply remove the line altogether.
I have a problem submitting a HIT to Amazon Mechanical Turk sandbox.
I'm using the following code to submit a HIT:
external_content = """"
<ExternalQuestion xmlns="http://mechanicalturk.amazonaws.com/AWSMechanicalTurkDataSchemas/2006-07-14/ExternalQuestion.xsd">
<ExternalURL>https://MY_HOST_GOES_HERE/</ExternalURL>
<FrameHeight>400</FrameHeight>
</ExternalQuestion>
"""
import boto3
import os
region_name = 'us-east-1'
aws_access_key_id = 'MYKEY'
aws_secret_access_key = 'MYSECRETKEY'
endpoint_url = 'https://mturk-requester-sandbox.us-east-1.amazonaws.com'
# Uncomment this line to use in production
# endpoint_url = 'https://mturk-requester.us-east-1.amazonaws.com'
client = boto3.client('mturk',
endpoint_url=endpoint_url,
region_name=region_name,
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key,
)
# This will return $10,000.00 in the MTurk Developer Sandbox
print(client.get_account_balance()['AvailableBalance'])
response = client.create_hit(Question=external_content,
LifetimeInSeconds=60 * 60 * 24,
Title="Answer a simple question",
Description="Help research a topic",
Keywords="question, answer, research",
AssignmentDurationInSeconds=120,
Reward='0.05')
# The response included several helpful fields
hit_group_id = response['HIT']['HITGroupId']
hit_id = response['HIT']['HITId']
# Let's construct a URL to access the HIT
sb_path = "https://workersandbox.mturk.com/mturk/preview?groupId={}"
hit_url = sb_path.format(hit_group_id)
print(hit_url)
The error message I get is:
botocore.exceptions.ClientError: An error occurred (ParameterValidationError) when calling the CreateHIT operation: There was an error parsing the XML question or answer data in your request. Please make sure the data is well-formed and validates against the appropriate schema. Details: Content is not allowed in prolog. (1493572622889 s)
What might be the reason here? The xml fully agrees with xml schema located on amazon servers.
The html returned by the external host is:
<!DOCTYPE html>
<head>
<meta http-equiv='Content-Type' content='text/html; charset=UTF-8'/>
<script src='https://s3.amazonaws.com/mturk-public/externalHIT_v1.js' type='text/javascript'></script>
</head>
<body>
<!-- HTML to handle creating the HIT form -->
<form name='mturk_form' method='post' id='mturk_form' action='https://workersandbox.mturk.com/mturk/externalSubmit'>
<input type='hidden' value='' name='assignmentId' id='assignmentId'/>
<!-- This is where you define your question(s) -->
<h1>Please name the company that created the iPhone</h1>
<p><textarea name='answer' rows=3 cols=80></textarea></p>
<!-- HTML to handle submitting the HIT -->
<p><input type='submit' id='submitButton' value='Submit' /></p></form>
<script language='Javascript'>turkSetAssignmentID();</script>
</body>
</html>
Thank you
This message "Details: Content is not allowed in prolog." is the clue. It turns out that what this is saying is that you can't have content outside of where it is expected. This is what usually happens when a junk character (think smart-quotes or non-printable ASCII value) appears in there. These can be a real pain in the butt to diagnose.
In your case, it's a little easier to debug but still just as frustrating. Check out this line:
external_content = """"
It turns out that Python only needs three quotes (""") in order to acknowledge a multi-line string definition. Thus your fourth " was actually rendering as part of the XML. Change that line to this:
external_content = """
And you're golden. I just tested it and it works. Sorry for all the frustration, but hopefully this unblocks you. Happy Sunday!
I am trying to write a web service that performs some data processing. The requests contain a data vector as a binary file, and meta-data and processing parameters as form data. An example using python with requests:
import numpy as np
import requests
url = 'http://localhost:8080/pp'
payload = {'param_a': 4, 'param_b': -2.1}
file = {'binary_data': ('file_name.bin', bytes(np.random.randn(1000))}
r = requests.post(url, data=payload, files=file)
Now on the service side, I have:
import bottle
import numpy as np
#bottle.post('/pp')
def pp():
file_path = '/home/generic_user/howhaveyou.bin'
return_file_path = '/home/generic_user/finethanks.bin'
bin_file = bottle.request.files.get('binary_data')
bin_file.save(file_path, overwrite=True)
param_a = float(bottle.request.forms.get('param_a')
param_b = float(bottle.request.forms.get('param_b')
data_vector = np.fromfile(file_path)
processed_data_vector = (data_vector-param_a)*param_b
processed_data_mean = np.mean(processed_data_vector)
processed_data_samples = len(processed_data_vector)
return_metrics = {'mean': processed_data_mean,
'n_of_samples': processed_data_samples}
with open(return_file_path, 'wb') as return_file:
return_file.write(bytes(processed_data_vector))
return return_metrics, bottle.static_file(return_file_path, '/')
which doesn't quite work. Either of the returns work on their own, but together I get the following response:
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html>
<head>
<title>Error: 500 Internal Server Error</title>
<style type="text/css">
html {background-color: #eee; font-family: sans;}
body {background-color: #fff; border: 1px solid #ddd;
padding: 15px; margin: 15px;}
pre {background-color: #eee; border: 1px solid #ddd; padding: 5px;}
</style>
</head>
<body>
<h1>Error: 500 Internal Server Error</h1>
<p>Sorry, the requested URL <tt>'http://localhost:8080/pp'</tt>
caused an error:</p>
<pre>Unsupported response type: <class 'dict'></pre>
</body>
</html>
I have a complete lack of experience with Web Services, so I don't even know if I'm on the right track at all. The point is I wish to return some binary data, along with a few (preferably named) metrics of said data. Is it possible to do this using bottle only? Is there a best practice (with respect to web services, python or both) that I should follow when it comes to this kind of thing?
Note that the client will not be written in python, that is just my test case.
Thanks for any guidance!
The problem with the server-side code is that you cannot return a dictionary (return_metrics) and the contents of a file (return_file_path) with the same response. A solution is to encode the file contents in a string and include it in the returned dict (return_metrics). Then, the client will need to decode the content string to access the data.
I'm having some issues getting AJAX communication working using the Bottle framework. This is my first time using AJAX, so it's likely I just have the basics wrong. Hopefully a Bottle/AJAX guru can point this novice in the right direction. Here is the code I'm using:
#!/usr/bin/env python
from bottle import route, request, run, get
# Form constructor route
#route('/form')
def construct_form():
return '''
<html>
<head>
<script type="text/javascript">
function loadXMLDoc()
{
xmlhttp = new XMLHTTPRequest();
xmlhttp.onReadyStateChange = function()
{
if(xmlhttp.readyState == 4 && xmlhttp.status == 200)
{
document.getElementById("responseDiv").innerHTML = xmlhttp.responseText;
}
}
xmlhttp.open("GET", "/ajax", true);
xmlhttp.send();
}
</script>
</head>
<body>
<form>
<input name="username" type="text"/>
<input type="button" value="Submit" onclick="loadXMLDoc()"/>
</form>
<div id="responseDiv">Change this text to what you type in the box above.</div>
</body>
</html>
'''
# Server response generator
#route('/ajax', method='GET')
def ajaxtest():
inputname = request.forms.username
if inputname:
return 'You typed %s.' % (inputname)
return "You didn't type anything."
run(host = 'localhost', port = '8080')
There are a few issues here.
Javascript is case sensitive. XMLHTTPRequest should be XMLHttpRequest. You should have seen an error about this in your Javascript console.
onReadyStateChange should be onreadystatechange.
If you fix the above two issues your AJAX call will work, but you will only ever get the 'You didn't type anything.' response. This is because you are using GET. You need to change your code so the form values are posted using the POST method.
Also, why aren't you using jQuery to do AJAX? It would make your life much easier. :)