the code does not work with multiple keywords at the same time
Ex. keyword1 keyword2 keyword3 referral to bad URL formatting:
/get_all_related/keyword1%20keyword2%20keyword3
instead of this
/get_all_related/keyword1+keyword2+keyword3
How to get URL formatting as desired?
here the code:
from flask import Flask, render_template, jsonify
from urllib.parse import quote_plus
import json
import sys
import urllib.request
app = Flask(__name__)
url = 'http://suggestqueries.google.com/complete/search?client=firefox&q={}'
#app.route('/')
def index():
return render_template('index.html')
def get_related(entity):
search_term = quote_plus('{} vs '.format(entity))
request = urllib.request.Request(url.format(search_term))
result = urllib.request.urlopen(request)
suggestions = json.loads(result.read())
return [x.replace(suggestions[0],'') for x in suggestions[1] if 'vs' not in x.replace(suggestions[0],'')]
Thank's for help
if search_term has any spaces in it, they will be converted to %20 when calling a function to encode for URI (url.format()). You will have to replace them with + before encoding into a URL safe format:
request = urllib.request.Request(url.format(search_term.replace(' ','+')))
This will replace all spaces with +, then encode the entire string as URI safe.
Related
What is the correct way to handle character encoding when processing a POST request containing "text/plain" data in Flask?
Test application:
from flask import Flask
from flask import request
app = Flask(__name__)
#app.route("/put-text", method=['POST'])
def hello_world():
print("data", request.data)
return {}
Example request
import requests
url = 'http://127.0.0.1:5000/put-text'
data = 'The quick brown fox jumped over the lazy dog."
headers = {
"Content-Type:":"text/plain",
}
res = requests.post(url, data=data, headers=headers)
With this, the request.data is bytes, and I could decode it relatively easily, but how do I make this code robust to support unicode or different text encodings?
According to the Flask docs, request.get_data(as_text=True) probably does what you want:
#app.route("/put-text", method=['POST'])
def hello_world():
text = request.get_data(as_text=True)
# do something with the `text`...
return {}
Presumably, Flask will decode the text with the correct encoding, as specified by the client in the respective header. Provided the client specifies the correct encoding, of course...
I am trying to pull the data using get request through the below URL and I'm getting below error.
I'm not understanding what is the issue with the URL.
Any help would be appreciated
import requests
from requests.auth import HTTPBasicAuth
import json
import urllib.parse
url = """https://msi.abc.com/admin/ui/feedbackCSV/reports
/5d14de32309baf0001501fb7 ?reports[]=pageviews&reports[]=
searches&from=Oct 1 2020&to=Oct 2 2020"""
encode_url = urllib.parse.quote(url,encoding='utf-8')
response = requests.get(encode_url,auth = HTTPBasicAuth('admin#abc.com', 'Summer2020'))
print(response.content)
Error
raise MissingSchema(error)
requests.exceptions.MissingSchema: Invalid URL 'https%3A%2F%2Fmsi.abc.com%2Fadmin%2Fui%2FfeedbackCSV%2Freports%0D%0A%2F5d14de32309baf0001501fb7%20%3Freports%5B%5D%3Dpageviews%26reports%5B%5D%3D%0D%0Asearches%26from%3DOct%201%202020%26to%3DOct%202%202020': No schema supplied. Perhaps you meant http:https%3A%2F%2Fmsi.abc.com%2Fadmin%2Fui%2FfeedbackCSV%2Freports%0D%0A%2F5d14de32309baf0001501fb7%20%3Freports%5B%5D%3Dpageviews%26reports%5B%5D%3D%0D%0Asearches%26from%3DOct%201%202020%26to%3DOct%202%202020?
This might be due to that you pass the whole URL to the urllib.parse.quote function. Try passing the params only to the urllib.parse.quote or use requests params like in the below example:
import requests
from requests.auth import HTTPBasicAuth
import json
import urllib.parse
url = "https://msi.abc.com/admin/ui/feedbackCSV/reports/5d14de32309baf0001501fb7"
payload = {'reports[]':'pageviews', 'reports[]':'searches','from':'Oct 1 2020', 'to':'Oct 2 2020' }
authParams = HTTPBasicAuth('admin#abc.com', 'Summer2020')
response = requests.get(url,params=payload, auth =authParams )
print(response.content)
I'm trying to invoke my sagemaker model using aws chalice, a lambda function, and an API Gateaway.
I'm attempting to send the image over POST request but I'm having problem receiving it on the lambda function.
My code looks like:
from chalice import Chalice
from chalice import BadRequestError
import base64
import os
import boto3
import ast
import json
app = Chalice(app_name='foo')
app.debug = True
#app.route('/', methods=['POST'], content_types=['application/json'])
def index():
body = ''
try:
body = app.current_request.json_body # <- I suspect this is the problem
return {'response': body}
except Exception as e:
return {'error': str(e)}
It's just returning
<Response [200]> {'error': 'BadRequestError: Error Parsing JSON'}
As I mentioned before, my end goal is to receive my image and make a sagemaker request with it. But I just can't seem to read the image.
My python test client looks like this:
import base64, requests, json
def test():
url = 'api_url_from_chalice'
body = ''
with open('b1.jpg', 'rb') as image:
f = image.read()
body = base64.b64encode(f)
payload = {'data': body}
headers = {'Content-Type': 'application/json'}
r = requests.post(url, data=payload, headers=headers)
print(r)
r = r.json()
# r = r['response']
print(r)
test()
Please help me, I spent way to much time trying to figure this out
That error message is because you're not sending a JSON body over to your Chalice app. One way to check this is by using the .raw_body property to confirm:
#app.route('/', methods=['POST'], content_types=['application/json'])
def index():
body = ''
try:
#body = app.current_request.json_body # <- I suspect this is the problem
return {'response': app.current_request.raw_body.decode()}
except Exception as e:
return {'error': str(e)}
You'll see that the body is form-encoded and not JSON.
$ python client.py
<Response [200]>
{'response': 'data=c2FkZmFzZGZhc2RmYQo%3D'}
To fix this, you can use the json parameter in the requests.post() call:
r = requests.post(url, json=payload, headers=headers)
We can then confirm we're getting a JSON body in your chalice app:
$ python client.py
<Response [200]>
{'response': '{"data": "c2FkZmFzZGZhc2RmYQo="}'}
So I was able to figure it out with the help of an aws engineer (i got lucky I suppose). I'm including the complete lambda function. Nothing changed on the client.
from chalice import Chalice
from chalice import BadRequestError
import base64
import os
import boto3
import ast
import json
import sys
from chalice import Chalice
if sys.version_info[0] == 3:
# Python 3 imports.
from urllib.parse import urlparse, parse_qs
else:
# Python 2 imports.
from urlparse import urlparse, parse_qs
app = Chalice(app_name='app_name')
app.debug = True
#app.route('/', methods=['POST'])
def index():
parsed = parse_qs(app.current_request.raw_body.decode())
body = parsed['data'][0]
print(type(body))
try:
body = base64.b64decode(body)
body = bytearray(body)
except e:
return {'error': str(e)}
endpoint = "object-detection-endpoint_name"
runtime = boto3.Session().client(service_name='sagemaker-runtime', region_name='us-east-2')
response = runtime.invoke_endpoint(EndpointName=endpoint, ContentType='image/jpeg', Body=body)
print(response)
results = response['Body'].read().decode("utf-8")
results = results['predictions']
results = json.loads(results)
results = results['predictions']
return {'result': results}
Im trying to create a script, either by Python or PHP, that can receive JSON from Dialogflow webhooks.
The data value will be from resolvedQuery.
I want to parse the data then send to the below URL:
https://joinjoaomgcd.appspot.com/_ah/api/messaging/v1/sendPush?text=***JSON GOES HERE ***&deviceId=18c972b753ad&apikey=5b48aed7
The data from resolvedQuery needs to be sent to the about URL where *JSON GOES HERE * is.
Here is the Python code I have been trying:
from flask import Flask
from flask import request
from flask import make_response
import json
import logging as l
import requests
app = Flask(__name__)
#app.route('/')
def hello():
"""Return a friendly HTTP greeting."""
l.info("reached the hello() module...")
return 'Hello Beautiful World!\n'
#app.route('/apiai', methods=['POST'])
def apiai_response():
requests_session = requests.session()
requests_session.headers.update({'Content-Type': 'application/json'})
requests_session.headers.update({'charset':'utf-8'})
post_data = '[{ "resolvedQuery": "" }]'
requests_response = requests_session.post(url="https://joinjoaomgcd.appspot.com/_ah/api/messaging/v1/sendPush?text=***JSON GOES HERE ***&deviceId=18c972b753ad&apikey=5b48aed7", data=post_data)
print requests_response.content
#app.errorhandler(404)
def page_not_found(e):
"""Return a custom 404 error."""
return 'Sorry, nothing at this URL.', 404
I would try something like this:
import json
import requests
json_string = json.dumps({"hello": "world"})
these_params = {"text": json_string,
"deviceId": "18c972b753ad&apikey=5b48aed7"}
this_url = "https://joinjoaomgcd.appspot.com/_ah/api/messaging/v1/sendPush"
r.get(url=this_url, params=these_params)
notice it is a get request and not a post request
It is a matter of opinion however using url encoded params on a GET request differ in how stuff gets read. As far as the data field is concerned on a post request, the handling is a bit different.
On a get request it will look something like part of the url as :
text=%7B%22hello%22%3A%22world%22%7D
Where as a post request the server will try to read in a application/json encoded body which expects the "data" to be valid json text.
also note:
How to urlencode a querystring in Python?
I have two servers where one is trying to get a file from the other. I am using Flask get requests to send simple data back and forth (strings, lists, JSON objects, etc.).
I also know how to send just a file, but I need to send an error code with my data.
I'm using something along the following lines:
Server 1:
req = requests.post('https://www.otherserver.com/_download_file', data = {'filename':filename})
Server 2:
#app.route('/_download_file', methods = ['POST'])
def download_file():
filename = requests.form.get('filename')
file_data = codecs.open(filename, 'rb').read()
return file_data
Server 1:
with codecs.open('new_file.xyz', 'w') as f:
f.write(req.content)
...all of which works fine. However, I want to send an error code variable along with file_data so that Server 1 knows the status (and not the HTTP status, but an internal status code).
Any help is appreciated.
One solution that comes to my mind is to use a custom HTTP header.
Here is an example server and client implementation.
Of course, you are free to change the name and the value of the custom header as you need.
server
from flask import Flask, send_from_directory
app = Flask(__name__)
#app.route('/', methods=['POST'])
def index():
response = send_from_directory(directory='your-directory', filename='your-file-name')
response.headers['my-custom-header'] = 'my-custom-status-0'
return response
if __name__ == '__main__':
app.run(debug=True)
client
import requests
r = requests.post(url)
status = r.headers['my-custom-header']
# do what you want with status
UPDATE
Here is another version of the server based on your implementation
import codecs
from flask import Flask, request, make_response
app = Flask(__name__)
#app.route('/', methods=['POST'])
def index():
filename = request.form.get('filename')
file_data = codecs.open(filename, 'rb').read()
response = make_response()
response.headers['my-custom-header'] = 'my-custom-status-0'
response.data = file_data
return response
if __name__ == '__main__':
app.run(debug=True)