My end goal is to have an external website POST data to my site using DRF. The data they send is in XML Format.
python 2.7
djangorestframework 3.3.2
djangorestframework-xml 1.3.0
My issues is whenever the external site posts data, it returns a HTTP 400 error. But I can CURL the data fine. I am not able to find where I can see or dump the data that is in the rest framework POST. I have tried working through the parser_context dictionary being sent to CustomXMLParse Parse. But I always return None, even with the successful CURL
I overrode the default XMLParser so I could retrieve an attribute. I will say I am able to successfully use the API when I CURL.
Successful Curl
curl http:/website/post/ -H "Content-Type: text/xml" -d "<registrationreport format='summary' regid='abc123' instanceid='0'><complete>complete</complete><success>failed</success><totaltime>19</totaltime><score>0</score></registrationreport>"
Views.py
#api_view(['POST'])
#authentication_classes((BasicAuthentication,))
#parser_classes((CustomXMLParser, ))
#permission_classes((IsAuthenticated,))
def coursePostBack(request):
...
#Error from parsers.py STOPS before running anything in View
Parsers.py
class CustomXMLParser(XMLParser):
"""
Custom XML parser.
"""
logger.debug(' coursePostBack | PARSER HIT ')
media_type = 'text/xml'
def parse(self, stream, media_type=None, parser_context=None):
logger.debug(' coursePostBack | PARSER--PARSE HIT')
assert etree, 'XMLParser requires defusedxml to be installed'
parser_context = parser_context or {}
encoding = parser_context.get('encoding', settings.DEFAULT_CHARSET)
parser = etree.DefusedXMLParser(encoding=encoding)
try:
## This is where it fails! How can I SEE the data being sent in POST ##
tree = etree.parse(stream, parser=parser, forbid_dtd=True)
except (etree.ParseError, ValueError) as exc:
logger.error('XML Parse error - %s' % six.text_type(exc))
raise ParseError('XML parse error - %s' % six.text_type(exc))
data = self._xml_convert(tree.getroot())
attribs = tree.getroot().attrib
data['regid'] = attribs['regid']
return data
Log.py
ERROR [training:36] XML Parse error - not well-formed (invalid token): line 1, column 1
Related
Hi I am new to writing web APIs in python. And my understanding of REST is limited
I have a simple Flask API that takes in a python dict {'pdf':pdf_as_bytes, 'filename':string}
The below is my server script:
#app.route("/predict", methods=["POST"])
def predict():
data = {"success": False}
if flask.request.method == "POST":
pdf = flask.request.files["pdf"].read()
filename = flask.request.files["filename"].read().decode("utf-8")
assert isinstance(filename, str), "Got {}".format(type(filename))
assert isinstance(pdf, bytes), "Got {}".format(type(pdf))
# further processing happens and returns a json
This works as intended when I write a python client as follows:
import requests
import os
ip = "localhost"
port = 8605
url = "http://{}:{}/predict".format(ip,port)
path_to_pdf = "./617339931.pdf"
with open(path_to_pdf, "rb") as f:
pdf = f.read() # pdf is a bytes
# the payload must have the following fields: "pdf": bytes, "filename": string object
payload = {"pdf":pdf,"filename":os.path.basename(path_to_pdf).split(".")[0]}
# post request
result = requests.post(url=url, files=payload).json()
# the resulting json always has a field called as success, which returns True or False accordingly
if result["success"] == True:
print(result["data"].keys())
But, When I send a request using Postman I get a 400 Error! Below is the screen shot of the error
I don't understand. How can I change my server code so that it works with Postman and also Python client programs
I just did the same thing, and I think it's because of the double quotes you are putting in key and value, try to take them out.
I'm getting a POST request inside a Flask app from Slack. The request is sent when a user presses on an interactive message button. According to Slack docs I must extract the body of the request to verify the signature.
My computed signature doesn't match the one sent by Slack, though.
In fact, the body of the request comes as some encoded string. The string is actually an encoded dictionary instead of a query str parameters, as expected.
Here's the beginning of my view:
#app.route('/register', methods=['POST'])
def register_visit():
data = request.get_data()
signature = request.headers.get('X-Slack-Signature', None)
timestamp = request.headers.get('X-Slack-Request-Timestamp', None)
signing_secret = b'aaaaaaaaaaaaaaaa'
# old message, ignore
if round(actual_time.time() - float(timestamp)) > 60 * 5:
return
concatenated = ("v0:%s:%s" % (timestamp, data)).encode('utf-8')
computed_signature = 'v0=' + hmac.new(signing_secret, msg=concatenated, digestmod=hashlib.sha256).hexdigest()
if hmac.compare_digest(computed_signature, signature):
...
I've tried to format the received data to make it look like:
token=fdjkgjl&user_id=1234... but I am not aware of all of the necessary parameters that have to be present in the data.
Any ideas are highly appreciated.
The body of the message is following - after being URL decoded (note I've modified possibly sensitive data):
b'payload={"type":"interactive_message","actions":
[{"name":"yes_button","type":"button","value":"236"}],"callback_id":"visit_button","team":{"id":"fffff","domain":"ffff"},"channel":{"id":"ffff","name":"directmessage"},"user":{"id":"ffffff","name":"fffft"},"action_ts":"1540403943.419120","message_ts":"1541403949.000100","attachment_id":"1","token":"8LpjBuv13J7xAjhl2lEajoBU","is_app_unfurl":false,"original_message":{"text":"Test","bot_id":"DDDDDDDDD","attachments":[{"callback_id":"visit_button","text":"Register","id":1,"color":"3AA3E3","actions":[{"id":"1","name":"yes_button","text":"Yes","type":"button","value":"236","style":""}],"fallback":"Register"}],"type":"message","subtype":"bot_message","ts":"1540413949.000100"},"response_url":"https://hooks.slack.com/actions/ffffff/ffffff/tXJjx1XInaUhrikj6oEzK08e","trigger_id":"464662548327.425084163429.dda35a299eedb940ab98dbb9386b56f0"}'
The reason you are getting the "garbled" data is that you are using request.get_data(). That method will return the raw data of a request, but not do any decoding for you.
Much more convenient is to use request.form.get('payload'), which will directly give you the JSON string of the request object. You can then convert that into a dict object with json.loads() to process it further in your app.
Note that the format you received is the correct format for interactive messages. You will not get a query string (e.g. "token=abc;user_id?def...") as you suggested (like for slash command requests). Interactive message request will always contain the request as JSON string in a payload form property. See here for reference.
Here is a simple working example, which will reply a greeting to the user that pressed the button. It will work directly with Slack, but I recommend using Postman to test it.
#app.py
from flask import Flask, request #import main Flask class and request object
import json
app = Flask(__name__) #create the Flask app
#app.route('/register', methods=['POST'])
def register_visit():
slack_req = json.loads(request.form.get('payload'))
response = '{"text": "Hi, <#' + slack_req["user"]["id"] + '>"}'
return response, 200, {'content-type': 'application/json'}
if __name__ == '__main__':
app.run(debug=True, port=5000) #run app in debug mode on port 5000
OK, the issue wasn't related to how Slack sends me the message. It was about misunderstanding which data comes as bytes and which data is unicode. The culprit was string formatting in my case - the line concatenated = ("v0:%s:%s" % (timestamp, data)).encode('utf-8') should have been concatenated = (b"v0:%b:%b" % (timestamp.encode("utf-8"), data)). Data is already bytes, timestamp meanwhile is unicode.
Cannot believe I've banged my head on this for hours -_-
#app.route('/register', methods=['POST'])
def register_visit():
data = request.get_data()
signature = request.headers.get('X-Slack-Signature', None)
timestamp = request.headers.get('X-Slack-Request-Timestamp', None)
signing_secret = b'aaaaaaaaaaaaaaaa'
# old message, ignore
if round(actual_time.time() - float(timestamp)) > 60 * 5:
return
concatenated = (b"v0:%b:%b" % (timestamp.encode("utf-8"), data))
computed_signature = 'v0=' + hmac.new(signing_secret, msg=concatenated,
digestmod=hashlib.sha256).hexdigest()
if hmac.compare_digest(computed_signature, signature):
...
This worked for me
from urllib import parse
parsed_text = parse.unquote('your bytes text here')
I am trying to develop as a service where someone can send the csv file to my REST API which I dump in the database. My cURL request is reading the data but flask_restful is not able to process it. Can you please tell me what am I doing wrong and how can I fix it?
[EDITED BELOW]
I found after reading the docs that request.files lets you read the files from the POST request from a form. I also found a way to send a csv file through cURL as a form.
class ForBetaAndUpload(Resource):
def post(self, kind='quotes'):
# parser = reqparse.RequestParser()
file = request.files['file']
print(file)
# kind = parser.add_argument('kind').parse_args()['kind']
if kind:
if file and file[-3:]=='csv':
if kind == 'quotes':
try:
df = pd.read_csv(file)
df.to_sql('QUOTES', helper.conx, index=False, if_exists='append')
return jsonify({'message':'%s rows inserted in the databasee table successfully' %(df.shape[0])})
except Exception as e:
return jsonify({'exception': e})
if kind == 'trace':
try:
df = pd.read_csv(todos)
df.iloc[:10].to_sql('TRACE', helper.conx, index=False, if_exists='append')
return jsonify({'message':'%s rows inserted in the databasee table successfully' %(df.shape[0])})
except Exception as e:
return jsonify({'message': e})
else:
return jsonify({'message': 'Please provide a csv file'})
else:
return jsonify({'message':'Please provide the file for to update relevant table in the databse'})
api.add_resource(ForBetaAndUpload, '/upload', endpoint='upload')
if __name__ == "__main__":
app.run(debug=True)
cURL Request:
curl "https://localhost:5000/upload" -X POST -H 'Content-Type: txt/csv' -d trace.csv --insecure
I'm getting the following message:
curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown
protocol
API Error Message
code 400, message Bad HTTP/0.9 request type
('\x16\x03\x01\x02\x00\x01\x00\x01ü\x03\x03\x08Ú:ü^¢Ù~ö7W\x9fDyy\x16j\x7fõ>½\x82\x90uÎ&3ÿZ\x08êE\x00\x00')
How can I send a csv file to the flask restful_api. Is it right what I am doing or is there any other way to do it?
My solution to read csv from Flask is:
in Flask:
f = request.files['file']
f will be a handler to the file, you can then use csv_reader.
and to send the file:
curl -s "http://localhost:5000" \
-F file=#./myfile.csv \
-X POST \
-H 'enctype:multipart/form-data ; Content-Type:multipart/form-data'
And that should works. Let me know.
I'm trying to make a universal script in Python that can be used by anybody to import/export all sorts of information from/to Work Etc CRM platform. It has all the documentation here: http://admin.worketc.com/xml.
However, I am now a bit stuck. Authentication works, I can call different API methods, but only the ones without parameters. I am new to Python and that's why I can't figure out how to pass the parameters onto that specific method in the API. Specifically I need to export all time sheets. I'm trying to call this method specifically: http://admin.worketc.com/xml?op=GetDraftTimesheets. For obvious reasons I cannot disclose the login information so it might be a bit hard to test for you.
The code itself:
import xml.etree.ElementTree as ET
import urllib2
import sys
email = 'email#domain.co.uk'
password = 'pass'
#service = 'GetEmployee?EntityID=1658'
#service = 'GetEntryID?EntryID=23354'
#service = ['GetAllCurrenciesWebSafe']
#service = ['GetEntryID', 'EntryID=23354']
service = ['GetDraftTimesheets','2005-08-15T15:52:01+00:00','2014-08-15T15:52:01+00:00' ]
class workEtcUniversal():
sessionkey = None
def __init__(self,url):
if not "http://" in url and not "https://" in url:
url = "http://%s" % url
self.base_url = url
else:
self.base_url = url
def authenticate(self, user, password):
try:
loginurl = self.base_url + email + '&pass=' + password
req = urllib2.Request(loginurl)
response = urllib2.urlopen(req)
the_page = response.read()
root = ET.fromstring(the_page)
sessionkey = root[1].text
print 'Authentication successful!'
try:
f = self.service(sessionkey, service)
except RuntimeError:
print 'Did not perform function!'
except RuntimeError:
print 'Error logging in or calling the service method!'
def service(self, sessionkey, service):
try:
if len(service)<2:
retrieveurl = 'https://domain.worketc.com/xml/' + service[0] + '?VeetroSession=' + sessionkey
else:
retrieveurl = 'https://domain.worketc.com/xml/' + service[0,1,2] + '?VeetroSession=' + sessionkey
except TypeError as err:
print 'Type Error, which means arguments are wrong (or wrong implementation)'
print 'Quitting..'
sys.exit()
try:
responsefile = urllib2.urlopen(retrieveurl)
except urllib2.HTTPError as err:
if err.code == 500:
print 'Internal Server Error: Permission Denied or Object (Service) Does Not Exist'
print 'Quitting..'
sys.exit()
elif err.code == 404:
print 'Wrong URL!'
print 'Quitting..'
sys.exit()
else:
raise
try:
f = open("ExportFolder/worketcdata.xml",'wb')
for line in responsefile:
f.write(line)
f.close()
print 'File has been saved into: ExportFolder'
except (RuntimeError,UnboundLocalError):
print 'Could not write into the file'
client = workEtcUniversal('https://domain.worketc.com/xml/AuthenticateWebSafe?email=')
client.authenticate(email, password)
Writing a code Consuming API requires resolving few questions:
what methods on API are available (get their list with names)
how does a request to such method looks like (find out url, HTTP method to use, requirements to body if used, what headers are expected)
how to build up all the parts to make the request
What methods are available
http://admin.worketc.com/xml lists many of them
How does a request looks like
GetDraftTimesheet is described here http://admin.worketc.com/xml?op=GetDraftTimesheets
and it expects you to create following HTTP request:
POST /xml HTTP/1.1
Host: admin.worketc.com
Content-Type: text/xml; charset=utf-8
Content-Length: length
SOAPAction: "http://schema.veetro.com/GetDraftTimesheets"
<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
<soap:Body>
<GetDraftTimesheets xmlns="http://schema.veetro.com">
<arg>
<FromUtc>dateTime</FromUtc>
<ToUtc>dateTime</ToUtc>
</arg>
</GetDraftTimesheets>
</soap:Body>
</soap:Envelope>
Building up the request
The biggest task it to build properly shaped XML document as shown above and having elements FromUtc and ToUtc filled with proper values. I guess, the values shall be in format of ISO datetime, this you shall find yourself.
You shall be able building such an XML by some Python library, I would use lxml.
Note, that the XML document is using namespaces, you have to handle them properly.
Making POST request with all the headers shall be easy. The library you use to make HTTP requests shall fill in properly Content-Length value, but this is mostly done automatically.
Veerto providing many alternative methods
E.g. for "http://admin.worketc.com/xml?op=FindArticlesWebSafe" there is set of different methods for the same service:
SOAP 1.1
SOAP 1.2
HTTP GET
HTTP POST
Depending on your preferences, pick the one which fits your needs.
The simplest is mostly HTTP GET.
For HTTP requests, I would recommend using requests, which are really easy to use, if you get through tutorial, you will understand what I mean.
I am having problems with pycurl in conjunction with Twitter's Streaming API filter stream.
What is happening when I run the code below it seems to barf on the perform call. I know this because I placed print statements before and after the perform call. I am using Python 2.6.1 and I am on a Mac if that matters.
#!/usr/bin/python
print "Content-type: text/html"
print
import pycurl, json, urllib
STREAM_URL = "http://stream.twitter.com/1/statuses/filter.json?follow=1&count=100"
USER = "user"
PASS = "password"
print "<html><head></head><body>"
class Client:
def __init__(self):
self.buffer = ""
self.conn = pycurl.Curl()
self.conn.setopt(pycurl.POST,1)
self.conn.setopt(pycurl.USERPWD, "%s:%s" % (USER,PASS))
self.conn.setopt(pycurl.URL, STREAM_URL)
self.conn.setopt(pycurl.WRITEFUNCTION, self.on_receive)
try:
self.conn.perform()
self.conn.close()
except BaseException:
traceback.print_exc()
def on_receive(self,data):
self.buffer += data
if data.endswith("\r\n") and self.buffer.strip():
content = json.loads(self.buffer)
self.buffer = ""
print content
if "text" in content:
print u"{0[user][name]}: {0[text]}".format(content)
client = Client()
print "</body></html>"
First, try turning on verbosity to help debug:
self.conn.setopt(pycurl.VERBOSE ,1)
It looks like you aren't setting the basic auth mode:
self.conn.setopt(pycurl.HTTPAUTH, pycurl.HTTPAUTH_BASIC)
Also according to the documentation, you need to provide a POST of the parameters to the API, not pass them in as a GET parameter:
data = dict( track='stack overflow' )
self.conn.setopt(pycurl.POSTFIELDS,urlencode(data))
You are trying to use basic authentication.
Basic Authentication sends user
credentials in the header of the HTTP
request. This makes it easy to use,
but insecure. OAuth is the Twitter
preferred method of authentication
moving forward - come August 2010,
we'll be turning off Basic Auth from
the API. --Authentication, Twitter