I'm writing an application REACT frontend and Flask backend (with Flask-cord installed). When I make a call from the frontend I get an error
Access to fetch at 'http://127.0.0.1:5000/get' from origin 'http://127.0.0.1:3000' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
So I followed this post
https://www.arundhaj.com/blog/definitive-guide-to-solve-cors-access-control-allow-origin-python-flask.html
and configured my application following the instructions.
If I run
$ curl -v -X OPTIONS -H "Origin: http://127.0.0.1:5000" -H "Access-Control-Request-Method: PUT" -H "Access-Control-Request-Headers: Authorization" http://127.0.0.1:5000
I get this response with the right Access-Control-Allow
Trying 127.0.0.1:5000...
* Connected to 127.0.0.1 (127.0.0.1) port 5000 (#0)
> OPTIONS / HTTP/1.1
> Host: 127.0.0.1:5000
> User-Agent: curl/7.77.0
> Accept: */*
> Origin: http://127.0.0.1:5000
> Access-Control-Request-Method: PUT
> Access-Control-Request-Headers: Authorization
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 NOT FOUND
< Server: Werkzeug/2.1.1 Python/3.10.4
< Date: Sat, 23 Apr 2022 09:36:22 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 232
< Access-Control-Allow-Origin: http://127.0.0.1:5000
< Access-Control-Allow-Headers: Authorization
< Access-Control-Allow-Methods: GET, OPTIONS, POST, PUT
If I run the same on http://127.0.0.1:3000 I get this
Trying 127.0.0.1:3000...
* Connected to 127.0.0.1 (127.0.0.1) port 3000 (#0)
> OPTIONS / HTTP/1.1
> Host: 127.0.0.1:3000
> User-Agent: curl/7.77.0
> Accept: */*
> Origin: http://127.0.0.1:3000
> Access-Control-Request-Method: PUT
> Access-Control-Request-Headers: Authorization
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< X-Powered-By: Express
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: *
< Access-Control-Allow-Headers: *
< Content-Security-Policy: default-src 'none'
< X-Content-Type-Options: nosniff
< Content-Type: text/html; charset=utf-8
< Content-Length: 143
< Vary: Accept-Encoding
< Date: Sat, 23 Apr 2022 09:50:15 GMT
< Connection: keep-alive
< Keep-Alive: timeout=5
<
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Cannot OPTIONS /</pre>
</body>
</html>
* Connection #0 to host 127.0.0.1 left intact
Of corse if I run the application the same CORS error is popping up. I have the impression that flask-cors is not seen by React.
Here is the flask-cors configuration
api_config = {
"origins": ["http://127.0.0.1:3000"],
"methods": ["OPTIONS", "GET", "POST", "PUT"],
"allow_headers": ['Content-Type', 'Authorization']
}
CORS(app, resources={
r"/*":api_config
})
And I have this in my js file
useEffect(() => {
fetch("http://127.0.0.1:5000/get", {
mode: 'cors',
method: "GET",
headers: {
"Content-Type": "application/json",
"Accept": "application/json",
},
})
.then(resp => resp.json())
.then(resp => console.log(resp))
.catch(error => console.log(error))
}, []);
I have this issue from an angular + flask app run with python3.10 in a macOS.
The same app works perfectly when run with python3.8 on ubuntu.
I have to try to run it with a lower version of python in the macOS computer to check wether is the os or the python version.
Happy to hear I am not the only one with this issue, I hope I will come back to you with some news.
Related
I am trying to send an image from my flutter mobile app to my ComputerVision flask app.
Everything is also working normally on my local.
After deploying my Flask app using Azure Web service, everything seems to be working fine, using the browser works, POSTMAN works, but whenever I am posting from the Flutter app, I always get this 500 error code.
Peering into the SUKU console, it seems that there is a problem with the PIL package?
2022-11-03T04:22:27.232127482Z: [ERROR] File "/tmp/8dabcd6400b99b6/antenv/lib/python3.9/site-packages/PIL/TiffImagePlugin.py", line 878, in tobytes
2022-11-03T04:22:27.232130983Z: [ERROR] data = ifd.tobytes(offset)
2022-11-03T04:22:27.232134283Z: [ERROR] File "/tmp/8dabcd6400b99b6/antenv/lib/python3.9/site-packages/PIL/TiffImagePlugin.py", line 887, in tobytes
2022-11-03T04:22:27.232137883Z: [ERROR] "<table: %d bytes>" % len(data) if len(data) >= 16 else str(values)
2022-11-03T04:22:27.232142983Z: [ERROR] TypeError: object of type 'int' has no len()
This is the code I'm using to send the POST request (this works fine for the Flask app in local though)
var stream = http.ByteStream(DelegatingStream.typed(imageFile.openRead()));
// get file length
var length = await imageFile.length();
// string to uri
var uri = Uri.parse(ai_url);
// create multipart request
var request = http.MultipartRequest("POST", uri);
// multipart that takes file
var multipartFile = http.MultipartFile('file', stream, length,
filename: basename(imageFile.path));
request.files.add(multipartFile);
Map<String, String> headers = {
"Content-Type": "multipart/form-data",
"Accept-Encoding": "gzip, deflate, br",
"Accept": "*/*",
"Accept": "application/json",
"Connection": "keep-alive"
};
// add file to multipart
request.headers.addAll(headers);
// send
var response = await request.send();
return await response.stream.bytesToString();
And this is the POSTMAN logs (which is working for both deployed and local FLask app)
POST / HTTP/1.1
User-Agent: PostmanRuntime/7.29.0
Accept: */*
Postman-Token: 6e9a5808-7b9a-44a9-a6e3-58509552da11
Host: [...].azurewebsites.net
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Type: multipart/form-data; boundary=--------------------------040251306687332340508582
Content-Length: 87777
----------------------------040251306687332340508582
Content-Disposition: form-data; name="file"; filename="img.jpg"
<img.jpg>
----------------------------040251306687332340508582--
HTTP/1.1 200 OK
Content-Length: 400
Content-Type: text/html; charset=utf-8
Date: Thu, 03 Nov 2022 04:21:17 GMT
Server: gunicorn
[Response...]
Can anyone help me?
I maintain an API client for the TRIAS API (German Link) to retrieve local public transport information for various states / cities in Germany. Recently one of the TRIAS servers (Baden-Württemberg) started responding with an error message to requests.
When I try to send the request via curl, the server responds just fine:
$ curl -vH "Content-Type: text/xml; charset=utf-8" -d#trias-req.xml http://www.efa-bw.de/trias
* Trying 94.186.213.206:80...
* Connected to www.efa-bw.de (94.186.213.206) port 80 (#0)
> POST /trias HTTP/1.1
> Host: www.efa-bw.de
> User-Agent: curl/7.83.1
> Accept: */*
> Content-Type: text/xml; charset=utf-8
> Content-Length: 652
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Tue, 31 May 2022 09:35:35 GMT
< Server: EFAController/10.4.25.9/BW-WW33
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Headers: Authorization, Content-Type
< Access-Control-Allow-Methods: GET
< Access-Control-Expose-Headers: Content-Security-Policy, Location
< Access-Control-Max-Age: 600
< Content-Type: text/xml
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Accept-Ranges: none
< Content-Length: 1520
< Last-Modified: Tue, 31 May 2022 09:35:35 GMT
< Set-Cookie: ServerID=bw-ww33;Path=/
<
<?xml version="1.0" encoding="UTF-8"?>
<trias:Trias xmlns:siri="http://www.siri.org.uk/siri" xmlns:trias="http://www.vdv.de/trias" xmlns:acsb="http://www.ifopt.org.uk/acsb" xmlns:ifopt="http://www.ifopt.org.uk/ifopt" xmlns:datex2="http://datex2.eu/schema/1_0/1_0" version="1.1"><trias:ServiceDelivery><siri:ResponseTimestamp>2022-05-31T09:35:35Z</siri:ResponseTimestamp><siri:ProducerRef>de:nvbw</siri:ProducerRef><siri:Status>true</siri:Status><trias:Language>de</trias:Language><trias:CalcTime>30</trias:CalcTime><trias:DeliveryPayload><trias:LocationInformationResponse><trias:Location><trias:Location><trias:Address><trias:AddressCode>streetID:1500001248::8222000:-1:T 1:Mannheim:T 1::T 1: 68161:ANY:DIVA_STREET:942862:5641376:MRCV:B_W:0</trias:AddressCode><trias:AddressName><trias:Text>Mannheim, T 1</trias:Text><trias:Language>de</trias:Language></trias:AddressName><trias:PostalCode> 68161</trias:PostalCode><trias:LocalityName>Mannheim</trias:LocalityName><trias:LocalityRef>8222000:-1</trias:LocalityRef><trias:StreetName>T 1</trias:StreetName></trias:A* Connection #0 to host www.efa-bw.de left intact
ddress><trias:LocationName><trias:Text>Mannheim</trias:Text><trias:Language>de</trias:Language></trias:LocationName><trias:GeoPosition><trias:Longitude>8.46987</trias:Longitude><trias:Latitude>49.49121</trias:Latitude></trias:GeoPosition></trias:Location><trias:Complete>true</trias:Complete><trias:Probability>0.763999999</trias:Probability></trias:Location></trias:LocationInformationResponse></trias:DeliveryPayload></trias:ServiceDelivery></trias:Trias>
However, it fails with requests.post(), while it works with urllib.request.urlopen():
$ python efabw.py
502 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Proxy Error</title>
</head><body>
<h1>Proxy Error</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
The proxy server could not handle the request<p>Reason: <strong>Error reading from remote server</strong></p></p>
</body></html>
653
200 b'<?xml version="1.0" encoding="UTF-8"?>\n<trias:Trias xmlns:siri="http://www.siri.org.uk/siri" xmlns:trias="http://www.vdv.de/trias" xmlns:acsb="http://www.ifopt.org.uk/acsb" xmlns:ifopt="http://www.ifopt.org.uk/ifopt" xmlns:datex2="http://datex2.eu/schema/1_0/1_0" version="1.1"><trias:ServiceDelivery><siri:ResponseTimestamp>2022-05-31T09:37:02Z</siri:ResponseTimestamp><siri:ProducerRef>de:nvbw</siri:ProducerRef><siri:Status>true</siri:Status><trias:Language>de</trias:Language><trias:CalcTime>30</trias:CalcTime><trias:DeliveryPayload><trias:LocationInformationResponse><trias:Location><trias:Location><trias:Address><trias:AddressCode>streetID:1500001248::8222000:-1:T 1:Mannheim:T 1::T 1: 68161:ANY:DIVA_STREET:942862:5641376:MRCV:B_W:0</trias:AddressCode><trias:AddressName><trias:Text>Mannheim, T 1</trias:Text><trias:Language>de</trias:Language></trias:AddressName><trias:PostalCode> 68161</trias:PostalCode><trias:LocalityName>Mannheim</trias:LocalityName><trias:LocalityRef>8222000:-1</trias:LocalityRef><trias:StreetName>T 1</trias:StreetName></trias:Address><trias:LocationName><trias:Text>Mannheim</trias:Text><trias:Language>de</trias:Language></trias:LocationName><trias:GeoPosition><trias:Longitude>8.46987</trias:Longitude><trias:Latitude>49.49121</trias:Latitude></trias:GeoPosition></trias:Location><trias:Complete>true</trias:Complete><trias:Probability>0.763999999</trias:Probability></trias:Location></trias:LocationInformationResponse></trias:DeliveryPayload></trias:ServiceDelivery></trias:Trias>'
The respective code is:
#! /usr/bin/env python3
from urllib.request import Request, urlopen
from requests import post
URL = 'http://www.efa-bw.de/trias'
HEADERS = {'Content-Type': 'text/xml'}
def main():
with open('trias-req.xml', 'rb') as file:
xml = file.read()
response = post(URL, data=xml, headers=HEADERS)
print(response.status_code, response.text, len(xml))
request = Request(URL, data=xml, headers=HEADERS)
with urlopen(request) as response:
print(response.status, response.read())
if __name__ == '__main__':
main()
Why does the request fail with requests.post() only?
What can I do to debug this further?
Other API servers respond fine to the request with requests.post()
It turned out to be the user agent. After inspecting the headers with tcpdump, I found, that the request fails with user agent python-requests/2.27.1 but succeeds with Python-urllib/3.10 and curl/7.83.1:
#! /usr/bin/env python3
from urllib.request import Request, urlopen
from requests import post
URL = 'http://www.efa-bw.de/trias'
HEADERS_REQUESTS = {'Content-Type': 'text/xml', 'User-Agent': 'Python-urllib/3.10'}
HEADERS_URLLIB = {'Content-Type': 'text/xml'}
def main():
with open('trias-req.xml', 'rb') as file:
xml = file.read()
response = post(URL, data=xml, headers=HEADERS_REQUESTS)
print(response.status_code, response.text, len(xml))
request = Request(URL, data=xml, headers=HEADERS_URLLIB)
with urlopen(request) as response:
print(response.status, response.read())
if __name__ == '__main__':
main()
$ python efabw.py
200 <?xml version="1.0" encoding="UTF-8"?>
<trias:Trias xmlns:siri="http://www.siri.org.uk/siri" xmlns:trias="http://www.vdv.de/trias" xmlns:acsb="http://www.ifopt.org.uk/acsb" xmlns:ifopt="http://www.ifopt.org.uk/ifopt" xmlns:datex2="http://datex2.eu/schema/1_0/1_0" version="1.1"><trias:ServiceDelivery><siri:ResponseTimestamp>2022-05-31T10:09:06Z</siri:ResponseTimestamp><siri:ProducerRef>de:nvbw</siri:ProducerRef><siri:Status>true</siri:Status><trias:Language>de</trias:Language><trias:CalcTime>45</trias:CalcTime><trias:DeliveryPayload><trias:LocationInformationResponse><trias:Location><trias:Location><trias:Address><trias:AddressCode>streetID:1500001248::8222000:-1:T 1:Mannheim:T 1::T 1: 68161:ANY:DIVA_STREET:942862:5641376:MRCV:B_W:0</trias:AddressCode><trias:AddressName><trias:Text>Mannheim, T 1</trias:Text><trias:Language>de</trias:Language></trias:AddressName><trias:PostalCode> 68161</trias:PostalCode><trias:LocalityName>Mannheim</trias:LocalityName><trias:LocalityRef>8222000:-1</trias:LocalityRef><trias:StreetName>T 1</trias:StreetName></trias:Address><trias:LocationName><trias:Text>Mannheim</trias:Text><trias:Language>de</trias:Language></trias:LocationName><trias:GeoPosition><trias:Longitude>8.46987</trias:Longitude><trias:Latitude>49.49121</trias:Latitude></trias:GeoPosition></trias:Location><trias:Complete>true</trias:Complete><trias:Probability>0.763999999</trias:Probability></trias:Location></trias:LocationInformationResponse></trias:DeliveryPayload></trias:ServiceDelivery></trias:Trias> 653
200 b'<?xml version="1.0" encoding="UTF-8"?>\n<trias:Trias xmlns:siri="http://www.siri.org.uk/siri" xmlns:trias="http://www.vdv.de/trias" xmlns:acsb="http://www.ifopt.org.uk/acsb" xmlns:ifopt="http://www.ifopt.org.uk/ifopt" xmlns:datex2="http://datex2.eu/schema/1_0/1_0" version="1.1"><trias:ServiceDelivery><siri:ResponseTimestamp>2022-05-31T10:09:06Z</siri:ResponseTimestamp><siri:ProducerRef>de:nvbw</siri:ProducerRef><siri:Status>true</siri:Status><trias:Language>de</trias:Language><trias:CalcTime>19</trias:CalcTime><trias:DeliveryPayload><trias:LocationInformationResponse><trias:Location><trias:Location><trias:Address><trias:AddressCode>streetID:1500001248::8222000:-1:T 1:Mannheim:T 1::T 1: 68161:ANY:DIVA_STREET:942862:5641376:MRCV:B_W:0</trias:AddressCode><trias:AddressName><trias:Text>Mannheim, T 1</trias:Text><trias:Language>de</trias:Language></trias:AddressName><trias:PostalCode> 68161</trias:PostalCode><trias:LocalityName>Mannheim</trias:LocalityName><trias:LocalityRef>8222000:-1</trias:LocalityRef><trias:StreetName>T 1</trias:StreetName></trias:Address><trias:LocationName><trias:Text>Mannheim</trias:Text><trias:Language>de</trias:Language></trias:LocationName><trias:GeoPosition><trias:Longitude>8.46987</trias:Longitude><trias:Latitude>49.49121</trias:Latitude></trias:GeoPosition></trias:Location><trias:Complete>true</trias:Complete><trias:Probability>0.763999999</trias:Probability></trias:Location></trias:LocationInformationResponse></trias:DeliveryPayload></trias:ServiceDelivery></trias:Trias>'
I'm setting up the Google Contacts CardDAV API client.
OAuth 2.0 using oauth2client.
Request using requests.
from oauth2client import file, client, tools
import requests
SCOPES = 'https://www.googleapis.com/auth/carddav'
store = file.Storage('credentials.json')
creds = store.get()
if not creds or creds.invalid:
flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
creds = tools.run_flow(flow, store)
print(creds.access_token)
hed = {'Authorization': 'Bearer ' + creds.access_token}
response = requests.request('PROPFIND', 'https://www.googleapis.com/.well-known/carddav', headers=hed, allow_redirects=False)
if response.status_code == 301:
location = response.headers['location']
response = requests.request('PROPFIND', 'https://www.googleapis.com' + location, headers=hed)
print(response.text)
But when I request url for get the address book (I get it from Location header of first request), it returns error:
{
"error": {
"code": 400,
"message": "Request contains an invalid argument.",
"status": "INVALID_ARGUMENT"
}
}
Full requests information
First request
requests.request('PROPFIND', 'https://www.googleapis.com/.well-known/carddav', headers=hed, allow_redirects=False)
REQUEST
=======
endpoint: PROPFIND https://www.googleapis.com/.well-known/carddav
headers:
User-Agent: python-requests/2.22.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Authorization: Bearer ya29.***********************************************
Content-Length: 0
=======
RESPONSE
========
status_code: 301
headers:
Content-Type: text/plain; charset=UTF-8
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
X-Frame-Options: SAMEORIGIN
Location: /carddav/v1/principals/<my_email>/lists/default/
Pragma: no-cache
Vary: Origin, X-Origin, Referer
Date: Fri, 21 Jun 2019 11:43:23 GMT
Server: ESF
Content-Length: 0
Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39"
========
Second request
response = requests.request('PROPFIND', 'https://www.googleapis.com' + location, headers=hed)
REQUEST
=======
endpoint: PROPFIND https://www.googleapis.com/carddav/v1/principals/<my_email>/lists/default/
headers:
User-Agent: python-requests/2.22.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Authorization: Bearer ya29.***********************************************
Content-Length: 0
=======
RESPONSE
========
status_code: 400
headers:
Vary: Origin, X-Origin, Referer
Content-Type: application/json; charset=UTF-8
Date: Fri, 21 Jun 2019 11:43:23 GMT
Server: ESF
Content-Length: 127
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39"
body:
{
"error": {
"code": 400,
"message": "Request contains an invalid argument.",
"status": "INVALID_ARGUMENT"
}
}
========
The short answer: The PROPFIND method is generic and must include a body that specifies what information the server should return. You must pass an XML payload in the body of the request that identifies the properties that you are requesting.
Getting the addressbooks URIs
According to Google's CardDav API docs, your first request is perfect and will redirect you to the Address Book Resource for the current user. Here is Google's description of the next step:
Your client program can then discover the principal address book by performing a PROPFIND on the addressbook-home-set and looking for the addressbook and collection resources.
Let me unpack this: your second request should query a list of the user's resources found at the location you got from the first request. To do this query properly, you'll need to pass an XML body with the PROPFIND request, like so:
PROPFIND https://www.googleapis.com/carddav/v1/principals/<my_email>/lists/default/
Authorization: Bearer ya29.***********************************************
Depth: 1
Content-Type: application/xml; charset=utf-8
<D:propfind xmlns:D="DAV:">
<D:prop>
<D:resourcetype />
<D:displayname />
</D:prop>
</D:propfind>
Here you specify the properties that you want the server to respond. You specify the resourcetype property because you are only interested in addressbook or collection resources which contain contacts.
This request will return a list of URIs to resources from which you can select any that have a resource type of addressbook or collection.
At this point, you don't have any contacts or even URIs of contacts. You have a list of URIs for the user's address books or collections of contacts. (Typically, there's just one of these, but there could be many.)
You didn't ask how to get the user's contacts, but I'm assuming it is your end goal and will continue with the subsequent steps.
Getting the contacts URIs
Your next set of requests will query each of the address book URIs for the URIs of their contacts. Loop over each result from the previous query and issue another PROPFIND request on the URI with this payload:
REPORT <addressbook_uri>
Authorization: Bearer ya29.***********************************************
Content-Type: application/xml; charset=utf-8
<D:propfind xmlns:D="DAV:">
<D:prop>
<D:getetag />
<D:getcontenttype />
</D:prop>
</D:propfind>
Here we query the content type of each item so we can determine if it is a VCard type. VCards are legit contact records.
Now you can filter this set of results by contenttype == 'text/vcard' to get a new list of URIs pointing to each contact in the user's address book.
Oh man, we are getting so close.
Get the contact VCards
Finally, assemble your list of URIs to actual contact data and query the data from the server.
Here you will make an addressbook-multiget REPORT request to retrieve a batch of the contacts in your list. Google doesn't say how many contact URIs you may include in your request. I've typically limited my request to a few hundred at a time.
E.g.
REPORT <addressbook_uri>
Authorization: Bearer ya29.***********************************************
Content-Type: application/xml; charset=utf-8
<C:addressbook-multiget xmlns:D="DAV:" xmlns:C="urn:ietf:params:xml:ns:carddav">
<D:prop>
<D:getetag/>
<C:address-data>
<C:allprop/>
</C:address-data>
</D:prop>
<D:href>/carddav/v1/principals/<my_email>/lists/default/contact1.vcf</D:href>
<D:href>/carddav/v1/principals/<my_email>/lists/default/contact2.vcf</D:href>
...
</C:addressbook-multiget>
The response will contain the VCard data for each of the contacts, packed inside XML. Parse out the XML text and then parse the VCard data to finally retrieve your contact details.
Done!
Resources:
Google's limited CardDav docs
The best resource I've found is this description of building a CardDav client by Sabre
When others fail, go to the source: RFC6352
I'm using following infrastructure for scraping a web site:
Scrapy <--> Splash <--> Scrapoxy <--> web site
I'm doing requests via Splash execute endpoint, with a Lua script like this:
function main(splash)
local host = "..."
local port = "..."
local username = "..."
local password = "..."
splash:on_request(function (request)
request:set_proxy{host, port, username=username, password=password}
end)
splash:go(splash.args.url)
return splash:html()
end
I want to detect bans and remove banned proxies. According to Scrapoxy documentation:
Scrapoxy adds to the response an HTTP header x-cache-proxyname
But I don't see this header in response.headers. The only headers are:
{b'Content-Type': b'text/html; charset=utf-8',
b'Date': b'Wed, 18 Apr 2018 19:02:21 GMT',
b'Server': b'TwistedWeb/16.1.1'}
What am I doing wrong? Should I add something to the Lua script to properly return headers?
UPDATE: Actually, it doesn't seem to be a Splash problem. Scrapoxy doesn't return x-cache-proxyname even if used via HTTPie.
http -v --proxy=https:http://<user>:<password>#<scrapoxy-server>:8888 https://<site>
GET / HTTP/1.1
User-Agent: HTTPie/0.9.9
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Host: <site>
HTTP/1.1 200 OK
Server: nginx
Date: Thu, 28 Jun 2018 08:14:26 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
Set-Cookie: <...>
X-Powered-By: Express
ETag: W/"5a31b-faPJ7bjKH24S/3EvHU/8IoJHyxw"
Vary: Cookie, User-Agent
Content-Security-Policy: default-src https:; child-src https:; connect-src https: wss:; form-action https:; frame-ancestors https: http://webvisor.com; media-src https:; object-src https:; img-src https: data: blob:; script-src https: data: 'unsafe-inline' 'unsafe-eval'; style-src https: 'unsafe-inline'; font-src https: data:; report-uri /ajax/csp-report/
Content-Encoding: gzip
I managed to get x-cache-proxyname with this lua script
function main(splash)
local host = "..."
local port = "..."
local username = "..."
local password = "..."
local proxy = ""
splash:on_request(function (request)
request:set_proxy{host, port, username=username, password=password}
end)
splash:on_response_headers(function(response)
proxy = response.headers["x-cache-proxyname"]
end)
splash.images_enabled = false
splash:go(splash.args.url)
splash:set_result_header("x-cache-proxyname", proxy)
splash:go(splash.args.url)
return splash:html()
end
UPDATE:
When you use HTTPs scrapoxy cannot edit headers and add x-cache-proxyname to response
I'm having some difficulties parsing multipart form data when using the django-rest-framework. I've set up a minimal view to just echo back the request data:
from rest_framework.views import APIView
from rest_framework.response import Response
from rest_framework.parsers import MultiPartParser, FormParser
class FileUpload(APIView):
parser_classes = (MultiPartParser, FormParser, )
def post(self, request, format=None, *args, **kwargs):
return Response({'raw': request.data, 'data': request._request.POST,
'files': str(request._request.FILES)})
I am expecting that raw (slightly badly named I admit) contains effectively the same data as request._request.POST and request._request.FILES.
If I POST to the view with Content-Type= application/x-www-form-urlencoded this works as expected:
$ http -v --form POST http://localhost:8000/upload/api/ course=3 name=name
POST /upload/api/ HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 18
Content-Type: application/x-www-form-urlencoded; charset=utf-8
Host: localhost:8000
User-Agent: HTTPie/0.9.2
course=3&name=name
HTTP/1.0 200 OK
Allow: POST, OPTIONS
Content-Type: application/json
Date: Thu, 17 Dec 2015 16:52:37 GMT
Server: WSGIServer/0.2 CPython/3.4.3
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN
{
"data": {
"course": "3",
"name": "name"
},
"files": "<MultiValueDict: {}>",
"raw": {
"course": "3",
"name": "name"
}
}
However if I post with Content-Type=multipart/form-data I get the following:
$ http -v --form POST http://localhost:8000/upload/api/ file#~/Projects/lms/manage.py course=3 name=name
POST /upload/api/ HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 577
Content-Type: multipart/form-data; boundary=634ec7c7e89a487b89c1c07c0d24744c
Host: localhost:8000
User-Agent: HTTPie/0.9.2
--634ec7c7e89a487b89c1c07c0d24744c
Content-Disposition: form-data; name="course"
3
--634ec7c7e89a487b89c1c07c0d24744c
Content-Disposition: form-data; name="name"
name
--634ec7c7e89a487b89c1c07c0d24744c
Content-Disposition: form-data; name="file"; filename="manage.py"
#!/usr/bin/env python
import os
import sys
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings.local")
from django.core.management import execute_from_command_line
execute_from_command_line(sys.argv)
--634ec7c7e89a487b89c1c07c0d24744c--
HTTP/1.0 200 OK
Allow: POST, OPTIONS
Content-Type: application/json
Date: Thu, 17 Dec 2015 16:55:44 GMT
Server: WSGIServer/0.2 CPython/3.4.3
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN
{
"data": {
"course": "3",
"name": "name"
},
"files": "<MultiValueDict: {'file': [<InMemoryUploadedFile: manage.py ()>]}>",
"raw": {}
}
Am I missing something here? I am using HTTPIE to generate the requests here but the same behaviour exists with curl so I'm pretty sure that is not the problem. I am using djangorestframework==3.3.0 and Django==1.8.4
EDIT:
It seems that PUTing to the url (with an otherwise identical request) achieves the desired result:
$ http -v --form PUT http://localhost:8000/upload/api/ file#~/Projects/lms/manage.py course=3 name=name
PUT /upload/api/ HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 577
Content-Type: multipart/form-data; boundary=98feb59a8abe4bfb95a7321f536ed800
Host: localhost:8000
User-Agent: HTTPie/0.9.2
--98feb59a8abe4bfb95a7321f536ed800
Content-Disposition: form-data; name="course"
3
--98feb59a8abe4bfb95a7321f536ed800
Content-Disposition: form-data; name="name"
name
--98feb59a8abe4bfb95a7321f536ed800
Content-Disposition: form-data; name="file"; filename="manage.py"
#!/usr/bin/env python
import os
import sys
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings.local")
from django.core.management import execute_from_command_line
execute_from_command_line(sys.argv)
--98feb59a8abe4bfb95a7321f536ed800--
HTTP/1.0 200 OK
Allow: POST, PUT, OPTIONS
Content-Type: application/json
Date: Thu, 17 Dec 2015 18:10:34 GMT
Server: WSGIServer/0.2 CPython/3.4.3
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN
{
"data": {},
"files": "<MultiValueDict: {}>",
"raw": "<QueryDict: {'name': ['name'], 'course': ['3'], 'file': [<InMemoryUploadedFile: manage.py ()>]}>"
}
So I could just use PUT. That is however not ideal as the client does not control what the file is named or where it is located on the server. POST is more appropriate in that sense. In any sense, I don't see why PUT works when POST doesn't.
It is a known issue for the version you're using. Upgrading django rest framework to the latest version will solve the problem. However, you can PUT the request as a workaround.