I use Requests and python2.7 in order to fill some values in a form and upload (submit) an image to the server.
I actually execute the script from the server pointing to the file I need to upload. The file is located in the /home directory and I have made sure it has full permissions.
Although I get a 200 Response, nothing is uploaded. This is part of my code:
import requests
try:
headers = {
"Referer": 'url_for_upload_form',
"sessionid": sessionid # retrieved earlier
}
files = {'file': ('doc_file', open('/home/test.png', 'rb'))}
payload = {
'title': 'test',
'source': 'source',
'date': '2016-10-26 02:13',
'csrfmiddlewaretoken': csrftoken # retrieved earlier -
}
r = c.post(upload_url, files=files, data=payload, headers=headers)
print r.headers
print r.status_code
except:
print "Error uploading file"
As I said I get a 200 Response and the headers returned are:
{'Content-Language': 'en', 'Transfer-Encoding': 'chunked', 'Set-Cookie': 'csrftoken=fNfJU8vrvOLAnJ5h7QriPIQ7RkI755VQ; expires=Tue, 17-Oct-2017 08:04:58 GMT; Max-Age=31449600; Path=/', 'Vary': 'Accept-Language, Cookie', 'Server': 'nginx/1.6.0', 'Connection': 'keep-alive', 'Date': 'Tue, 18 Oct 2016 08:04:58 GMT', 'X-Frame-Options': 'SAMEORIGIN', 'Content-Type': 'text/html; charset=utf-8'}
Does anyone have any idea what I am doing wrong? Am I missing something basic here?
Related
I'm currently using the python requests module to perform automated HTTP tasks on a website.
The problem is that I don't get the same results on my console as on my browser.
This is what I get when making a POST request on my browser:
This is what I get when making the POST request through the python requests module and running the .headers method on the request:
{
'Date': 'Fri, 14 Jul 2017 15:19:22 GMT',
'Content-Type': 'text/html; charset=utf-8',
'Transfer-Encoding': 'chunked',
'Connection': 'keep-alive',
'Cache-Control': 'private',
'Location': '/cart/view',
'Set-Cookie': 'png.notice=9Hz8GWQ38JQZqTrqcsnn1J5nfgIZt71orHtf71mI+rwqFpQg4RnV7BqZni/GgIS/SmUnC4jgnhjQuDhZNW2adxeLctG+bToT0wTTbgxe40t5RmbVv1viuH2gkL1eH2xN3IavOUBhVXm+JlQrmVnHLocqjgvWi8wAClLYmrShY1U2ege9; expires=Fri, 14-Jul-2017 15:34:03 GMT; path=/; HttpOnly',
'X-Powered-By': 'ASP.NET',
'X-UA-Compatible': 'IE=Edge,chrome=1',
'Server': 'cloudflare-nginx',
'CF-RAY': '37e575befbf43c35-CDG'
}
Notice how the two results are completely different.
I'm trying to get the "Location" header inside the Response headers (the one beginning with "https://live.adyen.com/hpp...".
What am I doing wrong here?
EDIT: This is my source code:
request = session.post('https://www.nakedcph.com/cart/process', data=user_info)
request.url
# outputs 'https://www.nakedcph.com/cart/view' (probably the issue)
request.headers
# outputs the headers (but not all of them?)
PS: After making the POST request, the website redirects to the URL inside the "Location" header from the Response headers.
I figured it out. Missed some parameters in the post request. My bad.
I tried requesting an RSS feed on Treasury Direct using Python. In the past I've used urllib, or requests libraries to serve this purpose and it's worked fine. This time however, I continue to get the 406 status error, which I understand is the page's way of telling me it doesn't accept my header details from the request. I've tried altering it however to no avail.
This is how I've tried
import requests
url = 'https://www.treasurydirect.gov/TA_WS/securities/announced/rss'
user_agent = {'User-agent': 'Mozilla/5.0'}
response = requests.get(url, headers = user_agent)
print response.text
Environments: Python 2.7 and 3.4.
I also tried accessing via curl with the same exact error.
I believe this to be page specific, but can't figure out how to appropriately frame a request to read this page.
I found an API on the page which I can read the same data in json so this issue is now more of a curiosity to me than a true problem.
Any answers would be greatly appreciated!
Header Details
{'surrogate-control': 'content="ESI/1.0",no-store', 'content-language': 'en-US', 'x-content-type-options': 'nosniff', 'x-powered-by': 'Servlet/3.0', 'transfer-encoding': 'chunked', 'set-cookie': 'BIGipServerpl_www.treasurydirect.gov_443=3221581322.47873.0000; path=/; Httponly; Secure, TS01598982=016b0e6f4634928e3e7e689fa438848df043a46cb4aa96f235b0190439b1d07550484963354d8ef442c9a3eb647175602535b52f3823e209341b1cba0236e4845955f0cdcf; Path=/', 'strict-transport-security': 'max-age=31536000; includeSubDomains', 'keep-alive': 'timeout=10, max=100', 'connection': 'Keep-Alive', 'cache-control': 'no-store', 'date': 'Sun, 23 Apr 2017 04:13:00 GMT', 'x-frame-options': 'SAMEORIGIN', '$wsep': '', 'content-type': 'text/html;charset=ISO-8859-1'}
You need to add accept to headers request:
import requests
url = 'https://www.treasurydirect.gov/TA_WS/securities/announced/rss'
headers = {'accept': 'application/xml;q=0.9, */*;q=0.8'}
response = requests.get(url, headers=headers)
print response.text
Totally newbie on python, I had to try to upload a gzip file using requests module and python3.5 launched from a freeBSD bash.
Here is my upload.py after gathering the Internet information I managed to get:
#!/usr/bin/env python3.5
import requests, os.path, sys, glob, time, re, datetime
urlPydio ='https://remote.pydio.server.fr'
certPydio = 'remote.pydio.server.ssl.crt'
depotDistant = 'ws'
urlComplet = urlPydio + "/api/" + depotDistant + "/"
loginPydio = 'user.name'`
passwordPydio = 'PASSWORD'
repLocal = os.path.abspath('/var/squid/logs/access_log_gz/')
yesterday = datetime.date.today() - datetime.timedelta(days=1)
listFichiers = os.listdir(repLocal)
for fichier in listFichiers:`
if re.search ('^\w{5}_\w{4}_\w+_'+ str(yesterday) + '.gz$', fichier):
nomFichierComplet = repLocal +'/'+ fichier
headers = {'x-File-Name': fichier}
urlAction = urlComplet + "upload/input_stream" + "/remote_folder/remote_folder/"
print(nomFichierComplet, urlAction)
files = {fichier: open(nomFichierComplet, 'rb')}
try:
rest = requests.put(urlAction, files=files, auth=(loginPydio,passwordPydio), headers=headers, verify=False)
codeRetour = rest.status_code
if codeRetour == 200:
print('file sent succcessfully', codeRetour, rest.headers)
else:
print('error sending file ', codeRetour, rest.text)
except requests.exceptions.RequestException as e:
print(e)
As a result, I managed to get the log.gz file on remote pydio server but there is some headers stuff added in the file, making it impossible to unzip. Open with a notepad, the 3 lines to remove are :
--223ef42df08f4792ba6ef6e71cdf749c
Content-Disposition: form-data; name="log.gz"; filename="log.gz"
I tried some request.post stuff, unsuccessful, I tried to change headers values to None, no way to send the file.
I also tried to rest.prepare() to del headers['Content-Disposition'] unsuccessfully too
rest.headers returns :
{'Content-Type': 'text/html; charset=UTF-8', 'Date': 'Thu, 05 Jan 2017 10:10:16 GMT', 'Transfer-Encoding': 'chunked', 'Set-Cookie': 'AjaXplorer=i6m1ud1cgocokaehc58gelc8m2; path=/; HttpOnly', 'Vary': 'Accept-Encoding', 'Connection': 'keep-alive', 'Server': 'nginx', 'Cache-Control': 'no-store, no-cache, must-revalidate, post-check=0, pre-check=0', 'Content-Encoding': 'gzip', 'Expires': 'Thu, 19 Nov 1981 08:52:00 GMT', 'Pragma': 'no-cache'}
Now, I'm here, asking you some help or some papers/posts to read.
I just want to upload a log.gz file without any change to the file itself so it can be downloaded and unzipped.
I'm trying to implement a small function to verify possible phishing URL's and thought that using Google Safe Browsing API would be a good start.
After reading the API documentation I thought I had a handle on things and cobbled together the following code:
import requests
import json
url = "https://safebrowsing.googleapis.com/v4/threatMatches:find?key=<REDACTED>"
headers = {'content-type': 'application/json'}
payload = {'client': {'clientId': "mycompany", 'clientVersion': "0.1"},
'threatInfo': {'threatTypes': ["SOCIAL_ENGINEERING", "MALWARE"],
'platformTypes': ["ANY_PLATFORM"],
'threatEntryTypes': ["URL"],
'threatEntries:': [{'url': "http://www.urltocheck1.org"}]}}
print (json.dumps(payload, indent=4))
r = requests.post(url, headers=headers, json=payload)
If I do a
print (json.dumps(payload, indent=4)
it all looks ok. However, the reply I get back from Google doesn't agree.
{'error': {'message': 'Invalid JSON payload received. Unknown name
"threat_entries:" at \'threat_info\': Cannot find field.', 'status':
'INVALID_ARGUMENT', 'code': 400, 'details': [{'#type':
'type.googleapis.com/google.rpc.BadRequest', 'fieldViolations':
[{'field': 'threat_info', 'description': 'Invalid JSON payload
received. Unknown name "threat_entries:" at \'threat_info\': Cannot
find field.'}]}]}} {'X-Frame-Options': 'SAMEORIGIN',
'Transfer-Encoding': 'chunked', 'Cache-Control': 'private', 'Date':
'Tue, 25 Oct 2016 07:55:30 GMT', 'Content-Type': 'application/json;
charset=UTF-8', 'Alt-Svc': 'quic=":443"; ma=2592000;
v="36,35,34,33,32"', 'X-Content-Type-Options': 'nosniff',
'Content-Encoding': 'gzip', 'X-XSS-Protection': '1; mode=block',
'Server': 'ESF'} application/json; charset=UTF-8
I can't - as usual - spot my mistake. Can someone else spot it and possibly put me on the correct track?
Just remove unnecessary colon after threatEntries
and it should work just fine.
Also if you are using requests version 2.4.2 or newer you do not need to insert content-type header to your code, instead you could move your key to the params section:
import requests
import json
api_key='your_key'
url = "https://safebrowsing.googleapis.com/v4/threatMatches:find"
payload = {'client': {'clientId': "mycompany", 'clientVersion': "0.1"},
'threatInfo': {'threatTypes': ["SOCIAL_ENGINEERING", "MALWARE"],
'platformTypes': ["ANY_PLATFORM"],
'threatEntryTypes': ["URL"],
'threatEntries': [{'url': "http://www.urltocheck1.org"}]}}
params = {'key': api_key}
r = requests.post(url, params=params, json=payload)
# Print response
print(r)
print(r.json())
I am using python-pipedrive to wrap Pipedrive's API though it doesn't quite work out of the box on python3 (which I'm using) so I modified it. I'm having trouble with just the Http requests portion.
This is what taught me how to use Httplib2: https://github.com/jcgregorio/httplib2/wiki/Examples-Python3
Basically, I just want to send a GET request to this:
https://api.pipedrive.com/v1/persons/123?api_token=1234abcd1234abcd
This works:
from httplib2 import Http
from urllib.parse import urlencode
PIPEDRIVE_API_URL = "https://api.pipedrive.com/v1/persons/123?api_token=1234abcd1234abcd"
response, data = http.request(PIPEDRIVE_API_URL, method='GET',
headers={'Content-Type': 'application/x-www-form-urlencoded'})
However, Pipedrive returns an error 401 with 'You need to be authorized to make this request.' if I do this:
PIPEDRIVE_API_URL = "https://api.pipedrive.com/v1/"
parameters = 'persons/123'
api_token = '1234abcd1234abcd'
response, data = http.request(PIPEDRIVE_API_URL + parameters,
method='GET', body=urlencode(api_token),
headers={'Content-Type': 'application/x-www-form-urlencoded'})
The actual response is:
response =
{'server': 'nginx',
'status': '401',
'connection': 'keep-alive',
'set-cookie': 'pipe-session=7b6ddadbc67abdadb6a67dbadcb; path=/; domain=.pipedrive.com; secure; httponly',
'date': 'Sat, 11 Jun 2016 06:50:13 GMT',
'transfer-encoding': 'chunked',
'x-frame-options': 'SAMEORIGIN',
'content-type': 'application/json, charset=UTF-8',
'x-xss-protection': '1; mode=block'}
data =
{'success': False,
'error': 'You need to be authorized to make this request.'}
How do I properly provide the api_token as a parameter (body) to the GET request? Anyone know what I'm doing wrong?
You need to provide the api_token as a query parameter. Concatenate the stings like this
PIPEDRIVE_API_URL = "https://api.pipedrive.com/v1/"
route = 'persons/123'
api_token = '1234abcd1234abcd'
response, data = http.request(PIPEDRIVE_API_URL + route + '?api_token=' + api_token,
method='GET',
headers={'Content-Type': 'application/x-www-form-urlencoded'})