I'm using scraperAPI.com to handle IP rotation for a scraping job I'm working on and I'm trying to implement their new post request method, But I keep receiving a 'HtmlResponse' object has no attribute 'dont_filter' error. Here is the custom start_requests function:
def start_requests(self):
S_API_KEY = {'key':'eifgvaiejfvbailefvbaiefvbialefgilabfva5465461654685312165465134654311'
}
url = "XXXXXXXXXXXXXX.com"
payload={}
headers = {
'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
'x-requested-with': 'XMLHttpRequest',
'Access-Control-Allow-Origin': '*',
'accept': 'application/json, text/javascript, */*; q=0.01',
'referer': 'XXXXXXXXXXX.com'
}
client = ScraperAPIClient(S_API_KEY['key'])
resp = client.post(url = url, body = payload, headers = headers)
yield HtmlResponse(resp.url, body = resp.text,encoding = 'utf-8')
The weird part is that when I execute this script piecewise in scrapy shell it works fine and returns the proper data, Any insight into this issue would be GREATLY appreciated? currently 4 hours into this problem.
Notes:
Client.post returns a response object
Not my real API key
client.post doesn't have a body method
The error you get is caused by returning the wrong type (a Response).
From the docs for start_requests:
This method must return an iterable with the first Requests to crawl for this spider.
It seems the easiest solution would be using a scrapy request (probably a FormRequest) to the API url, instead of using ScraperAPIClient.post().
You should be able to use ScraperAPIClient.scrapyGet() to generate the correct url, but I have not tested this.
If you would prefer to continue using the official api library, a slightly more complicated option is Writing your own downloader middleware.
I am trying to download a series of trading history files from www.mql5.com. I want to automate this using Python which I have never done before. The website requires login so I have been following the tutorial at this page regarding the login/session. What I am trying to achieve is the equivalent of going to this page:
https://www.mql5.com/en/auth_login
and logging in, and then going to a link such as this:
https://www.mql5.com/en/signals/552592/export/history
If I do this in Chrome, pasting that link into the browser (once logged in) downloads a csv file immediately to my Downloads folder without opening any real page in the browser.
The code that I have written is:
import requests
loginurl = 'https://www.mql5.com/en/auth_login'
fileurl = 'https://www.mql5.com/en/signals/552592/export/history'
loginpayload = {
'Login': '<mylogingoeshere>',
'Password': '<mypasswordgoeshere>'
}
session = requests.Session()
post = session.post(loginurl, data=loginpayload)
print post.status_code
myfile = session.get(fileurl)
open('<pathtowhereIwantfilestogo>\\420560.history.csv', 'wb').write(myfile.content)
When I run the code, the print statement prints "200", but it is hard to tell whether it's really "logged in". I was sort of expecting that just "visiting" the file URL in python might be enough to make it appear on my computer somewhere, but that doesn't seem to be the case. I decided to use the first example on this page regarding getting the actual file. From my mode, a csv file is created at the path specified, but it appears to contain lines of HTML code from the website rather than the file data I was expecting. The real file when downloaded properly is a list of trading data.
Please would it be possible for someone to help me identify where I might have gone wrong? It looks like "myfile" is getting a webpage rather than the real file that I am after. There are millions of pages about requesting files/logging in on Google but many of them seem to be much more complicated than what I am trying to achieve, or I simply don't understand them! Any help would be much appreciated.
You have to add headers and proper Payload wile making login request.
import requests
login_url = 'https://www.mql5.com/en/auth_login'
file_url = 'https://www.mql5.com/en/signals/552592/export/history'
login_payload = {
"RedirectAfterLoginUrl":"https://www.mql5.com/",
"RegistrationUrl":"",
"ShowOpenId":"True",
"ViewType":"0",
"Login":"USER_ID",
"Password":"PASSWORD"
}
headers = {
'Host': 'www.mql5.com',
'User-Agent': 'Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language': 'en-US,en;q=0.5', 'Accept-Encoding': 'gzip, deflate, br',
'Referer': 'https://www.mql5.com/en/auth_login',
'Content-Type': 'application/x-www-form-urlencoded',
'Origin': 'https://www.mql5.com'
}
session = requests.Session()
post = session.post(login_url, data=login_payload, headers=headers)
print post.status_code
my_file = session.get(file_url)
with open('<pathtowhereIwantfilestogo>\\420560.history.csv', 'wb') as file:
file.write(my_file.content)
Using Python 3, I am trying to download a file (xlsx) from a https ASP.Net form page using Python requests. I am creating a session and at first trying to login to the site. It is HTTPS but I do not have access to SSL cert, so I am using Verify=False, which I am happy with for this purpose.
I have manually set the User-Agent header with help from here,
to the same as the browser in Network traffic capturing under IE F12 feature, as this page seems to need a browser user-agent, as the python requests user-agent may be forbidden.
I am also capturing __VIEWSTATE and __VIEWSTATEGENERATOR from the response text as advised in this answer and adding this to my POST data along with Username & Password.
import requests
import bs4
login_payload = {'ctl00_txtEmailAddr':my_login, 'ctl00_txtPwd': pwd}
headers = {'User-Agent': user_agent,
'Accept':r'*/*',
'Accept-Encoding':r'gzip, deflate',
'Connection': r'Keep-Alive'}
s = requests.Session()
req = requests.Request('GET', my_url, headers=headers)
prep0 = s.prepare_request(req)
s.headers.update(headers)
resp = s.send(
prep0,
verify=False,
allow_redirects=True,
)
soup = bs4.BeautifulSoup(resp.text)
login_payload["__VIEWSTATE"] = soup.select_one("#__VIEWSTATE")["value"]
login_payload["__VIEWSTATEGENERATOR"] =
soup.select_one("#__VIEWSTATEGENERATOR")["value"]
req_login = requests.Request('POST', juvo_url, headers=s.headers,
data=login_payload)
prep1 = s.prepare_request(req_login)
login_resp = s.send(prep1, verify=False)
Here is the rest of the request body if this helps, I am not using this.
__EVENTTARGET=&__EVENTARGUMENT=&forErrorMsg=&ctl00%24txtEmailAddr=*MYLOGIN*&ctl00%24txtPwd=*MYPASSWORD*&ctl00%24ImgBtnLoging.x=0&ctl00%24ImgBtnLoging.y=0
With other attempts with more code additional to the above, every page, including trying to get the file from the direct hyperlink copied from IE, returns "Object moved to here" (with a direct link to the file I need which works in browser) or redirects me to the login page.
If I try to download this, in Python using this direct link from requests.history, I download a html file with the same, depending on the response either "Object moved to here" or the html of the login page.
My request status is always 302 or 200 as seen from urllib3 debugging being enabled, but I am yet to see any response other than login/object moved to here.
Closest I can get is with this header after doing a GET request after modifying in Python the copied browser URL to the date I am interested in:
(which may actually be a website vulnerability if I can get this far without being logged in...)
{'Cache-Control': 'private', 'Content-Length': '873', 'Content-Type': 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet; charset=utf-8', 'Location': 'redacted login page with a whole load of params', 'Server': 'Microsoft-IIS/7.5', 'content-disposition': 'attachment;filename='redacted filename', 'X-AspNet-Version': '2.0.50727', 'X-Powered-By': 'ASP.NET'}
With almost every SO hyperlink now purple, any clues/suggestions would be greatly appreciated.
Many thanks.
I am dealing with this little error but I can not get the solution. I authenticate into a page and I had opened the "inspect/network" chrome tool to see what web service is called and how. I found out this is used:
I have censored sensitive data releated to the site. So, I have to do this same request using python, but I am always getting error 500 and the log on the server side is not showing helpful information (only java traceback).
This is the code of the request
response = requests.post(url,data = 'username=XXXXX&password=XXXXXXX')
URL has the same string that you see in the image under "General/Request URL" label.
Data has the same string that you see in the image under "Form Data".
It looks very simple request but I can not get it to work :( .
Best regards
If you want your request appears like coming from Chrome, other than sending correct data you need to specify headers as well. The reason you got 500 error is probably there're certain settings on your server side disallowing traffic from "non-browsers".
So in your case, you need to add headers:
headers = {'Accept': 'application/json, text/plain, */*',
'Accept-Encoding': gzip, deflate,
...... # more
'User-Agent': 'Mozilla/5.0 XXXXX...' # this line tells the server what browser/agent is used for this request
}
response = requests.post(url,data = 'username=XXXXX&password=XXXXXXX', headers=headers)
P.S. If you are curious, default headers from requests are:
>>> import requests
>>> session = requests.Session()
>>> session.headers
{'Connection': 'keep-alive', 'Accept-Encoding': 'gzip, deflate',
'Accept': '*/*', 'User-Agent': 'python-requests/2.13.0'}
As you can see the default User-Agent is python-requests/2.13.0, and some websites do block such traffic.
I'm attempting to programmatically submit some data to a form on our company's admin page rather than doing it by hand.
I've written numerous other tools which scrape this website and manipulate data. However, for some reason, this particular one is giving me a ton of trouble.
Walking through with a browser:
Below are the pages I'm attempting to scrape and post data to. Note, that these pages usually show up in js shadowboxes, however, it functions fine with Javascript disabled, so I'm assuming that javascript is not an issue with regards to the scraper trouble.
(Note, since this is a company page, I've filled I've replaced all the form fields with junk titles, so, for instance, the client numbers are completely made-up)
Also, being that it is a company page behind a username/password wall, I can't give out the website for testing, so I've attempted in inject as much detail as possible into this post!
Main entry point is here:
From this page, I click "Add New form", which opens this next page in a new tag (since javascript is disabled).
On this page, I fill out the small form, click submit, which then gets the next page displaying a success message.
Should be simple, right?
Code attempt 1: Mechanize
import mechanize
import base64
import cookielib
br = mechanize.Browser()
username = 'USERNAME'
password = 'PASSWORD'
br.addheaders.append(('Authorization',
'Basic %s' % base64.encodestring('%s:%s' % (username, password))))
br.addheaders = [('User-agent',
'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.22 (KHTML,'
' like Gecko) Chrome/25.0.1364.172 Safari/537.22')]
br.open('www.our_company_page.com/adm/add_forms.php')
links = [link for link in br.links()]
# Follow "Add a form" Link
response = br.follow_link(links[0])
br.select_form(nr=0)
br.form.set_all_readonly(False)
br.form['formNumber'] = "FROM_PYTHON"
br.form['RevisionNumber'] = ['20']
br.form['FormType'] = ['H(num)']
response = br.submit()
print response.read() #Shows the exact same page! >:(
So, as you can see, I attempt to duplicate the steps that I would take in a browser. I load the initial /adm/forms page, follow the first link, which is Add a Form, and fill out the form, and click the submit button. But here's where it get screwy. The response that mechanize returns is the exact same page with the form. No error messages, no success messages, and when I manually check our admin page, no changes have been made.
Inspecting Network Activity
Frustrated, I opened Chrome and watched the network tab as I manually filed out and submitted the form in the browser.
Upon submitting the form, this is the network activity:
Seems pretty straight forward to me. There's the post, and then a get for the css files, and another get for the jquery library. There's another get for some kind of image, but I have no idea what that is for.
Inspecting the details of the POST request:
After some Googling about scraping problems, I saw a suggestion that the server may be expecting a certain header, and the I should simply copy everything that gets made in the POST request and then slowly take away headers until I figure out which one was the important one. So I did just that, copied every bit of information in the Network tab and stuck in my post request.
Code Attempt 2: Urllib
I had some trouble figuring out all of the header stuff with Mechanize, so I switched over to urllib2.
import urllib
import urllib2
import base64
url = 'www.our_company_page.com/adm/add_forms.php'
values = {
'SID':'', #Hidden field
'FormNumber':'FROM_PYTHON1030PM',
'RevisionNumber':'5',
'FormType':'H(num)',
'fsubmit':'Save Page'
}
username = 'USERNAME'
password = 'PASSWORD'
headers = {
'Accept' : 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Charset' : 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
'Accept-Encoding' : 'gzip,deflate,sdch',
'Accept-Language' : 'en-US,en;q=0.8',
'User-Agent' : 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)',
'Authorization': 'Basic %s' % base64.encodestring('%s:%s' % (username, password)),
'Cache-Control' : 'max-age=0',
'Connection' : 'keep-alive',
'Content-Type' : 'application/x-www-form-urlencoded',
'Cookie' : 'ID=201399',
'Host' : 'our_company_page.com',
'Origin' : 'http://our_company_page.com',
'Referer' : 'http://our_company_page.com/adm/add_form.php',
'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, '
'like Gecko) Chrome/26.0.1410.43 Safari/537.31'
}
data = urllib.urlencode(values)
req = urllib2.Request(url, data, headers)
response = urllib2.urlopen(req)
print response.read()
As you can see, I added header present in Chrome's Network tab to the POST request in urllib2.
One addition change from the Mechainze version is that I now access the add_form.php page directly by adding the relevant cookie to my Request.
However, even with duplication everything I can, I still have the exact same issue: The response is the exact same page I started on -- no errors, no success messages, no changes on the server, just returned to a blank form.
Final Step: Desperation sits in, I install WireShark
Time to do some traffic sniffing. I'm determined to see WTF is going on in this magical post request!
I download, install, and fire up Wireshark. I filter for http, and then first submit the form manually in the browser, and then run my code with attempts to submit the form programmatically.
This is the network traffic:
Browser:
Python:
Aside from the headers being in a slightly different order (does that matter), they look exactly the same!
So that's where I am, completely confused as to why a post request, which is (as far as I can tell) nearly identical to the one made by the browser, isn't making any changes on the server.
Has anyone ever encountered anything like this? Am I missing something obvious? What's going on here?
Edit
As per Ric's suggestion, I replicated the POST data exactly. I copies it directly from the Network Source tab in Chrome.
Modified code looks as follows
data = 'SegmentID=&Segment=FROMPYTHON&SegmentPosition=1&SegmentContains=Sections&fsubmit=Save+Page'
req = urllib2.Request(url, data, headers)
response = urllib2.urlopen(req)
print response.read()
The only thing I changed was the Segment value from FROMBROWSER to FROMPYTHON.
Unfortunately, this still yields the same result. The response is the same page, I started from.
Update
working, but not solved
I checked out the requests library, duplicated my efforts using their API, and lo' magically it worked! The POST actually went through. The question remains: why!? I again took another snapshot with wireshark, and as near as I can tell it is exactly the same as the POST made from the browser.
The Code
def post(eventID, name, pos, containsID):
segmentContains = ["Sections", "Products"]
url = 'http://my_site.com/adm/add_page.php'
cookies = dict(EventID=str(eventID))
payload = { "SegmentID" : "",
"FormNumber" : name,
"RevisionNumber" : str(pos),
"FormType" : containsID,
"fsubmit" : "Save Page"
}
r = requests.post(
url,
auth=(auth.username, auth.password),
allow_redirects=True,
cookies=cookies,
data=payload)
Wireshark output
Requests
Browser
So, to summarize the current state of the question. It works, but I nothing has really changed. I have no idea why attempts with both Mechanize and urllib2 failed. What is going on that allows that requests POST to actually go through?
Edit -- Wing Tang Wong suggestion:
At Wing Tand Wongs suggestion, I created a cookie handler, and attached that to the urllib.opener. So no more cookies are being send manually in the headers -- in fact, I don't assign anything at all now.
I first connect to the adm page with has the link to the form, rather than connecting to the form right away.
'http://my_web_page.com/adm/segments.php?&n=201399'
This gives the ID cookie to my urllib cookieJar. From this point I follow the link to the page that has the form, and then attempt to submit to it as usual.
Full Code:
url = 'http://my_web_page.com/adm/segments.php?&n=201399'
post_url = 'http://my_web_page.com/adm/add_page.php'
values = {
'SegmentID':'',
'Segment':'FROM_PYTHON1030PM',
'SegmentPosition':'5',
'SegmentContains':'Products',
'fsubmit':'Save Page'
}
username = auth.username
password = auth.password
headers = {
'Accept' : 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Charset' : 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
'Accept-Encoding' : 'gzip,deflate,sdch',
'Accept-Language' : 'en-US,en;q=0.8',
'User-Agent' : 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)',
'Authorization': 'Basic %s' % base64.encodestring('%s:%s' % (username, password)),
'Cache-Control' : 'max-age=0',
'Connection' : 'keep-alive',
'Content-Type' : 'application/x-www-form-urlencoded',
'Host' : 'mt_site.com',
'Origin' : 'http://my_site.com',
'Referer' : 'http://my_site.com/adm/add_page.php',
'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.43 Safari/537.31'
}
COOKIEFILE = 'cookies.lwp'
cj = cookielib.LWPCookieJar()
if os.path.isfile(COOKIEFILE):
cj.load(COOKIEFILE)
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
urllib2.install_opener(opener)
data = urllib.urlencode(values)
req = urllib2.Request(url, headers=headers)
handle = urllib2.urlopen(req)
req = urllib2.Request(post_url, data, headers)
handle = urllib2.urlopen(req)
print handle.info()
print handle.read()
print
if cj:
print 'These are the cookies we have received so far :'
for index, cookie in enumerate(cj):
print index, ' : ', cookie
cj.save(COOKIEFILE)
Same thing as before. No changes get made on the server. To verify that the cookies are indeed there, I print them to the console after submitting the form, which gives the output:
These are the cookies we have received so far :
<Cookie EventID=201399 for my_site.com/adm>
So, the cookie is there, and has been sent along side the request.. so still not sure what's going on.
Read and re-read your post and the other folks answers a few times. My thoughts:
When you implemented in mechanize and urllib2, it looks like the cookies were hard coded into the header response. This would most likely cause the form to kick you out.
When you switched to using the web broswer and using the python 'requests' library, the cookies and sessions handling was being taken care of behind the scenes.
I believe that if you change your code to take into account the cookie and session states, ie. presume an automated session at start, has an empty cookie for the site and no session data, but properly tracks and manages it during the session, it should work.
Simple copying and substituting the header data will not work, and a properly coded site should bounce you back to the beginning.
Without seeing the backend code for the website, the above is my observation. Cookies and Session data are the culprit.
Edit:
Found this link: http://docs.python-requests.org/en/latest/
Which describes accessing a site with authentication/etc. The format of the authentication is similar to the Requests implementation you are using. They link to a git source for a urllib2 implementation that does the same thing and I noticed that the authentication bits are different from how you are doing the auth bits:
https://gist.github.com/kennethreitz/973705
from the page:
password_manager = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_manager.add_password(None, gh_url, 'user', 'pass')
auth_manager = urllib2.HTTPBasicAuthHandler(password_manager)
opener = urllib2.build_opener(auth_manager)
I wonder if you change the way you are implementing the authentication bits for the urllib2 implementation, that it would work.
I think that the PHP script is erroring out and not displaying anything because your form data is not exactly identical. Try replicating a post request to be completely identical including all the values. I see that the line-based text data on your Wireshark screenshot for the browser includes parameters such as SegmentPosition which is 0, but in your Python screenshot does not have a value for SegmentPosition. The format for some of the parameters such as Segment seem different between the Browser and the Python request which may be causing an error as it tries to parse it.