Develop Reference

Python 3, filling out a form with request (library) returns same page HTML without inputting parameters

I am trying to use requests to fill out a form on https://www.doleta.gov/tradeact/taa/taa_search_form.cfm and return the HTML of the new page that this opens and extract information from the new page.
Here is the relevant HTML
<form action="taa_search.cfm" method="post" name="number_search" id="number_search" onsubmit="return validate(this);">
<label for="input">Petition number</label>
:
<input name="input" type="text" size="7" maxlength="7" id="input">
<input type="hidden" name="form_name" value="number_search" />
<input type=submit value="Get TAA information" />
</form>
Here is the python code I am trying to use.
url = 'https://www.doleta.gov/tradeact/taa/taa_search.cfm'
payload = {'number_search':'11111'}
r = requests.get(url, params=payload)
with open("requests_results1.html", "wb") as f:
f.write(r.content)
When you perform the query manually, this page opens https://www.doleta.gov/tradeact/taa/taa_search.cfm.
However, when I use the above Python code, it returns the HTML of https://www.doleta.gov/tradeact/taa/taa_search_form.cfm (the first page) and nothing is different.
I cannot perform similar code on https://www.doleta.gov/tradeact/taa/taa_search.cfm because it redirects to the first URL and thus, running the code returns the HTML of the first URL.
Because of the permissions setup of my computer, I cannot redirect the path of my PC (which means Selenium is off the table) and I cannot install Python 2 (which means mechanize is off the table). I am open to using urllib but do not know the library very well.
I need to perform this action ~10,000 times to scrap the information. I can build the iteration part myself, but I cannot figure out how to get the base function to work properly.

The first observation is that you seem to be using a get request in your example code instead of a post request.
<form action="taa_search.cfm" method="post" ...>
^^^^^^^^^^^^^
After changing to a post request, I was still getting the same results as you though (html from the main search form page). After a bit of experimentation, I seem to be able to get the proper html results by adding a referer to the header.
Here is the code (I only commented out the writing to file part for example purposes):
import requests
BASE_URL = 'https://www.doleta.gov/tradeact/taa'
def get_case_decision(case_number):
headers = {
'referer': '{}/taa_search_form.cfm'.format(BASE_URL)
}
payload = {
'form_name': 'number_search',
'input': case_number
}
r = requests.post(
'{}/taa_search.cfm'.format(BASE_URL),
data=payload,
headers=headers
)
r.raise_for_status()
return r.text
# with open('requests_results_{}.html'.format(case_number), 'wb') as f:
# f.write(r.content)
Testing:
>>> result = get_case_decision(10000)
>>> 'MODINE MFG. COMPANY' in result
True
>>> '9/12/1980' in result
True
>>> result = get_case_decision(10001)
>>> 'MUSKIN CORPORATION' in result
True
>>> '2/27/1981' in result
True
Since you mentioned that you need to perform this ~10,000 times, you will probably want to look into using requests.Session as well.

I am trying to create POST data for web scraping using python requests

I have a form-input like:
<PRE>
<FORM>
<INPUT name="field_RecordingDateID-StartDate" id="field_RecordingDateID-StartDate" placeholder="mm/dd/yyyy" type="date" value="">
<INPUT name="field_RecordingDateID-EndDate" id="field_RecordingDateID-EndDate" placeholder="mm/dd/yyyy" type="date" value="">
<input type="submit" value="search">
</FORM>
</PRE>
I am try to create dictionary as
payload = dict(field_RecordingDateID-StartDate = "10-08-2016",
field_RecordingDateID-EndDate ="12-08-2016")
And I need to send post request so that I can get the result through Python requests.
I am using
r = requests.get(url,data=payload)
Currently I am getting error as 'Can't assigned the value to operator' in console.
The error is at creating dictionary.
Can you help me?

It is a small problem in your code. Actually the correct way to define a dictionary is will be like this.
payload = {"field_RecordingDateID-StartDate" :"10-08-2016",
"field_RecordingDateID-EndDate" :"12-08-2016"}

You should use a valid identifier. Try to create payload dict as:
StartDate = 'field_RecordingDateID-StartDate'
EndDate = 'field_RecordingDateID-EndDate'
payload = dict(StartDate = '10-08-2016', EndDate = '12-08-2016')
and post your data as:
r = requests.post(url,data=payload)

Send data and get the response

I'm sure that it's pretty easy, but I really can understand it.
I need to write a script with python. The script has to take a link and send this it to http://archive.org/web/, to be more precise the script has to put this link to form "Save page now":
<form id='wwmform_save' name="wwmform_save" method="get" action="#" onsubmit="if (''==$('#web_save_url').val()){$('#web_save_url').attr('placeholder', 'enter a web address')} else {document.location.href='//web.archive.org/save/'+$('#web_save_url').val();} return false;" style="display:inline;">
<input id='web_save_url' class="web_input web_text" type="text" name="url" placeholder="http://" />
<button id='web_save_button' type="submit" class="web_button web_text">SAVE PAGE</button>
</form>
And get an achieved link.
I would like to use the "Requests" library, but can't understan how.
Should I make a request first?
I think I use request.post but I don't realize what parameters I have to use.
Edited:
I did like n1c9 has written, it words and saves links but also I need the link where page was saved. When send request to http://web.archive.org/save/(link) it's loadind few seconds and then redirect to the needed link.

url = 'urlyouwanttoarchive.com'
archive = 'http://web.archive.org/save/'
requests.get(archive + url)
and if you want the URL to the newly archived page:
print(archive + url)
edit: if you had a list of URLs you wanted to archive, this would work too:
urls = ['url1.com','url2.com','url3.com','url4.com']
for i in urls:
requests.get(archive + i)

use python requests to post a html form to a server and save the reponse to a file

I have exactly the same problem as this post
Python submitting webform using requests
but your answers do not solve it. When I execute this HTML file called api.htm in the browser, then for a second or so I see its page.
Then the browser shows the data I want with the URL https://api.someserver.com/api/ as as per the action below. But I want the data written to a file so I try the Python 2.7 script below.
But all I get is the source code of api.htm Please put me on the right track!
<html>
<body>
<form id="ID" method="post" action="https://api.someserver.com/api/ ">
<input type="hidden" name="key" value="passkey">
<input type="text" name="start" value ="2015-05-01">
<input type="text" name="end" value ="2015-05-31">
<input type="submit" value ="Submit">
</form>
<script type="text/javascript">
document.getElementById("ID").submit();
</script>
</body>
</html>
The code:
import urllib
import requests
def main():
try:
values = {'start' : '2015-05-01',
'end' : '2015-05-31'}
req=requests.post("http://my-api-page.com/api.htm",
data=urllib.urlencode(values))
filename = "datafile.csv"
output = open(filename,'wb')
output.write(req.text)
output.close()
return
main()

I can see several problems:
Your post target URL is incorrect. The form action attribute tells you where to post to:
<form id="ID" method="post" action="https://api.someserver.com/api/ ">
You are not including all the fields; type=hidden fields need to be posted too, but you are ignoring this one:
<input type="hidden" name="key" value="passkey">
Do not URL-encode your POST variables yourself; leave this to requests to do for you. By encoding yourself requests won't recognise that you are using an application/x-www-form-urlencoded content type as the body. Just pass in the dictionary as the data parameters and it'll be encoded for you and the header will be set.
You can also stream the response straight to a file object; this is helpful when the response is large. Switch on response streaming, make sure the underlying raw urllib3 file-like object decodes from transfer encoding and use shutil.copyfileobj to write to disk:
import requests
import shutil
def main():
values = {
'start': '2015-05-01',
'end': '2015-05-31',
'key': 'passkey',
}
req = requests.post("http://my-api-page.com/api.htm",
data=values, stream=True)
if req.status_code == 200:
with open("datafile.csv", 'wb') as output:
req.raw.decode_content = True
shutil.copyfileobj(req.raw, output)
There may still be issues with that key value however; perhaps the server sets a new value for each session, coupled with a cookie, for example. In that case you'd have to use a Session() object to preserve cookies, first do a GET request to the api.htm page, parse out the key hidden field value and only then post. If that is the case then using a tool like robobrowser might just be easier.

Retrieving HTML form data and storing in csv with Flask & Python

I have a small .py program, rendering 2 HTML pages. One of those HTML pages has a form in it. A basic form requesting a name, and a comment. I can not figure out how to take the name and the comment from the form and store it into the csv file. I have got the coding so that the very little I already manually input into the csv file is printed/returned on the HTML page, which is one of the goals. But I can't get the data I input into the form into the csv file, then back n the HTML page. I feel like this is a simple fix, but the Flask book makes absolutely no sense to me, I'm dyslexic and I find it impossible to make sense of the examples and the written explanations.
This is the code I have for reading the csv back onto the page;
#app.route('/guestbook')
def guestbook():
with open('nameList.csv','r') as inFile:
reader=csv.reader(inFile)
names=[row for row in reader]
return render_template('guestbook.html',names=names[1:])
And this is my form coding;
<h3 class="tab">Feel free to enter your comments below</h3>
<br />
<br />
<form action="" method="get" enctype="text/plain" name="Comments Form">
<input id="namebox" type="text" maxlength="45" size="32" placeholder="Name"
class="tab"/>
<br />
<textarea id="txt1" class="textbox tab" rows="6" placeholder="Your comment"
class="tab" cols="28"></textarea>
<br />
<button class="menuitem tab" onclick="clearComment()" class="tab">Clear
comment</button>
<button class="menuitem" onclick="saveComment()" class="tab">Save comment</button>
<br>
</div>

By what I understand all you need is to save the data into the file and you don't know how to handle this in Flask, I'll try to explain it with code as clear as possible:
# request is a part of Flask's HTTP requests
from flask import request
import csv
# methods is an array that's used in Flask which requests' methods are
# allowed to be performed in this route.
#app.route('/save-comment', methods=['POST'])
def save_comment():
# This is to make sure the HTTP method is POST and not any other
if request.method == 'POST':
# request.form is a dictionary that contains the form sent through
# the HTTP request. This work by getting the name="xxx" attribute of
# the html form field. So, if you want to get the name, your input
# should be something like this: <input type="text" name="name" />.
name = request.form['name']
comment = request.form['comment']
# This array is the fields your csv file has and in the following code
# you'll see how it will be used. Change it to your actual csv's fields.
fieldnames = ['name', 'comment']
# We repeat the same step as the reading, but with "w" to indicate
# the file is going to be written.
with open('nameList.csv','w') as inFile:
# DictWriter will help you write the file easily by treating the
# csv as a python's class and will allow you to work with
# dictionaries instead of having to add the csv manually.
writer = csv.DictWriter(inFile, fieldnames=fieldnames)
# writerow() will write a row in your csv file
writer.writerow({'name': name, 'comment': comment})
# And you return a text or a template, but if you don't return anything
# this code will never work.
return 'Thanks for your input!'