I am trying to use urllib to fill out a form and then retrieve the results from the submitted data.The app I'm posting to is on a server that I have to login through appgate to access. The POST form (that I see in the response in the terminal) is like this:
<form action="accuracy-upload3" method="post"
enctype="multipart/form-data">
Human: <input type="file" id="human" name="human"><br>
Machine: <input type="file" id="machine" name="machine"><br>
<input type="submit" class="submit" value="Submit" />
</form>
But even if the method is "POST" I seem to be doing a GET instead since the html from the url is all that returns, not the response from the url on the 'action' parameter. My code is as follows:
url = "https://accuracy3.html"
filename1 = 'human.txt'
filename2 = 'machine.txt'
filedata1 = open(filename1).read()
filedata2 = open(filename2).read()
values = {
"Human": filedata1,
"id": "human",
"name": "human",
"Machine": filedata2,
"id": "machine",
"name": "machine",
"value": "Submit"
}
req = urllib2.Request(url, data = urllib.urlencode(values), headers
=
{"Content-type": "application/x-www-form-urlencoded"})
response = urllib2.urlopen(req)
print response.read()
From Making a POST call instead of GET using urllib2 I see that it should be sending a POST since I've sent data, but it isn't doing that.
Can anyone see what the problem is with my script?
Related
I am trying to build a Lambda that displays a simple html form, where you fill your name (Mary for example) and the output should be "welcome Mary" but I dont know how to do it without .php
some information:
1.i am using python.
2.the first if (==GET) works fine.
3.action="lambda's URL", omitted in the code below.
4.my problem is on the second if(==POST).I dont know how to collect form data after submitting my HTML form.
Thanks in advance;)
here is the code:
import json
def lambda_handler(event, context):
if event['requestContext']['http']['method'] == 'GET':
content='''
<html>
<body>
<form action="my lambda's URL here" method="POST">
Name: <input type="text" name="fname"><br>
<input type="submit">
</form>
</body>
</html>
'''
if event['requestContext']['http']['method'] == 'POST':
content='''
<html>
<body>
<p>
"I would like to see:"Welcome Mary" here but i don't know how!
</p>
</body>
</html>
'''
# TODO implement
response = {
"statusCode": 200,
"headers": {
'Content-Type': 'text/html',
},
If your code you're accessing parts of the request like the HTTP method.
The event actually holds a heap of different bits of information about the request. You can see an example in the Lambda developer guide.
I would suggest printing out the entire event to start with while you get used to the format. Then work out how to access the form data.
I have been currently working with Flask & MongoDB for experimenting around with a small application. I'm currently having the user get all of the laws in a MongoDB collection and create a new law via a POST method to the MongoDB collection by sending JSON to the server.
However, the output below shows that sending a POST request to /post/law will send out a 400 error.
127.0.0.1 - - [12/Aug/2020 09:21:50] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [12/Aug/2020 09:21:50] "GET /laws HTTP/1.1" 200 -
127.0.0.1 - - [12/Aug/2020 09:21:54] "POST /post/law HTTP/1.1" 400 -
I've been a bit confused about why this error is happening. I know that 400 errors are usually connected to bad requests, however, I'm not sure where the bad request error is happening?
# This handles the POST request
#app.route('/post/law', methods = ["POST"])
def post_law():
if request.is_json:
content = request.get_json()
print(content)
Client-side:
<!--This tries to send a request to /post/law-->
<h1>Submit a Law</h1>
<form action="", method="GET">
<div id="law_name_section">
<label>Name of the law:</label>
<input type="text" required id="law_name">
</div>
<div id="law_description_section">
<label>Description of the law:</label>
<input type="text" required id="law_description">
</div>
<div id="law_url_section">
<label>URL to the law:</label>
<input type="text" required id="law_url">
</div>
<input type="submit" value="Submit law" onclick="submitLaw()">
</form>
<script>
// submitLaw() tries to send a request to /post/law in a JSON request
async function submitLaw() {
let name = await document.getElementById('law_name').value
let description = await document.getElementById('law_description').value
let url = await document.getElementById('law_url').value
let options = {
method: "POST",
headers: { "Content-Type": "application/json" },
body: {
name: name,
description: description,
url: url,
status: "Bill"
}
}
let response = await fetch("http://127.0.01:8000/post/law", options)
if (response.ok) {
alert("Successfully sent data to /post/law")
} else {
alert(`Couldn't send data to /post/law`)
}
}
</script>
Could be because your view does not return a response. Try:
#app.route('/post/law', methods = ["POST"])
def post_law():
if request.is_json:
content = request.get_json()
print(content)
return "hello"
Also, your URL is malformed. Should be:
await fetch("http://127.0.0.1:8000/post/law", options)
And, why all the async calls? The only thing that should be async is your fetch()
You also have 2 submits occurring. Try this:
<!--This tries to send a request to /post/law-->
<h1>Submit a Law</h1>
<form action="", method="POST">
<div id="law_name_section">
<label>Name of the law:</label>
<input type="text" required id="law_name">
</div>
<div id="law_description_section">
<label>Description of the law:</label>
<input type="text" required id="law_description">
</div>
<div id="law_url_section">
<label>URL to the law:</label>
<input type="text" required id="law_url">
</div>
<input type="button" value="Submit law" onclick="submitLaw();">
</form>
<script>
// submitLaw() tries to send a request to /post/law in a JSON request
function submitLaw() {
let name = document.getElementById('law_name').value;
let description = document.getElementById('law_description').value;
let url = document.getElementById('law_url').value;
let options = {
method: "POST",
headers: { "Content-Type": "application/json" },
body: {
name: name,
description: description,
url: url,
status: "Bill"
}
}
let response = await fetch("http://127.0.0.1:8000/post/law", options)
if (response.ok) {
alert("Successfully sent data to /post/law")
} else {
alert(`Couldn't send data to /post/law`)
}
}
</script>
I am trying to fill a form in a webpage that has a single text box and a send button the html looks like this
<form class="form-horizontal">
<div class="row">
<div class="col-md-12">
<div id="TextContainer" class="textarea-container">
<textarea id="Text" rows="5" maxlength="700" class="form-control remove-border" style="background:none;"></textarea>
</div><button id="Send" class="btn btn-primary-outline" type="button" onclick="SendMessage()" style="margin-top:10px" data-loading-text="Loading..."><span class="icon icon-pencil"></span> Send</button>
</div>
</div>
</form>
I tried to use mechanize to submit the form with this code
import re
from mechanize import Browser
br = Browser()
response=br.open("https://abcd.com/")
for f in br.forms():
if f.attrs['class'] == 'form-horizontal':
br.form = f
text = br.form.find_control(id="Text")
text.value = "something"
br.submit()
The code runs without an error, but no submission is happening , how do I do it?
Here is the SendMessage function
function SendMessage() {
var text = $('#Text').val();
var userId = $('#RecipientId').val();
if (text.trim() === "")
{
$('#TextContainer').css('border-color', 'red');
}
else if (new RegExp("([a-zA-Z0-9]+://)?([a-zA-Z0-9_]+:[a-zA-Z0-9_]+#)?([a-zA-Z0-9.-]+\\.[A-Za-z]{2,4})(:[0-9]+)?(/.*)?").test(text))
{
$('#TextContainer').css('border-color', 'red');
$('#message').html("Links are not allowed in messages");
}
else
{
$('#Send').button('loading');
$.ajax(
{
url: '/Messages/SendMessage',
type: 'POST',
cache: false,
data:
{
__RequestVerificationToken: $('<input name="__RequestVerificationToken" type="hidden" value="CfDJ8MQSRebrM95Pv2f7WNJmKQWGnVR66zie_VVqFsquOCZLDuYRRBPP1yzk_755VDntlD3u0L3P-YYR0-Aqqh1qIjd09HrBg8GNiN_AU48MMlrOtUKDyJyYCJrD918coQPG0dmgkLR3W85gV6P4zObdEMw" />').attr('value'),
userId: userId,
text: text
}
});
}
}
I suspect the issue is that the submit button in the HTML form is not of type=submit - so mechanise won't know what to do when you call br.submit(). The fix is to either change the button type on the HTML website, or tell Browser which button to use for submitting the form:
br.submit(type='button', id='Send')
The submit method takes the same arguments as the HTML Forms API, so I recommend taking a look at the documentation for more details.
Update
The problem here seems to be the JavaScript method attached to the button. Mechanize does not support calling JavaScript functions, hence you won't be able to just use the .submit() method to submit the form. Instead, the best option would probably be to read in the SendMessage() JavaScript function, which gets called if someone clicks on the Send button, and translate it to Python manually. In the best case it consists of a simple AJAX POST request which is very easy to implement in Python. Please look here for a related question.
Second Update
Given the new information in your question, in particular the JavaScript function, you can now manually implement the POST request inside your Python script. I suggest the use of the Requests module which will make the implementation much easier.
import requests
data = {
"__RequestVerificationToken": "CfDJ8MQSRebrM95Pv2f7WNJmKQWGnVR66zie_VVqFsquOCZLDuYRRBPP1yzk_755VDntlD3u0L3P-YYR0-Aqqh1qIjd09HrBg8GNiN_AU48MMlrOtUKDyJyYCJrD918coQPG0dmgkLR3W85gV6P4zObdEMw",
"userId": "something",
"text": "something else"
}
response = requests.post("https://example.com/Messages/SendMessage", data=data)
response will now consist of the response which you can use to check if the request was successfully made. Please note that you might need to read out the __RequestVerificationToken with mechanize as I suspect it is generated each time you open the website. You could just read out the HTML source with html_source = br.read() and then search for __RequestVerificationToken and try to extract the corresponding value.
You can give name attribute to your text area like:
<form class="form-horizontal">
<div class="row">
<div class="col-md-12">
<div id="TextContainer" class="textarea-container">
<textarea id="Text" name="sometext" rows="5" maxlength="700" class="form-control remove-border" style="background:none;"></textarea>
</div><button id="Send" class="btn btn-primary-outline" type="button" onclick="SendMessage()" style="margin-top:10px" data-loading-text="Loading..."><span class="icon icon-pencil"></span> Send</button>
</div>
</div>
</form>
Then try this out:
import re
from mechanize import Browser
br = mechanize.Browser()
br.open("https://abcd.com/")
br.select_form(nr=0) #in case of just single form you can select form passing nr=0
br["sometext"] = "something"
response = br.submit()
print(response.read())
If it successfully submits form then you can read your response body.
I am working on my first project using python to submit a form and retrieve weather data from http://www.weather.gov
I am brand new to HTTP form submission and as such it is highly possible I am going about this all wrong. I've read that mechanize and/or selenium are more efficient for this type of job, but I am limited to these modules by my school's server.
import requests
r = requests.get('http://www.weather.gov')
location = raw_input("Enter zipcode: ")
payload = {'key1' : location}
q = requests.post('http://forecast.weather.gov/', data = payload)
print q.text
My attempts to search a given zipcode have been unsuccessful, I am not reaching the local weather for the given zipcode.
Note: I have also tried this form submission using urllib & urllib2 without success.
import urllib
import urllib2
location = raw_input("Enter Zip Code here: ")
url = 'http://forecast.weather.gov/'
values = {'inputstring' : location}
data = urllib.urlencode(values)
req = urllib2.Request(url, data = data)
response = urllib2.urlopen(req)
html = response.read()
print html
Form as seen from inspecting the page:
<form name="getForecast" id="getForecast" action="http://forecast.weather.gov/zipcity.php" method="get">
<label for="inputstring">Local forecast by <br>"City, St" or ZIP code</label>
<input id="inputstring" name="inputstring" type="text" value="Enter location ..." onclick="this.value=''" autocomplete="off">
<input name="btnSearch" id="btnSearch" type="submit" value="Go">
<div id="txtError">
<div id="errorNoResults" style="display:none;">Sorry, the location you searched for was not found. Please try another search.</div>
<div id="errorMultipleResults" style="display:none">Multiple locations were found. Please select one of the following:</div>
<div id="errorChoices" style="display:none"></div>
<input id="btnCloseError" type="button" value="Close" style="display:none">
</div>
<div id="txtHelp"><a style="text-decoration: underline;" href="javascript:void(window.open('http://weather.gov/ForecastSearchHelp.html','locsearchhelp','status=0,toolbar=0,location=0,menubar=0,directories=0,resizable=1,scrollbars=1,height=500,width=530').focus());">Location Help</a></div>
</form>
<input id="inputstring" name="inputstring" type="text" value="Enter location ..." onclick="this.value=''" autocomplete="off">
Post to the URL http://forecast.weather.gov/zipcity.php
That is where the php script lies for receiving post data from the form.
I am trying to use the Python requests module to authenticate to a website and then retrieve some information from it. This is the login part of the page:
<div>
<label class="label-left" for="username"> … </label>
<input id="username" class="inputbox" type="text" size="18" alt="username" name="username"></input>
</div>
<div>
<label class="label-left" for="passwd"> … </label>
<input id="passwd" class="inputbox" type="password" alt="password" size="18" name="passwd"></input>
</div>
<div> … </div>
<div class="readon">
<input class="button" type="submit" value="Login" name="Submit"></input>
What I am doing now is:
payload = {
'username': username,
'passwd': password,
'Submit':'Login'
}
with requests.Session() as s:
s.post(login, data=payload)
ans = s.get(url)
print ans.text
The problem is that I get the same login page, even after the authentication. The response code is 200, so everything should be ok. Am I missing something?
UPDATE
Thanks to the comment, I have analyzed the post requests and I've seen that there are some hidden parameters. Among them, there are some parameters whose values vary between different requests. For this reason, I am simply getting them with BeautifulSoup and then updating the payload of the post request as follows:
with requests.Session() as s:
login_page = s.get(login)
soup = BeautifulSoup(login_page.text)
inputs = soup.findAll(name='input',type='hidden')
for el in inputs:
name = el['name']
value = el['value']
payload[name]=value
s.post(login, data=payload)
ans = s.get(url)
Nevertheless, I am still getting the login page. There can be some other influencing elements?