I am working on a project and I need to validate a piece of data using a third party site. I wrote a python script using the lxml package that successfully checks if a specific piece of data is valid.
Unfortunately, the site does not have a convenient url scheme for their data and therefor I can not predict the specific url that will contain the data for each unique request. Instead the third party site has a query page with a standard html text input that redirects to the proper url.
My question is this: is there a way to input a value into the html input and submit it all from my python script?
Yes there is.
Mechanize
Forms
List the forms
import mechanize
br = mechanize.Browser()
br.open(url)
for form in br.forms():
print "Form name:", form.name
print form
select form
br.select_form("form1")
br.form = list(br.forms())[0]
login form example
br.select_form("login")
br['login:loginUsernameField'] = user
br['login:password'] = password
br.method = "POST"
response = br.submit()
Selenium
Sending input
Given an element defined as:
<input type="text" name="passwd" id="passwd-id" />
you could find it using any of:
element = driver.find_element_by_id("passwd-id")
element = driver.find_element_by_name("passwd")
element = driver.find_element_by_xpath("//input[#id='passwd-id']")
You may want to enter some text into a text field:
element.send_keys("some text")
You can simulate pressing the arrow keys by using the “Keys” class:
element.send_keys("and some", Keys.ARROW_DOWN)
These are the two packages I'm aware of that can do what you've asked.
Related
I'm trying to write a Python code to submit a simple form.
http://stulish.com/soumalya01
[Edit : http://travelangkawi.com/soumalya01/
When you use this link it returns a different page on form submit. Is good for debugging]
Any code would do
Tried both mechanize and mechanical soup. Both are unable to handle the text fields. It does not have a name only ID. But we are unable to get the element by ID
Any Code would do as long as it works. (Fill ABC in the text box and hit submit)
I just followed the documentations of mechanize. See sample code below:
from mechanize import Browser
br = Browser()
br.open('http://stulish.com/soumalya01')
br.select_form(nr=0)
form.set_all_readonly(False) #add this
br.form.set_value('ABC', nr=1)
print(br.form.controls[1])
br.submit()
First of all I was not really sure how to formulate a proper question due to an insufficient knowledge of webprogramming in general, therefore my apologies.
I created a Flask app where an email in generated after a button click and the email contains HTML code (similar to http://code.activestate.com/recipes/473810/). Now the email itself also contains a button. Here's what I want to do:
When the email receiver clicks the button in the email, I want to generate an url and have acces to the database similar to what happens in e.g. the following webpage:
Html (e.g. validation.html):
<body>
<input type="button" class="validate" onclick="location.href='{{ url_for('validate.afterValidation') }}';" value="Validate" />
</body>
Python:
#validate_blueprint.route('/validate/',methods=['GET'])
def afterValidation():
return render_template('afterValidation.html')
So I want to use a similar way for rendering a template after the email button is clicked, using python to acces the database etc.
How can I achieve this?
Thanks!
Hi I am trying to make this basic scraper work, where it should go to a website fill "City" and "area" ,search for restaurants and return the html page.
This is the code i'm using
payload = OrderedDict([('cityId','NewYork'),('area','Centralpark')])
req = requests.get("http://www.somewebsite.com",params=payload)
f = req.content
soup = BeautifulSoup((f))
And Here is how the Source HTML looks like
When I'm checking the resulting soup variable it doesn't have the search results , instead it contains the data from the first page only,which has the form for entering city and area value (i.e. www.somewebsite.com, what i want is results of www.somewebsite.com?cityId=NewYork&area=centralPark).So Is there anything that i have to pass with that params to explicitly press the search button or is there any other way to make it work.
You need first check whether you can visit the URL by web browser and get the correct result.
I am writing a script to login to a website and I would like to fill out a form. I can fill out the login form with no problems, but I am unsure how to fill out the actual form once I login. Below is the code I use to get to the form: The form I would like to access is made up of three tabs that navigates between pages.
import mechanize
op = mechanize.Browser() # use mecahnize's browser
op.set_handle_robots(False)
url= "http://example.com"
op.open(url)
email = raw_input("Username: ") #ask for username
password = raw_input("Password: ") #ask for password
op.select_form(nr=0) #python labels forms 0,1,2,3 etc. since there is only 1 form, just select 0
op.form["loginForm:user"] = email
op.form["loginForm:pw"] = password
op.submit()
for i in op.forms():
print i
this code prints out: I changed the url and and replace the actual javax.faces thing with x's to protect the original code. When I inspect elements with Chrome on the form, the fields are not shown here. What is going on here?
<NavForm POST https://someurl.xhtml application/x-www-form- urlencoded
<HiddenControl(NavForm=NavForm) (readonly)>
<HiddenControl(javax.faces.ViewState=XXXX) (readonly)>>
I'm trying to submit a form on an .asp page but Mechanize does not recognize the name of the control. The form code is:
<form id="form1" name="frmSearchQuick" method="post">
....
<input type="button" name="btSearchTop" value="SEARCH" class="buttonctl" onClick="uf_Browse('dledir_search_quick.asp');" >
My code is as follows:
br = mechanize.Browser()
br.open(BASE_URL)
br.select_form(name='frmSearchQuick')
resp = br.click(name='btSearchTop')
I've also tried the last line as:
resp = br.submit(name='btSearchTop')
The error I get is:
raise ControlNotFoundError("no control matching "+description) ControlNotFoundError: no control matching name 'btSearchTop', kind 'clickable'
If I print br I get this: IgnoreControl(btSearchTop=)
But I don't see that anywhere in the HTML.
Any advice on how to submit this form?
The button doesn't submit the form - it calls some javascript function.
Mechanize can't run javascript, so you can't use it to click that button.
The easy way out is to read that function yourself, and see what it does - if it just submits the form, then maybe you can get around it by submitting the form without clicking on anything.
you need to inspect element first, did mechanize recognize the form ?
for form in br.forms():
print form