Question:
Hi. I am a beginner trying to learn Python, and for one of my first projects I want to write a script that will fill out a survey automatically for me. I am familiar with coding, and I have most of the code written to solve this problem. What I am struggling is to write a method that will hit button #1 in question #1, or that will push any given button. One way I've realized I can do this, is perhaps by writing a script to press tab-> up-> down-> tab-> up -> down in the order needed to answer all of the questions.
Here is an image of what the survey looks like (CSS was disable for clarity).
http://i.imgur.com/Tn94KFA.jpg
What is one way to go about writing a method to push a radio button?
[Disclaimer]: I have checked out the following questions but they were of no use to me in my current situation:
fill out a webform that uses javascript with python (question was framed very strangely and was about Javascript forums, which I don't understand how it is relevant/ I didn't understand it, and nobody answered the question).
Script to take web survey for me (answers were only about Java, however something like this tool called HtmlUnit seems cool, if there were a comparable library in Python).
How to fill out form data on a website (question is about Java).
There are Python libraries and tools for automatizing browser actions. StackOverflow.com is not a place to ask for an recommendation for such a tool and thus moderators will close this question (SO is usually asking a help for particular problem, not for broad help and tutoriing requests). However here are some starting points for you
Splinter - automate browser actions in Python
Mechanize - Python library for stateful programmatic web browsing
Selenium automizing framework Python bindings - using full installed browser
Headless web browsing listing - includes Python ones
Scrapy - web content scraping framework in Python
For installing Python packages please refer to official package installation tutorial.
Related
In my company intranet, any request to an external website X in Internet will be redirected to an internal page containing a button that I have to click on. Then the external website X in Internet will be loaded.
I want to write a program that automatically clicks this button for me (so I don't have to click it manually). After that, the program will make the browser redirect to a re-configured website Y (not X) for the purpose of security testing.
I don't have much experience with Python. So I would be really thankful if someone can tell me how I can write such a program.
Many thanks
unfortunately it can get a little complicated once you're interacting with javascript elements like buttons. However, the best way to approach this would be with selenium. There's a slight learning curve but thankfully the documentation is good and there are many resources online to show you how to get started.
Python has Selenium and BS4 library to help You out, but if You are not experienced with python, You might as well pick up node.js and puppeteer, its far superior in my opinion.
I have created a really cool Python program that I want all of my friends to try out. I really want to put the running code onto a website so people can try it all around the world. Is it possible to run Python code on a HTML page? If so, how?
In answer to your question: Yes, this is possible.
If you merely want to share your code with your friends, and allow them to try it out (even without them having Python installed), in browser, then there are a number of tools that you can use.
For example, with https://trinket.io/ , you can embed a snippet of python code in an HTML webpage or share it via a link.
Update:
Another alternative online python site is http://repl.it/. Repl.it has compiled the CPython interpreter into Javascript, using an LLVM -> Javascript compiler. It is opensource, so you could even self-host if you wanted. Caveat: Some of the libraries still have bugs in them.
The simple pedestrian solution is to create a WSGI wrapper. You can configure your web server to accept input through a form (typically) and feed that to your Python program as input, then display the program's output as the response to the form submission. Thus your program runs on the server -- which needs to have the required services and resources -- but the user interaction happens simply using the client's web browser.
There are more-complex interaction models but this is how the entire web 1.0 was set up and is quick and easy to get going. (The spec back then was the similar platform-independent CGI API.)
hi everyone I would like to create a small bot to help me on binary option.
i am not an expert on python but actualy I can read a web page and
retrieve a precise value in a tag,
but the information what I need is on a web application
and not in the source code of the web page. I am not an expert of eb application and I want to know if I retrieve a value displayed on the application with python.
here is a link to the picture of the application:
"http://comparatif-options-binaires.fr/wp-content/uploads/2014/05/optionweb-analyse-technique-ow-school.jpg"
I think the problem you face here is the value you need is being loaded via Javascript of some sort (though without access to the web application and no visible effort from your code I can't be sure).
Expanding on #sabhirams answer (and agreeing that requests and BeautifulSoup are excellent libraries for static text) I would recommend having a look at the following:
Selenium - automates web browser usage in python (so will run the full javascript).
Webkit - Again another headless browser for python that has some excellent SO questions on the matter.
Ghost.py - attempts to make the Webkit experience a little smoother.
pyv8 - something a bit more barebones, pyv8 is a python wrapper for the Google V8 Javascript engine and can be used to run the javascript on the page and, hopefully, extract the element you need.
And if you're really not settled with python why not look at using a Javascript headless browser to run the javascript like PhantomJS.
As mentioned before; Respect others when scraping and be aware there may be consequences if you are caught.
I think you mean you want to build a script which can scrape a given webpage, and extract a certain value out of a given target DOM element.
I dont currently have the time to write this code for you, but it should be rather simple to put together. Here are some modules which might help you:
Request - Use this to fetch a given webpage into your py script
BeautifulSoup - Feed the above "DOM text" to beautiful soup, and you will be able to more easily manipulate the HTML page (fetch your var of interest etc...).
EDIT:
As pointed out in the comments above, please consider the Terms and Conditions of the web-service you are trying to scrape info from.
I'm new to web programming, and have recently began looking into using Python to automate some manual processes. What I'm trying to do is log into a site, click some drop-down menus to select settings, and run a report.
I've found the acclaimed requests library: http://docs.python-requests.org/en/latest/user/advanced/#request-and-response-objects
and have been trying to figure out how to use it.
I've successfully logged in using bpbp's answer on this page: How to use Python to login to a webpage and retrieve cookies for later usage?
My understanding of "clicking" a button is to write a post() command that mimics a click: Python - clicking a javascript button
My question (since I'm new to web programming and this library) is how I would go about pulling the data I need to figure out how I would construct these commands. I've been looking into [RequestObject].headers, .text, etc. Any examples would be great.
As always, thanks for your help!
EDIT:::
To make this question more concrete, I'm having trouble interacting with different aspects of a web-page. The following image shows what I'm actually trying to do:
I'm on a web-page that looks like this. There is a drop-down menu with click-able dates that can be changed. My goal is to automate changing the date to the most recent date, "click"'Save and Run', and download the report when it's finished running.
The only solution to this I have found is Selenium. If it werent a javascript heavy website you could try mechanize but for this you need to render the javascript and then inject javascript...like Selenium does.
Upside: You can record actions in Firefox (using selenium) then export those actions to python. The downside is that this code has to open a browser window to run.
Is there a way to determine programmatically, using Python, which web page is currently active in Google Chrome?
Google Chrome for Mac has added the AppleScripting method for getting the URL.
Here's the Chromium AppleScript SDK
https://sites.google.com/a/chromium.org/dev/developers/design-documents/applescript
Example from the page linked below:
tell application "Google Chrome"
get URL of active tab of window 1
end tell
More examples here:
http://laclefyoshi.blogspot.com/2010/10/google-chrome-ver.html
Not yet, it seems. Such tasks are done, with other browsers, via the Applescript interface, but, as this thread shows, while hotly requested for Chrome on the Mac this feature is not there yet.
(Once a Mac app does get a good Applescript interface, there are several ways to use that interface from Python, without needing to actually involve Applescript as a language -- this has been touched upon on other SO questions about other apps -- but the problem is that Chrome in particular doesn't yet support this quintessentially Mac-ish functionality).
Given the insistence of the request I'd expect somebody to eventually roll up their sleeves and contribute a patch for the purpose, but it doesn't seem to have occurred yet.