In my company intranet, any request to an external website X in Internet will be redirected to an internal page containing a button that I have to click on. Then the external website X in Internet will be loaded.
I want to write a program that automatically clicks this button for me (so I don't have to click it manually). After that, the program will make the browser redirect to a re-configured website Y (not X) for the purpose of security testing.
I don't have much experience with Python. So I would be really thankful if someone can tell me how I can write such a program.
Many thanks
unfortunately it can get a little complicated once you're interacting with javascript elements like buttons. However, the best way to approach this would be with selenium. There's a slight learning curve but thankfully the documentation is good and there are many resources online to show you how to get started.
Python has Selenium and BS4 library to help You out, but if You are not experienced with python, You might as well pick up node.js and puppeteer, its far superior in my opinion.
Related
I currently scrape a website and take a screenshot when a certain case happens.
I want to consume less bandwidth so im trying to do it via Requests.
I cant figure out how will i take screenshots but i thought of a workaround which:
Once the certain case happens - it will open chrome as usual - take screenshot - close chrome.
Any smarter way im not thinking about?
Thanks!
Request is a library for making HTTP requests. You can't "take a screenshot" with it, it makes no sense.
Maybe try Selenium instead.
I am searching for a way which allows me to interact with a webrowser (Firefox,Chrome/Chromium,Edge are the most important).
I am currently using pyautogui, to locate login,password fields to put the login data into them. But since you can extract much easier informations when you can use IDs or xPath or other identifiers on webpages, it would make sense to use that.
I tried Firefox with selenium but I run in some problems. Can I attache it to a user created session (do I need the processID or something like that?). (Can I choose between the normal private session of the current profile?
I need a solution which works on Windows and Linux(it would be nice if the major Linux distros would support it. But the most important distros are Fedora/Ubuntu for me.) mac would be optional but since I do not got any mac I am not able to test it anyway.
The way with debugger mode or similar does not work really well for me since the browser needs to get started in a special way.
Would it possible to use something like this:
Can Selenium interact with an existing browser session? ,
When I can retrieve the this information some how form the existing browser?
driver.command_executor._url
driver.session_id
(But when I understand that currently it only works with browsers started with selenium?)
When I use Selenum and start a browserwindow with it can I login to a website and the user is logged in on the webside on his browser window too(if they us the same profile)? (Or does selenium separate cookies?)
If you need additional information or have some hints please post them so I can see them.
Thank you in advance for your help
It seems that it is not possible to connect to a web browser which was opened by the user to my understanding. How ever I found two possible solutions which I am currently trying to evaluate.
Using pyautogui to access the web browser over scanned images and control it with keyboard and mouse. (It is possible to access the console with the right combinations too).
The other solution is maybe more stable. Writing an browser extension which controls the browser.
Question:
Hi. I am a beginner trying to learn Python, and for one of my first projects I want to write a script that will fill out a survey automatically for me. I am familiar with coding, and I have most of the code written to solve this problem. What I am struggling is to write a method that will hit button #1 in question #1, or that will push any given button. One way I've realized I can do this, is perhaps by writing a script to press tab-> up-> down-> tab-> up -> down in the order needed to answer all of the questions.
Here is an image of what the survey looks like (CSS was disable for clarity).
http://i.imgur.com/Tn94KFA.jpg
What is one way to go about writing a method to push a radio button?
[Disclaimer]: I have checked out the following questions but they were of no use to me in my current situation:
fill out a webform that uses javascript with python (question was framed very strangely and was about Javascript forums, which I don't understand how it is relevant/ I didn't understand it, and nobody answered the question).
Script to take web survey for me (answers were only about Java, however something like this tool called HtmlUnit seems cool, if there were a comparable library in Python).
How to fill out form data on a website (question is about Java).
There are Python libraries and tools for automatizing browser actions. StackOverflow.com is not a place to ask for an recommendation for such a tool and thus moderators will close this question (SO is usually asking a help for particular problem, not for broad help and tutoriing requests). However here are some starting points for you
Splinter - automate browser actions in Python
Mechanize - Python library for stateful programmatic web browsing
Selenium automizing framework Python bindings - using full installed browser
Headless web browsing listing - includes Python ones
Scrapy - web content scraping framework in Python
For installing Python packages please refer to official package installation tutorial.
I'm writing a python script to do some screen scraping of a public website. This is going fine, until I want to interact with an AJAX-implemented tree control. Evidently, there is a large amount of javascript controlling the AJAX requests. It seems that the tree control is a JBoss RichFaces RichTree component.
How should I interact with this component programatically?
Are there any tricks I should know about?
Should I try an implement a subset of the RichFaces AJAX?
Or would I be better served wrapping some code around an existing web-browser? If so, is there a python library that can help with this?
PhantomJS Is probably the most interesting project letting you do javascript stuff in a headless environment with a decent API. While it doesn't support python natively anymore, there are options for interacting with it. Check out the discussion here for more info.
There is also vanilla webkit (wrapped by Qt and then PyQT). Check out an example here.
Hope that helps :)
You need to make the AJAX calls from your client to the server and interpret the data. Interpreting the AJAX data is easier and less error-prone than scraping HTML any way.
Although it can be a bit tricky to figure out the AJAX API if it isn't documented. A network sniffer tool like wireshark can be helpful there, there may also be useful plugins for your browser to do the same nowadays. I haven't needed to do that for years. :-)
My python program basically submits a form by loading a URL. There is a security code that seems to change every second so that you have to actually be on the website to enter the form.
For example,
http://www.locationary.com/prizes/index.jsp?ACTION_TOKEN=index_jsp$JspView$BetAction&inTickets=125000000&inSecureCode=091823021&inCampaignId=3060745
The only solution I can think of is using something like Selenium...I don't know any other way of kind of simulating a web browser but not really having it be as heavy and slow as a web browser...any ideas? Or is there a way I can do this without browser automation?
EDIT:
Response to first answer: I DID get the security code using urllib...the problem is that it seems to already have changed by the time I try to load my submission url...so I'm just guessing/assuming that you have to do it in realtime...
Yes, you'll need to get the security code programmatically since it changes every time. You can do this manually with urllib, or you can use mechanize or Selenium to make things easier.