I am trying to get the current url opened in my pc in any browser using python.
I tried selenium it worked but it opens up a new browser window. I only want to get the current url opened in the browser.
Any help would be appreciated.
use keylogger in python, that would easiest solution, orelse you should catch each and every packets in network.
Selenium isn't made for such purposes. It can be used to automate testing of websites or for website scraping. Selenium associated with a driver (like Chrome's driver for example) is just a fancy browser that let's you interact with it in a non-intrusive way.
If you need to interact with already running apps, you could attach to another process's memory and read stuff directly from there but it might be very messy.
Also, see: Interact with other programs using Python
Related
So, I made a web page on which I have a button that when clicked runs Selenium code (I'm writing in Python). I want my Selenium code to run in this existing, manually opened browser(in which I pre-opened this web page) rather than open new browser window. I already looked through all the similar questions here, but every answer requires opening chrome in debugging mode (which I want to avoid) or doesn't use this manually opened browser but some else. My idea is that there needs to be a way for me to get session id of my existing browser and "give" it to Selenium so it knows where to run. Any ideas, solutions? :)
It's officially not supported.
However, there is some working code which claims to support this:, Also this is in java I am sure you can make in your desire language
https://web.archive.org/web/20171214043703/http://tarunlalwani.com/post/reusing-existing-browser-session-selenium-java/
I am searching for a way which allows me to interact with a webrowser (Firefox,Chrome/Chromium,Edge are the most important).
I am currently using pyautogui, to locate login,password fields to put the login data into them. But since you can extract much easier informations when you can use IDs or xPath or other identifiers on webpages, it would make sense to use that.
I tried Firefox with selenium but I run in some problems. Can I attache it to a user created session (do I need the processID or something like that?). (Can I choose between the normal private session of the current profile?
I need a solution which works on Windows and Linux(it would be nice if the major Linux distros would support it. But the most important distros are Fedora/Ubuntu for me.) mac would be optional but since I do not got any mac I am not able to test it anyway.
The way with debugger mode or similar does not work really well for me since the browser needs to get started in a special way.
Would it possible to use something like this:
Can Selenium interact with an existing browser session? ,
When I can retrieve the this information some how form the existing browser?
driver.command_executor._url
driver.session_id
(But when I understand that currently it only works with browsers started with selenium?)
When I use Selenum and start a browserwindow with it can I login to a website and the user is logged in on the webside on his browser window too(if they us the same profile)? (Or does selenium separate cookies?)
If you need additional information or have some hints please post them so I can see them.
Thank you in advance for your help
It seems that it is not possible to connect to a web browser which was opened by the user to my understanding. How ever I found two possible solutions which I am currently trying to evaluate.
Using pyautogui to access the web browser over scanned images and control it with keyboard and mouse. (It is possible to access the console with the right combinations too).
The other solution is maybe more stable. Writing an browser extension which controls the browser.
Are there any alternatives to Selenium that don't require a web driver or browser to operate? I recently moved my code over to a Google Cloud VM instance, and when I run it there are multiple errors. I've been trying to get it to work for hours but just can't (no luck with PhantomJS, Chrome and GeckoDriver - tried re-downloading browsers, editing the sources.list file e.c.t.).
The page I'm web scraping uses JavaScript to load in numbers, which I was I initially chose Selenium. Everything else works perfectly though!
You could simply use the request library.
https://requests.readthedocs.io/en/master/
https://anaconda.org/anaconda/requests
You would then need to send a GET or POST request to the server.
If you do not know how to generate a proper POST request, simply try to "record" it.
If you have chrome, got to the page you want to navigate, press F12, navigate to the "Network" section and write method:POST into the filter.
Further info here:
https://stackoverflow.com/a/39661536/11971785
At first it is a bit more confusing than selenium, but once you understand it its waaaay better in my opinion.
Also the Java values shown on the page can usually be simply read out of the java code which is returned by your request.
No web driver or anything required and a lot more stable and customizable.
How can I bypass the Google CAPTCHA using Selenium and Python?
When I try to scrape something, Google give me a CAPTCHA. Can I bypass the Google CAPTCHA with Selenium Python?
As an example, it's Google reCAPTCHA. You can see this CAPTCHA via this link: https://www.google.com/recaptcha/api2/demo
To start with using Selenium's Python clients, you should avoid solving/bypass Google CAPTCHA.
Selenium
Selenium automates browsers. Now, what you want to achieve with that power is entirely up to individuals, but primarily it is for automating web applications through browser clients for testing purposes and of coarse it is certainly not limited to that.
CAPTCHA
On the other hand, CAPTCHA (the acronym being ...Completely Automated Public Turing test to tell Computers and Humans Apart...) is a type of challenge–response test used in computing to determine if the user is human.
So, Selenium and CAPTCHA serves two completely different purposes and ideally shouldn't be used to achieve any interrelated tasks.
Having said that, reCAPTCHA can easily detect the network traffic and identify your program as a Selenium driven bot.
Generic Solution
However, there are some generic approaches to avoid getting detected while web scraping:
The first and foremost attribute a website can determine your script/program by is through your monitor size. So it is recommended not to use the conventional Viewport.
If you need to send multiple requests to a website, keep on changing the User Agent on each request. Here you can find a detailed discussion on Way to change Google Chrome user agent in Selenium?
To simulate humanlike behavior, you may require to slow down the script execution even beyond WebDriverWait and expected_conditions inducing time.sleep(secs). Here you can find a detailed discussion on How to sleep Selenium WebDriver in Python for milliseconds
This use case
However, in a couple of use cases we were able to interact with the reCAPTCHA using Selenium and you can find more details in the following discussions:
How to click on the reCAPTCHA using Selenium and Java
CSS selector for reCAPTCHA checkbok using Selenium and VBA Excel
Find the reCAPTCHA element and click on it — Python + Selenium
References
You can find a couple of related discussion in:
How can I make a Selenium script undetectable using GeckoDriver and Firefox through Python?
Is there a version of Selenium WebDriver that is not detectable?
tl; dr
How does reCAPTCHA 3 know I'm using Selenium/chromedriver?
In order to bypass the CAPTCHA when scraping Google, you have to manually solve a CAPTCHA and export the cookies Google gives you. Now, every time you open a Selenium WebDriver, make sure you add the cookies you exported. The GOOGLE_ABUSE_EXEMPTION cookie is the one you're looking for, but I would save all cookies just to be on the safe side.
If you want an additional layer of stability in your scrapes, you should export several cookies and have your script randomly select one of them each time you ping Google.
These cookies have a long expiration date so you wouldn't need to get new cookies every day.
For help on saving and loading cookies in Python and Selenium, you should check out this answer: How to save and load cookies using Python + Selenium WebDriver
Clear Browsing History, cached data, cookies and other site data
First Create an Google Account while you are in browser window opened by selenium.
Sign in to your account
wd.get("https://accounts.google.com/signin/v2/identifier?hl=en&passive=true&continue=https%3A%2F%2Fwww.google.com%2F%3Fgws_rd%3Dssl&ec=GAZAmgQ&flowName=GlifWebSignIn&flowEntry=ServiceLogin");
Thread.sleep(2000);
wd.findElement(By.name("identifier")).sendKeys("Email"+Keys.ENTER);
Thread.sleep(3000);
wd.findElement(By.name("password")).sendKeys("Password"+Keys.ENTER);
Thread.sleep(5000);
Then Open any website that uses recaptcha tick on checkmark using this code
String framename=wd.findElement(By.tagName("iframe")).getAttribute("name");
wd.switchTo().frame(framename);
wd.findElement(By.xpath("//span[#id='recaptcha-anchor']")).click();
You won't find any Puzzles or anything.
Bypass as in solve it or bypass as in never get it at all?
To solve it:
sign up with 2captcha, capmonster cloud, deathbycaptcha, etc. and follow their instructions. They will give you a token that you pass with the form.
To never get it at all:
Make sure you have good IP reputation (most important for Cloudflare).
Make sure you have a good browser fingerprint (most important for Distil) - I recommend puppeteer + the stealth plugin.
Ok, so there is a simple python script to solve captcha for you.
It basically read the audio and then use google assistant to convert it into text and paste it.
It is only workable in audio captchas which is given the most case with imahe captcha V2
https://github.com/ohyicong/recaptcha_v2_solver
Disclaimer!
I do not write the script, i just get an idea of doing this but got this brother project so, thought to help others through this.
The simple solution is suspend the program for 10 seconds or more and then when the automated browser opens solve the reCAPTCHA on your own and then the program starts after 10 seconds and execute rest of the program like clicking submit button or other things
The basic use case I want to do is opening a web page from python, after which the user will perform some work on the web page (while python waits), and then the web page will return some info (essentially redirecting to another URL), and python will be able to capture the new web page - and continue executing.
After googling, I have not been able to find a way to do it. Essentially all examples are assuming I am working in batch and I do not need to perform any operation in the background. Does anybody have a hint for me?
Use Selenium with Python.You can also choose the browser with help of web drivers. Refer more on http://selenium-python.readthedocs.io/