I wanted to write a bot, which will automatically find canceled driving tests and rescheduled your test date to this date, in order to decrease the time of waiting.
I decided to use selenium on python to perform this. However, despite all my tries, I couldn’t get the browser to stop detecting the bot, so there is always CAPTHCA for this bot.
Can someone suggest a potential solution to this problem?
Selenium-powered browser windows are by default (required by law in some regions) marked automated, and websites that have bots disabled on their platforms can easily detect it and prevent any kind of automation. There are some workarounds but these workarounds are not very reliable as these get patched quite often.
You can try these resources if they still work:
Webpage Is Detecting Selenium Webdriver with Chromedriver as a bot
http://php8legs.com/en/php-web-scraper/51-how-to-avoid-selenium-webdriver-from-being-detected-as-bot-or-web-spider
Can a website detect when you are using Selenium with chromedriver?
Related
I am currently writing a Spotify Player in Gnome/GTK design, using Python/PyGObject/Selenium.
I am using the WebPlaybackSDK to play Spotify in a browser window controlled by Selenium, and a UI that is shown to the user.
When Spotify is playing, naturally it reports that the firefox (or whatever browser I am using with Selenium) is playing audio.
However I want Linux to show that something like "SpotifyGTK" or something is playing audio, with a different icon as well.
And I am unsure how to go about that. Can I maybe combine the processes alltogether, so that my main process shows up? Or can I at least change the name of the browser process?
I am writing a test bot for an app. Generally I need it to be in headless mode. But at some arbitrary buggy occasions it is a comfort to see the browser window and decide if what a wrong issue is happening there. So I need a way to connect a browser window to an active headless session. In an other word to literally convert it to a head-full one.
P.S. Every thing is in Python. Both Firefox and Chrome solutions(or any sort of guiding) are welcomed.
I am attempting to change the proxy settings on a browser without restarting it for the settings to take effect. Is this possible, and if so, how?
EDIT: Figured out a solution that works for me, however, it's not doing what's in the question and still requires a lot more time to complete requests. (Stopping the browser whenever it encounters Cloudflare, swapping proxies, moving on, etc. Stuff I'd quite like to avoid, as this task is extremely time-sensitive and consistency reliant.)
from selenium import webdriver
option = webdriver.ChromeOptions()
option.add_argument('--proxy-server=socks5://ip:ports')
driver = Chrome(options=option)
Login_page(driver).login()
I think no need to restart
I have a Python program that uses APScheduler and Selenium to automate webscraping. Basically it scrapes a particular website once every hour, and then schedules certain more detailed scrapes to happen at intervals.
The issue is that while I want to start the scraper, and then be free to use my computer for other work, Selenium will then automatically focus on the opened chrome tabs whenever a new scrape is started by APScheduler. Because of this, I am trying to find a way to have the new chrome windows open - but I don't have to focus on them. I've already tried headless and phantomJS, but the site is dynamically generated so these don't really work.
My current solution is to open the new window, minimise it, and then immediately shift back to the old window. To do this I want to perform a keyboard shortcut using ActionChains. I currently have this test code:
driver = webdriver.Chrome(ChromeDriverManager().install())
ActionChains(driver).key_down(Keys.CONTROL)
ActionChains(driver).send_keys(Keys.RIGHT)
ActionChains(driver).key_up(Keys.CONTROL)
Currently this code does nothing however. Is there anyway to fix this? I am using Jupyter Notebook and I am on a mac for reference.
NOTE: this code is being run apart from the main program, I am just using it to test if the script will work.
Try this out. You never .perform() it.
ActionChains(driver).key_down(Keys.CONTROL).send_keys(Keys.RIGHT).key_up(Keys.CONTROL).perform()
is there a way i could control(move mouse, send input) without using selenium and NO browsers necessary. I need it to be fast, i saw some programs like have proxies and they basically control them and move mouse to specific locations, and go through google like that, is there a way i could do this? I need to solve a 1 click captcha ( it's just a click, nothing special ) but selenium is way too slow and i need something faster.
You could try a hacky way of automation with PyAutoGUI, a tool which automates mouse movement and keyboard inputs. I used this in combination with a regular Chrome browser to scrape a few pages. Although I suspect this will not be faster than Selenium. Alternatively you could try requests-html.