Develop Reference

GoogleCaptcha roadblock in website scraper

I am currently working on a scraper for aniworld.to.
My goal is it to enter the anime name and get all of the Episodes downloaded.
I have everything working except one thing...
The websites has a Watch button. That Button redirects you to https://aniworld.to/redirect/SOMETHING and that Site has a captcha which means the link is not in the html...
Is there a way to bypass this/get the link in python? Or a way to display the captcha so I can solve it?
Because the captcha only appears every lightyear.
The only thing I need from that page is the redirect link. It looks like this:
https://vidoza.net/embed-something.html
My very very wip code is here if it helps: https://github.com/wolfswolke/aniworld_scraper

Mitchdu showed me how to do it.
If anyone else needs help here is my code: https://github.com/wolfswolke/aniworld_scraper/blob/main/src/logic/captcha.py
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.ui import WebDriverWait
from threading import Thread
import os
def open_captcha_window(full_url):
working_dir = os.getcwd()
path_to_ublock = r'{}\extensions\ublock'.format(working_dir)
options = webdriver.ChromeOptions()
options.add_argument("app=" + full_url)
options.add_argument("window-size=423,705")
options.add_experimental_option('excludeSwitches', ['enable-logging'])
if os.path.exists(path_to_ublock):
options.add_argument('load-extension=' + path_to_ublock)
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
driver.get(full_url)
wait = WebDriverWait(driver, 100, 0.3)
wait.until(lambda redirect: redirect.current_url != full_url)
new_page = driver.current_url
Thread(target=threaded_driver_close, args=(driver,)).start()
return new_page
def threaded_driver_close(driver):
driver.close()

ChromeWebdriver sees website differently than I do (Python)

I'm trying to make a script that logs into my online grade book to look for any changes (new grades, etc). This is my code so far.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from webdriver_manager.chrome import ChromeDriverManager
import time
def main():
options = Options()
# options.add_argument("--headless")
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
driver.maximize_window()
# goes to the desired website
driver.get('https://portal.librus.pl/rodzina')
# searches for and clicks a button that drops down a menu in which link for login form is visible
button = driver.find_element(By.CLASS_NAME, 'btn.btn-third.btn-synergia-top.btn-navbar.dropdown-toggle')
button.click()
# searches and clicks login link
agree = driver.find_element(By.CLASS_NAME, 'zmdi.zmdi-account.dropdown-item__icon')
agree.click()
time.sleep(10)
driver.quit()
if __name__ == '__main__':
main()
And there is a problem, I cannot seem to find a way to make webdriver see what I see. What I mean is that I see the webpage like this and webdriver sees the same webpage like this also the source code is different. I've tried using undetected ChromeDriver with no success. This is my code using UC.
import undetected_chromedriver as uc
import time
from selenium.webdriver.common.by import By
def main():
driver = uc.Chrome()
driver.maximize_window()
# goes to the desired website
driver.get('https://portal.librus.pl/rodzina/home')
# searches for and clicks a button that drops down a menu in which link for login form is visible
button = driver.find_element(By.CLASS_NAME, 'btn.btn-third.btn-synergia-top.btn-navbar.dropdown-toggle')
button.click()
# searches and clicks login link
agree = driver.find_element(By.CLASS_NAME, 'zmdi.zmdi-account.dropdown-item__icon')
agree.click()
time.sleep(5)
driver.execute_script("window.print();")
if __name__ == '__main__':
main()
Has anyone had a similar problem and managed to solve it?

How to restart webdriver Selenium when script is finished

I am experimenting with Selenium and I have a code that load some URL and click on the button, so far my code looks like this:
import time
from selenium import webdriver
driver = webdriver.Chrome()
url = "https://example.org/"
driver.get(url)
button = driver.find_element_by_class_name("cool-button")
button.click()
driver.quit()
This code works as expected, but what am trying to achieve is this:
when this script finished it should start again, and again, let's say like infinite loop, so click on the button, close, open URL again, close, and so on:
I am new to Python and I try something like this:
import time
from selenium import webdriver
while True:
driver = webdriver.Chrome()
url = "https://example.org/"
driver.get(url)
button = driver.find_element_by_class_name("cool-button")
button.click()
driver.quit()
Can anybody try to help me with this?

We do the same thing here where we test when we want to attempt a "retry" of a test. One thing to note in your code above; you give no wait time for the "cool-button" to appear, become clickable or it's presence known; that will often lead to a failing result. I added support for that; as well as encompassed the entire code in an keyboard interrupted while loop.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
try:
while True:
driver = webdriver.Chrome()
url = "https://example.org/"
driver.get(url)
try:
button = WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.CLASS_NAME, "cool-button")))
button.click()
except:
print('could not find the "cool" button')
pass
driver.quit()
print("Press CTRL+C to escape!")
time.sleep(3)
except KeyboardInterrupt:
quit()
I'd also suggest this youtube as a very good resource on the topic of selenium:
https://www.youtube.com/watch?v=b5jt2bhSeXs

Multikey press not working in python selenium

I am trying to press Ctrl+s and then click enter on a webpage to save its html. But the multikey pressing functionality is not working out. I have tried way1 and way2 but both didn't work. However, If I do action.send_keys('s') by without executing action.key_down(Keys.CONTROL) before it, it works fine. This is my full code:
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.keys import Keys
import time
driver = webdriver.Chrome(executable_path="chromedriver.exe")
# get method to launch the URL
driver.get("https://www.google.ca/")
time.sleep(3)
action = ActionChains(driver)
# click google search bar (to make sure driver is working)
driver.find_element_by_name("q").click()
print("passed1")
time.sleep(3)
# way 1
action.key_down(Keys.CONTROL).send_keys('s').key_up(Keys.CONTROL).perform()
print("passed2")
# way 2
action.key_down(Keys.CONTROL)
action.send_keys('s')
action.key_up(Keys.CONTROL)
action.perform()
print("passed2")
time.sleep(100)
driver.close()
Can someone please explain to me what is he issue? I've been trying to figure it out for an hour now.

After looking around, I found that it is not advisable to interact with the browser for this purpose. It is better to get the page_source instead.
from selenium import webdriver
driver = webdriver.Chrome(executable_path=r"C:\Program Files (x86)\Selenium\chromedriver.exe")
driver.get("http://www.example.com")
with open('page.html', 'w', encoding='utf-8') as f:
f.write(driver.page_source)

Python script goes to website but doesn't click button intended to

As a test, I am trying to create a script that goes to my website and clicks on the learn more button, but am having trouble actually automatically clicking the button.
I've tried everything that I've found on stack overflow but nothing has worked.
from selenium import webdriver
import webbrowser
import time
url = 'https://www.mwstan.com'
driver = webbrowser.open_new_tab(url)
element = driver.find_element_by_id('learnmore')
element.click()

You are going to need to install a binary for whatever driver you are going to use
import os
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--window-size=1920x1080")
chrome_driver = os.getcwd() + "/chromedriver"
def get_url_example(url):
driver = webdriver.Chrome(chrome_options=chrome_options, executable_path=chrome_driver)
driver.get(url)
button = driver.find_element_by_id("learnmore")
button.click()
# you can access the page source here using driver.page_source
if __name__ == '__main__':
get_url_page_source("https://www.mwstan.com")
This code works for me and hits your button.
This is using chrome webdriver but you can use another webdriver. JUst makesure you move the driver and access the path correctly like in line
chrome_driver = os.getcwd() + "/chromedriver"