I am having problems with my selenium script.
I want to webscrape a website that is running on javascript.
I have been gathering a lot of information on the internet but i can't find the solution.
picture of the html code
picture of my code
In this post i have also submitted a printscreen of the HTML code.
Basically: i want click on the accept button when i load the website but i can't figure how to do this.
searching on different websites for the solution.
Can you guys help me with my script, i have been trying and testing a lot but i can't figure it out.
thank you.
code:
from config import keys
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import time
def order(k):
chrome_path =
r"C:\Users\ltewo\PycharmProjects\livebetting\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.get(keys['url'])
xpath_button_accept = "//div[#class='cookieButtons']//a[#class='button
accept']"
button_accept = driver.find_element_by_xpath(xpath_button_accept)
xpath_button_accept.click()
if __name__ == '__main__':
order(keys)
You should wait for that element to load on the page, so use:
WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.XPATH, 'XpathOfMyElement')))
I noticed that the cookie panel was inside an iframe and was not directly accessible, I have updated the below, please check it's working now.
Updated your code:
from config import keys
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import time
def order(keys):
chrome_path = r"C:\Users\ltewo\PycharmProjects\livebetting\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.get(keys['url'])
driver.switch_to.frame('r42CookieBar')
button_accept = driver.find_element_by_class_name('accept')
button_accept.click()
if __name__ == '__main__':
order(keys)
Try this:
#update
driver.implicity_wait(10)
xpath_button_accept = "//div[#class='cookieButtons']//a[#class='button accept']"
button_accept = your_browser.find_element_by_xpath(xpath_button_accept)
button_accept.click()
Related
How can I take screenshot of only relevant content of any webpage using Selenium and Python?
I want to take the screenshot of the marked content (specifications) in this photo instead of whole page
Example webpage link
Currently I'm taking screenshot of the whole page. Also I want avoid referencing any class or id while taking the screenshot. Please let me know if I can achieve this (if yes, HOW?) or have to change my requirements. If there is any workaround such as cropping the relevant content, please do share too. Thanks.
Chrome appears to be a bit temperamental when screenshotting - only takes what's visible on screen, so I would advise Firefox/geckodriver for these kind of jobs. The following will take a full screenshot of that element:
from selenium import webdriver
from selenium.webdriver.firefox.service import Service
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.firefox.options import Options as Firefox_Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support import expected_conditions as EC
import time as t
firefox_options = Firefox_Options()
firefox_options.add_argument("--width=1280")
firefox_options.add_argument("--height=720")
# firefox_options.headless = True
driverService = Service('chromedriver/geckodriver')
browser = webdriver.Firefox(service=driverService, options=firefox_options)
actions = ActionChains(browser)
url = 'https://www.startech.com.bd/benq-gw2480-fhd-monitor'
browser.get(url)
elem = WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.XPATH, "//section[#id='specification']")))
elem.screenshot('fullspec.png')
print('screenshotted specs')
I am new to python and selenium. Trying to automate download of files from the dropdown menu on the below link for the first time. The code gets stuck after page loads i.e the drop down does not work and gives me an error. Sorry if my code looks clunky. Any help is appreciated.
Thanks
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.select import Select
import time
s=Service("C:\\python\\chromedriver.exe")
driver = webdriver.Chrome(service=s)
driver.maximize_window()
driver.get('http://www.amfiindia.com/research-information/aum-data/classified-average-aum')
#monthname=driver.find_element(By.XPATH,"//*[#id='AaumDate']")
monthdd=Select(monthname)
monthdd.select_by_value('01-Jan-22')
time.sleep(3)
typename=driver.find_element(By.XPATH,"//*[#id='AumType']")
typedd=Select(typename)
typedd.select_by_visible_text('Scheme category wise')
time.sleep(3)
mfname=driver.find_element(By.XPATH,"//*[#id='AumMFName']")
mfdd=Select(mfname)
mfdd.select_by_visible_text('All')
time.sleep(3)
driver.find_element("class name","sprite-inter go-btn").click()
wait=WebDriverWait(driver,20)
driver.get('http://www.amfiindia.com/research-information/aum-data/classified-average-aum')
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,"#divAumPeriod > span > a > span.ui-button-text"))).click()
wait.until(EC.element_to_be_clickable((By.XPATH,"//*[#class='ui-menu-item']/a[.='January - 2022']"))).click()
Most of these tags open like so and have another list with them that's not using the select tag.
Imports:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
I am using Python selenium chromedriver and I want to add Capmonster to my code. Basically if my Code gets the URL, a ReCaptcha appears and I want to get it solved.
My Code
from threading import Thread
from selenium.webdriver.chrome.options import Options
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
from selenium.webdriver.support.ui import Select
from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import TimeoutException
import os
chromedriver = 'C:\\Users\\yvesb\\OneDrive\\Desktop\\chromedriver\\chromedriver.exe'
options = webdriver.ChromeOptions()
options.add_argument('headless')
options.add_argument('window-size=1200x600') # optional
driver = webdriver.Chrome(executable_path=chromedriver, chrome_options=options)
def test(driver, url):
driver.get(url)
try:
driver.save_screenshot('C:\\Users\\yvesb\\Downloads\\headless_chrome_test7.png')
finally:
print("screenshot done")
url = "https://www.snipes.com/login"
Thread(target=test, args=(driver, url)).start()
And when it gets the URL a ReCaptcha appears, which I want to get solved with my Capmonster Key.
I suggest you to try this library I've developed some time ago. If you have a set of labelled captchas that service would fit you. Take a look: https://github.com/punkerpunker/captcha_solver
In README there is a section "Train model on external data" that you might be interested in.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains
import time
browser=webdriver.Chrome('C:/Users/Dell/Downloads/chromedriver')
browser.get('https://www.screener.in/')
sbox = browser.switch_to.active_element
sbox.send_keys('Infosys Ltd')
sbox.send_keys(Keys.RETURN)
The enter key is not working. I have tried using .submit() too but still isn't working. Please let me know if there is any other way to get it.
Try using Keys.ENTER instead of Keys.RETURN
url = "https://www.foodpanda.pk/restaurants/new?lat=24.9414896&lng=67.1676002&vertical=restaurants"
browser = webdriver.Chrome()
browser.get('https://www.screener.in/')
sbox = browser.switch_to.active_element
sbox.send_keys('Infosys Ltd')
WebDriverWait(browser, 10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, '[class="dropdown-content visible"]')))
sbox.send_keys(Keys.ENTER)
wait for the dropdown to be visible before sending the enter
imports required:
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
switch_to may be unstable depending on browsers. Also, make sure sbox is visible before you interact with it. Try:
sbox['value'].send_keys('Infosys Ltd')
sbox['value'].send_keys(Keys.RETURN)
I had this issue on 2 different systems, on first PC it went away after a reboot, on the other it had not gone away
Below is a portion of the code in question:
I masked some things on purpose
The edge browser is stuck on openign page with data;, shown in the address bar. It does not go beyond this.
I checked that Edge and Edgedriver are the same exact version, if it matters.
I cannot use Chrome or any other drivers for this project
Why doesn't selenium advance past the opening of the program?
I also have an error in CMD when it launches: [9704:11992:0305/080544.278:ERROR:storage_reserve.cc(164)] Failed to open file to mark as storage reserve: C:\Users**\AppData\Local\Temp\scoped_dir3468_1483298328\Default\Code Cache\js on selenium
How do I fix this?
# importing required package of webdriver
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from time import sleep
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.opera.options import Options
from selenium.webdriver.support.wait import WebDriverWait
import schedule
import time
from random import randint
def job():
while True:
# Instantiate the webdriver with the executable location of MS Edge
browser = webdriver.Edge(r"C:\Users\*****\Desktop\msedgedriver.exe")
sleep(2)
browser.maximize_window()
sleep(2)
browser.get('https://********/) #masked the name of website on purpose
try:
# Get the text box to insert Email using selector ID
myElem_1 = WebDriverWait(browser, 10).until(EC.presence_of_element_located((By.ID, 'anti-myhub')))
sleep(3)
# Entering the email address
myElem_1.click()