Develop Reference

How to comment on blogspot within an iframe with Selenium(Python)

I would like to comment on a blogspot with selenium, chromedriver and python. I tried many methods but failed. How can I run my code below?
driver.get(url)
iframe = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.NAME, 'comment-editor')))
driver.switch_to.frame(iframe)
element=WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.NAME, 'commentBody')))
actionChains = ActionChains(driver)
actionChains.move_to_element(element).click().perform()
actionChains.move_to_element(element).send_keys(text).perform()
I'm getting an error on this line:
element=WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.NAME, 'commentBody')))
Please help to comment with selenium.
Edit..
test url : https://lf2011b8308.blogspot.com/2011/12/macronutrients-carbohydrates-proteins.html
Error stacktrace:
Traceback (most recent call last):
File "C:/Users/Hotto/PycharmProjects/blogspot/chromes.py", line 51, in <module>
element=WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.NAME, 'commentBody')))
File "C:\Users\Hotto\AppData\Local\Programs\Python\Python38\lib\site-packages\selenium\webdriver\support\wait.py", line 89, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
Stacktrace:
Backtrace:
Ordinal0 [0x00405FD3+2187219]
Ordinal0 [0x0039E6D1+1763025]
Ordinal0 [0x002B3E78+802424]
Ordinal0 [0x002E1C10+990224]
Ordinal0 [0x002E1EAB+990891]
Ordinal0 [0x0030EC92+1174674]
Ordinal0 [0x002FCBD4+1100756]
Ordinal0 [0x0030CFC2+1167298]
Ordinal0 [0x002FC9A6+1100198]
Ordinal0 [0x002D6F80+946048]
Ordinal0 [0x002D7E76+949878]
GetHandleVerifier [0x006A90C2+2721218]
GetHandleVerifier [0x0069AAF0+2662384]
GetHandleVerifier [0x0049137A+526458]
GetHandleVerifier [0x00490416+522518]
Ordinal0 [0x003A4EAB+1789611]
Ordinal0 [0x003A97A8+1808296]
Ordinal0 [0x003A9895+1808533]
Ordinal0 [0x003B26C1+1844929]
BaseThreadInitThunk [0x7697343D+18]
RtlInitializeExceptionChain [0x77729812+99]
RtlInitializeExceptionChain [0x777297E5+54]

To send a character sequence to the commentBody field as the elements are within an <iframe> so you have to:
Induce WebDriverWait for the desired frame to be available and switch to it.
Induce WebDriverWait for the desired element to be clickable.
You can use the following locator strategies:
driver.get('https://clearing.apcs.at/emwebapcsem/startApp.do')
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe[name='comment-editor']")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.NAME, "commentBody"))).send_keys("Akif")
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Try to click the "accept-cookies" button Error: "no such element: Unable to locate element"

i want to click the "accept Cookies" Button, but everytime i get the following error "Message: no such element: Unable to locate element:".
I think this is because this is a popup-window. I try to click the Button via Xpath.
How can i handle this?
Picture of the Website
My Code:
from selenium import webdriver
from selenium.webdriver.common.by import By
driver = webdriver.Chrome('./chromedriver')
link = 'https://www.ticket-onlineshop.com/ols/fckoeln/de/heim/channel/shop/index/index/event/424413'
#Open Website
driver.get(link)
#Timeout
time.sleep(5)
#Cookies
driver.find_element(By.XPATH,'//*[#id="uc-center-container"]/div[2]/div/div/div/div[2]/button[2]').click()
Error:
driver = webdriver.Chrome('./chromedriver')
Traceback (most recent call last):
File "C:\Users\Manu\PycharmProjects\SampleSale\FC.py", line 12, in <module>
driver.find_element(By.XPATH, '//*[#id="uc-center-container"]/div[2]/div/div/div/div[2]/button[2]').click()
File "C:\Users\Manu\PycharmProjects\SampleSale\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 1248, in find_element
return self.execute(Command.FIND_ELEMENT, {
File "C:\Users\Manu\PycharmProjects\SampleSale\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 425, in execute
self.error_handler.check_response(response)
File "C:\Users\Manu\PycharmProjects\SampleSale\venv\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 247, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[#id="uc-center-container"]/div[2]/div/div/div/div[2]/button[2]"}
(Session info: chrome=101.0.4951.41)
Stacktrace:
Backtrace:
Ordinal0 [0x0067B8F3+2406643]
Ordinal0 [0x0060AF31+1945393]
Ordinal0 [0x004FC748+837448]
Ordinal0 [0x005292E0+1020640]
Ordinal0 [0x0052957B+1021307]
Ordinal0 [0x00556372+1205106]
Ordinal0 [0x005442C4+1131204]
Ordinal0 [0x00554682+1197698]
Ordinal0 [0x00544096+1130646]
Ordinal0 [0x0051E636+976438]
Ordinal0 [0x0051F546+980294]
GetHandleVerifier [0x008E9612+2498066]
GetHandleVerifier [0x008DC920+2445600]
GetHandleVerifier [0x00714F2A+579370]
GetHandleVerifier [0x00713D36+574774]
Ordinal0 [0x00611C0B+1973259]
Ordinal0 [0x00616688+1992328]
Ordinal0 [0x00616775+1992565]
Ordinal0 [0x0061F8D1+2029777]
BaseThreadInitThunk [0x761D6739+25]
RtlGetFullPathName_UEx [0x77C78E7F+1215]
RtlGetFullPathName_UEx [0x77C78E4D+1165]
Process finished with exit code 1

The element is present in the shadow root, to click the button we need to use querySelector() ..... with the below code i can able to click the button
driver.execute_script('''return document.querySelector('div#usercentrics-root').shadowRoot.querySelector('button[data-testid="uc-accept-all-button"]')''').click()
it is already explained in one of the StackOverflow question Link

Try:
driver.find_element(By.XPATH,”//*[#id=‘uc-center-container’]/div[2]/div/div/div/div[2]/button[2]”).click()

When i try to scrape this website with selenium and python

When i try and scrape the website it just throws some errors
I think it may have something to do with my webdriver but idk
I am trying to get this data so i can put it in a spreadsheet to get some cool staticstics
from selenium import webdriver
url = 'https://prosettings.net/cs-go-pro-settings-gear-list/'
driver = webdriver.Chrome(executable_path="C:\WebDrivers\chromedriver.exe")
driver.get(url)
names = driver.find_elements_by_class_name(" column-player")
for name in names:
title = name.find_element_by_xpath('.//a').text
print(name)
Here are the error i get in terminal
d:\downloads\PythonScraping\Test.py:5: DeprecationWarning: executable_path has been deprecated, please pass in a Service object
driver = webdriver.Chrome(executable_path="C:\WebDrivers\chromedriver.exe")
DevTools listening on ws://127.0.0.1:53131/devtools/browser/73ca0453-352e-47a0-a98a-fb539150d6f9
d:\downloads\PythonScraping\Test.py:8: DeprecationWarning: find_elements_by_* commands are deprecated. Please use find_elements() instead
names = driver.find_elements_by_class_name(" column-player")
Traceback (most recent call last):
File "d:\downloads\PythonScraping\Test.py", line 8, in <module>
names = driver.find_elements_by_class_name(" column-player")
File "C:\Users\terk0\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\selenium\webdriver\remote\webdriver.py", line 783, in
find_elements_by_class_name
return self.find_elements(by=By.CLASS_NAME, value=name)
File "C:\Users\terk0\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\selenium\webdriver\remote\webdriver.py", line 1279, in find_elements
return self.execute(Command.FIND_ELEMENTS, {
File "C:\Users\terk0\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\selenium\webdriver\remote\webdriver.py", line 424, in
execute
self.error_handler.check_response(response)
File "C:\Users\terk0\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\selenium\webdriver\remote\errorhandler.py", line 247,
in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.InvalidSelectorException: Message: invalid selector: An invalid or illegal selector was specified
(Session info: chrome=98.0.4758.102)
Stacktrace:
Backtrace:
Ordinal0 [0x00EF69A3+2582947]
Ordinal0 [0x00E8A6D1+2139857]
Ordinal0 [0x00D83A98+1063576]
Ordinal0 [0x00D862B7+1073847]
Ordinal0 [0x00D8617E+1073534]
Ordinal0 [0x00D863F0+1074160]
Ordinal0 [0x00DAFCB2+1244338]
Ordinal0 [0x00DB013B+1245499]
Ordinal0 [0x00DD9F8C+1417100]
Ordinal0 [0x00DC8594+1344916]
Ordinal0 [0x00DD834A+1409866]
Ordinal0 [0x00DC8366+1344358]
Ordinal0 [0x00DA5176+1200502]
Ordinal0 [0x00DA6066+1204326]
GetHandleVerifier [0x0109BE02+1675858]
GetHandleVerifier [0x0115036C+2414524]
GetHandleVerifier [0x00F8BB01+560977]
GetHandleVerifier [0x00F8A8D3+556323]
Ordinal0 [0x00E9020E+2163214]
Ordinal0 [0x00E95078+2183288]
Ordinal0 [0x00E951C0+2183616]
Ordinal0 [0x00E9EE1C+2223644]
BaseThreadInitThunk [0x7586FA29+25]
RtlGetAppContainerNamedObjectPath [0x77957A9E+286]
RtlGetAppContainerNamedObjectPath [0x77957A6E+238]

There are 2 problems here:
Instead of
names = driver.find_elements_by_class_name(" column-player")
it should be
names = driver.find_elements_by_class_name("column-player")
(I know, there are spaces before column-player class name there, but you still should not put them inside the locator)
2) You should add a delay to access these elements only after the page have been completely loaded.
This should work better:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
url = 'https://prosettings.net/cs-go-pro-settings-gear-list/'
driver = webdriver.Chrome(executable_path="C:\WebDrivers\chromedriver.exe")
wait = WebDriverWait(driver, 20)
driver.get(url)
#wait for at least 1 element visibility
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".column-player")))
#short additional wait for all the other elements to complete loading
time.sleep(0.5)
names = driver.find_elements_by_class_name("column-player")
for name in names:
title = name.find_element_by_xpath('.//a').text
print(name)

Why getting selenium.common.exceptions.NoSuchElementException in selenium python Script?

I am new to selenium and written a script to login, and when I am running the script getting the following error:-
Error:-
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="SignIn Form_email"]"}
Script:-
url = "https://someurl.com"
driver = webdriver.Chrome(executable_path="C:\CustomDrivers\chromedriver_win32\chromedriver")
driver.get(url)
print(driver.title)
user_id = driver.find_element_by_id("SignIn Form_email")
user_id.send_keys("admin#web.com")
user_password = driver.find_element_by_id("SignIn Form_password")
user_password.send_keys("web$$DD")
driver.find_element_by_xpath("//*[#id='SignIn Form']/div[3]/div/div/div/button").click()
print(driver.title)
time.sleep(5)
Full Error Trace:-
C:\Users\Lenovo\Desktop\convin\automation_test\logintest.py:6: DeprecationWarning: executable_path has been deprecated, please pass in a Service object
driver = webdriver.Chrome(executable_path="C:\CustomDrivers\chromedriver_win32\chromedriver")
Convin
C:\Users\Lenovo\Desktop\convin\automation_test\logintest.py:9: DeprecationWarning: find_element_by_* commands are deprecated. Please use find_element() instead
user_id = driver.find_element_by_id("SignIn Form_email")
Traceback (most recent call last):
File "C:\Users\Lenovo\Desktop\convin\automation_test\logintest.py", line 9, in <module>
user_id = driver.find_element_by_id("SignIn Form_email")
File "C:\Users\Lenovo\Desktop\convin\automation_test\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 466, in find_element_by_id
return self.find_element(by=By.ID, value=id_)
File "C:\Users\Lenovo\Desktop\convin\automation_test\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 1238, in find_element
return self.execute(Command.FIND_ELEMENT, {
File "C:\Users\Lenovo\Desktop\convin\automation_test\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 418, in execute
self.error_handler.check_response(response)
File "C:\Users\Lenovo\Desktop\convin\automation_test\venv\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 243, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="SignIn Form_email"]"}
(Session info: chrome=95.0.4638.69)
Stacktrace:
Backtrace:
Ordinal0 [0x00780C43+2493507]
Ordinal0 [0x0071A4B1+2073777]
Ordinal0 [0x00622608+1058312]
Ordinal0 [0x0064CAA4+1231524]
Ordinal0 [0x00676C62+1404002]
Ordinal0 [0x0066597A+1333626]
Ordinal0 [0x00675038+1396792]
Ordinal0 [0x0066580B+1333259]
Ordinal0 [0x00642314+1188628]
Ordinal0 [0x0064316F+1192303]
GetHandleVerifier [0x00907BF6+1548950]
GetHandleVerifier [0x009B461C+2256060]
GetHandleVerifier [0x0080C13B+518107]
GetHandleVerifier [0x0080B1E0+514176]
Ordinal0 [0x0071F53D+2094397]
Ordinal0 [0x00723418+2110488]
Ordinal0 [0x00723552+2110802]
Ordinal0 [0x0072CE81+2150017]
BaseThreadInitThunk [0x7602FA29+25]
RtlGetAppContainerNamedObjectPath [0x77467A9E+286]
RtlGetAppContainerNamedObjectPath [0x77467A6E+238]
Thanks in advance
Hope to here from you soon.

Where is the HTML? Without it, we can't really help you.
By the way, your user_id and user_password look like a class name, are you sure about find_element_by_id method?
Shouldn't you use find_element_by_class instead?

By seeing this error
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="SignIn Form_email"]"}
You will have to make sure
This [id="SignIn Form_email"] is not under an iframe.
This [id="SignIn Form_email"] is not under a shadow-root.
You should not be on the new tab/windows launched by selenium.
You may be trying to interact with the element and it has not rendedered completely.
Also,
Please check in the dev tools (Google chrome) if we have unique entry in HTML DOM or not.
xpath that you should check :
[id="SignIn Form_email"]
Steps to check:
Press F12 in Chrome -> go to element section -> do a CTRL + F -> then paste the xpath and see, if your desired element is getting highlighted with 1/1 matching node.
Now, if it is iframe, please switch to iframe first.
wait = WebDriverWait(driver, 30)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.XPATH, "")))
if it is element not rendered issue, Please induce web driver wait like
wait = WebDriverWait(driver, 30)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "")))
Imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Implicit delay with Selenium

I am practicing with selenium. I tried this page scraping with beautifulsoup, but as the page gets some time to load, I tried it with selenium with adding a delay time. I have tried drive.delay and timeout too. But nothing works fine with me. I get same alike error with all the tries, and I have attached my latest try and the error I get with it.
This is my ty:
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
url = 'https://www.checkatrade.com/search?page=1&categoryId=1164&location=London'
driver =webdriver.Chrome(executable_path='D:\Web_Scraping\chromedriver_win32\chromedriver.exe')
driver.get(url)
driver.implicitly_wait(20)
items = driver.find_element_by_class_name('sc-8keyp7-0 kbPUFb sc-1a2v1qv-0 gwBNSh sc-3jeiz1-1 bfuahU')
for item in items:
company = item.find_element_by_xpath('//*[#id="1052063"]/div/div[2]/div[2]/a/div[1]/h2').text
print(company)
But it throws me this error, which I can not figure out.
DeprecationWarning: executable_path has been deprecated, please pass in a Service object
driver =webdriver.Chrome(executable_path='D:\Web_Scraping\chromedriver_win32\chromedriver.exe')
DevTools listening on ws://127.0.0.1:1090/devtools/browser/9ed2753e-5d6e-4382-a9ac-a0ed2fbb8379
d:/Web_Scraping/seleniumtest.py:21: DeprecationWarning: find_element_by_* commands are deprecated. Please use find_element() instead
items = driver.find_element_by_class_name('sc-8keyp7-0 kbPUFb sc-1a2v1qv-0 gwBNSh sc-3jeiz1-1 bfuahU')
[5536:25384:1014/115806.197:ERROR:chrome_browser_main_extra_parts_metrics.cc(228)] crbug.com/1216328: Checking Bluetooth availability started. Please report if there is no report that this ends.
[5536:25384:1014/115806.198:ERROR:chrome_browser_main_extra_parts_metrics.cc(231)] crbug.com/1216328: Checking Bluetooth availability ended.
[5536:25384:1014/115806.199:ERROR:chrome_browser_main_extra_parts_metrics.cc(234)] crbug.com/1216328: Checking default browser status started. Please report if there is no report that this ends.
[5536:23016:1014/115806.200:ERROR:device_event_log_impl.cc(214)] [11:58:06.200] USB: usb_device_handle_win.cc:1048 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)
[5536:25384:1014/115806.733:ERROR:chrome_browser_main_extra_parts_metrics.cc(238)] crbug.com/1216328: Checking default browser status ended.
Traceback (most recent call last):
File "d:/Web_Scraping/seleniumtest.py", line 21, in <module>
items = driver.find_element_by_class_name('sc-8keyp7-0 kbPUFb sc-1a2v1qv-0 gwBNSh sc-3jeiz1-1 bfuahU')
File "C:\Python38\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 754, in find_element_by_class_name
return self.find_element(by=By.CLASS_NAME, value=name)
File "C:\Python38\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 1238, in find_element
return self.execute(Command.FIND_ELEMENT, {
File "C:\Python38\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 418, in execute
self.error_handler.check_response(response)
File "C:\Python38\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 243, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".sc-8keyp7-0 kbPUFb sc-1a2v1qv-0 gwBNSh sc-3jeiz1-1 bfuahU"}
(Session info: chrome=94.0.4606.81)
Stacktrace:
Backtrace:
Ordinal0 [0x0107BDE3+2473443]
Ordinal0 [0x01016661+2057825]
Ordinal0 [0x00F22438+1057848]
Ordinal0 [0x00F4C7DB+1230811]
Ordinal0 [0x00F76712+1402642]
Ordinal0 [0x00F6571A+1333018]
Ordinal0 [0x00F74ABF+1395391]
Ordinal0 [0x00F655AB+1332651]
Ordinal0 [0x00F42104+1188100]
Ordinal0 [0x00F42F59+1191769]
GetHandleVerifier [0x01202266+1549718]
GetHandleVerifier [0x012AD4A7+2250711]
GetHandleVerifier [0x0110718B+521403]
GetHandleVerifier [0x01106229+517465]
Ordinal0 [0x0101B79D+2078621]
Ordinal0 [0x0101FB58+2095960]
Ordinal0 [0x0101FC92+2096274]
Ordinal0 [0x01029541+2135361]
BaseThreadInitThunk [0x764DFA29+25]
RtlGetAppContainerNamedObjectPath [0x77A77A9E+286]
RtlGetAppContainerNamedObjectPath [0x77A77A6E+238]
What causes this error?

You are using a wrong locator.
find_element_by_class_name method is intended to receive a SINGLE class name while you are passing a multiple class names value to it "find_element_by_class_name".
To pass multiple class names you should use css selector or XPath.
So instead of
items = driver.find_element_by_class_name('sc-8keyp7-0 kbPUFb sc-1a2v1qv-0 gwBNSh sc-3jeiz1-1 bfuahU')
It could be
items = driver.find_element_by_css_selector('.sc-8keyp7-0.kbPUFb.sc-1a2v1qv-0.gwBNSh.sc-3jeiz1-1.bfuahU')
Or
items = driver.find_element_by_xpath("//div[#class='sc-8keyp7-0 kbPUFb sc-1a2v1qv-0 gwBNSh sc-3jeiz1-1 bfuahU']")