Develop Reference

How to locate the email address element with a Facebook page using Selenium

So I am trying to first pull the body where the email address lies on this specific facebook page and then pull the email address plus other attributes within that body. However, I have tried every way to locate these elements using Chropath, and SelectorHub but nothing has worked. Any idea why I am unable to use these tools?
Code trials:
from ast import Return
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
driver = webdriver.Chrome("C:/Users/Carson/Desktop/chromedriver.exe")
driver.get("https://www.facebook.com/TheVillageAtGracyFarms")
driver.maximize_window()
try:
WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.CSS_SELECTOR, "body._6s5d._71pn.system-fonts--body.segoe:nth-child(2) div.rq0escxv.l9j0dhe7.du4w35lb div.rq0escxv.l9j0dhe7.du4w35lb:nth-child(6) div.du4w35lb.l9j0dhe7.cbu4d94t.j83agx80 div.j83agx80.cbu4d94t.l9j0dhe7 div.j83agx80.cbu4d94t.l9j0dhe7.jgljxmt5.be9z9djy.qfz8c153 div.j83agx80.cbu4d94t.d6urw2fd.dp1hu0rb.l9j0dhe7.du4w35lb:nth-child(1) div.j83agx80.cbu4d94t.dp1hu0rb:nth-child(1) div.j83agx80.cbu4d94t.buofh1pr.dp1hu0rb.hpfvmrgz div.l9j0dhe7.dp1hu0rb.cbu4d94t.j83agx80 div.bp9cbjyn.j83agx80.cbu4d94t.d2edcug0:nth-child(4) div.rq0escxv.d2edcug0.ecyo15nh.k387qaup.r24q5c3a.hv4rvrfc.dati1w0a.tr9rh885:nth-child(2) div.rq0escxv.l9j0dhe7.du4w35lb.pfnyh3mw.gs1a9yip.j83agx80.btwxx1t3.lhclo0ds.taijpn5t.sv5sfqaa.o22cckgh.obtkqiv7.fop5sh7t div.rq0escxv.l9j0dhe7.du4w35lb.hpfvmrgz.g5gj957u.aov4n071.oi9244e8.bi6gxh9e.h676nmdw.aghb5jc5.o387gat7.g1e6inuh.fhuww2h9.rek2kq2y:nth-child(1) div.lpgh02oy:nth-child(2) div.sjgh65i0 div.j83agx80.l9j0dhe7.k4urcfbm div.rq0escxv.l9j0dhe7.du4w35lb.hybvsw6c.io0zqebd.m5lcvass.fbipl8qg.nwvqtn77.k4urcfbm.ni8dbmo4.stjgntxs.sbcfpzgs div.sej5wr8e div.rq0escxv.l9j0dhe7.du4w35lb.j83agx80.pfnyh3mw.i1fnvgqd.gs1a9yip.owycx6da.btwxx1t3.hv4rvrfc.dati1w0a.discj3wi.b5q2rw42.lq239pai.mysgfdmx.hddg9phg:nth-child(2) > div.rq0escxv.l9j0dhe7.du4w35lb.j83agx80.cbu4d94t.d2edcug0.hpfvmrgz.rj1gh0hx.buofh1pr.g5gj957u.p8fzw8mz.pcp91wgn.iuny7tx3.ipjc6fyt")))
print("Dub")
except:
print("Failed")
driver.quit()

The Email Address field within the webpage contains dynamic elements.
To locate the Email Address field instead of presence_of_element_located() you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following locator strategies:
Using XPATH and the text #:
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//span[contains(., '#')]"))).text)
Using XPATH and the texts # & com:
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//span[contains(., '#') and contains(., 'com')]"))).text)
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

How to locate the first channel points button/icon on twitch.tv/esl_csgo using Selenium and Python

On twitch.tv/esl_csgo I want to click on the channel points button/icon but it keeps giving me the Error
Message: no such element: Unable to locate element
I have searched and tried various methods of finding the element for over 4 hours and I have not found a way to click on the element
This is my code but it can not click on the button I want to, help would really be appreciated.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
import time
options = Options()
options.add_argument("--user-data-dir=C:\\Users\\Me\\Desktop\\UserData")
options.page_load_strategy = 'normal'
driver = webdriver.Chrome(options=options)
driver.get("https://twitch.tv/esl_csgo")
time.sleep(10)
element = driver.find_element_by_xpath('//*[#id="c7037441c8fd58e7e0ac6326babcf03d"]/div/div[1]/div/div/div/div/div/section/div/div[5]/div[2]/div[2]/div[1]/div/div/div/div[1]/div[2]/button/div/div/div/div[2]/span')
element.click()

To click on the first channel points button/icon you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div[class^='InjectLayout-sc'] > div span[data-test-selector]"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[starts-with(#class, 'InjectLayout-sc')]/div//span[#data-test-selector]"))).click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Getting id or xpath of parts of a website

I want to use selenium to login in the website https://www.winamax.es/account/login.php?redir=/apuestas-deportivas. The case is that I don't find the xpath/id/text to get te next code running successfully:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
options = webdriver.ChromeOptions()
options.add_argument("window-size=1920,1080")
options.add_argument('--disable-blink-features=AutomationControlled')
driver=webdriver.Chrome(options=options,executable_path=r"chromedriver.exe")
driver.get("https://www.winamax.es/account/login.php?redir=/apuestas-deportivas")
WebDriverWait(driver=driver, timeout=15).until(
lambda x: x.execute_script("return document.readyState === 'complete'")
)
upload_field = driver.find_element_by_xpath("//input[#type='email']")
I don't only want the specific xpath for this example, but I prefer a method to obtain the xpath or something similar to get the code working for other parts of the website

<iframe id="iframe-login" data-node="iframe" name="login" scrolling="auto" frameborder="1" style="min-height: 280px; width: 100%;"></iframe>
Your element is in an iframe. Switch to it.
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.ID, "iframe-login")))
Import
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
Full working code.
wait = WebDriverWait(driver, 10)
driver.get("https://www.winamax.es/account/login.php?redir=/apuestas-deportivas")
wait.until(EC.frame_to_be_available_and_switch_to_it((By.ID, "iframe-login")))
upload_field = wait.until(EC.element_to_be_clickable((By.XPATH, "//input[#type='email']")))
upload_field.send_keys("stuff")

The email element is within an <iframe> so you have to:
Induce WebDriverWait for the desired frame to be available and switch to it.
Induce WebDriverWait for the desired element to be clickable.
You can use either of the following Locator Strategies:
Using CSS_SELECTOR:
driver.get("https://www.winamax.es/account/login.php?redir=/apuestas-deportivas")
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe#iframe-login")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[type='email']"))).send_keys("scraper#stackoverflow.com")
Using XPATH:
driver.get("https://www.winamax.es/account/login.php?redir=/apuestas-deportivas")
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[#id='iframe-login']")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[#type='email']"))).send_keys("scraper#stackoverflow.com")
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Browser Snapshot:
Reference
You can find a couple of relevant discussions in:
Ways to deal with #document under iframe
Switch to an iframe through Selenium and python
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element while trying to click Next button with selenium
selenium in python : NoSuchElementException: Message: no such element: Unable to locate element

How to click the button to crawl the reviews using python

I am trying to crawl the reviews on this website: https://www.bol.com/nl/p/Matras-140x200-7-zones-koudschuim-premium-plus-tijk-15-cm-medium/9200000118425897/.
However, I have to click a button ( Toon meer) to show all the reviews.
<div class="load-more load-more--divider load-more--reviews js-review-load-more-container">
<a data-href="/nl/rnwy/productPage/reviews?productId=9200000118425896&offset=5&limit=10&loadMore=true" class="review-load-more__button js-review-load-more-button" data-test="review-load-more"><div class="css-loader css-loader--reviews"></div>
Toon meer</a>
</div>
I use the below code :
import requests
import pandas as pd
from selenium import webdriver
from bs4 import BeautifulSoup
from datetime import datetime
start_time = datetime.now()
data = []
link = "https://www.bol.com/nl/p/Matras-140x200-7-zones-koudschuim-premium-plus-tijk-15-cm-medium/9200000118425897/"
op = webdriver.ChromeOptions()
op.add_argument('--ignore-certificate-errors')
op.add_argument('--incognito')
op.add_argument('--headless')
driver = webdriver.Chrome(executable_path='D:/Desktop/work/real/chromedriver.exe',options=op)
driver.get(link)
driver.find_element_by_css_selector('div.review-load-more__button js-review-load-more-button').click()
However, it throws an error:
No such element: Unable to locate element: {"method":"css selector","selector":"div.review-load-more__button js-review-load-more-button"} .
Is there any solution?

Css selectors cannot select an element by containing text.
Try using xpath. The last line of your script should look something like:
wait = WebDriverWait(driver, 10)
wait.until(expected_conditions.element_to_be_clickable((By.XPATH, "//a[contains(., 'Toon meer')]")).click()

To click on Toon meer you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
driver.get('https://www.bol.com/nl/p/Matras-140x200-7-zones-koudschuim-premium-plus-tijk-15-cm-medium/9200000118425896/')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[data-test='consent-modal-confirm-btn']>span"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.review-load-more__button.js-review-load-more-button"))).click()
Using XPATH:
driver.get('https://www.bol.com/nl/p/Matras-140x200-7-zones-koudschuim-premium-plus-tijk-15-cm-medium/9200000118425896/')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[#data-test='consent-modal-confirm-btn']/span"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[#class='review-load-more__button js-review-load-more-button' and contains(., 'Toon meer')]"))).click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

When you get the page a popup comes with an accept button click it and then proceed with clicking your element.
driver.get('https://www.bol.com/nl/p/Matras-140x200-7-zones-koudschuim-premium-plus-tijk-15-cm-medium/9200000118425896/')
wait=WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.XPATH, "//button[#class='js-confirm-button']"))).click()
wait.until(EC.element_to_be_clickable((By.XPATH, "//a[#data-test='review-load-more']"))).click()
Import
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

Python Selenium, help me locate an element in a website

I want to click on "new order" icon in mt4 web terminal using selenium module in python
This is the code:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome('./chromedriver')
driver.get("https://www.mql5.com/en/trading")
new_order = driver.find_element_by_xpath('/html/body/div[3]/div[1]/a[1]/span[1]')
new_order.click()
And this is the error that I get:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"/html/body/div[3]/div[1]/a[1]/span[1]"}
(Session info: chrome=86.0.4240.198)
What is the correct way to locate that button, I searched and found some ways to locate elements for selenium but I couldn't get any of them work for me.

Looks like your page is dealing with iframes. So while the above answer has good practices, you also need to switch to the iframe:
driver.switch_to.iframe(self,frame reference)
Look for more details at https://www.techbeamers.com/switch-between-iframes-selenium-python/ or https://stackoverflow.com/a/24286392/1387701

The element with tooltip as New Order is within an <iframe> so you have to:
Induce WebDriverWait for the desired frame to be available and switch to it.
Induce WebDriverWait for the desired element to be clickable.
You can use the following xpath based Locator Strategies:
driver.get('https://www.mql5.com/en/trading')
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[#id='webTerminalHost']")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[text()='Connect to an Account']//following-sibling::div[1]/span"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[#title='New Order']/span"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

You can use a different xpath:
new_order = driver.find_element_by_xpath('//a[#title="New Order"]')
But I would suggest By, WebDriverWait, and expected_conditions:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
import time
driver = webdriver.Chrome('./chromedriver')
driver.get("https://www.mql5.com/en/trading")
time.sleep(5)
iframe = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, '//iframe[#id="webTerminalHost"]')))
driver.switch_to.frame(iframe)
new_order = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//a[#title="New Order"]')))
new_order.click()