Develop Reference

How to generate the search results in fbref.com sending text to the search field using Python Selenium and send_keys method

I am unable to retrieve any search results in fbref.com when using either of send_keys and execute_script in selenium for python using chrome web driver
This is the code ive used so far:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver import ActionChains
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from bs4 import BeautifulSoup
import pandas as pd
import numpy as np
import csv
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.action_chains import ActionChains
s=Service(ChromeDriverManager().install())
driver = webdriver.Chrome(service=s)
driver.get("https://fbref.com/en/")
element = driver.find_element(by=By.CLASS_NAME, value="ac-hint")
action = ActionChains(driver)
element1= driver.find_element(by=By.CSS_SELECTOR, value=("input[type='search']"))
action.click(on_element=element1)
action.perform()
#element.send_keys("lionel messi")
#driver.execute_script("arguments[0].value='lionel messi'",element)
element2=driver.find_element(by=By.CSS_SELECTOR, value=("input[type='submit']"))
action.click(on_element=element2)
action.perform()```
The code is able to interact with the search button and the text is typed and the search button is clicked without any trouble but the search result is as follows:
which basically means that the search was invalid ,ive tried to search manually in the browser window opened by the driver and that gives me a successful result

You are doing your player name input in the wrong field, if you look closely at the html, there are 2 input fields for the search.
instead of the "ac-hint", use "ac-input":
element = driver.find_element(by=By.CLASS_NAME, value="ac-input")

The locator strategy you have used to identify the search field
doesn't identifies the desired element uniquely within the HTML DOM
Solution
To send a character sequence to the element you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following solution:
Code Block:
driver.get("https://fbref.com/en/")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[type='search'][placeholder='Enter Person, Team, Section, etc']"))).send_keys("lionel messi" + Keys.RETURN)
Note: You have to add the following imports :
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Browser Snapshot:

Can't find an element using name or text shown on website - python and selenium

I have a sample code here which keeps saying that it cant find the element im looking for. Please help me, I want to find the element by the name and not the absolute Xpath
from selenium import webdriver
from time import sleep
browser = webdriver.Chrome()
browser.get('https://www.instagram.com')
sleep(5)
x = browser.find_element_by_xpath("//span[text() = 'Sign Up']").click()

When you get the instagram page you need to induce waits for the page to load and then click the parent a tag of that span.
WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.XPATH, "//span[text()='Sign up']/parent::a"))).click()
Import
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

Getting error:(Element not interactable) when sending keys into the input field in selenium python

I am getting error (element not interactable) when I am trying to send keys into the input field. When I am trying to only click the input filed I am able to do but when giving text it is showing error I have so many things to solve this but I am getting this error only.
My code:
from selenium import webdriver
Driver=webdriver.Chrome()
Driver=get('https://YouTube.com')
Box=Driver.find_element_by_xpath('//*[#id="search-input"]')
Box.send_keys('music') ```

The searchbar is input(id=search) in div class(search-input). Try this;
from selenium import webdriver
Driver=webdriver.Chrome()
Driver.get('https://YouTube.com')
Box=Driver.find_element_by_id('search-input').find_element_by_id('search')
Box.send_keys('music')

To send elements to the search box. First induce a wait for the element to be clickable due to page load.
Box=WebDriverWait(Driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input#search")))
Box.send_keys('music')
Import
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

Try with that:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
Driver=webdriver.Chrome()
Driver=get('https://YouTube.com')
Box=Driver.find_element_by_xpath('/html/body/ytd-app/div/div/ytd-masthead/div[3]/div[2]/ytd-searchbox/form/div/div[1]/input')
Box.send_keys('music')

Why does this selenium not click "next page" until end?

I am writing a scraping code for the website Upwork, and need to click through each page for job listings. Here is my python code, which I used selenium to web crawl.
from bs4 import BeautifulSoup
import requests
from os.path import basename
from selenium import webdriver
import time
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
driver = webdriver.Chrome("./chromedriver")
driver.get("https://www.upwork.com/o/jobs/browse/c/design-creative/")
link = driver.find_element_by_link_text("Next")
while EC.elementToBeClickable(By.linkText("Next")):
wait.until(EC.element_to_be_clickable((By.linkText, "Next")))
link.click()

There are couple of problems:
EC has no attribute elementToBeClickable. In Python you should use element_to_be_clickable
Your link defined on the first page only, so using it on the second page should give you StaleElementReferenceException
There is no wait variable defined in your code. I guess you mean something like
wait = WebDriverWait(driver, 10)
By has no attribute linkText. Try LINK_TEXT instead
Try to use below code to get required behavior
from selenium.common.exceptions import TimeoutException
while True:
try:
wait(driver, 10).until(EC.element_to_be_clickable((By.LINK_TEXT, Next"))).click()
except TimeoutException:
break
This should allow you to click Next button while it's available

HTTP selector of a link (xpath or css)

I'm trying to grab the href element of each shoe in this site:
http://www.soccerpro.com/Clearance-Soccer-Shoes-c168/
But I can't get the proper selectors right.
response.xpath('.//*[#class="newnav itemnamelink"]')
[]
Anyone know how would I do this in xpath or css?

Required links generated dynamically, so you wouldn't be able to scrape them from HTML source that you get like requests.get("http://www.soccerpro.com/Clearance-Soccer-Shoes-c168/")
You might use selenium to get required values via browser session:
from selenium import webdriver as web
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait as wait
driver = web.Chrome()
driver.get('http://www.soccerpro.com/Clearance-Soccer-Shoes-c168/')
wait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//table[#class='getproductdisplay-innertable']")))
links = [link.get_attribute('href') for link in driver.find_elements_by_xpath('//a[#class="newnav itemnamelink"]')]