So I trying to automate the scrolling of my google play music playlist but its not picking up and I don't what other solutions to use. Normal scrolling like on stack overflow works but I think that Google Music webapp uses something different. So here is my my code so far
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
from bs4 import BeautifulSoup
from selenium.webdriver.common.action_chains import ActionChains
profile = webdriver.FirefoxProfile(r"C:\Users\Richard\AppData\Roaming\Mozilla\Firefox\Profiles\puom7wwz.default-1468625172145")
driver = webdriver.Firefox(executable_path=r'geckodriver.exe',firefox_profile=profile)
driver.get("https://play.google.com/music/listen?u=1#/pl/AMaBXymw0YGY7Y-z1B-FdrwkDCm7WdO_kyK4x4xQqn_vZsg56-VzwjsRKaQPypMs88oIKHn4qZd5aDbFbh5ojt6DpoZZCyHgCA%3D%3D")
driver.find_element_by_id("gb_70").click()
time.sleep(2)
element = driver.find_element_by_xpath("//*[contains(text(), 'William Thomas')]").click()
time.sleep(2)
password_field = driver.find_element_by_name("password")
password_field.clear()
password_field.send_keys("IcewindDale1995%")
password_field.send_keys(u'\ue007') #unicode for enter key
time.sleep(10)
driver.get("https://stackoverflow.com/questions/44370208/clicking-on-link-through-google-with-selenium-web-driver")
#driver.find_element_by_id("gba_70").click()
source = driver.page_source
soup = BeautifulSoup(source, "html.parser")
time.sleep(10)
#driver.execute_script("document.getElementById('descriptionWrapper').focus();")
#time.sleep(10)
driver.execute_script("window.scrollTo(0, 400)")
# i = 0
# while i<1000:
# scroll = driver.find_element_by_id('descriptionWrapper').send_keys(Keys.PAGE_DOWN)
# i+=1
driver.find_element_by_id("gba_70").click()
with open('page1.html', 'w', encoding="utf-8") as fid:
fid.write(str(soup))
driver.close()
Edit:
There is no scrolling taking place with that code but it works on the Stack overflow website.
Anyone with a solution?
The issue is that the Google Play Music Player on itself does not scroll, you need to have the browser focus on the internal scrollable element, something along the lines of this:
driver.find_element_by_css_selector('#music-content > div.detail-view.g-content.view-transition').click()
Hope this answer serves you well!
Related
I'm beginner on programming. I trying get my Instagram follower list but i have just 12 follower. I tried firstly click to box and scroll down but it didn't work.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
import time
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
url= "https://www.instagram.com/"
driver.get(url)
time.sleep(1)
kullaniciAdiGir = driver.find_element(By.XPATH, "//*[#id='loginForm']/div/div[1]/div/label/input"")
kullaniciAdiGir.send_keys("USERNAME")
sifreGir = driver.find_element(By.XPATH, "//input[#name='password']")
sifreGir.send_keys("PASS")
girisButonu = driver.find_element(By.XPATH, "//*[#id='loginForm']/div/div[3]/button/div").click()
time.sleep(5)
driver.get(url="https://www.instagram.com/USERNAME/")
time.sleep(3)
kutucuk= driver.get(url="https://www.instagram.com/USERNAME/followers/")
time.sleep(5)
box =driver.find_element(By.XPATH, "//div[#class='xs83m0k xl56j7k x1iy3rx x1n2onr6 x1sy10c2 x1h5jrl4 xieb3on xmn8rco x1hfn5x7 x13wlyjk x1v7wizp x1l0w46t xa3vuyk xw8ag78']")
box.click()
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(5)
takipciler = driver.find_elements(By.CSS_SELECTOR, "._ab8y._ab94._ab97._ab9f._ab9k._ab9p._abcm")
for takipci in takipciler:
print(takipci.text)
time.sleep(10)
How can i fix it? How can scroll down in box? Thanks
You can select multiple elements with this.
#get all followers
followers = driver.find_elements(By.CSS_SELECTOR, "._ab8y._ab94._ab97._ab9f._ab9k._ab9p._abcm")
# loop each follower
for user in followers:
#do something here.
Using css selectors, in my opinion, is much easier.
Also note I used find_elemets not find_element, as the latter only returns a single result.
As the data is loaded dynamically, you will, infact have to scroll to make the site load more results. Then compare what you have agaisnt whats loaded. Probably execute some javascrtipt like scroll into view on the last element in that container etc.
or take a look here for an alternative solution
https://www.folkstalk.com/2022/10/how-to-get-a-list-of-followers-on-instagram-python-with-code-examples.html
or look at instragrams API, probably something in there for getting your followers.
Disclaimer: I'm coming back to scripting after more than a decade (and I was a novice to begin with) so apologies if the question is mundane but help is much needed and appreciated.
I'm trying to modify a python script to log me into a job portal, search job vacancies based on attributes and then apply to said jobs.
Since the portal opens new vacancies on separate tabs, the code is supposed to go to the next tab and test the criteria on the job description
The code snippet is as below:
for i in range(1,6):
driver.get('https://www.naukri.com/'+role+'-jobs-in-'+location+'-'+str(i)+'?ctcFilter='+str(LL)+'to'+str(UL))
driver.switch_to.window(driver.window_handles[1])
url = driver.current_url
driver.get(url)
try:
test = driver.find_element_by_xpath('//*[#id="root"]/main/div[2]/div[2]/section[2]')
if all(word in test.text.lower() for word in Skillset):
driver.find_element_by_xpath('//*[#id="root"]/main/div[2]/div[2]/section[1]/div[1]/div[3]/div/button[2]').click()
time.sleep(2)
driver.close()
driver.switch_to.window(driver.window_handles[0])
else:
driver.close()
driver.switch_to.window(driver.window_handles[0])
except:
driver.close()
driver.switch_to.window(driver.window_handles[0])
However, when I run the script, it just logs me in to the portal and goes to the correct listings page but just stays there. Plus, it pops this error:
> line 43, in <module> driver.switch_to.window(driver.window_handles[1])
> IndexError: list index out of range
Not able to understand what list this is referring to and how to fix this code. Any help is appreciated.
The complete code for those interested:
import selenium
from selenium import webdriver as wb
import pandas as pd
import time
from time import sleep
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
driver = wb.Chrome("/Users/shashank/Desktop/Naukri Auto Apply/chromedriver")
Skillset = ['Marketing','Communications','Sales']
Skillset = [x.lower() for x in Skillset]
LL = 15 #lower limit of expected CTC
UL = 25 #upper limit of expected CTC
location = 'Bangalore'
location = location.lower().replace(" ","-")
role = 'Marketing Manager'
role = role.lower().replace(" ","-")
driver.get("https://www.naukri.com")
driver.find_element_by_xpath('//*[#id="login_Layer"]/div').click()
time.sleep(5)
driver.find_element_by_xpath('//*[#id="root"]/div[3]/div[2]/div/form/div[2]/input').send_keys("email_address_here")
driver.find_element_by_xpath('//*[#id="root"]/div[3]/div[2]/div/form/div[3]/input').send_keys("password_here")
time.sleep(5)
driver.find_element_by_xpath('//*[#id="root"]/div[3]/div[2]/div/form/div[6]/button').click()
time.sleep(20)
driver.find_element_by_xpath('/html/body/div[3]/div/div[1]/div[1]/div').click()
for i in range(1,6):
driver.get('https://www.naukri.com/'+role+'-jobs-in-'+location+'-'+str(i)+'?ctcFilter='+str(LL)+'to'+str(UL))
driver.switch_to.window(driver.window_handles[1])
url = driver.current_url
driver.get(url)
try:
test = driver.find_element_by_xpath('//*[#id="root"]/main/div[2]/div[2]/section[2]')
if all(word in test.text.lower() for word in Skillset):
driver.find_element_by_xpath('//*[#id="root"]/main/div[2]/div[2]/section[1]/div[1]/div[3]/div/button[2]').click()
time.sleep(2)
driver.close()
driver.switch_to.window(driver.window_handles[0])
else:
driver.close()
driver.switch_to.window(driver.window_handles[0])
except:
driver.close()
driver.switch_to.window(driver.window_handles[0])
Thanks in advance for answering! It will really help with the job search!
No need to switch to another window
When you are opening the URL with that specific details in the for loop, the page is getting loaded one by one in the same window. Switch to window when there is a New window tab opened. Link to Refer
Choose Explicit waits instead of time.sleep(). You have refined WebdriverWait but never used it.
Try to come up with good locators. Go for Relative Xpath instead of Absolute Xpath.Link to Refer
Not sure what you are trying to do in try block. The locators does not highlight any elements in the page.
Refer below code:
# Imports
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
driver = webdriver.Chrome(service= Service("path to chromedriver.exe"))
driver.maximize_window()
driver.get("https://www.naukri.com")
wait = WebDriverWait(driver,30)
login_btn = wait.until(EC.element_to_be_clickable((By.XPATH,"//div[text()='Login']")))
login_btn.click()
email = wait.until(EC.element_to_be_clickable((By.XPATH,"//form[#name='login-form']//input[contains(#placeholder,'Email')]")))
email.send_keys("abc#gmail.com")
password = wait.until(EC.element_to_be_clickable((By.XPATH,"//form[#name='login-form']//input[contains(#placeholder,'password')]")))
password.send_keys("password")
password.submit()
role = "marketing-manager"
loc = "bangalore"
llimt = "15"
ulimit = "25"
for i in range(1,6):
driver.get(f"https://www.naukri.com/{role}-jobs-in-{loc}-{i}?ctcFilter={llimt}to{ulimit}")
time.sleep(2)
# Update for Chat bot
try:
chatbot_close = wait.until(EC.element_to_be_clickable((By.XPATH,"//div[#class='chatbot_Nav']/div")))
chatbot_close.click()
except:
print("Chat bot did not appear")
im quite noob in python and right now building up a web scraper in Selenium that would take all URL's for products in the clicked 'tab' on web page. But my code take the URL's from the first 'tab'. Code below. Thank you guys. Im starting to be kind of frustrated lol.
Screenshot
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
from lxml import html
PATH = 'C:\Program Files (x86)\chromedriver.exe'
driver = webdriver.Chrome(PATH)
url = 'https://www.alza.sk/vypredaj-akcia-zlava/e0.htm'
driver.get(url)
driver.find_element_by_xpath('//*[#id="tabs"]/ul/li[2]').click()
links = []
try:
WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CLASS_NAME, 'blockFilter')))
link = driver.find_elements_by_xpath("//a[#class='name browsinglink impression-binded']")
for i in link:
links.append(i.get_attribute('href'))
finally:
driver.quit()
print(links)
To select current tab:
current_tab = driver.current_window_handle
To switch between tabs:
driver.switch_to_window(driver.window_handles[1])
driver.switch_to.window(driver.window_handles[-1])
Assuming you have the new tab url as TAB_URL, you should try:
from selenium.webdriver.common.action_chains import ActionChains
action = ActionChains(driver)
action.key_down(Keys.CONTROL).click(TAB_URL).key_up(Keys.CONTROL).perform()
Also, apparently the li doesn't have a click event, are you sure this element you are getting '//*[#id="tabs"]/ul/li[2]' has the aria-selected property set to true or any of these classes: ui-tabs-active ui-state-active?
If not, you should call click on the a tag inside this li.
Then you should increase the timeout parameter of your WebDriverWait to guarantee that the div is loaded.
I have a python script that goes through the UI and fills out a lot of pages. I take a screenshot, but if the page is longer than the screen, then I miss half of the page. I tried a longer length, but with the normal pages, it's ridiculous and unusable. I tried to use ashot, but can't figure out how to add it to my project in pycharm.
Any wisdom or other ideas? Thanks.
UPDATE on things tried that didn't work (nothing happened on the page:
actions = ActionChains(driver)
actions.move_to_element(continuebtn)
Also:
chrome_options.add_argument('--enable-javascript')
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
driver.execute_script("window.scrollTo(0, 1080);")
You could take a multiple screenshots and scroll through the whole page.
A simple example would be:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.firefox.options import Options
import time
d = webdriver.Firefox(executable_path="PATH TO YOUR DRIVER")
d.get("https://en.wikipedia.org/wiki/HTTP_cookie")
time.sleep(1)
scrollHeight = d.execute_script("return window.scrollMaxY")
index = 0
while d.execute_script("return window.pageYOffset") < scrollHeight:
d.save_screenshot(PICTURE_PATH+"/"+ FILENAME + str(index).zfill(2) + ".png")
d.execute_script("window.scrollByPages(1)")
# maybe call time.sleep in here somewhere to load lazyloaded images
time.sleep(0.2)
index += 1
d.quit()
As the title said, I'd like to performa a Google Search using Selenium and then open all results of the first page on separate tabs.
Please have a look at the code, I can't get any further (it's just my 3rd day learning Python)
Thank you for your help !!
Code:
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
import pyautogui
query = 'New Search Query'
browser = webdriver.Chrome('/Users/MYUSERNAME/Desktop/Desktop-Files/Chromedriver/chromedriver')
browser.get('http://www.google.com')
search = browser.find_element_by_name('q')
search.send_keys(query)
search.send_keys(Keys.RETURN)
element = browser.find_element_by_class_name('LC20lb')
element.click()
The reason why I imported pyautogui is because I tried simulating a right click and then open in new tab for each result but it was a little confusing :)
Forget about pyautogui as what you want to do can be done in Selenium. Same with most of the rest. You just do not need it. See if this code meets your needs.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
query = 'sins of a solar empire' #my query about a video game
browser = webdriver.Chrome()
browser.get('http://www.google.com')
search = browser.find_element_by_name('q')
search.send_keys(query)
search.send_keys(Keys.RETURN)
links = browser.find_elements_by_class_name('r') #I went on Google Search and found the container class for the link
for link in links:
url = link.find_element_by_tag_name('a').get_attribute("href") #this code extracts the url of the HTML link
browser.execute_script('''window.open("{}","_blank");'''.format(url)) # this code uses Javascript to open a new tab and open the given url in that new tab
print(link.find_element_by_tag_name('a').get_attribute("href"))