there are multiple windows to get the info
I have a csv with keys to search in the first column, sometimes the website doesn't contain the element so I want driver quit then it can send the next key.
Here I give a specific example key I wanna search (and a normal situation):
url = 'http://biz.finance.sina.com.cn/suggest/lookup_n.php?country=&q=21%BD%F0%BF%C602'
# the following is a normal situation with all elements exist
# url = 'http://money.finance.sina.com.cn/bond/quotes/sz149373.html'
driver.get(url)
try:
driver.switch_to.window(driver.window_handles[-1])
driver.find_element_by_xpath('//div[#class="title tit06"]/a').click()
driver.switch_to.window(driver.window_handles[-1])
driver.find_element_by_link_text('基本资料').click()
driver.switch_to.window(driver.window_handles[-1])
driver.find_element_by_link_text('发行信息').click()
except:
try:
driver.switch_to.window(driver.window_handles[-1])
driver.find_element_by_link_text('基本资料').click()
driver.switch_to.window(driver.window_handles[-1])
driver.find_element_by_link_text('发行信息').click()
except:
#error after quitting, cannot continue to search next key
**driver.quit()**
driver.switch_to.window(driver.window_handles[-1])
driver.find_element_by_link_text('基本资料').click()
driver.switch_to.window(driver.window_handles[-1])
driver.find_element_by_link_text('发行信息').click()
ERROR:MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=63963): Max retries exceeded with url:
When the chrome quit in the highlighted line, then the error shows as above.
Many thanks!!
driver.quit() will shutdown all of driver operation in your computer. The error you receive mean it able to find your driver because it has been shutdown by your code.
No further interaction can be execute.
If you want to close the current website, in your current window.
Switch back to your parent window and using driver.close() instead (Remember to keep track of your parent window):
driver.switch_to.window(parent_window)
driver.close()
It only close the current window your driver is focusing on.Code example for your case:
for key in read_key_from_csv:
# declare your driver inside loop
driver = webdriver.Chrome()
driver.get(key)
# put all your code in side the loop
# in case it not find element you want close the driver, the loop will jump to next key and start new driver
driver.close()
# no action relate to driver can perform after you close it
Related
I am a new learner for python and selenium. I have written a code to extract data from multiple pages but there is certain problem in the code.
I am not able to break the a while loop function which clicks on next page until there is an option. The next page element disables after reaching the last page but code sill runs.
xpath: '//button[#aria-label="Next page"]'
Full SPAN: class="awsui_icon_h11ix_31bp4_98 awsui_size-normal-mapped-height_h11ix_31bp4_151 awsui_size-normal_h11ix_31bp4_147 awsui_variant-normal_h11ix_31bp4_219"
I am able to get the list of data which I want to extract from the webpage but I am getting on the last page data when I close the webpage from my end, ending the while loop.
Full Code:
opts = webdriver.ChromeOptions()
opts.headless = True
driver = webdriver.Chrome(ChromeDriverManager().install())
base_url = "XYZ"
driver.maximize_window()
driver.get(base_url)
driver.set_page_load_timeout(50)
element = WebDriverWait(driver, 50).until(EC.presence_of_element_located((By.ID, 'all-my-groups')))
driver.find_element(by=By.XPATH, value = '//*[#id="sim-issueListContent"]/div[1]/div/div/div[2]/div[1]/span/div/input').send_keys('No Stock')
dfs = []
page_counter = 0
while True:
wait = WebDriverWait(driver, 30)
wait.until(EC.visibility_of_all_elements_located((By.XPATH, "//div[contains(#class, 'alias-wrapper sim-ellipsis sim-list--shortId')]")))
cards = driver.find_elements_by_xpath("//div[contains(#class, 'alias-wrapper sim-ellipsis sim-list--shortId')]")
sims = []
for card in cards:
sims.append([card.text])
df = pd.DataFrame(sims)
dfs.append(df)
print(page_counter)
page_counter+=1
try:
wait.until(EC.element_to_be_clickable((By.XPATH,'//button[#aria-label="Next page"]'))).click()
except:
break
driver.close()
driver.quit()
I am also, attaching the image of the class and sorry I cannot share the URL as it of private domain.
The easiest option is to let your wait.until() fail via timeout when the "Next page" button is missing. Right now your line wait = WebDriverWait(driver, 30) is setting the timeout to 30 seconds; assuming the page normally loads much faster than that, you could change the timeout to be 5 seconds and then the loop will end faster once you're at the last page. If your page load times are sometimes slow then you should make sure the timeout won't accidentally cut off too early; if the load times are consistently fast then you might be able to get away with an even shorter timeout interval.
Alternatively, you could look through the specific target webpage more carefully to find some element that a) is always present and b) can be used to determine whether we're on the final page or not. Then you could read the value of that element and decide whether to break the loop before trying to find the "Next page" button. This could save a couple of seconds of waiting on the final loop iteration (avoid waiting for timeout) but may not be worth the trouble.
Change the below condtion
try:
wait.until(EC.element_to_be_clickable((By.XPATH,'//button[#aria-label="Next page"]'))).click()
except:
break
as shown in the below pseduocode #disabled is the diff that will make sure to exit the while loop if the button is disabled.
if(driver.find_elements_by_xpath('//button[#aria-label="Next page"][#disabled]'))).size()>0)
break
else
driver.find_element_by_xpath('//button[#aria-label="Next page"]').click()
driver.find_element(By.XPATH, "//input[#name = 'SelectAll' and #value='Move All >>>']").click()
driver.find_element(By.NAME, "Submit1").click()
For background, this is part of a script that goes to a website, completes a few actions/prompts, and pulls a CSV file. The code above works most of the time but sometimes gives an error of,
UnexpectedAlertPresentException: unexpected alert open: {Alert text : No contracts have been selected.}
(Session info: headless chrome=97.0.4692.99)
This is from a page where you have to move list items over from one dialogue box to another. You receive this error when nothing has been pulled over. Also as a note, I do have an implicitly_wait(100) before this code.
I am trying to get the above code to run through a loop where it will keep repeating/retrying the code until there is a success.
If I clearly understnd what goes there:
You want to continue performing the 2 click actions you mentioned until no alert appeared.
I.e. in case alert appearing - close the alert and perform the clicks again.
In case no alert appeared - no more need to click these 2 elements, continue further with the next code.
If so you can do something like this:
while True:
driver.find_element(By.XPATH, "//input[#name = 'SelectAll' and #value='Move All >>>']").click()
driver.find_element(By.NAME, "Submit1").click()
try:
alert = driver.switch_to.alert
alert.accept()
except:
break
I want to automate filling google forms with the images in google drive so I am using python with selenium.
The first image I can automate, but I got the error when I repeated the same process for the second image.
I created a sample google form to test.
https://forms.gle/fZ439KckkbpNU68v5
time.sleep(5)
driver.find_element_by_xpath('x path to first add file button').click()
frame= driver.find_element_by_class_name('picker-frame')
driver.switch_to.frame(frame)
*the first time it is working time*
driver.switch_to_default_content()
time.sleep(5)
driver.find_element_by_xpath('xpath to secondadd file button').click()
frame= driver.find_element_by_class_name('picker-frame')
driver.switch_to.frame(frame)
*after this, it causes an error*
I could not use the send_keys() method because this is button type.
Full code:
from selenium import webdriver
import random,time
option=webdriver.ChromeOptions()
driver = webdriver.Chrome("path to chrome driver")
driver.get("https://forms.gle/RvBYTZG5xuWoD8QA7")
driver.find_element_by_xpath('//*[#id="identifierId"]').send_keys("gmail")
driver.find_element_by_xpath('//*[#id="identifierNext"]/div/button').click()
time.sleep(3)
driver.find_element_by_xpath('//*[#id="password"]/div[1]/div/div[1]/input').send_keys("password")
driver.find_element_by_xpath('//*[#id="passwordNext"]/div/button').click()
time.sleep(5)
driver.find_element_by_xpath('//*[#id="mG61Hd"]/div[2]/div/div[2]/div[1]/div/div/div[2]/div/div[2]/span').click()
frame1 = driver.find_element_by_class_name('picker-frame')
driver.switch_to.frame(frame1)
driver.find_element_by_xpath('//*[#id=":6"]').click()
time.sleep(5)
driver.find_element_by_xpath('//*[#id=":1a.docs.0.1oe7tESa9uNnUC75FdYuqzPtPbLYz7QFi"]/div[2]').click()
driver.find_element_by_xpath('//*[#id="picker:ap:2"]').click()
driver.switch_to_default_content()
driver.find_element_by_xpath('//*[#id="mG61Hd"]/div[2]/div/div[2]/div[2]/div/div/div[2]/div/div[2]').click()
frame2 = driver.find_element_by_class_name('picker-frame')
driver.switch_to.frame(frame2)
driver.find_element_by_xpath('//*[#id=":6"]').click()
time.sleep(3)
This is a guess since no HTML was provided but maybe the location in the iframe of the element isnt correct since if you give it the global iframe path it want work i suggest to check if the button has a specifique i frame and switch to that iframe and give it the path in that one.
Or the element path isn't unique that makes problems too.
I think you are typing it wrongly. You should try this code.
driver.switch_to.default_content()
The solution is to put the refresh after switching the frame. Without having to use the switch_to_default_content() at all.
driver.refresh()
driver.switch_to.alert.accept()
It works.
I have seen other questions about this error but my case is that in my program the other element should receive the click. In details: the webdriver is scrolling through google search and it must click every website it finds but the program is preventing that. How can I make it NOT search the previous site it clicked?
This is the function. The program is looping it and after the first loop it scrolls down and the error occurs:
def get_info():
browser.switch_to.window(browser.window_handles[2])
description = WebDriverWait(browser, 10).until(
EC.presence_of_element_located((By.TAG_NAME, "h3"))
).text
site = WebDriverWait(browser, 10).until(
EC.presence_of_element_located((By.TAG_NAME, "cite"))
)
site.click()
url=browser.current_url
#removes the https:// and the / of the url
#to get just the domain of the website
try:
link=url.split("https://")
link1=link[1].split("/")
link2=link1[0]
link3=link2.split("www.")
real_link=link3[1]
except IndexError:
link=url.split("https://")
link1=link[1].split("/")
real_link=link1[0]
time.sleep(3)
screenshot=browser.save_screenshot("photos/"+"(" + real_link + ")" + ".png")
global content
content=[]
content.append(real_link)
content.append(description)
print(content)
browser.back()
time.sleep(5)
browser.execute_script("window.scrollBy(0,400)","")
time.sleep(5)
You can create a list of clicked website and check every time if that link is clicked or not. Here's the demo code :
clicked_website=[]
url=browser.current_url
clicked_website.append(url)
# Now while clicking
if <new_url> not in clicked_website:
<>.click()
This is just an idea how to implement. Your code is mess, I didn't understand clearly so, implement in your code by yourself.
I am working on my first scraper and ran into an issue. My scraper accesses a website and saves links from the each result page. Now, I only want it to go through 10 pages. The problem comes when the search results has less than 10 pages. I tried using a while loop along with a try statement, but it does not seem to work. After the scraper goes through the first page of results, it does not return any links on the successive pages; however, it does not give me an error and stops once it reaches 10 pages or the exception.
Here is a snippet of my code:
links = []
page = 1
while(page <= 10):
try:
# Get information from the propertyInfo class
properties = WebDriverWait(driver, 10).until(lambda driver: driver.find_elements_by_xpath('//div[#class = "propertyInfo item"]'))
# For each listing
for p in properties:
# Find all elements with a tags
tmp_link = p.find_elements_by_xpath('.//a')
# Get the link from the second element to avoid error
links.append(tmp_link[1].get_attribute('href'))
page += 1
WebDriverWait(driver, 10).until(lambda driver: driver.find_element_by_xpath('//*[#id="paginador_siguiente"]/a').click())
except ElementNotVisibleException:
break
I really appreciate any pointers on how to fix this issue.
You are explicitely catching ElementNotVisibleException exception and stopping on it. This way you won't see any error message. The error is probably in this line:
WebDriverWait(driver, 10).until(lambda driver:
driver.find_element_by_xpath('//*[#id="paginador_siguiente"]/a').click())
I assume lambda here should be a test, which is run until succeeded. So it shouldn't make any action like click. I actually believe that you don't need to wait here at all, page should be already fully loaded so you can just click on the link:
driver.find_element_by_xpath('//*[#id="paginador_siguiente"]/a').click()
This will either pass to next page (and WebDriverWait at the start of the loop will wait for it) or raise exception if no next link is found.
Also, you better minimize try ... except scope, this way you won't capture something unintentionally. E.g. here you only want to surround next link finding code not the whole loop body:
# ...
while(page <= 10):
# Scrape this page
properties = WebDriverWait(driver, 10).until(...)
for p in properties:
# ...
page += 1
# Try to pass to next page
try:
driver.find_element_by_xpath('//*[#id="paginador_siguiente"]/a').click()
except ElementNotVisibleException:
# Break if no next link is found
break