I am trying to get followers with python selenium. But sometimes python clicks by itself.
I want to make an error-free program. I try to I've tried "try catch" constructs but it didn't work. Here is my code:
def getFollowers(self):
try:
self.browser.get(f"https://www.instagram.com/{self.username}")
time.sleep(2)
followers=self.browser.find_element_by_xpath("//*[#id='react-root']/section/main/div/header/section/ul/li[2]/a").click()
time.sleep(2)
dialog=self.browser.find_element_by_xpath("/html/body/div[5]/div/div/div[2]")
followerCount=len(dialog.find_elements_by_tag_name("li"))
print(f"first count:{followerCount}")
action=webdriver.ActionChains(self.browser)
//*******************************************Probly my problem is here****************************************
while True:
dialog.click()
action.key_down(Keys.SPACE).key_up(Keys.SPACE).perform()
time.sleep(3)
newCount=len(dialog.find_elements_by_tag_name("li"))
if followerCount!=newCount or newCount==24:
print(f"New count:{newCount}")
time.sleep(3)
followerCount=newCount
else:
break
//**********************************************************************************************************
followers=dialog.find_elements_by_tag_name("li")
followersList=[]
for user in followers:
link=user.find_element_by_css_selector("a").get_attribute("href")
# print(link)
followersList.append(link)
with open("followers.txt","w",encoding="UTF-8") as file:
for item in followersList:
file.write(item+"\n")
time.sleep(5)
except:
pass
I also have def getfollowing and it works flawlessly. If you want I can show it too. But they are almost same.
EDIT: #RohanShah solved my problem. At the bottom of the page you can see the solution.
Edit: I am new here thats why sometimes my questions could be meanless.But please dont decrease my points. Stackoverflow not gonna accept my questions anymore. Please increase my points.
I've had this exact same problem while scrolling the popups. What happens is your dialog.click(), while attempting to focus your key down on the popup, occasionally clicks a user and loads their profile. Your script then crashes as the popup is no longer on the screen.
After a lot of research into solving this problem, I noticed it only happens with usernames that are long. Regardless, I implemented a simple hack to get around this problem.
First we get the url of what the standard scroll looks like. When opening and scrolling the popup, this is the url we are on.
https://www.instagram.com/some_username/followers/
2.Now I have created a function to hold the code for opening the popup. This will be very useful so trap the necessary code into a function. (I don't have the classnames or xpath's on me so please customize the function for yourself)
def openPopup():
self.browser.get(f"https://www.instagram.com/{self.username}")
global popup # we will need to access this variable outside of the function
popup = driver.find_element_by_class_name('popupClass') #you don't have to use class_name
popup.click()
Now we have to tell our while loop to not scan when Selenium accidentally clicks on a user. We will use our URL from step 1. Please make sure the following if-statement is inserted at the TOP of your loop so if there is a break, it will handle it first before trying to access the popup.
while True:
check_url = self.browser.current_url #returns string with current_url
if check_url != 'https://www.instagram.com/some_username/followers/':
#if this code is executed, this means there has been an accidental click
openPopup() #this will bring back to the page and reopen popup
#the rest of your code
popup.click() # variable from our function
action.key_down(Keys.SPACE).key_up(Keys.SPACE).perform()
time.sleep(3)
newCount=len(dialog.find_elements_by_tag_name("li"))
if followerCount!=newCount or newCount==24:
print(f"New count:{newCount}")
time.sleep(3)
followerCount=newCount
else:
break
check_url = self.browser.current_url #we must recheck the current_url every time the loop runs to see if there has been a misclick
Now, whenever your loop detects the URL is no longer one of the popup, it will automatically call openPopup() which will get you back to the page and back in the popup, and your loop will continue as if nothing happened.
Related
in my scraping script in python I'm in a situation where, while collapsing multiple buttons on a page, randomically a couple of pop up appear in the page and automatically the script fails.
These two pop ups are already managed and the beginning of the script but the website in a non systematic way dedices to show these two.
This is the part of the script interested where the script sleeps for 3 secs between one click to the other:
collapes = driver.find_elements_by_css_selector('.suf-CompetitionMarketGroup suf-CompetitionMarketGroup-collapsed ')
for collapes in collapes:
collapes.click()
sleep(3)
These are the two lines of the scipt where I click on the pop ups at the beginning
wait.until(EC.presence_of_element_located((By.XPATH, '/html/body/div[3]/div/div[2]/div[2]'))).click()
driver.find_element(By.XPATH, '/html/body/div[1]/div/div[4]/div[1]/div/div[3]/div[4]/div[3]/div').click()
DO you think there's a way to continue running the process being ready to click on these two without going on error?
You can try to close the popups every time your code in for loop fails:
def try_closing_popups():
try:
wait.until(EC.presence_of_element_located((By.XPATH, '/html/body/div[3]/div/div[2]/div[2]'))).click()
driver.find_element(By.XPATH, '/html/body/div[1]/div/div[4]/div[1]/div/div[3]/div[4]/div[3]/div').click()
except:
pass
collapes = driver.find_elements_by_css_selector('.suf-CompetitionMarketGroup suf-CompetitionMarketGroup-collapsed ')
for collapes in collapes:
try:
collapes.click()
except:
try_closing_popups()
sleep(3)
When visiting some websites with selenium there are times where the page doesn't load correctly. So I want to write a code that checks if the website loaded correctly by searching for a particular button. If it can't find the button it needs to refresh until it finds the button and if the code does find the button it needs to execute the rest of the code.
So for example: Button not there: >>> refresh >>> (checks again if button is not there) Button not there >>> refresh >>> (checks again if button is not there) Button is there >>> rest of code
The loop that I currently have looks like this but after refreshing the loop doesn't restart and runs the else: function.
So the question is how do I make a loop that restarts the loop after it refreshes.
while not (driver.find_elements(By.CLASS_NAME, "button")):
driver.refresh()
else:
Rest of code
Help would be much appreciated, thanks in advance
You can have an infinite while loop, and an if condition with find_elements, Please note that find_elements does not return any exception, it returns a list of web elements or either 0.
Code:
while True:
try:
if len(driver.find_elements(By.XPATH, "xpath of the button")) >0:
print("Button is present, since find elements list is non empty")
# execute the code, may be click on the button or whatever it is.
#make sure to exit from infinite loop as well
break
else:
driver.refresh()
#may be put some delay here to let button available to click again.
except:
print("There was some problem trying to find the button element, tearing it down")
break
So I am trying to fill out a form on this site. Every time I try to click the submit button at the end, using what I believe is the correct id, it just gives me an error. Here is a code snippet:
from selenium import webdriver
thePassword = "asdf123"
print("Desired name: ")
name = raw_input()
print("Desired Last Name: ")
userLastName = raw_input()
browser = webdriver.Firefox()
browser.get('https://www.panerabread.com/en-us/mypanera/registration-page.html')
firstName = browser.find_element_by_id('join_first_name')
firstName.send_keys(name)
lastName = browser.find_element_by_id('join_last_name')
lastName.send_keys(userLastName)
emailElem = browser.find_element_by_id('join_email')
emailElem.send_keys("asdafasda" + "#gmail.com")
emailConfirm = browser.find_element_by_id("join_confirm_email")
emailConfirm.send_keys("asdafasda" + "#gmail.com")
password = browser.find_element_by_id("join_password")
password.send_keys("thePassword")
passwordConfirm = browser.find_element_by_id("join_confirm_password")
passwordConfirm.send_keys("thePassword")
phoneA = browser.find_element_by_id("phone_number_a")
phoneA.send_keys("231")
phoneB = browser.find_element_by_id("phone_number_b")
phoneB.send_keys("123")
phoneC = browser.find_element_by_id("phone_number_c")
phoneC.send_keys("2310")
tos = browser.find_element_by_id("join_i_agree")
tos.click()
browser.execute_script("$('#join_password').get(0).scrollIntoView();")
#browser.implicitly_wait(10)
# And then perform the click
browser.find_element_by_id("join_card_not_available").click()
browser.find_elements_by_css_selector("#join-now-primary")[1].click()
print "Your email is: " + "asdafasda" + "#gmail.com"
print "Your password is: " + thePassword
My question is, how can I submit the form at the end of my script?
Edit: There is no error. The problem is that it doesn't click the button I want it to at all. I tried running the below code on a seperate file and it worked, however when you run it with this entire script it does not work.
This was a weird one... it took me a minute to figure out what was going on. The problem is that there are actually two elements that have that id on the page (which is a no-no according to the HTML standard... but alas it happens). One on the bottom of the page that you are looking at and another on the Sign In popup. If you click the Sign In button at the top of the page, you will see the (first) Sign In button on the popup. Because it is hidden, your code wouldn't click on it. Anyway... to the solution.
There are a few ways you can handle this, any of them valid. I would do this.
browser.find_elements_by_css_selector("#join-now-primary")[1].click()
What this is doing is using a CSS selector to get all the elements with ID=join-now-primary. The CSS selector is #join-now-primary which means id (#) of join-now-primary. Because it uses .find_elements (plural), it will get both. We then use [1] to get the 2nd element (0-based index, so 1 is the 2nd) and then click on it.
EDIT
My guess is that it's a timing issue that is causing the code to work on its own but not in your script. Put a breakpoint on the first line and step through each line and make sure that it executes. Does it work? If I were to guess again... it's likely the line right before the Join Now click. That click has an animation that closes the credit card picture. I would wait for that section to become invisible using the code below
element = WebDriverWait(browser, 5).until(EC.invisibility_of_element_located(By.ID('panera-card-section')))
browser.find_elements_by_css_selector("#join-now-primary")[1].click()
You didn't really ask a question but it's likely you need to look at the WebElement class's methods and properties.
Looks like the button might not be in the visible portion of the window based on your code.
WebElement Has a property call that scrolls until an element moves into view.
If an element is not visible by Selenium definition, it is not clickable.
Even though you use the guts of the page to drive it, selenium wants to pretend it is testing human like interaction and so provides an artificial constraint.
You can bypass that by executing JavaScript click() on the element.
I have the following issue with Selenium Webdriver. There are two dropdown menus on a page i am testing "brand" and "items". The options of "items" depend on which brand you choose. I am trying to iterate through all possible choices and print brand-item pairs. I use two possible ways to pick an option from each dropdown menu
Using Select():
def retryingSelectOption(name,n):
result=False
attempts=0
while attempts<5:
try:
element=Select(driver.find_element_by_name(name))
element.select_by_index(n)
print element.all_selected_options[0].text
result=True
break
except StaleElementReferenceException:
pass
attempts+=1
return result
And using .click():
def retryingClickOption(name,n):
result=False
attempts=0
while attempts<5:
try:
driver.find_element_by_name(name).find_elements_by_tag_name("option")[n].click()
result=True
break
except StaleElementReferenceException:
pass
attempts+=1
return result
My problem is that at ,what seem to me as random moments (sometimes it works sometimes it does not), even though the above functions return True and printing out the selected option shows me the correct answer, the browser still displays the previous option. So basically Selenium tells me i have picked the right option but the browser displays the previous one.No idea what is wrong.
After checking it a bit more it seems that if i manually click somewhere in the browser the option actually refreshes to the correct value. It seems like the browser hangs.
I have created a test to create a ticket. When I click 'new ticket', it brings up a popup form. I fill in the title, and body, and click 'save and new'. At this point I created a loop. While i<10, repeat the test. I can get to the first ticket, but during the second ticket, it just exits out and says the error in my title. I have gotten to the point where it almost starts the 2nd ticket, but it enters the text in the wrong area, which I don't understand. It enters text in the field one above my 'title'. I have the send_keys based on the ID as well. here is the code:
while i<2:
driver.implicitly_wait(10)
title = f.name()
driver.find_element_by_name('txtsummary')
driver.find_element_by_name('txtsummary').send_keys(title)
driver.find_element_by_name('txtcontactorg').send_keys('unnamed')
body = f.name()
page = driver.find_elements_by_xpath('//td[#class="mceIframeContainer mceFirst mceLast"]/iframe')[1]
page.click()
page.send_keys(body)
driver.find_element_by_xpath('//span[text()="Save and New"]').click()
If I take out the implicitly wait, it enters text right after saving the first one, into the wrong box. If I leave it in, it just waits 10 seconds and closes out. My first ticket gets created always, and it always takes forever than fails for the next ticket. I mean that in my last line of code, the save and new, it takes like 8 seconds for that line to run, when it should be instantly after sending body. I have a feeling this is a factor.
I believe the problem has to do with the fact that you are selecting an iframe. Once you are in the iframe, it is unable to re-find 'txtsummary'.
The reason this happens is because the focus has been changed to a place where 'txtsummary' doesnt exist as soon as you clicked the iframe. To return focus to the topmost frame insert.
driver.switch_to_default_content()
at the top of your loop, and every time that your loop is run, it will start at the top most frame.