I am trying to make an Instagram bot that can perform various functions - InstaPy kept timing out on me so I decided to use selenium BUT the issue is: I can't seem to get the past the first hurdle of actually logging into IG.
I am not getting any errors on the console but it won't let me past the past additional cookies acceptance page. I have played with the xpath and done a few tweeks but still nothing - any ideas on a fix here ?
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time, urllib.request
import requests
PATH = r"/Users/PycharmProjects/pythonProject13/chromedriver"
driver = webdriver.Chrome(PATH)
driver.get('https://www.instagram.com')
#login
time.sleep(5)
notnow = driver.find_element_by_xpath("/html/body/div[4]/div/div/button[2], 'Allow Essential and Optional Cookies')]").click()
username=driver.find_element_by_css_selector("input[name='username']") #arialabelondevtools = #Phone number, username or email address
password=driver.find_element_by_css_selector("input[name='password']")
username.clear()
password.clear()
username.send_keys("testacct1")
password.send_keys("testpassword123")
login = driver.find_element_by_css_selector("button[type='submit']").click()
One of the most common mistake that people do is write absolute xpath or probably you are copying xpath from browser it self so instead use smarter xpath use id, class and other attributes to write xpath..
I recently did login to Instagram and here is the simple go
driver.get('https://www.instagram.com/')
wait = WebDriverWait(driver, 30)
wait.until(EC.visibility_of_element_located((By.XPATH, '//input[#name="username"]')))
driver.find_element_by_xpath('//input[#name="username"]').send_keys('your_login')
driver.find_element_by_xpath('//input[#type="password"]').send_keys('your_password')
driver.find_element_by_xpath('//input[#type="password"]').submit()
once you past login page you can
driver.get('https://instagram.com/')
it will reload to your home page...
Related
I'm using Selenium and ChromeDriver to scrape data from a website.
I need to keep my account logged in after closing the Driver: for this purpose I use every time the default Chrome profile.
Here you can see my code:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
urlpage = 'https://example.com/'
options = webdriver.ChromeOptions()
options.add_argument("user-data-dir=C:\\Users\\MyName\\AppData\\Local\\Google\\Chrome\\User Data")
driver = webdriver.Chrome(options=options)
driver.get(urlpage)
The problem is that for some websites (e.g. https://projecteuler.net/) it works, so I'm logged in also the following session, but for other (like https://www.fundraiso.ch, the one I need) it doesn't, although in the "normal" browser I'm still logged in after I close the window.
Does anyone know how to fix this problem?
EDIT:
I didn't mention that I can't automate the login because the website has a maximum login number, and if I breach it the website will block my account.
I am learning python and selenium right now by making a small script that posts birthday comments to my facebook friends. I am able to successfully login and navigate to the "friends' birthday" modal, but then I am unable to comment in the textbox.
Here is the error I am getting:
selenium.common.exceptions.NoSuchElementException:
Message: no such element: Unable to locate element:
{"method":"xpath","selector":"//*[#id="u_i_6"]"}
I haven't had any issues finding other page elements via XPath so I am not sure what is causing the issue. I also tried finding the textbox to comment via classname, but had the same result. Can anyone offer some thoughts?
EDIT: forgot to paste code
from selenium import webdriver
from pynput.keyboard import Key, Controller
import time
import config
# fetch facebook password from separate file in the same directory
pw = config.secret['pw']
keyboard = Controller()
driver = webdriver.Chrome()
options = webdriver.ChromeOptions()
options.add_argument('--disable-extensions')
options.add_argument('--disable-notifications')
# navigate to facebook url
driver.get('https://facebook.com')
# input email address
email_input = driver.find_element_by_xpath('//*[#id="email"]')
email_input.send_keys(<omitted>)
# input pw
password_input = driver.find_element_by_xpath('//*[#id="pass"]')
password_input.send_keys(pw)
# click login button element
login_button = driver.find_element_by_xpath('//*[#id="u_0_b"]')
login_button.click()
home_button = driver.find_element_by_xpath('//*[#id="u_0_c"]/a')
# wait 8 seconds for browser popup before pressing esc to get out
time.sleep(8)
keyboard.press(Key.esc)
# check if there are any birthdays today
birthday_section = driver.find_element_by_xpath('//*[#id="home_birthdays"]/div/div/div/div/a/div/div/span/span[1]')
birthday_section.click()
first_comment = driver.find_element_by_xpath('//*[#id="u_i_6"]')
first_comment.send_keys('happy birthday!')
# click post button to post comment
# close browser window after actions are complete
time.sleep(5)
driver.close()
It is hard to answer without the code, but here are a couple of ideas:
Make sure you are using some sort of Selenium wait, implicit or explicit, so your code does not search for an element before this element appears on the page. You can add this code after driver.get() for implicit wait:
driver.implicitly_wait(5)
Also, it looks like IDs are dynamic on that page, I get a different one on my FB page. Try using this xpath to find the textarea:
"//form[contains(#action, 'birthday')]//textarea"
Hope this is helpful, good luck!
I need to download a massive amount of excel-files (estimated: 500 - 1000) from sellercentral.amazon.de. Manually downloading is not an option, as every download needs several clicks until the excel pops up.
Since amazon cannot provide me a simple xml with its structure, I decided to automate this on my own. The first thing coming to mind was Selenium and Firefox.
The Problem:
A login to sellercentral is required, as well as 2-factor-authentication (2FA). So if I login once, i can open another tab, enter sellercentral.amazon.de and am instantly logged in.
I can even open another instance of the browser, and be instantly logged in there too. They might be using session-cookies. The target URL to "scrape" is https://sellercentral.amazon.de/listing/download?ref=ag_dnldinv_apvu_newapvu .
But when I open the URL from my python-script with selenium webdrive, a new instance of the browser is launched, in which I am not logged in. Even though, there are instances of firefox running at the same time, in which I am logged in. So I guess the instances launched by selenium are somewhat different.
What I've tried:
I tried setting a timedelay after the first .get() (to open site), then I'll manually login, and after that redoing the .get(), which makes the script go on for forever.
from selenium import webdriver
import time
browser = webdriver.Firefox()
# Wait for website to fire onload event
browser.get("https://sellercentral.amazon.de/listing/download?ref=ag_dnldinv_apvu_newapvu")
time.sleep(30000)
browser.get("https://sellercentral.amazon.de/listing/download?ref=ag_dnldinv_apvu_newapvu")
elements = browser.find_elements_by_tag_name("browse-node-component")
print(str(elements))
What am I looking for?
Need solution to use the two factor authentication token from google authenticator.
I want the selenium to be opened up as a tab in the existing instance of the firefox browser, where I will have already logged in beforehand. Therefore no login (should be) required and the "scraping" and downloading can be done.
If there's no direct way, maybe someone comes up with a workaround?
I know selenium cannot download the files itself, as the popups are no longer part of the browser. I'll fix that when I get there.
Important Side-Notes:
Firefox is not a given! I'll gladly accept a solution for any browser.
Here is the code that will read the google authenticator token and used in the login. Used js to open the new tab.
Install pyotp package before running the test code.
pip install pyotp
Test code:
from pyotp import *
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Firefox()
driver.get("https://sellercentral.amazon.de/listing/download?ref=ag_dnldinv_apvu_newapvu")
wait = WebDriverWait(driver,10)
# enter the email
email = wait.until(EC.presence_of_element_located((By.XPATH, "//input[#name='email']")))
email.send_keys("email goes here")
# enter password
driver.find_element_by_xpath("//input[#name='password']").send_keys("password goes here")
# click on signin button
driver.find_element_by_xpath("//input[#id='signInSubmit']").click()
#wait for the 2FA feild to display
authField = wait.until(EC.presence_of_element_located((By.XPATH, "xpath goes here")))
# get the token from google authenticator
totp = TOTP("secret goes here")
token = totp.now()
print (token)
# enter the token in the UI
authField.send_keys(token)
# click on the button to complete 2FA
driver.find_element_by_xpath("xpath of the button goes here").click()
# now open new tab
driver.execute_script("""window.open("https://sellercentral.amazon.de/listing/download?ref=ag_dnldinv_apvu_newapvu")""")
# continue with your logic from here
I'm working on trying to automate a game I want to get ahead in called pokemon vortex and when I login using selenium it works just fine, however when I attempt to load a page that requires a user to be logged in I am sent right back to the login page (I have tried it outside of selenium with the same browser, chrome).
This is what I have
import time
from selenium import webdriver
from random import randint
driver = webdriver.Chrome(r'C:\Program Files (x86)\SeleniumDrivers\chromedriver.exe')
driver.get('https://zeta.pokemon-vortex.com/dashboard/');
time.sleep(5) # Let the user actually see something!
usernameLoc = driver.find_element_by_id('myusername')
passwordLoc = driver.find_element_by_id('mypassword')
usernameLoc.send_keys('mypassword')
passwordLoc.send_keys('12345')
submitButton = driver.find_element_by_id('submit')
submitButton.submit()
time.sleep(3)
driver.get('https://zeta.pokemon-vortex.com/map/10')
time.sleep(10)
I'm using python 3.6+ and I literally just installed selenium today so it's up to date, how do I force selenium to hold onto cookies?
Using a pre-defined user profile might solve your problem. This way your cache will be saved and will not be deleted.
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("--user-data-dir=C:/Users/user_name/AppData/Local/Google/Chrome/User Data")
driver = webdriver.Chrome(options=options)
driver.get("xyz.com")
I am just beginner in Python, so I don't know much about it.
For my research project I have to get the friend list of users(already defined) from Facebook and Twitter by crawling webpages by using Python.
I don't how to start like open account then go to friend, save its webpage, then go to another webpage and do the same.
Can anyone please tell me how to do it?
Use the Google API.
https://towardsdatascience.com/how-to-use-facebook-graph-api-and-extract-data-using-python-1839e19d6999
Or use this link for Code
https://codereview.stackexchange.com/questions/167486/parser-for-facebook-friend-list
You May Can Use this Code of Python For that task get from the upper link...
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
class FacebookCrawler:
LOGIN_URL = 'https://www.facebook.com/login.php?login_attempt=1&lwv=111'
def __init__(self, login, password):
chrome_options = webdriver.ChromeOptions()
prefs = {"profile.default_content_setting_values.notifications": 2}
chrome_options.add_experimental_option("prefs", prefs)
self.driver = webdriver.Chrome(chrome_options=chrome_options)
self.wait = WebDriverWait(self.driver, 10)
self.login(login, password)
def login(self, login, password):
self.driver.get(self.LOGIN_URL)
# wait for the login page to load
self.wait.until(EC.visibility_of_element_located((By.ID, "email")))
self.driver.find_element_by_id('email').send_keys(login)
self.driver.find_element_by_id('pass').send_keys(password)
self.driver.find_element_by_id('loginbutton').click()
# wait for the main page to load
self.wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "a#findFriendsNav")))
def _get_friends_list(self):
return self.driver.find_elements_by_css_selector(".friendBrowserNameTitle > a")
def get_friends(self):
# navigate to "friends" page
self.driver.find_element_by_css_selector("a#findFriendsNav").click()
# continuous scroll until no more new friends loaded
num_of_loaded_friends = len(self._get_friends_list())
while True:
self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
try:
self.wait.until(lambda driver: len(self._get_friends_list()) > num_of_loaded_friends)
num_of_loaded_friends = len(self._get_friends_list())
except TimeoutException:
break # no more friends loaded
return [friend.text for friend in self._get_friends_list()]
if __name__ == '__main__':
crawler = FacebookCrawler(login='login', password='password')
for friend in crawler.get_friends():
print(friend)
You can use Facebook's Graph API to fetch friend lists for those who gave permissions to your website only if Facebook approves your website to access this data (You need to request permission). I think the chances of getting approval for a personal website is not very high.
Another way of getting this data is that crawling friend lists via an automated code or application. For this to work:
That person should be your friend on Facebook
That person's friend list should be open to friends (Some people change the privacy setting to "Only me")
You need their Facebook profile URLs
If everything is setup, the crawler goes to profile URLs one by one and visit friend lists to collect data.
Please note that crawling data on Facebook may cause legal issues depending on where you live.
You want to start by looking at the requests library.