Develop Reference

Getting to my profile in Facebook for a self web scrapping project using selenium

I am trying to get to my Facebook profile using selenium for a self-web scraping project I am working on. I have already figured out how to log in, but every time I have selenium click on the profile button it sends me back to the homepage. I am using a chrome web driver btw.
I have tried this so far.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
import os
import wget
import time
# bypass alerts
chrome_options = webdriver.ChromeOptions()
prefs = {"profile.default_content_setting_values.notifications" : 2}
chrome_options.add_experimental_option("prefs", prefs)
# specify the path to chromdriver.exe
# will have to be changed on each computer
driver = webdriver.Chrome('-enter path to chrome driver here-', chrome_options=chrome_options)
# open the webpage
driver.get("https://www.facebook.com")
# target username
username = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[name='email']")))
password = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[name='pass']")))
#enter username and password
username.clear()
username.send_keys("username")
password.clear()
password.send_keys("password")
#target the login button and click it
button = WebDriverWait(driver, 2).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[type='submit']"))).click()
# We are logged in
# target account button and click it
accountButton = WebDriverWait(driver, 2).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div[aria-label='Account']"))).click()
# target profile pick and click it
profileButton = WebDriverWait(driver, 2).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a[role='link']"))).click()
It does click the profile button but I just get sent back to the home page.

Script will not locate Twitter's password field

In this super basic python script using Selenium, I am just trying to automate my twitter login so I can begin scraping. When the chrome session opens, the username is filled out, but the password field is left blank.
import bs4
from selenium import webdriver
driver = webdriver.Chrome();
url = "https://twitter.com/login"
driver.get(url)
assert "Twitter" in driver.title
username = driver.find_element_by_class_name('js-username-field')
username.send_keys('example_username')
password = driver.find_element_by_class_name('js-password-field')
password.clear()
password.send_keys('exmaple_password')
login_button = driver.find_element_by_css_selector("button.submit.EdgeButton.EdgeButton--primary.EdgeButtom--medium")
login_button.submit()

It's possible you need to add a wait on the password field. This can help if you are seeing intermittent issues with your script. I prefer to wait on an element rather than sleep.
import bs4
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome();
url = "https://twitter.com/login"
driver.get(url)
assert "Twitter" in driver.title
username = driver.find_element_by_class_name('js-username-field')
username.send_keys('example_username')
# wait on password field
WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'js-password-field')))
password = driver.find_element_by_class_name('js-password-field')
password.clear()
password.send_keys('exmaple_password')
login_button = driver.find_element_by_css_selector("button.submit.EdgeButton.EdgeButton--primary.EdgeButtom--medium")
login_button.submit()
Python moves through steps pretty quickly, and it takes a while for web pages to catch up sometimes. Adding a wait helps slow things down a bit, and can fix some intermittent errors you might be seeing.

How to execute all javascript content on webpage with selenium to find and send login form info on fully loaded webpage

I've been trying to make a Python script to login into a certain website, navigate through the menu, fill out a form and save the file it generates to a folder.
I've been using Selenium trying to make the website fully load so i can find the elements for the login, but i'm being unsucessful, maybe because the website does a lot of JavaScript content before it fully loads, but i can't make it fully load and show me the data i want.
I tried Robobrowser, Selenium, Requests and BeautifulSoup to get it done.
import requests
from bs4 import BeautifulSoup
from selenium import webdriver
url = "https://directa.natal.rn.gov.br/"
driver = webdriver.Chrome(executable_path="C:\\webdrivers\\chromedriver.exe")
driver.get(url)
html = driver.execute_script("return document.documentElement.outerHTML")
sel_soup = BeautifulSoup(html, 'html.parser')
senha = driver.find_element_by_xpath('//*[#id="senha"]')
senha.send_keys("123")
I expected to have filled the password (senha) field with "123" but i can't even find the element.

It seems like what's needed here is a little bit of a scroll, wait and switch, incase the login fields just aren't ready for input :) The below should work, whereby we actually scroll to the element, having switch to the iframe, before we interact with the rest of the login form. You're able to adjust the delay from 5 seconds to anything of your preference.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
""" Variables """
url = "https://directa.natal.rn.gov.br/"
delay = 5 # seconds
""" Initiate driver """
driver = webdriver.Chrome(executable_path="C:\\webdrivers\\chromedriver.exe")
""" Go to url """
driver.get(url)
""" Iframe switch """
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"frame[name='mainsystem'][src^='main']")))
""" Attempt to get all our elements """
try:
""" Username """
usuario = WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.ID, 'usuario')))
""" Password """
senha = WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.ID, 'senha')))
print("All elements located!")
except TimeoutException:
print("Loading took too much time!")
exit(0)
"""Scroll to our element """
driver.execute_script("arguments[0].scrollIntoView();", usuario)
""" Input data into our fields """
usuario.send_keys("username")
senha.send_keys("password")
""" Locate our login element """
login = WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.ID, 'acessar')))
""" Click Login """
login.click()

To send the character sequence 123 to the password (senha) field, as the the desired elements are within a <frame> so you have to:
Induce WebDriverWait for the desired frame to be available and switch to it.
Induce WebDriverWait for the desired element to be clickable.
You can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_argument('disable-infobars')
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://directa.natal.rn.gov.br/")
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"frame[name='mainsystem'][src^='main']")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input.input[name='usuario']"))).send_keys("Tads")
driver.find_element_by_css_selector("input.input[name='senha']").send_keys("123")
Browser Snapshot:
Here you can find a relevant discussion on Ways to deal with #document under iframe

http 405 while login with selenium

I'm trying to use auto login in to a site using selenium through pyhton but it throws
http: 405 error "Pardon our Interruption. something about your browser made us think you were a bot"
What can I do to avoid it? I would like to see the execution live to check if the code is working correctly which I can't do if I use it in headless mode. Am I wrong?
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
usernameStr = 'bucmi1#yandex.com'
passwordStr = 'pmz4'
browser = webdriver.Chrome()
browser.get(('https://www.milanuncios.com/mis-anuncios/'))
# fill in username and strike a subsequent button
username = WebDriverWait(browser, 10).until(
EC.presence_of_element_located((By.ID, 'email')))
username.send_keys(usernameStr)
# wait for transition then continue to fill items
password = WebDriverWait(browser, 10).until(
EC.presence_of_element_located((By.ID, 'contra')))
password.send_keys(passwordStr)
signInButton = browser.find_element_by_class_name('submit btnSend')
signInButton.click()
Thanks in advance.

It is not clear why you got an error message as :
http: 405 error "Pardon our Interruption. something about your browser made us think you were a bot"
But I was able to successfully able to login through the credentials you provided with your own code adding some minor tweaks through chrome.options in incognito mode as follows :
Code Block :
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
usernameStr = 'bucmi1#yandex.com'
passwordStr = 'pmz4'
options = Options()
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
options.add_argument('--incognito')
browser = webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
browser.get(('https://www.milanuncios.com/mis-anuncios/'))
# fill in username and strike a subsequent button
username = WebDriverWait(browser, 10).until(EC.presence_of_element_located((By.ID, 'email')))
username.send_keys(usernameStr)
# wait for transition then continue to fill items
password = WebDriverWait(browser, 10).until(EC.presence_of_element_located((By.ID, 'contra')))
password.send_keys(passwordStr)
signInButton = browser.find_element_by_css_selector("div.btnEnviarFrm>input.submit.btnSend[value^='INICIAR']")
signInButton.click()
Browser Snapshot :

Thanks for the answers! The problem had something to do with my default profile folder. Don't know exactly what since there are many files I don't understand. But once I created a new one using options.add_argument("user-data-dir=C:\\dir\\of\\example\\profile")
could access without problems.

Python selenium issue: able to find element but not send_keys

I am trying to login into the ESPN footytips website so that I can scrape information for one of my leagues.
I am having no issues opening an instance of Chrome and navigating to the homepage (which contains the login form) and can even select the username field but I cannot for the life of me send my login details to the form.
In debugging I know I can find and select the form submit button and the issue seems to be in passing my login details using send_keys as my exception rule always triggers after I attempt call send_keys.
Any suggestions on how to resolve would be welcomed! My script is below:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
login_address = "http://www.footytips.com.au/home"
me_login = "test#test.com"
me_password = "N0TMYR3#LP#S5W0RD"
browser = webdriver.Chrome()
browser.get(login_address)
try:
login_field = browser.find_element_by_id("ft_username")
password_field = browser.find_element_by_id("ft_password")
print("User login fields found")
login_field.send_keys(me_login)
password_field.send_keys(me_password)
print("Entered login data")
submit_button = browser.find_element_by_id("signin-ft")
print("Submit button found")
submit_button.submit()
except:
print("Error: unable to enter form data")

The locators you have used doesn't uniquely identifies the login_field and the password_field. Additionally you need to wait for the respective WebElements to be visible. Here is your own code with some tweaks :
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
#lines of code
login_address = "http://www.footytips.com.au/home"
me_login = "test#test.com"
me_password = "N0TMYR3#LP#S5W0RD"
browser.get(login_address)
login_field = WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.XPATH,"//div[#class='login-form']//input[#id='ft_username']")))
password_field = browser.find_element_by_xpath("//div[#class='login-form']//input[#id='ft_password']")
login_field.send_keys(me_login)
password_field.send_keys(me_password)
print("Entered login data")