Develop Reference

Trying to read data from Kaspersky website with MechanicalSoup or Selenium

Currently, I'm trying to scrape data from a Website (https://account.kaspersky.com/). Before I can read the data I need to login to the website.
But for some reason, it is not working. I read through the internet to get it to work, but unfortunately, I wasn't able to solve the issue.
import mechanicalsoup
import csv
import xlsxwriter
from time import sleep
# create stateful browser
browser = mechanicalsoup.StatefulBrowser(
soup_config={'features': 'lxml'},
raise_on_404=True,
user_agent='MyBot/0.1: mysite.example.com/bot_info',
)
# use browser to open link
browser.open("https://account.kaspersky.com/")
sleep(2)
# check url
print(browser.get_url())
# get first form available
form = browser.select_form()
browser.submit()
# check url
print(browser.get_url())
The script always ends at the selct_form() method. No matter what I try I always get the same error. Even when I specify it.
Traceback (most recent call last):
File "c:\Python\Cloud.Kaspersky\read_from_website_mechanicalsoup.py", line 23, in <module>
form = browser.select_form()
File "C:\Users\dw.FROMMEDV\AppData\Local\Programs\Python\Python39\lib\site-packages\
mechanicalsoup\stateful_browser.py", line 220, in select_form
raise LinkNotFoundError()
mechanicalsoup.utils.LinkNotFoundError
After a few hours of trying, I wanted to try a different tool. Selenium. But I have kind of the same problem here as well. But here I can't submit the login. Selenium can't find the button.
from selenium import webdriver
import time
driver = webdriver.Firefox()
driver.get("https://account.kaspersky.com/")
file = open('login.txt', 'r')
username_file = file.readline()
password_file = file.readline()
time.sleep(1)
username = driver.find_element_by_id("EMail")
username.clear()
username.send_keys(username_file)
time.sleep(1)
password = driver.find_element_by_name("Password")
password.clear()
password.send_keys(password_file)
time.sleep(2)
driver.find_element_by_class_name("assets-button primary").click()
Is it possible that this website is protected or something? Or is anyone seeing my issue?
Here the Error Message with Selenium:
Traceback (most recent call last):
File "c:\Python\Cloud.Kaspersky\read_from_website_selenium.py", line 26,
in
<module>
driver.find_element_by_class_name("assets-button primary").click()
File "C:\Users\dw.FROMMEDV\AppData\Local\Programs\Python\Python39
\lib\site-
packages\selenium\webdriver\remote\webdriver.py", line 564, in
find_element_by_class_name
return self.find_element(by=By.CLASS_NAME, value=name)
File "C:\Users\dw.FROMMEDV\AppData\Local\Programs\Python\Python39
\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 976, in
find_element
return self.execute(Command.FIND_ELEMENT, {
File "C:\Users\dw.FROMMEDV\AppData\Local\Programs\Python\Python39
\lib\site-
packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Users\dw.FROMMEDV\AppData\Local\Programs\Python\Python39
\lib\site-
packages\selenium\webdriver\remote\errorhandler.py", line 242, in
check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: Unable to
locate
element: .assets-button primary

Try using the following identifiers. The last line is the error using Java instead of Python and multiple class names instead of singular.
driver.find_element_by_class_name("assets-button").click()
driver.find_element_by_xpath("//button[#class='assets-button primary']").click()
You could also try
from selenium.webdriver.common.keys import Keys
password.send_keys(Keys.ENTER)

Updating facebook post using Selenium: error- element not found but i can find in my browser

i am trying to post on facebook wall using selenium in python. I am able to login but after login it cant find class name of status box which i copied from browser
here is my code-
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
user_name = "email"
password = "password"
msg = "hi i am new here"
driver = webdriver.Firefox()
driver.get("https://www.facebook.com")
element = driver.find_element_by_id("email")
element.send_keys(user_name)
element = driver.find_element_by_id("pass")
element.send_keys(password)
element.send_keys(Keys.RETURN)
time.sleep(5)
post_box = driver.find_element_by_class_name("a8c37x1j ni8dbmo4 stjgntxs l9j0dhe7")
post_box.click()
time.sleep(5)
post_box.send_keys(msg)
the snapshot of code i copied from browser is attached as image here
here is error i recived-
Traceback (most recent call last):
File "C:/Users/rosha/Desktop/facebook bot.py", line 17, in <module>
post_box = driver.find_element_by_class_name("a8c37x1j ni8dbmo4 stjgntxs l9j0dhe7")
File "C:\ProgramData\Anaconda3\envs\facebook bot.py\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 564, in find_element_by_class_name
return self.find_element(by=By.CLASS_NAME, value=name)
File "C:\ProgramData\Anaconda3\envs\facebook bot.py\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 976, in find_element
return self.execute(Command.FIND_ELEMENT, {
File "C:\ProgramData\Anaconda3\envs\facebook bot.py\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\ProgramData\Anaconda3\envs\facebook bot.py\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: .a8c37x1j ni8dbmo4 stjgntxs l9j0dhe7

try to find element by Xpath for example:
driver.find_element(By.XPATH, '//button[text()="Some text"]')
to find the xpath from the browser, just right click on something in the webpage and press inspect after that right click, a menu will appear, navigate to copy then another menu will appear, press copy fullpath.
check this https://selenium-python.readthedocs.io/locating-elements.html

The problem is that driver.find_element_by_class_name() can be used for one class, and not multiple classes as you have: a8c37x1j ni8dbmo4 stjgntxs l9j0dhe7 which are multiple classes separated by spaces.
Refer to the solution suggested here, it suggests using find_elements_by_xpath or find_element_by_css_selector.

Attempting to select a field for username and password using Selenium in Python

I'm trying to figure out how to select an ID in a website that has a username and password using selenium so that I can login using a python script. The problem is the fields on the website don't seem to have IDs for their username and password fields in the HTML code and I'm not really sure as to how to get the fields I need.
from selenium import webdriver
import time
#from selenium.webdriver.common.keys import Keys
link = "https://logistics.vendini.com/"
login = "e-mail"
password = "pass"
chromedriver = "D:\Downloads\chromedriver"
driver = webdriver.Chrome(chromedriver)
driver.get(link)
username = driver.find_element_by_class_name('form-control')
print(username)
I tried this just to see if finding it by class would work but all I get is the webpage opening and then I get this error
DevTools listening on ws://127.0.0.1:64343/devtools/browser/8a74989c-0f07-442c-ba50-077d3ec005bc
Traceback (most recent call last):
File "d:/marko/Programming/RavensHouseCup/webscrape.py", line 13, in <module>
username = driver.find_element_by_class_name('form-control')
File "D:\Program Files (x86)\Python\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 564, in find_element_by_class_name
return self.find_element(by=By.CLASS_NAME, value=name)
File "D:\Program Files (x86)\Python\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 976, in find_element
return self.execute(Command.FIND_ELEMENT, {
File "D:\Program Files (x86)\Python\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "D:\Program Files (x86)\Python\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".form-control"}
(Session info: chrome=79.0.3945.88)
If you go to the website that is in the link variable and inspect the email address and password fields is there something I'm missing as to how I'd be able to access them?
So I added an explicit wait of 10 seconds which seems to allow me to input the email, but for some reason the password isn't working. The code I'm using is as follows
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.XPATH, "/html/body/div/div/div/div/div/form/div[1]/input"))
)
username = driver.find_element_by_xpath("/html/body/div/div/div/div/div/form/div[1]/input")
password = driver.find_element_by_xpath("/html/body/div/div/div/div/div/form/div[2]/input")
username.send_keys(login)
password.send_keys(password)
The e-mail gets entered correctly, but the password is giving me an error saying
File "d:/marko/Programming/RavensHouseCup/webscrape.py", line 24, in <module>
password.send_keys(password)
File "D:\Program Files (x86)\Python\lib\site-packages\selenium\webdriver\remote\webelement.py", line 478, in send_keys
{'text': "".join(keys_to_typing(value)),
File "D:\Program Files (x86)\Python\lib\site-packages\selenium\webdriver\common\utils.py", line 150, in keys_to_typing
for i in range(len(val)):
TypeError: object of type 'WebElement' has no len()

When the page first loads the element is not present, there appears to be some JS that loads the form. You need to wait for the element to be present
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver, 10)
username = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'form-control')))
print(username)

I am the stupidest coder alive. I realized my mistake, I had 2 different variables both named password, sorry everyone, I'm a moron. I changed the actual password to the website to be password2 and it all works!

How to edit chromes search and homepage with selenium/python?

I am trying to edit chrome browsers search and homepage using selenium/python. After navigating to chrome://settings/searchEngines and targeting the 'add' button with the ID 'addSearchEngine', I get an error when I run a .click function. How do I target this element correctly, or is there another way to update chromes search/startpage with python?
I'm guessing this element is trapped inside an iframe but I'm unable to find one on the page using the dev tools, xpath noted the following about the absolute xpath: "It might be a child of iframe from different src & it is not supported currently."
from selenium import webdriver
driver = webdriver.Chrome()
driver.set_page_load_timeout(10)
driver.get("chrome://settings/searchEngines")
driver.find_element_by_id("addSearchEngine").click()
Traceback (most recent call last):
File "C:/Users/Jonathan/PycharmProjects/test_project/test_project/Main.py", line 20, in <module>
driver.find_element_by_id("addSearchEngine").click()
File "C:\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 360, in find_element_by_id
return self.find_element(by=By.ID, value=id_)
File "C:\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 978, in find_element
'value': value})['value']
File "C:\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Python37-32\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="addSearchEngine"]"}
(Session info: chrome=75.0.3770.142)

chrome://settings/searchEngines has Shadow DOM elements . You will need to use the driver.execute_script() to get handle of the shadowRoot element and ultimately get to the 'addSearchEngine' element.
Example python : shadowRoot python
Example Java : For chrome://downloads/ shadowRoot java

Refer to this post for detailed explanation.
In your case you can do the below.
url = "chrome://settings/searchEngines"
driver.get(url)
addButton = driver.execute_script("return document.querySelector('settings-ui')"
".shadowRoot.querySelector('#main')"
".shadowRoot.querySelector('settings-basic-page.showing-subpage')"
".shadowRoot.querySelector('settings-search-page')"
".shadowRoot.querySelector('settings-search-engines-page')"
".shadowRoot.querySelector('#addSearchEngine')")
addButton.click()
Screenshot:

Python Selenium cannot find an element on a page by xpath

I am using python and selenium to test some things with fantasy football. Here is my code so far (I just started).
import time
from selenium import webdriver
driver = webdriver.Chrome('C:\\Users\\202300Fontenot\\Desktop\\3\\chromedriver.exe')
driver.get('http://games.espn.com/ffl/signin?redir=http%3A%2F%2Fgames.espn.com%2Fffl%2Fclubhouse%3FseasonId%3D2018%26leagueId%3D49607%26teamId%3D4');
driver.implicitly_wait(10)
time.sleep(10)
search_box = driver.find_element_by_xpath('//*[#id="did-ui-view"]/div/section/section/form/section/div[1]/div/label/span[2]/input')
search_box.send_keys('email#icloud.com')
search_box.submit()
time.sleep(5)
driver.quit()
This just tries to enter an email address into the box. I am getting this error every time:
Traceback (most recent call last):
File "C:\Users\202300Fontenot\Desktop\3\ESPN.py", line 8, in <module>
search_box = driver.find_element_by_xpath('//*[#id="did-ui-view"]/div/section/section/form/section/div[1]/div/label/span[2]/input')
File "C:\Users\202300Fontenot\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 393, in find_element_by_xpath
return self.find_element(by=By.XPATH, value=xpath)
File "C:\Users\202300Fontenot\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 966, in find_element
'value': value})['value']
File "C:\Users\202300Fontenot\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 320, in execute
self.error_handler.check_response(response)
File "C:\Users\202300Fontenot\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[#id="did-ui-view"]/div/section/section/form/section/div[1]/div/label/span[2]/input"}
(Session info: chrome=69.0.3497.92)
(Driver info: chromedriver=2.42.591088 (7b2b2dca23cca0862f674758c9a3933e685c27d5),platform=Windows NT 10.0.16299 x86_64)
Thanks for your help.

Your problem is that the input you are trying to select exists inside of an iframe. You must first tell the driver to switch to the iframe and then execute the xpath selection.
driver.switch_to.frame("disneyid-iframe")
driver.find_element_by_xpath("//input[#type='email']").send_keys('email#icloud.com')

First of all you need to switch to your iframe using its name:
disneyid-iframe
Also if XPATH is not the only choice, you may use the CSS Selector:
div.field-username-email input[type='email']
This will get the driver to find & fill your field.

Using XPath in python You need to provide the string for
driver.find_element_by_xpath
Example Code below
("//*[#id=\"did-ui-view\"]/div/section/section/form/section/div[1]/div/label/span[2]/input")
Please refer below document regarding String
String Gude first Para