Develop Reference

elements from selenium does not show specific documents

not very familiar with interacting with page source code, I was trying to get the table on a stock screening website, but I just can't seem to get the table in the #document from inspect element to show up in any of my attempts.
the table I'm trying to reach
the inspect element
just the table it seems
Then I tried with selenium:
from time import sleep
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
username = 'yemingsang'
password = MY_PASSWORD
url = 'https://www.esignal.com/members/login'
options = Options()
options.add_experimental_option("detach",True)
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()),options=options)
driver.get(url)
sleep(1)
driver.find_element(By.NAME,'user').send_keys(username)
driver.find_element(By.NAME,'password').send_keys(password)
driver.find_element(By.CSS_SELECTOR,'input[type=\"SUBMIT\" i]').click()
sleep(1)
get_source = driver.page_source
print(get_source)
does not have the #document element either using selenium

When I start my program (Selenium) it opens web page then it closes immediately after whithout error in log

My code is very short. When I try it, the program opens web page but program closes immediately after.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import Select
from selenium.common.exceptions import NoSuchElementException
driver = webdriver.Chrome()
driver.get("https://google.com")
I believe the problem is coming from my environment, so I have not tried any changes to the code to resolve this.

You have to add the below chrome options:
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_experimental_option("detach", True)
driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()), options=options)
driver.get("https://google.com")

Wait for cookie with Selenium python webdriver

I basically here the same question as here, only with Python.
I want to use Selenium to get the user to log in with a browser, then grab the session cookie that will contain the log in information, and finally close the browser. In an interactive session, it's easy to just wait until the authentication is complete before calling get_cookie(). But in run mode, I'm stuck.
From the Selenium documentation, I get that it is about defining a wait strategy. So I tried to repurpose their code example as follows:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from selenium.webdriver.support.ui import WebDriverWait
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()))
driver.get('https://examplecom/')
sessionId = WebDriverWait(driver, timeout=120).until(lambda d: d.get_cookie('session_cookie')['value'])
driver.quit()
Unfortunately, the following error is immediately raised:
TypeError: 'NoneType' object is not subscriptable
What should I do instead?

Duh! I was calling the value key as part of the wait function, thus even before it was available! This works:
sessionId = WebDriverWait(driver, timeout=120).until(lambda d: d.get_cookie('TK'))['value']

To save cookies in Selenium, you can do as follows:
import os
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
expanduser = os.path.expanduser('~')
options = webdriver.ChromeOptions()
and then add option of user-data-dir:
options.add_argument(f'--user-data-dir={expanduser}\\AppData\\Local\\Google\\Chrome\\any_Name_You_Want')
and then create your browser with these option:
browser = webdriver.Chrome(ChromeDriverManager().install(), options=options)

Why is scrolling page down in selenium not working?

I have the problem with scroll page with selenium and, to be precise, with scrolling to the bottom of the page. This two line of code does not work for me. Who can explain, why?:
browser.execute_script("window.scrollBy(0, document.body.scrollHeight)")
browser.execute_script("window.scrollTo(0, document.body.scrollHeight)")
This is my properties for WebDriver:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
browser = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
But this options works, but I need to scroll page without connect to some element:
flag = browser.find_element(By.XPATH, ".....")
browser.execute_script("arguments[0].scrollIntoView();", flag)

You can find the body and try:
body = driver.find_element_by_xpath('/html/body')
body.send_keys(Keys.PAGE_DOWN)
You can find more information, e.g. here:
enter link description here

Selenium send_keys not working on wappalyzer?

I'm using Selenium to automate searching on wappalyzer.com. Below is my code:
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
def handleWappalyzer(driver):
driver.find_element(By.CSS_SELECTOR, "div[class*='v-select__selections']").click()
search = driver.find_element(By.CSS_SELECTOR, "div[class*='v-text-field__slot'] > input[type='text']")
search.send_keys("https://www.facebook.com//")
if __name__ == '__main__':
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get("https://www.wappalyzer.com/")
handleWappalyzer(driver)
driver.quit()
However, whenever I run the script, the key is not getting written in the search bar. What did I do wrong?

#Warrier
The locator you have provided for search looks incorrect.
Can you try this "#input-360"
Also, add some wait before it.