I unfortunately can not stop a page from loading using Selenium in Python.
I have tried:
driver.execute_script("window.stop();")
driver.set_page_load_timeout(10)
webdriver.ActionChains(driver).send_keys(Keys.ESCAPE).perform()
The page is a .cgi that constantly loads. I would like to either scrape data from a class on the page or the page title, however neither works with the 3 methods above.
When I try to manually press ESC, or click the cross, it works perfectly.
Thank you for reading.
You didn't share your code and a page you are working on, so we can only guess.
So, in case you really tried all the above correctly and it still not helped try adding Eager page loading strategy to your driver options.
Eager page loading strategy will make WebDriver wait until the initial HTML document has been completely loaded and parsed, and discards loading of stylesheets, images and subframes (DOMContentLoaded event fire is returned).
With it your code will look something like this:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.page_load_strategy = 'eager'
driver = webdriver.Chrome(options=options)
# Navigate to url
driver.get(your_page_url)
UPD
You are trying to upload a file with Selenium and doing it wrong.
To upload the file with Selenium you need to send a full file path to that element.
So, if the file you want to upload is located by C:/Model.lp your code should be:
driver.find_element_by_xpath("//input[#name='field.1']").send_keys("C:/Model.lp")
Related
I am trying to web scrape a dynamically loaded page with Selenium. I can copy and paste the below url into a normal Chrome browser and it works perfectly fine but when I use selenium, it return the wrong page of horse races for a different day. It seems to work the first time you run the code but retains some sort of memory and you cannot run it again with a different date as it just returns the original date?
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
url = "https://www.tab.com.au/racing/meetings/2021-06-11"
driver = webdriver.Chrome('xxxxxxxx')
driver.get(url)
Has anyone every come across something like this with Selenium?
I am working on an automation task that opens a webpage, screenshots an element from the page, then closes the page and moves on to the next. I had previously set this up using AutoHotKey and, although it worked technically, I wanted to create a more refined version. Selenium has worked well up to now for automating the navigation of pages, but when I would be ready to take my screenshot, I can't seem to get the console open to issue the command. I tried using the
from selenium.webdriver.common.keys import Keys
browser.find_element_by_tag_name('body').send_keys(Keys.CONTROL + Keys.SHIFT + 'K')
command but it seems that firefox doesn't receive the input. I also tried
from selenium.webdriver.common.action_chains import ActionChains
actions = ActionChains(browser)
actions.send_keys(Keys.CONTROL + Keys.SHIFT + 'k')
which, again, didn't seem to work.
Lastly I tried using
browser.execute_script(":screenshot")
but I kept getting a JavaScript error which makes sense since the screenshot command isn't js.
If there's anything you can think of that I am overlooking please let me know! Thanks in advance :)
Selenium screenshot
Example
You can take a screenshot of a webpage with the method get_screenshot_as_file() with as parameter the filename.
The program below uses firefox to load a webpage and take a screenshot, but any web browser will do.
from selenium import webdriver
from time import sleep
driver = webdriver.Firefox()
driver.get('https://www.python.org')
sleep(1)
driver.get_screenshot_as_file("screenshot.png")
driver.quit()
print("end...")
The screenshot image will be stored in the same directory as your Python script. Unless you explicitly define the path where the screenshot has to be stored.
I'm trying to make a program that downloads pdfs after I've done a search. The web is an aspx and with selenium I can enter the information in the fields correctly:
input_user=driver.find_element(By.XPATH, '//*[#name="ctl00$SPWebPartManager1$g_93bc4c3a_0f69_4097_bed1_978c8b545335$freetext"]')
textolibre="jamaica"
input_user.send_keys(textolibre)
but the page that returns the results is the same (aspx) and I can't download the pdfs.
I would like to be able to enter the fields in the form without selenium opening a browser. I tried with PhantomJS but it says:
UserWarning: Selenium support for PhantomJS has been deprecated, please use headless versions of Chrome or Firefox instead
And when you return the results, I can download them, considering that the url doesn't change.
While using PhantomJS just add 2 extra lines to your code
import warnings
warnings.filterwarnings("ignore")
Or else if you are using gekodriver for the firefox browser you can use this in your code
opts = webdriver.FirefoxOptions()
opts.headless = True
driver = webdriver.Firefox(options=opts, executable_path='Your gekdriver exe path')
If you could share a more detailed complete code i could help you more effeciently
I am trying to use Selenium to log into a printer and conduct some tests. I really do not have much experience with this process and have found it somewhat confusing.
First, to get values that I may have wanted I opened the printers web page in Chrome and did a right click "view page source" - This turned out to be not helpful. From here I can only see a bunch of <script> tags that call some .js scripts. I am assuming this is a big part of my problem.
Next I selected the "inspect" option after right clicking. From here I can see the actual HTML that is loaded. I logged into the site and recorded the process in Chrome. With this I was able to identify the variables which contain the Username and Password. I went to this part of the HTML did a right click and copied the Xpath. I then tried to use the Selenium find_element_by_xpath but still no luck. I have tried all the other methods to (find by ID, and name) however it returns an error that the element is not found.
I feel like there is something fundamental here that I am not understanding. Does anyone have any experience with this???
Note: I am using Python 3.7 and Selenium, however I am not opposed to trying something other than Selenium if there is a more graceful way to accomplish this.
My code looks something like this:
EDIT
Here is my updated code - I can confirm this is not just a time/wait issue. I have managed to successfully grab the first two outer elements but the second I go deeper it errors out.
def sel_test():
chromeOptions = Options()
chromeOptions.add_experimental_option("useAutomationExtension", False)
browser = webdriver.Chrome(chrome_options=chromeOptions)
url = 'http://<ip address>/'
browser.get(url)
try:
element = WebDriverWait(browser, 10).until(EC.presence_of_element_located((By.XPATH, '//*[#id="ccrx-root"]')))
finally: browser.quit()
The element that I want is buried in this tag - Maybe this has something to do with it? Maybe related to this post
<frame name="wlmframe" src="../startwlm/Start_Wlm.htm?arg11=">
As mentioned in this post you can only work with the current frame which is seen. You need to tell selenium to switch frames in order to access child frames.
For example:
browser.switch_to.frame('wlmframe')
This will then load the nested content so you can access the children
Your issue is most likely do to either the element not loading on the page until after your bot searches for it, or a pop-up changing the xpath of the element.
Try this:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
delay = 3 # seconds
try:
elementUsername = WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.xpath, 'element-xpath')))
element.send_keys('your username')
except TimeoutException:
print("Loading took too much time!")
you can find out more about this here
I am trying to crawl a website (with python) and get its users info. But when I download the source of the pages, it is different from what I see in inspect element in chrome. I googled and it seems I should use selenium, but I don't know how to use it. This is the code I have and when I see the driver.page_source it is still the source page as in chrome and doesn't look like the source in inspect element.
I really appreciate if someone can help me to fix this.
import os
from selenium import webdriver
chromedriver = "/Users/adam/Downloads/chromedriver"
os.environ["webdriver.chrome.driver"] = chromedriver
driver = webdriver.Chrome(chromedriver)
driver.get("http://www.tudiabetes.org/forum/users/Bug74/activity")
driver.quit()
It's called XHR.
Your page was loaded from another call, (your url only loads the strcuture of the page, and the meat of the page comes from a different source using XHR, json formatted string) not the pageload it self.
You should really consider using requests and bs4 to query this page instead.