I'm trying to make a program that downloads pdfs after I've done a search. The web is an aspx and with selenium I can enter the information in the fields correctly:
input_user=driver.find_element(By.XPATH, '//*[#name="ctl00$SPWebPartManager1$g_93bc4c3a_0f69_4097_bed1_978c8b545335$freetext"]')
textolibre="jamaica"
input_user.send_keys(textolibre)
but the page that returns the results is the same (aspx) and I can't download the pdfs.
I would like to be able to enter the fields in the form without selenium opening a browser. I tried with PhantomJS but it says:
UserWarning: Selenium support for PhantomJS has been deprecated, please use headless versions of Chrome or Firefox instead
And when you return the results, I can download them, considering that the url doesn't change.
While using PhantomJS just add 2 extra lines to your code
import warnings
warnings.filterwarnings("ignore")
Or else if you are using gekodriver for the firefox browser you can use this in your code
opts = webdriver.FirefoxOptions()
opts.headless = True
driver = webdriver.Firefox(options=opts, executable_path='Your gekdriver exe path')
If you could share a more detailed complete code i could help you more effeciently
Related
I unfortunately can not stop a page from loading using Selenium in Python.
I have tried:
driver.execute_script("window.stop();")
driver.set_page_load_timeout(10)
webdriver.ActionChains(driver).send_keys(Keys.ESCAPE).perform()
The page is a .cgi that constantly loads. I would like to either scrape data from a class on the page or the page title, however neither works with the 3 methods above.
When I try to manually press ESC, or click the cross, it works perfectly.
Thank you for reading.
You didn't share your code and a page you are working on, so we can only guess.
So, in case you really tried all the above correctly and it still not helped try adding Eager page loading strategy to your driver options.
Eager page loading strategy will make WebDriver wait until the initial HTML document has been completely loaded and parsed, and discards loading of stylesheets, images and subframes (DOMContentLoaded event fire is returned).
With it your code will look something like this:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.page_load_strategy = 'eager'
driver = webdriver.Chrome(options=options)
# Navigate to url
driver.get(your_page_url)
UPD
You are trying to upload a file with Selenium and doing it wrong.
To upload the file with Selenium you need to send a full file path to that element.
So, if the file you want to upload is located by C:/Model.lp your code should be:
driver.find_element_by_xpath("//input[#name='field.1']").send_keys("C:/Model.lp")
I am using Chrome/Chromedriver version: 93.0.4577 and Selenium version: 3.141.0 on Win 10. I'm trying to get Selenium to download PDF's without opening the file prompt.
I'm working with a work site unfortunately so I'm unable to share it but I've tried the libraries: urllib, request, as well as adjusting the chrome Options (which I'd like to use):
chrome_profile = webdriver.ChromeOptions()
profile = {"plugins.plugins_list": [{"enabled": False,
"name": "Chrome PDF Viewer"}],
"download.default_directory": "C:\\PDFDownload\\PDFs",
"download.prompt_for_download": False,
"download.directory_upgrade": True}
chrome_profile.add_experimental_option("prefs", profile)
driver = webdriver.Chrome(executable_path="C:\\Selenium\\chromedriver.exe",
chrome_options=chrome_profile)
I've also tried switching the chrome PDF site settings from 'Open PDFs in Chrome' to 'Download PDFs' when visiting them. However, everything still prompts me to save the file manually.
I feel like the chrome Options should be the winning ticket but it doesn't seem to have worked for me. Is there something I need to change/add in order to get past the pop up prompt?
I appreciate any advice, thank you!
You can try the following:
First get the 'href' or download link of the PDF you're looking for. There must be some button or link which you might want to catch.
use self.driver.get(the_link_you_extracted_in_step_1) and the PDF should be automatically downloaded
I am working on an automation task that opens a webpage, screenshots an element from the page, then closes the page and moves on to the next. I had previously set this up using AutoHotKey and, although it worked technically, I wanted to create a more refined version. Selenium has worked well up to now for automating the navigation of pages, but when I would be ready to take my screenshot, I can't seem to get the console open to issue the command. I tried using the
from selenium.webdriver.common.keys import Keys
browser.find_element_by_tag_name('body').send_keys(Keys.CONTROL + Keys.SHIFT + 'K')
command but it seems that firefox doesn't receive the input. I also tried
from selenium.webdriver.common.action_chains import ActionChains
actions = ActionChains(browser)
actions.send_keys(Keys.CONTROL + Keys.SHIFT + 'k')
which, again, didn't seem to work.
Lastly I tried using
browser.execute_script(":screenshot")
but I kept getting a JavaScript error which makes sense since the screenshot command isn't js.
If there's anything you can think of that I am overlooking please let me know! Thanks in advance :)
Selenium screenshot
Example
You can take a screenshot of a webpage with the method get_screenshot_as_file() with as parameter the filename.
The program below uses firefox to load a webpage and take a screenshot, but any web browser will do.
from selenium import webdriver
from time import sleep
driver = webdriver.Firefox()
driver.get('https://www.python.org')
sleep(1)
driver.get_screenshot_as_file("screenshot.png")
driver.quit()
print("end...")
The screenshot image will be stored in the same directory as your Python script. Unless you explicitly define the path where the screenshot has to be stored.
I am trying to crawl a website (with python) and get its users info. But when I download the source of the pages, it is different from what I see in inspect element in chrome. I googled and it seems I should use selenium, but I don't know how to use it. This is the code I have and when I see the driver.page_source it is still the source page as in chrome and doesn't look like the source in inspect element.
I really appreciate if someone can help me to fix this.
import os
from selenium import webdriver
chromedriver = "/Users/adam/Downloads/chromedriver"
os.environ["webdriver.chrome.driver"] = chromedriver
driver = webdriver.Chrome(chromedriver)
driver.get("http://www.tudiabetes.org/forum/users/Bug74/activity")
driver.quit()
It's called XHR.
Your page was loaded from another call, (your url only loads the strcuture of the page, and the meat of the page comes from a different source using XHR, json formatted string) not the pageload it self.
You should really consider using requests and bs4 to query this page instead.
I'm searching for a python module that simulates a webbrowser and can handle html frames. I want to use the chatbot Brain http://www.thebot.de/ with python. If you know any tutorials that explain how to use your suggested module in relation with forms and frames give me a links to them please.
Tim,
I suggest you take a look at Selenium. By default it opens and manipulates Firefox to navigate the web. Its primary use case is testing, however in a pinch I've used it in some of my scripts to get past sites with a lot of javascript or, as in this case, iframes.
The basic usage is:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
driver.get("http://www.python.org")
assert "Python" in driver.title
elem = driver.find_element_by_name("q")
elem.send_keys("pycon")
elem.send_keys(Keys.RETURN)
driver.close()
To access an iframe you can use the following code:
elm = driver.find_element_by_tag_name("iframe")
driver.switch_to.frame(elm)
Then when you want to switch out of the iframe:
driver.switch_to.default_content()
You can even utilize Selenium headless with xvfbwrapper like this (code from github.com/cgolberg/xvfbwrapper):
from xvfbwrapper import Xvfb
vdisplay = Xvfb()
vdisplay.start()
# launch stuff inside virtual display here
vdisplay.stop()