I need to open multiple links in separate tabs or sessions...
I already know how to do it, so what i would like to know is if it's possible to connect to an already open webpage instead of open every links every time i run the script.
What i used now in Python is:
from selenium import webdriver
driver.get(link)
The purpose would be once i run the first script (to load multiple links), the second should connect to the webpages, refresh them and continue with the code.
Is it possible? Anyone know how to do it?
Thanks a lot for the help!!!!
Connecting to the previously opened window is easy:
driver = webdriver.Firefox()
url = driver.command_executor._url
session_id = driver.session_id
driver2 = webdriver.Remote(command_executor=url,desired_capabilities={})
driver2.session_id = session_id
#You're all set to do whatever with the previously opened browser
driver2.get("http://www.stackoverflow.com")
Related
So I am trying to login programatically (python) to https://www.datacamp.com/users/sign_in using my email & password.
I have tried 2 methods of login. One using requests library & another using selenium (code below). Both time facing [403] issue.
Could someone please help me login programatically to it ?
Thank you !
Using Requests library.
import requests; r = requests.get("https://www.datacamp.com/users/sign_in"); r (which gives <response [403]>)
Using Selenium webdriver.
driver = webdriver.Chrome(executable_path=driver_path, options=option)
driver.get("https://www.datacamp.com/users/sign_in")
driver.find_element_by_id("user_email") # there is supposed to be form element with id=user_email for inputting email
Implicit wait at least should have worked, like this:
from selenium import webdriver
driver = webdriver.Chrome(executable_path='/snap/bin/chromium.chromedriver')
driver.implicitly_wait(10)
url = "https://www.datacamp.com/users/sign_in"
driver.get(url)
driver.find_element_by_id("user_email").send_keys("test#dsfdfs.com")
driver.find_element_by_css_selector("#new_user>button[type=button]").click()
BUT
The real issue is the the site uses anti-scraping software.
If you open Console and go to request itself you'll see:
It means that the site blocks your connection even before you try to login.
Here is similar question with different solutions: Can a website detect when you are using Selenium with chromedriver?
Not all answers will work for you, try different approaches suggested.
With Firefox you'll have the same issue (I've already checked).
You have to add a wait after driver.get("https://www.datacamp.com/users/sign_in") before driver.find_element_by_id("user_email") to let the page loaded.
Try something like WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, 'user_email')))
I need to scrape the CSV result of the website (https://requestmap.webperf.tools/). But I can’t.
This is the process I need to do:
1- load the website (https://requestmap.webperf.tools/)
2- enter a new website as an input (for example, https://stackoverflow.com/)
3- click submit
4- in the new page which opens after the submit, download the csv file at the end of the page
But I think the driver has the main page and doesn’t switch to the new page. That's why I can’t download the CSV file.
Would you please tell me how to do it?
here is my code:
options = webdriver.ChromeOptions()
options.add_argument('-headless')
options.add_argument('-no-sandbox')
options.add_argument('-disable-dev-shm-usage')
driver = webdriver.Chrome('chromedriver',options=options)
driver.get("https://requestmap.webperf.tools/")
driver.find_element_by_id("url").send_keys("https://stackoverflow.com/")
driver.find_element_by_id("submitter").click()
I have tested different ways to solve my issue:
First of all:
I got this code from here :
But it doesn't work.
window_after = driver.window_handles[1]
driver.switch_to.window(window_after)
I also tried this or this, but they are not working as well.
# wait to make sure there are two windows open
# it is not working
WebDriverWait(driver, 30).until(lambda d: len(d.window_handles) == 2)
# switch windows
# it is not working
driver.switch_to_window(driver.window_handles[1])
content = driver.page_source
soup = BeautifulSoup(content)
driver.find_elements_by_name("Download CSV")
So How can I get this issue solved?
Is there any other way in python to do so and switch to a new windows?
from selenium import webdriver
import random
url = "https://www.youtube.com/"
list_of_drivers = [webdriver.Firefox(), webdriver.Chrome(), webdriver.Edge()]
Driver = random.choice(list_of_drivers)
Driver.get(url)
I'm trying to cycle though a list of random webdrivers using selenium.
It does a good job at picking a random webdriver and opening the URL however, it also opens up other webdrivers with a blanck page.
How do I stop this from happening?
I am running python 2.7 in a virtualenv.
list_of_drivers = [webdriver.Firefox(), webdriver.Chrome(), webdriver.Edge()]
You created three instances already in this line, that's why all 3 browsers show up with a blank page at the very beginning.
Driver = random.choice(list_of_drivers)
Driver.get(url)
And then you randomly choose one to open a webpage, leaving the rest doing nothing.
Instead of creating three instances, just create one:
list_of_drivers = ['Firefox', 'Chrome', 'Edge']
Driver = getattr(webdriver, random.choice(list_of_drivers))()
Driver.get(url)
Here is my problem: I'm trying to use selenium to access a webpage and the special about this page is it is an auto redirecting page (you open that page and after few seconds, it automatically redirect to another page). When i use driver = webdriver.Firefox(), my IDM catched that link just perfectly after few seconds.
And because i don't want the browser to come up so i use Phantomjs instead, ut it not working. My application just can get the loading page url (bitdl-1336...) but not the redirected link. Please help!
This is my code:
link = 'http://torrent.ajee.sh/hash.php?hash=' + self.global_hash_code
driver = webdriver.PhantomJS('phantomjs.exe')
driver.get(str(link))
element = driver.find_element_by_link_text('Download Zip')
element.click()
time.sleep(10)
msg = QMessageBox.information(self, QString('Thành công'),QString(driver.current_url))
And this is the result:
Please help!
Sorry about my english
Not exactly an answer to your PhantomJS-specific question, but a workaround to the problem.
And because i don't want the browser to come up so i use Phantomjs instead
You can continue using Firefox, but start it in a Virtual Display, see more information at:
How do I run Selenium in Xvfb?
You may also need to let the browser automatically save the archive in a specified directory, see:
How do I automatically download files from a pop up dialog using selenium-python
Access to file download dialog in Firefox
I'm using Python to gather some information, construct a very simple html page, save it locally and display the page in my browser using webbrowser.open('file:///c:/testfile.html'). I check for new information every minute. If the information changes, I rewrite the local html file and would like to reload the displayed page.
The problem is that webbrowser.open opens a new tab in my browser every time I run it. How do I refresh the page rather than reopen it? I tried new=0, new=1 and new=2, but all do the same thing. Using controller() doesn't work any better.
I suppose I could add something like < META HTTP-EQUIV="refresh" CONTENT="60" > to the < head > section of the html page to trigger a refresh every minute whether or not the content changed, but would prefer finding a better way.
Exact time interval is not important.
Python 2.7.2, chrome 26.0.1410.64 m, Windows 7 64.
If you're going to need a refresh on the same tab, you'll need selenium webdriver.
After installing selenium using pip, you can use the following code :
from selenium import webdriver
import time
import urllib
import urllib2
x = raw_input("Enter the URL")
refreshrate = raw_input("Enter the number of seconds")
refreshrate = int(refreshrate)
driver = webdriver.Firefox()
driver.get("http://"+x)
while True:
time.sleep(refreshrate)
driver.refresh()
This will open the URL and refresh the tab every refreshrate seconds
I use pyautogui module to refresh the browser page. It's one liner:
import pyautogui
pyautogui.hotkey('f5') #Simulates F5 key press = page refresh
Keep it very short, as simple as:
from selenium import webdriver
import time
driver = webdriver.Firefox()
driver.get('URL')
while True:
time.sleep(20)
driver.refresh()
driver.quit()
It looks like several people have asked this in the past but here is a link that sums it up.
Python refresh HTML document
But webbrowser.open( url, new=0 ) should open the page in the current window and not initialize a new one.
The LivePage extension for Chrome. You can write to a file, then LivePage will monitor it for you. You can also optionally refresh on imported content like CSS. Chrome will require that you grant permissions on local file:// urls.
(I'm unaffiliated with the project.)
Or, you can use the easy auto refresh extension in chrome :D
It allows you to set your own refresh time, fastest being one second!
I have had same issues as you for some reason new=1 , new=0 doesn't work. U may try the code below. It simply open the url, waits for 10 sec then kill the process n repeat.
from subprocess import Popen
import time
while True:
link = ('https://www.google.com/')
Popen(['start', 'chrome', link], shell=True)
time.sleep(10)
Popen('taskkill /F /IM chrome.exe', shell=True)
After using pip install to install pyautogui use the following:
import pyautogui
import time
for i in range(number of times you want to refresh):
time.sleep(refreshrate in seconds)
pyautogui.hotkey('f5')