I am attempting to scrape websites using selenium-python. I am trying to use firefox driver (since PhantomJS doesn't work on me), but is there way to block the pop-ups when using the firefox driver?
thank you
Even I turn on "disable pop ups" in FireFox manually, it's not working for all website. A work around is switch to pop up window, close it, and switch back to the original window.
Here is the code in python:
# Switch to new window opened
driver.switch_to.window(driver.window_handles[-1])
# Close the new window
driver.close()
# Switch back to original browser (first window)
driver.switch_to.window(driver.window_handles[0])
Sorry for the late answer, but you can use the dom.popup_maximum profile preference, setting it to 0 like this:
from selenium import webdriver
fp = webdriver.FirefoxProfile()
fp.set_preference("dom.popup_maximum", 0)
driver = webdriver.Firefox(firefox_profile=fp)
url = "http://some.url.with.annoying.popups"
driver.get(url)
For a list of other available options, please refer to this answer
Use xvfb in the framebuffer and here is a simple usage.
It provides an X environment for selenium.
Related
I am trying to automate a process with Selenium, and am having troubles figuring out how to switch between open windows while the program is running.
After clicking on the button, it opens another website that has a separate url, which is unique each time it is opened. I need to switch Selenium from interacting with the original website to this new popup within the browser, caused by the original website. The new window shows that it is also controlled by Chromedriver with the bit at the top that says "Chrome is being controlled by automated test software." Additionally, the actual website opened will be the same, just the fine print after the '.com/' is different.
How would I go about doing this? Also, how would I switch back? (If this is even possible)
For example:
driver=webdriver.Chrome(service=s)
driver.get("https://originalwebsite.com/")
driver.find_element(By.XPATH, 'buttons-xpath').click()
# (popup opens up now)
# *switch to popup website here*
driver.find_element(By.XPATH, 'button-on-new-website-xpath').click()
driver.find_element(By.XPATH, 'second-button-on-new-website-xpath').click()
# *popup website closes*
# switch back to original website / window
Thanks!
I have tried to use driver.navigate in a variety of ways but generally have no clue what I am doing. Thanks again!
The comment from ALex Break led to the answer.
All I had to do was:
handles = driver.window_handles
driver.switch_to.window(handles[x])
#handles[x] is the index of the list handles that has the handle I want to switch
#to stored in it
Firefox (recently) has decided that we need to see an obnoxious list of downloads at the top right of the browser, even if we do not ask for it. There are work-arounds, which I have tried (going to about:config in the browser and adjusting settings). I have also tried this in my code itself:
from splinter import Browser
browser = Browser()
browser.driver.profile.DEFAULT_PREFERENCES['frozen']["dom.disable_open_during_load"] = True
browser.driver.profile.DEFAULT_PREFERENCES['frozen']['dom.webnotifications.enabled'] = False
browser.driver.firefox_profile.set_preference("browser.download.alwaysOpenPanel", False)
None of the above works. At this point, if I can just get Splinter to click on the "Display the progress of ongoing downloads" button at the top right, my life would be much easier. Is this possible?
I'm using Firefox 104.0.2, Splnter 0.16.0, and Python 3.10.6. I also have Selenium 3.141.0.
I am currently working on a auto log-on for a website, but I found that selenium opens a new browser window each time I run the script. I want to make it so that it opens a new tab in the browser window I am already using. Selenium opens a new browser window, like so:
However, I want it to detect a window I am already using, and place a new tab there, like so:
Can we do this with Selenium for Python or do I need another language?
Sorry if my question is a bit stupid.
Thanks!
There are no stupid questions.
You need to find the reason why your selenium is opening new browser each time you log in.
It is probably because you are initialize webdriver each time you run that method.
Or maybe you use some hooks or tags which are triggering creation of new webdriver.
The code form other user from this site:
driver.find_element_by_tag_name('body').send_keys(Keys.CONTROL + 't')
driver.find_element_by_tag_name('body').send_keys(Keys.CONTROL + Keys.TAB)
windows = driver.window_handles
time.sleep(3)
driver.switch_to.window(windows[1])
I am trying to click on open application alert using Selenium, and I am getting this error
NoAlertPresentException: Message: no such alert
So basically I am trying to open zoom application from the browser
And here is my code:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
driver = webdriver.Chrome(executable_path='chromedriver/chromedriver')
driver.get("https://zoom.us/j/93459172503?pwd=QkhnMEQ0ZTRZd0grUVJkT2NudmlFZz09")
try:
WebDriverWait(driver, 5).until(EC.alert_is_present(), 'Timed out waiting for alerts to appear')
alert = driver.switch_to.alert
alert.accept()
print("alert accepted")
except TimeoutException:
print("no alert")
Thanks to NetworkMeister, I end up using default app method, Like the following:
1- Open Firefox and go to zoom URL, when Launch application appear choose zoom and click on remember my choice and click open link
2- Go to firefox and paste this about:support then search for Profile Folder and copy the path
3- Go to your code and add profile parameter to selenium driver and use the path you have copied
fp = webdriver.FirefoxProfile('C:/Users/ASUS//AppData/Roaming/Mozilla/Firefox/Profiles/0rgewd47.default-release')
driver = webdriver.Firefox(executable_path='geckodriver', firefox_profile=fp)
driver.get('https://zoom.us/j/93459172503?pwd=QkhnMEQ0ZTRZd0grUVJkT2NudmlFZz09')
Now zoom application will open automatically when you execute this code
Note: Firefox profile contains all of your browsing data, If you want to share your code I suggest creating a new Firefox profile, For more information look here
Because this is not a browser Alert, rather OS App selector, you cannot interact with it within Selenium.
See: Selenium C# How to handle Alert "Open Pick an app"?
You can prevent these App selectors by default by using the --disable-default-apps flag when starting up Chrome.
I encountered this problem my self.
Apparently this is related to the alert being OS level rather than being at a browser level.
The simplest solution I found is https://pypi.org/project/PyAutoGUI/ which allows you to pass an img of the button you want to click and then it locates it on the screen. You can call python directly from java.
Native java solution: https://docs.oracle.com/javase/7/docs/api/java/awt/Robot.html
this allows you to automate stuff like "move mouse to position" -> "mouse click" etc.
EDIT: the main downside is that the webdriver cannot be run in headless mode as PyAutoGUI makes a screenshot of the screen to locate the button.
From what I noticed is that when you open any such link in the browser the URL changes a bit. With this you can figure out the link for opening it in the browser. For example I have this zoom meeting
https://us04web.zoom.us/ w/ 76919011107?tk=5Q_zikLZhvWlqc_nzVcYaHoTyo7JuDY6cvLB9y9t0zc.DQIAAAARDLr3IxZHM21mSHFzYlR6Q0xZdnNhcnUwbUV3AAAAAAAAAAAAAAAAAAAAAAAAAAAA&pwd=QANkcTdsNlpjYWY4czZvd3FHV0NLQT09
Now this would open a popup like you showed in the browser. (This link is not real. I changed 2-3 characters in the URL for security reasons). But now if you try this link
https://us04web.zoom.us/ wc/join/ 76919011107?tk=5Q_zikLZhvWlqc_nzVcYaHoTyo7JuDY6cvLB9y9t0zc.DQIAAAARDLr3IxZHM21mSHFzYlR6Q0xZdnNhcnUwbUV3AAAAAAAAAAAAAAAAAAAAAAAAAAAA&pwd=QANkcTdsNlpjYWY4czZvd3FHV0NLQT09
This will directly open the meeting in the browser. Note the change in URL has been shown in bold (I had to add spaces because of that on both sides). Now this may change over time but fundamentally by comparing the 2 links you should find a way to achieve this.
Similarly for Microsoft Teams if you add "_#" after the domain you can join the meeting through the browser
For example if I want to open this link in the browser
https://teams.microsoft.com/l/meetup-join/19%3ameeting_NTRlM2JiZWUtZjNiMC00ZjVhLTlmMWEtZDcxYjBmYjdhY2Nl%40thread.v2/0?context=%7b%22Tid%22%3a%22e85f2c00-2730-4ca5-b8d8-609b15bd4746%22%2c%22Oid%22%3a%223bf1c992-96b1-4a87-808f-dcc5bb2009c9%22%7d
I'll have to write
https://teams.microsoft.com/_#/l/meetup-join/19%3ameeting_NTRlM2JiZWUtZjNiMC00ZjVhLTlmMWEtZDcxYjBmYjdhY2Nl%40thread.v2/0?context=%7b%22Tid%22%3a%22e85f2c00-2730-4ca5-b8d8-609b15bd4746%22%2c%22Oid%22%3a%223bf1c992-96b1-4a87-808f-dcc5bb2009c9%22%7d
These patterns can be found by observing the link before and after opening it in the browser.
This is not the best way but I resolved the issue by sending keyboard commands. It works on Windows, I haven't tested on different os.
import pyautogui
sleep(3) # sleep until pop up shown
pyautogui.press('tab', presses=2) # navigate to open button
pyautogui.press('enter') # open application
For me using Robot class worked for teams.
Robot robot = new Robot();
robot.keyPress(KeyEvent.VK_ENTER);
I want to use selenium with Python to open multi-tabs in one browser and scraping the real-time betting odds simultaneously with multi-tabs.
The website home page generate a list of games. However, there is no way to get the link of game unless you find the game element and use click()(the website is ajax heavy), which open the game in the same tab. My solution to open multi-tabs is to get the list of game then manually open new tab with home-page first loaded and then click on the game with different index in the list. However, I find the driver.window_handles array always include only one item, which is the current tab instead of all the tabs I opened manually in the browser.
Can anybody tell me what goes wrong or if you can give a better solution to this issue?
The problem is simplified as the code in following:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
# create a new Firefox session
driver_temp = webdriver.Firefox()
driver_temp.implicitly_wait(30)
driver_temp.get("https://www.google.com")
body = driver_temp.find_element_by_tag_name('body')
# manually open second tab
body.send_keys(Keys.CONTROL + 't')
driver_temp.get("https://www.google.com")
body = driver_temp.find_element_by_tag_name('body')
# manually open third tab
body.send_keys(Keys.CONTROL + 't')
driver_temp.get("https://www.google.com")
body = driver_temp.find_element_by_tag_name('body')
#print the number of window_handles
print len(driver_temp.window_handles)
I have opened 3 tabs, however the len(driver_temp.window_handles) is always 1
Selenium does not provide an API to manipulate browser tabs. You've probably noticed that applying the CTRL/COMMAND+T "hack" to open a new tab.
See more at:
Controlling firefox tabs in selenium
Opening a new tab in the same window session of the browser through selenium web driver command?
Instead, open up new browser windows.
Well, to be fair, it is important to mention that the behavior is quite different in Firefox and in Chrome - if you open new tabs in Chrome, selenium would see each tab as a window with it's own handle and you'll switch between them using switch_to.window() easily.