Develop Reference

Use Selenium with Brave Browser pass service object written in python

#TLDR I want to use brave browser with selenium written in python but can't find any current solutions that work.
This code works
from selenium import webdriver
option = webdriver.ChromeOptions()
option.binary_location = r'C:\Program Files\BraveSoftware\Brave-
Browser\Application\brave.exe'
driver = webdriver.Chrome(executable_path=r'C:\WebDrivers\chromedriver.exe',
options=option)
driver.get("https://www.google.com")
driver.quit()
but executable_path is deprecated:
C:\Users\USER\PycharmProjects\pythonProject\sol2.py:5:
DeprecationWarning: executable_path has been deprecated, please pass in a Service object
driver = webdriver.Chrome(executable_path=r'C:\WebDrivers\chromedriver.exe', options=option)
Found this on youtube: https://www.youtube.com/watch?v=VMzmVFA-Gps
# import statements
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
# Declare variables and setup services
driverService = Service('C:/webdrivers/chromedriver.exe')
# 1. Passes the chromedriver path to the service object
# 2. stores the service object in the s variable
driver = webdriver.Chrome(service=driverService)
# 1. Passes service object driverSerice into the webdriver.Chrome
# 2. Stores object in driver variable
# Body (actually doing stuff)
driver.maximize_window() # maximizes the browser window
driver.get("https://www.google.com") # navigates to google.com
myPageTitle = driver.title
# gets the title of the web page stores in myPageTitle
print(myPageTitle) # prints myPageTitle to Console
assert "Google" in myPageTitle
# checks myPageTitle to ensure it contains Google
# clean up
driver.quit() # closes the browser
When I run this code I get:
selenium.common.exceptions.WebDriverException: Message: unknown error: cannot find Chrome binary
This code works as long as you allow Google Chrome onto your PC. I don't want Chrome on my PC.
The problem is that I can't figure out how to get selenium to use brave instead of Chrome.
As of this writing I am using the following:
Windows 11 Home
Selenium v4.0.0
Python v3.10
ChromeDriver 95.0.4638.69
Brave Browser Version 1.31.91 Chromium: 95.0.4638.69 (Official Build) (64-bit)
Can some one please explain how to make this work with the current (read nondeprecated) code on brave browser? Thanks for your time.

To initiate a brave browsing context you need to:
Use the binary_location attribute to point to the brave binary location.
Use the chromedriver executable to initiate the brave browser.
Code block:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
option = webdriver.ChromeOptions()
option.binary_location = r'C:\Program Files (x86)\BraveSoftware\Brave-Browser\Application\brave.exe'
driverService = Service('C:/Users/.../chromedriver.exe')
driver = webdriver.Chrome(service=driverService, options=option)
driver.get("https://www.google.com")
Note: The DeprecationWarning: executable_path has been deprecated is a harmless warning message which doesn't affects your test execution and you can still ignore it.
References
You can find a couple of relevant detailed discussions in:
How to use Brave web browser with python, selenium and chromedriver?
How to initiate Brave browser using Selenium and Python on Windows

you have to set your path to brave binary.
options.setBinary("Path to brave.exe");
Go through this website:
https://mundrisoft.com/tech-bytes/how-to-execute-selenium-script-in-brave-browser/

Selenium is not loading TikTok pages

I'm implementing a TikTok crawler using selenium and scrapy
start_urls = ['https://www.tiktok.com/trending']
....
def parse(self, response):
options = webdriver.ChromeOptions()
from fake_useragent import UserAgent
ua = UserAgent()
user_agent = ua.random
options.add_argument(f'user-agent={user_agent}')
options.add_argument('window-size=800x841')
driver = webdriver.Chrome(chrome_options=options)
driver.get(response.url)
The crawler open Chrome but it does not load videos.
Image loading
The same problem happens also using Firefox
No loading page using Firefox
The same problem using a simple script using Selenium
from selenium import webdriver
import time
driver = webdriver.Firefox()
driver.get("https://www.tiktok.com/trending")
time.sleep(10)
driver.close()
driver = webdriver.Chrome()
driver.get("https://www.tiktok.com/trending")
time.sleep(10)
driver.close()

Did u try to navigate further within the selenium browser window? If an error 404 appears on following sites, I have a solution that worked for me:
I simply changed my User-Agent to "Naverbot" which is "allowed" by the robots.txt file from Tik Tok
(Robots.txt)
After changing that all sites and videos loaded properly.
Other user-agents that are listed under the "allow" segment should work too, if you want to add a rotation.

You can use Windows IE. Instead of chrome or firefox
Videos will load in IE but IE's Layout of showing feed is somehow different from chrome and firefox.
Reasons, why your page, is not loading.
Few advance web apps check your browser history, profile data and cached to check the authentication of the user.
One other thing you can do is run your default profile within your selenium It would be helpfull.

How to open a fully functioning chrome browser using selenium WebDriver with python?

When I'm trying to open a web page, its opening in a new chrome window stripped of all the extensions and modules. I'm not able to emulate the certain behavior of the website using selenium chrome browser window but I'm able to do the same thing in a normal chrome window without any issues.
from selenium import webdriver
driver = webdriver.Chrome(r'C:\chromedriver.exe')
driver.get("remote_worksplace_link")
id_box = driver.find_element_by_id('Enter user name')
id_box.send_keys('123456')
pass_box = driver.find_element_by_id('passwd')
pass_box.send_keys('123abc')
login_button = driver.find_element_by_id('Log_On')
login_button.click()
driver.implicitly_wait(2)
launch_button = driver.find_element_by_class_name('storeapp-icon ui-sortable-handle')
launch_button.click()
driver.implicitly_wait(5)
driver.close()

all extentions has its .crx file just you need to add those path
chrome_options = Options()
chrome_options.add_extension('path_to_extension')
driver = webdriver.Chrome(executable_path=executable_path, chrome_options=chrome_options)
driver.get("url")
driver.quit()

Selenium Python - Webdriver

I am using selenium to crawl a javascript website, the issue is that, a Firefox browser opens up, but the call for the URL is not done. however, when I close the browser, it is then that call for URL is done and of course I get the missing driver exception. what do you think the issue comes from.
knowing that:
all programs are up-to-date
my solution works fine, in local, but when I try to deploy it on the server, I start having issues
Example: at my local machine, I run this script and everything goes smooth, however when I run it a server (Linux), only the browser opens up and no get URL is called
from selenium import webdriver
import time
geckodriver_path = r'.../geckodriver'
driver = webdriver.Firefox(executable_path= geckodriver_path)
time.sleep(3)
driver.get("http://www.stackoverflow.com")

I end up finding the solution :
from selenium import webdriver
import time
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
geckodriver_path = r'/path_to/geckodriver'
binary = FirefoxBinary(r'/usr/bin/firefox')
capabilities = webdriver.DesiredCapabilities().FIREFOX
capabilities["marionette"] = False
driver = webdriver.Firefox(firefox_binary=binary,
executable_path= geckodriver_path,
capabilities=capabilities)
time.sleep(3)
driver.get("https://stackoverflow.com/")
time.sleep(6)
driver.close()
# solution from:
# https://github.com/SeleniumHQ/selenium/issues/3884
# https://stackoverflow.com/questions/25713824/setting-path-to-firefox-binary-on-windows-with-selenium-webdriver

Using selenium at hosted app?

I want to click one button with my Django app that I have hosted at DigitalOcean.
Here it is how I do it offline:
import selenium.webdriver as webdriver
firefox = webdriver.Firefox()
firefox.get("http://www.hltv.org/match/2296366-gplay-gamers2-acer-predator-masters-powered-by-intel")
element = firefox.find_element_by_id("voteteam1")
element.click()
But can I use it online? Maybe there is other solution?

You will need to use firefox as headless on a Linux box. The following articles should help -
http://www.installationpage.com/selenium/how-to-run-selenium-headless-firefox-in-ubuntu/

If you tied up to Firefox or any other browser "with a head", the common approach is to start a "Virtual Display" with the help of PyVirtualDisplay which is a wrapper around Xvfb, Xephyr and Xvnc, see this answer for an example working code.
Another option would be to use a "headless" browser, such as PhantomJS. In this case, the change is usually very simple, replacing:
firefox = webdriver.Firefox()
with:
driver = webdriver.PhantomJS()
Assuming you have PhantomJS installed.
Demo:
>>> from selenium import webdriver
>>> driver = webdriver.PhantomJS()
>>> driver.get("http://www.hltv.org/match/2296366-gplay-gamers2-acer-predator-masters-powered-by-intel")
>>> driver.title
u'HLTV.org - Hot Match: GPlay vs Gamers2'
The third option (mine most favorite) would be use a remote selenium server, either your own, or provided by third-party services like BrowserStack or Sauce Labs. Example code:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
desired_cap = {'os': 'Windows', 'os_version': 'xp', 'browser': 'IE', 'browser_version': '7.0' }
driver = webdriver.Remote(
command_executor='http://username:key#hub.browserstack.com:80/wd/hub',
desired_capabilities=desired_cap)
driver.get("http://www.google.com")
if not "Google" in driver.title:
raise Exception("Unable to load google page!")
elem = driver.find_element_by_name("q")
elem.send_keys("BrowerStack")
elem.submit()
print driver.title
driver.quit()
In case of BrowserStack or Sauce Labs you have an enormous amount of browsers and operating systems to choose from. Note that these are not free services and you would need a username and a key for this code to work.