Develop Reference

How to install Python and Selenium wed driver without having access to the internet

I need to install Python and Selenium webdriver on a PC that does not have access to the internet, and don't how to accomplished that.
Sofar, I managed to download Python onto a thumb drive and was able to install it on the PC. But when it comes to Selenium webdriver, I don't know what to do.
Please help.
Regards,
Kiet

You can create a virtual environment with all dependencies that you need and transfer the venv folder.

Selenium for Chromium in Python

I've created a Python script to collect data from different websites using Selenium. On my Windows PC the script works fine and does exactly what it's supose to do. Now I'm trying to make my script run on my Raspberyy Pi. On my PC I use Google Chrome with selenium but ofcourse Chrome is not supported on a Raspberry Pi. Instead I have to use Chromium but I struggle to make this work. I use the following code to start my driver:
import requests
from selenium import webdriver
session_requests = requests.session()
ser = Service(r"/home/pi/Downloads/chromedriver")
op = webdriver.ChromeOptions()
driver = webdriver.Chrome(service=ser, options=op)
With this code I get the following error:
OSError: [Errno 8] Exec format error: '/home/pi/Downloads/chromedriver'
This error is because my chromedriver is for Google Chrome and not Chromium. When I look for the driver for Chromium, it automatically gives me the driver for Google Chrome instead.
I'm using Chromium version 98.0.4758.106. Is there a driver for Chromium or should I look for another solution? I found a work-around to download Google Chrome on my Raspberry Pi, but this does not look like it should be done. Any suggestions are appreciated, thanks in advance!

After a long search I found a solution for my own problem. People from the Raspbian project have compiled a chromium-chromedriver version for the armhf platform and added it to the repo. The following command line will add the Chromium-driver and make it ready to use:
sudo apt-get install chromium-chromedriver
With this solution you will no longer need to give the path to the driver.
Source: https://ivanderevianko.com/2020/01/selenium-chromedriver-for-raspberrypi

Not able to open website in google colab using selenium webdriver

I am trying to open website website in google colab using selenium but not able to do so. Installing chrome driver and setting path variable had issues, so I used Kora library. I am able to to execute wd.get('website') successfully and able to print source_code as well. Kindly let me know how to open website. Below is my code
!pip install kora -q
from kora.selenium import wd
wd.get('https://www.google.com')
wd.page_source

You can show the screen by display the wd.
display(wd)
What exactly do you want to do?

How to scrape without web browser driver?

I made a web scrape program using selenium.
This program is access target URL and download a file.
After updating Chrome, program does not work because chromedriver is old version.
How to do web scraping and file download not use chromedriver?
Thank all for reading.

I think it would be easier if you also updated ChromeDriver, that way your program would work again. Or you could install the previous Chrome version again.
But if you don't want that, you can use GeckoDriver with Firefox.

You can use a headless scrapper like Pupeppeteer. You also can update your Chrome driver to be compatible with your browser version that is the most recomended.

How to run Selenium Scripts in webservers?

I have written a few Selenium [Python] webdriver scripts, and while they are running fine on my system, they are not running on my website's server. It is showing errors with Firefox. Firefox is installed on the server. The webserver is Ubuntu. What do I have to do to run those scripts? Please help, I am a newbie.

Selenium requires a running browser and browsers need some kind of X server in order to run. There are many types of X servers and one of them is Xvfb aka X virtual framebuffer that does all the operations in memory and thus requires no screen.
In Wikipedia you could find very nice examples.
This is a nice example too.

You probably need to open the browser headless when executing your scripts on the server.
Here is the Java code for Firefox (Python code should be similar):
import java.io.File;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.firefox.FirefoxBinary;
WebDriver openHeadless() throws Exception
{
FirefoxBinary binary = new FirefoxBinary(new File("/usr/local/bin/firefox"));
binary.setEnvironmentProperty("DISPLAY",System.getProperty("lmportal.xvfb.id",":99"));
return new FirefoxDriver(binary,null); // or 'binary,profile' if you have a profile
}
Make sure that you have Firefox installed on the server at /usr/local/bin/firefox.

If you want simply do web browser testing you can use libraries like Casper JS this creates a server side browser for web browser testing, it doesn't need a display driver.

I ended up using pyvirtualdisplay which is a Python wrapper for Xvfb and Xephyr. If anyone is having same issues [and I am sure newbies will], You can try THIS. Also you can use xvfbwrapper. Tutorial for xvfbwrapper HERE.