I've created a Python script to collect data from different websites using Selenium. On my Windows PC the script works fine and does exactly what it's supose to do. Now I'm trying to make my script run on my Raspberyy Pi. On my PC I use Google Chrome with selenium but ofcourse Chrome is not supported on a Raspberry Pi. Instead I have to use Chromium but I struggle to make this work. I use the following code to start my driver:
import requests
from selenium import webdriver
session_requests = requests.session()
ser = Service(r"/home/pi/Downloads/chromedriver")
op = webdriver.ChromeOptions()
driver = webdriver.Chrome(service=ser, options=op)
With this code I get the following error:
OSError: [Errno 8] Exec format error: '/home/pi/Downloads/chromedriver'
This error is because my chromedriver is for Google Chrome and not Chromium. When I look for the driver for Chromium, it automatically gives me the driver for Google Chrome instead.
I'm using Chromium version 98.0.4758.106. Is there a driver for Chromium or should I look for another solution? I found a work-around to download Google Chrome on my Raspberry Pi, but this does not look like it should be done. Any suggestions are appreciated, thanks in advance!
After a long search I found a solution for my own problem. People from the Raspbian project have compiled a chromium-chromedriver version for the armhf platform and added it to the repo. The following command line will add the Chromium-driver and make it ready to use:
sudo apt-get install chromium-chromedriver
With this solution you will no longer need to give the path to the driver.
Source: https://ivanderevianko.com/2020/01/selenium-chromedriver-for-raspberrypi
Related
I am using Selenium to open a certain website (for example YouTube) on the server, but it can't seem to open the website. However, the code works just fine with a different website. This code also works fine without any problems on my local PC.
I don't know if I have problems with my Chrome Driver or Selenium but it can't open youtube.com as it only outputs: "Before getting the website" and that's it. There are no exceptions/errors that are shown but the script still runs and I have to manually end stop it.
Why can't Selenium open certain URLs on the server, but it works fine on my PC?
options = webdriver.ChromeOptions()
options.add_argument("no-sandbox")
options.add_argument('--headless')
options.add_argument("--start-maximized")
PATH = "./chromedriver"
global driver
driver = webdriver.Chrome(PATH, chrome_options=options)
print("Before getting the website")
driver.get("https://youtube.com")
print("opened", driver.current_url)
I had a exactly same problem. Perhaps i don't know why this happened.
NOTE:
When you are scraping working on Ubuntu ec2 with no GUI you have to provide some GUI interface for chrome to run and for me Xvfb solved it.
"Xvfb (short for X virtual framebuffer) is an in-memory display server for UNIX-like operating system (e.g., Linux). It enables you to run graphical applications without a display (e.g., browser tests on a CI server) while also having the ability to take screenshots."
SOLUTION
Install Xvfb for ubuntu: sudo apt install xvfb
Now execute your script as: xvfb-run python[version] script.py
IMPORTANT NOTE:
If your program stuck during initialization does not show any output just make sure that you did not add chrome_option.add_argument("disable-dev-shm-usage").
If this argument is added in your chrome header then it would disable the /dev/shm. Not sure but it is some shared memory and xvfb is in-memory display server which i think need it.
This worked for me.
I write some code with python selenium using chrome driver as web driver. When I run code in my system ( mac os ), the code works. Chrome browser was opened, and selenium works correctly. But when I want to run my code in Centos 7 without GUI, selenium can't match with web driver and run the code. Although I install google-chrome-stable and use the same web driver for it, selenium can't run web driver.
driver = webdriver.Chrome(driver_path=chromedrive75)
So, I don't know how to fix the code and running on my centos operating system.
#Vahid, please check the chrome web browser version which is compatible with server, also try running in headless mode.
I'm trying to open a web browser using Selenium Chromedriver on Windows 10 with python in jupyter notebooks through an ubuntu command prompt. I've read stack overflow posts and tried to solve based on their answers, but I'm stuck in a loop where I keep receiving the same 3 errors.
Here is what I have installed:
OS - Windows 10, 1709, 64-bit Selenium - 3.8.1 Chromedriver - 2.45
Chrome - Version 71.0.3578.98 Python - 3.5.2
I tried various websites. The goal is to eventually get to a social media login page, but i'm stuck at opening a new blank web browser.
Here is my starting code:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get("https://www.imdb.com/")
WebDriverException: Message: 'chromedriver' executable may have wrong
permissions.
Then I tried the following:
from selenium import webdriver
chromedriver = "C:/Users/xxxx/AppData/Local/lxss/home/xxxx/chromedriver.exe"
browser = webdriver.Chrome(chromedriver)
browser.get('https://www.imdb.com/')
WebDriverException: Message: 'chromedriver.exe' executable needs to be
in PATH.
Here are the steps I have taken:
I added a PATH under environment variables to the folder - (C:\Users\xxxxx\AppData\Local\lxss\home\xxxx),
I tried using \, and /, and even \
Once I added the PATH, I tried the following two codes (and various versions) and received the same error:
A.
from selenium import webdriver
driver = webdriver.Chrome(executable_path=r'C:\Users\xxxx\AppData\Local\lxss\home\xxxx)
driver.get("https://www.imdb.com/")
B.
from selenium import webdriver
chromedriver = r'C:\Users\xxxx\AppData\Local\lxss\home\xxxx\chromedriver.exe'
driver = webdriver.Chrome(chromedriver)
driver.get("https://www.imdb.com/")
WebDriverException: Message:
'C:\Users\xxxx\AppData\Local\lxss\home\xxxx' executable may have wrong
permissions.
Then I did the following:
- Went to file Properties, under General, took off Read-only (Windows permissions)
- Went to file Properties, under Security and changed permissions to Full Control
- In the C:\Users\xxxxx\AppData\Local\lxss\home\xxxx file, I changed the permissions
using chmod 777 -R in my command prompt. Then I tried the following code:
from selenium import webdriver
import os
chromedriver = r'C:\Users\xxxx\AppData\Local\lxss\home\xxxx\chromedriver.exe'
driver = webdriver.Chrome(os.path.join(os.getcwd(), 'chromedriver.exe'))
driver.get("https://www.imdb.com/")
WebDriverException: Message: Service /home/ariggs/chromedriver.exe
unexpectedly exited. Status code was: 1
I am stuck between these three error messages. Does anyone have another suggestion for a beginner?
You can actually start Windows executables from a linux subsystem as it is described here https://learn.microsoft.com/en-us/windows/wsl/interop.
But you have to keep in mind that Selenium and ChromeDriver communicate over a network connection. Actually chromedriver starts its own http server and Selenium sends requests and receives responses over http. (see https://sqa.stackexchange.com/questions/28358/how-does-chromedriver-exe-work-on-a-core-and-fundamental-level)
According to Microsoft, WSL and Windows share the same IP address and network connections via localhost are supported. But in your case there seems to be a problem during the startup.
You can start a remote webdriver on windows with Python and connect to that.
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
import subprocess
subprocess.run(["d:\\develop\\remotewebdriver.cmd", ""])
driver = webdriver.Remote(
command_executor='http://localhost:4444/wd/hub',
desired_capabilities=DesiredCapabilities.CHROME)
driver.get('http://www.google.in/')
driver.close()
You need a windows script remotewebdriver.cmd for the remote webdriver that is called from Python:
SET JAVA_HOME=D:\develop\Java\jdk-11.0.2
d:
cd \develop
start D:\develop\Java\jdk-11.0.2\bin\java -Dwebdriver.chrome.driver=d:\develop\chromedriver.exe -jar selenium-server-standalone-3.141.59.jar
You have to adapt the path to your own environment. This setup works for me.
You can do this way.
Step 1:
Download chrome driver from this link (download the specific version as chrome):
http://chromedriver.chromium.org/downloads
Important: check your chrome version first. Go to help -> about Google Chrome, to see the version of your chrome.
Step 2:
After downloading extract and save the chromedriver file in a specific folder like, C:\selenium
go to environment variable and add a new path, C:\selenium
Step 3
Double click on the chromedriver application and then restart your command prompt. (If you are using conda environment.)
I am wondering if there is a way within py2app to include the Firefox browser, or if there is a way for Selenium to use Firefox without having to install Firefox on the host machine.
I have created an app with py2app that uses Selenium, however, I have Firefox installed on my machine, but not everyone that will receive the app will have Firefox installed. I am looking for a way to either include Firefox in the distribution or go around this.
Script will not run if Firefox is not preinstalled.
You can test your script with other browser, for example Chrome. If it works on Chrome also, then you can edit script like this:
from selenium.common.exceptions import WebDriverException
try:
driver = webdriver.Firefox()
except WebDriverException:
driver = webdriver.Chrome()
You can add same for few more browsers (IE, Opera, Safari...) to be sure that script will run on users machine
I have written a few Selenium [Python] webdriver scripts, and while they are running fine on my system, they are not running on my website's server. It is showing errors with Firefox. Firefox is installed on the server. The webserver is Ubuntu. What do I have to do to run those scripts? Please help, I am a newbie.
Selenium requires a running browser and browsers need some kind of X server in order to run. There are many types of X servers and one of them is Xvfb aka X virtual framebuffer that does all the operations in memory and thus requires no screen.
In Wikipedia you could find very nice examples.
This is a nice example too.
You probably need to open the browser headless when executing your scripts on the server.
Here is the Java code for Firefox (Python code should be similar):
import java.io.File;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.firefox.FirefoxBinary;
WebDriver openHeadless() throws Exception
{
FirefoxBinary binary = new FirefoxBinary(new File("/usr/local/bin/firefox"));
binary.setEnvironmentProperty("DISPLAY",System.getProperty("lmportal.xvfb.id",":99"));
return new FirefoxDriver(binary,null); // or 'binary,profile' if you have a profile
}
Make sure that you have Firefox installed on the server at /usr/local/bin/firefox.
If you want simply do web browser testing you can use libraries like Casper JS this creates a server side browser for web browser testing, it doesn't need a display driver.
I ended up using pyvirtualdisplay which is a Python wrapper for Xvfb and Xephyr. If anyone is having same issues [and I am sure newbies will], You can try THIS. Also you can use xvfbwrapper. Tutorial for xvfbwrapper HERE.