I have written a few Selenium [Python] webdriver scripts, and while they are running fine on my system, they are not running on my website's server. It is showing errors with Firefox. Firefox is installed on the server. The webserver is Ubuntu. What do I have to do to run those scripts? Please help, I am a newbie.
Selenium requires a running browser and browsers need some kind of X server in order to run. There are many types of X servers and one of them is Xvfb aka X virtual framebuffer that does all the operations in memory and thus requires no screen.
In Wikipedia you could find very nice examples.
This is a nice example too.
You probably need to open the browser headless when executing your scripts on the server.
Here is the Java code for Firefox (Python code should be similar):
import java.io.File;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.firefox.FirefoxBinary;
WebDriver openHeadless() throws Exception
{
FirefoxBinary binary = new FirefoxBinary(new File("/usr/local/bin/firefox"));
binary.setEnvironmentProperty("DISPLAY",System.getProperty("lmportal.xvfb.id",":99"));
return new FirefoxDriver(binary,null); // or 'binary,profile' if you have a profile
}
Make sure that you have Firefox installed on the server at /usr/local/bin/firefox.
If you want simply do web browser testing you can use libraries like Casper JS this creates a server side browser for web browser testing, it doesn't need a display driver.
I ended up using pyvirtualdisplay which is a Python wrapper for Xvfb and Xephyr. If anyone is having same issues [and I am sure newbies will], You can try THIS. Also you can use xvfbwrapper. Tutorial for xvfbwrapper HERE.
Related
I'm new to programming, this being my first post, but I was wondering if there was a way to make selenium run while my computer is turned off? I made an instagram bot with selenium webdriver python but it only runs when I run it from my computer and turns off when I turn off my computer. I have seen post saying to host it in PythonAnywhere which I tried but selenium requires a webdriver (im using firefox so gekodriver). Is there a way of fixing this or is there another way of making an insagram bot without selenium webdriver?
Thank you
You can use virtual machine and host a server and then through DevOps pipelines (Azure Pipelines/ Jenkins) can make your scripts run without manual intervention.
I am using Selenium to open a certain website (for example YouTube) on the server, but it can't seem to open the website. However, the code works just fine with a different website. This code also works fine without any problems on my local PC.
I don't know if I have problems with my Chrome Driver or Selenium but it can't open youtube.com as it only outputs: "Before getting the website" and that's it. There are no exceptions/errors that are shown but the script still runs and I have to manually end stop it.
Why can't Selenium open certain URLs on the server, but it works fine on my PC?
options = webdriver.ChromeOptions()
options.add_argument("no-sandbox")
options.add_argument('--headless')
options.add_argument("--start-maximized")
PATH = "./chromedriver"
global driver
driver = webdriver.Chrome(PATH, chrome_options=options)
print("Before getting the website")
driver.get("https://youtube.com")
print("opened", driver.current_url)
I had a exactly same problem. Perhaps i don't know why this happened.
NOTE:
When you are scraping working on Ubuntu ec2 with no GUI you have to provide some GUI interface for chrome to run and for me Xvfb solved it.
"Xvfb (short for X virtual framebuffer) is an in-memory display server for UNIX-like operating system (e.g., Linux). It enables you to run graphical applications without a display (e.g., browser tests on a CI server) while also having the ability to take screenshots."
SOLUTION
Install Xvfb for ubuntu: sudo apt install xvfb
Now execute your script as: xvfb-run python[version] script.py
IMPORTANT NOTE:
If your program stuck during initialization does not show any output just make sure that you did not add chrome_option.add_argument("disable-dev-shm-usage").
If this argument is added in your chrome header then it would disable the /dev/shm. Not sure but it is some shared memory and xvfb is in-memory display server which i think need it.
This worked for me.
my problem is, that I wrote some python scripts, which are working fine. Now I have to get them to work on an ubuntu server. The problem is, that I need to use the chromedriver (selenium) and ofc there cant be an open browser at the server. So is there a way to use selenium with a server?
What you need is called 'Headless' editions of a browser.
These headless browsers don't open up as a browser but run in the background for you to perform scripts on.
Try searching for Headless + 'The browser driver you use'
Here is a quick tutorial to get you started: https://medium.com/#pyzzled/running-headless-chrome-with-selenium-in-python-3f42d1f5ff1d
At this point I'm very familiar with how to get Selenium and Behave to work together to interact with another website (like typing a phrase into an input box). However, because I have created a fullstack python application that I am able to run on my local port 8000, I would like to have Selenium and Behave interact with my site running on the local port the same as it would with an external website. For instance, if I try to run my application on port 8000 and then run Selenium with the following code:
from selenium.webdriver import Chrome
from selenium.webdriver.support.ui import Select
from selenium import webdriver
driver = webdriver.Chrome()
driver.get("localhost:8000/")
select = Select(driver.find_element_by_xpath('/html/body/header/div[1]/div/div/div/form/div/div[1]/div/input'))
select.select_by_value('1')
I am met with a 500 server error in the browser.
I have followed along with this helpful Medium article. But again, it uses an external website.
Is this a case to use java -jar in the command line? I don't see how this would work given I have not written anything in Java.
Thank you all so much for any guidance.
EDITED TO ADD
Selenium will now open the page (I accidentally had http before localhost)
My solution on this was setting it towards the local IP. In my case it was:
http://127.0.0.1:PORT
I have a python script that uses selenium RC; specifically webdriver.
I would love to have the script run as a post-commit hook, ideally through IE, Safari (if possible), Chrome & Firefox.
But I'm not sure what I'd need to do, since every time I run it on my local machine, a browser pops up.
I've heard of "saucelabs". Is it the best solution?
Yes, you need to have X server running because Selenium requires a running browser and browsers need some kind of X server in order to run.
Fortunately there are many types of X servers and one of them is Xvfb aka X virtual framebuffer that does all the operations in memory and thus requires no screen.
Wikipedia has also some nice examples.
Also there are lots of questions here on SO for keywords headless, xvfb and selenium. Choose wisely ;)
[Edit]: Here's a nice example.