I am working on a project that is built mostly in python it is using selenium to emulate user interaction with a website. Then it downloads the network history through chrome dev tools. I recently ran into a roadblock as I am unable to monitor network data directly through Selenium CDP when using python.(or at least can't find out how to do so.) Nor can I figure out to simply type in commands into the browser.
If anyone knows how to either send keys directly to the browser through selenium (I have not been able to get actionchains to work.) or how to access CDP network and history as .har file in python. Then their help would be greatly appreciated.
my problem is, that I wrote some python scripts, which are working fine. Now I have to get them to work on an ubuntu server. The problem is, that I need to use the chromedriver (selenium) and ofc there cant be an open browser at the server. So is there a way to use selenium with a server?
What you need is called 'Headless' editions of a browser.
These headless browsers don't open up as a browser but run in the background for you to perform scripts on.
Try searching for Headless + 'The browser driver you use'
Here is a quick tutorial to get you started: https://medium.com/#pyzzled/running-headless-chrome-with-selenium-in-python-3f42d1f5ff1d
At this point I'm very familiar with how to get Selenium and Behave to work together to interact with another website (like typing a phrase into an input box). However, because I have created a fullstack python application that I am able to run on my local port 8000, I would like to have Selenium and Behave interact with my site running on the local port the same as it would with an external website. For instance, if I try to run my application on port 8000 and then run Selenium with the following code:
from selenium.webdriver import Chrome
from selenium.webdriver.support.ui import Select
from selenium import webdriver
driver = webdriver.Chrome()
driver.get("localhost:8000/")
select = Select(driver.find_element_by_xpath('/html/body/header/div[1]/div/div/div/form/div/div[1]/div/input'))
select.select_by_value('1')
I am met with a 500 server error in the browser.
I have followed along with this helpful Medium article. But again, it uses an external website.
Is this a case to use java -jar in the command line? I don't see how this would work given I have not written anything in Java.
Thank you all so much for any guidance.
EDITED TO ADD
Selenium will now open the page (I accidentally had http before localhost)
My solution on this was setting it towards the local IP. In my case it was:
http://127.0.0.1:PORT
I am facing a very peculiar issue with Selenium Webdriver. All of sudden my selenium scripts written in python have started to execute very slow in both firefox and chrome.
The script execution returns to its normal speed when I manually turn off the network/internet through my laptop's wifi button after a webpage is loaded. The same scripts run at normal speed on another computer (on my office network) where I connect to internet through VPN.
Another example is when I run my scripts on a locally stored webpage file (htm). The scripts run superfast when internet is turned off but run equally slow when the internet is turned on.
I do not use a proxy server.
I have written a few Selenium [Python] webdriver scripts, and while they are running fine on my system, they are not running on my website's server. It is showing errors with Firefox. Firefox is installed on the server. The webserver is Ubuntu. What do I have to do to run those scripts? Please help, I am a newbie.
Selenium requires a running browser and browsers need some kind of X server in order to run. There are many types of X servers and one of them is Xvfb aka X virtual framebuffer that does all the operations in memory and thus requires no screen.
In Wikipedia you could find very nice examples.
This is a nice example too.
You probably need to open the browser headless when executing your scripts on the server.
Here is the Java code for Firefox (Python code should be similar):
import java.io.File;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.firefox.FirefoxBinary;
WebDriver openHeadless() throws Exception
{
FirefoxBinary binary = new FirefoxBinary(new File("/usr/local/bin/firefox"));
binary.setEnvironmentProperty("DISPLAY",System.getProperty("lmportal.xvfb.id",":99"));
return new FirefoxDriver(binary,null); // or 'binary,profile' if you have a profile
}
Make sure that you have Firefox installed on the server at /usr/local/bin/firefox.
If you want simply do web browser testing you can use libraries like Casper JS this creates a server side browser for web browser testing, it doesn't need a display driver.
I ended up using pyvirtualdisplay which is a Python wrapper for Xvfb and Xephyr. If anyone is having same issues [and I am sure newbies will], You can try THIS. Also you can use xvfbwrapper. Tutorial for xvfbwrapper HERE.