Develop Reference

Using Chrome Addons in a Selenium Session possible?

For a project of mine I decided to go with Selenium. It's a "scanner" for an web based online game, hence interaction with the page is needed. All things set and done I'm worrying for the security of my application and hence I took a look at well-known questions on SO. I found bunch of suggestions which may help securing the detectability of said programme.
What is left is something related to Chrome Plugins. In my Project File I got the chromedriver.exe and a dedicated resource folder for stuff like the cookies and settings it's supposed to run with. Now I want to run the Chromedriver.exe with Chrome Plugins.
So is there any way to download Chrome Plugins in the Chromedriver.exe and run them every time with the Selenium process?

First method is to create a chrome profile and load it with youre chrome plugins
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("user-data-dir=C:\\Path") #Path to your chrome profile
w = webdriver.Chrome(executable_path="C:\\Users\\chromedriver.exe",
chrome_options=options)
Read more here How to load default profile in chrome using Python Selenium Webdriver?
another method is to add it in the chrome options:
options.add_extension('./exampleOfExtensionDownloadedToFolder.crx')

To download the chrome extension, follow below steps.
add chrome extension.
Go to http://chrome://extensions/]
Enable developer mode.
Click on 'Details' of desired extension.
Click on 'Pack Extension'.
Browse extension root directory. Eg. C:\Users\Admin\AppData\Local\Google\Chrome\User Data\Default\Extensions\mooikfkahbdckldjjndioackbalphokd\3.17.0_0 ( Don't forget to sort the folders by date and select latest folder.)
Click on 'Pack Extension'.( If any error get encountered related to key browse manifest.json file from step 6 path)
Browse to extension path (Eg. C:\Users\Admin\AppData\Local\Google\Chrome\User Data\Default\Extensions\mooikfkahbdckldjjndioackbalphokd ) you will find .crx and .pem file.
Copy and paste both the files in resources folder of your project.
Below java code will help you to access the extension.
System.setProperty("webdriver.chrome.driver",
System.getProperty("user.dir") + "\\src\\test\\resources\\executables\\chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.addExtensions(new File(System.getProperty("user.dir") + "\\src\\test\\resources\\executables\\5.6_0.crx"));
options.addArguments("start-maximized");
WebDriver driver = new ChromeDriver(options);

Selenium on Amazon Ubuntu Server (EC2) is not opening certain links but works fine on local machine

I am using Selenium to open a certain website (for example YouTube) on the server, but it can't seem to open the website. However, the code works just fine with a different website. This code also works fine without any problems on my local PC.
I don't know if I have problems with my Chrome Driver or Selenium but it can't open youtube.com as it only outputs: "Before getting the website" and that's it. There are no exceptions/errors that are shown but the script still runs and I have to manually end stop it.
Why can't Selenium open certain URLs on the server, but it works fine on my PC?
options = webdriver.ChromeOptions()
options.add_argument("no-sandbox")
options.add_argument('--headless')
options.add_argument("--start-maximized")
PATH = "./chromedriver"
global driver
driver = webdriver.Chrome(PATH, chrome_options=options)
print("Before getting the website")
driver.get("https://youtube.com")
print("opened", driver.current_url)

I had a exactly same problem. Perhaps i don't know why this happened.
NOTE:
When you are scraping working on Ubuntu ec2 with no GUI you have to provide some GUI interface for chrome to run and for me Xvfb solved it.
"Xvfb (short for X virtual framebuffer) is an in-memory display server for UNIX-like operating system (e.g., Linux). It enables you to run graphical applications without a display (e.g., browser tests on a CI server) while also having the ability to take screenshots."
SOLUTION
Install Xvfb for ubuntu: sudo apt install xvfb
Now execute your script as: xvfb-run python[version] script.py
IMPORTANT NOTE:
If your program stuck during initialization does not show any output just make sure that you did not add chrome_option.add_argument("disable-dev-shm-usage").
If this argument is added in your chrome header then it would disable the /dev/shm. Not sure but it is some shared memory and xvfb is in-memory display server which i think need it.
This worked for me.

where to place the chromedriver in aws lambda

I am working in a web scraping project with python 3.7.
Done the code in python with selenium and chromediver.exe in windows, it's working fine.
we are adding the script in aws lambda.
the issue is we need to specify the chrome driver of Linux.
I followed the steps in https://github.com/yai333/Selenium-UI-testing-with-AWS-Lambda-Layers.
i am not using any serverless yml sript(i don't know the same).
doing the following
we have a Linux machine.
create a virtual python environment and add selenium module(as described in awshttps://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html#python-package-venv)
download chromdriver and headless into a folder(size is large, so upload to S3) add both (chrome driver and python lib) as layers.
paste the handler.py (in https://github.com/yai333/Selenium-UI-testing-with-AWS-Lambda-Layers) to a lambdahandler file.
create a sample test, and click on test.
shows error:
error message 'chromedriver' executable needs to be in path
can I upload chrome driver in S3 and show the path.

Having just fought this exact issue for a few hours, I think I can help.
Within your Lambda Layer, you need to include the chromedriver binary under /bin. Os it will look something like:
layerZip/
|- bin/
|- chromedriver
Within your lambda function's infrastructure, it will exist at /opt/bin/chromedriver. As such, we need to point our Python towards that as the executable. To get it to work, I had to add the following:
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.binary_location = "/opt/bin/chromedriver"
driver = webdriver.Chrome(executable_path="opt/bin/chromedriver", options=chrome_options)

How do you run Chrome while selenium headless script is executing on mac?

I have a python selenium script that uses chromedriver in headless mode to test links on a website. I'm running this script on MacOS High Sierra.
If I have Chrome open prior to running the script, I can continue to use the browser normally. Headless mode appears to isolate me from the testing. However, if Chrome is closed first, I cannot launch the browser while the selenium script is running.
Looking at Activity monitor in the first scenario, it appears to me that chromedriver creates a new process for Chrome to run. So I (perhaps incorrectly) assumed I should be able to launch the browser manually while the script is running.
What am I missing?
How do I launch Chrome while selenium is running on Mac?

Ah-HA!
This post I stumbled upon made it possible to open Chrome while selenium is running (although I don't understand why it works one way but not the other).
For most applications, launching with open -naF "App Name" or open -nF
/path/to/application.app should be sufficient (see Safari, Forklift,
Office apps, etc.).
-n opens a new instance
-a tells open to look in the list of registered apps (no need to specify the full path, but can go wrong if you have multiple versions
of an app. I sometimes see issues on my system with differentiating
between macOS apps and apps installed by Parallels OSes… I prefer to
specify the path directly.)
-F opens a fresh instance (ignores an app save state left over from previous instances)

How to access local files with chrome (--allow-access doesn't work)

I am running a python script for a webpage. The python script connects to localhost:5000. In a html I have some images that are in local storage I want to load.
I have set the file to src to file:///path/to/file/some_image.jpg
However I keep getting the error Not allowed to load local resource
I have tried to allow local access using:
"C:\Program Files (x86)\Google\Chrome\Application\chrome.exe" --allow-file-access-from-files
However I get the same error. I have tried restarting my computer, chrome and ran everything from admin but still no luck.
How can I run local files in chrome with localhost?

Well, have you checked whether you have permission to access those files?
Maybe you can try to use selenium to control the chrome through python, below is a simple python code that you can check it out(Replace the google.com to your html file).
http://selenium-python.readthedocs.io/installation.html#introduction
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("--allow-file-access-from-files")
driver = webdriver.Chrome("C:\Program Files (x86)\Google\Chrome\Application\chrome.exe", chrome_options=options)
driver.get("http://www.google.com")