I am working in a web scraping project with python 3.7.
Done the code in python with selenium and chromediver.exe in windows, it's working fine.
we are adding the script in aws lambda.
the issue is we need to specify the chrome driver of Linux.
I followed the steps in https://github.com/yai333/Selenium-UI-testing-with-AWS-Lambda-Layers.
i am not using any serverless yml sript(i don't know the same).
doing the following
we have a Linux machine.
create a virtual python environment and add selenium module(as described in awshttps://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html#python-package-venv)
download chromdriver and headless into a folder(size is large, so upload to S3) add both (chrome driver and python lib) as layers.
paste the handler.py (in https://github.com/yai333/Selenium-UI-testing-with-AWS-Lambda-Layers) to a lambdahandler file.
create a sample test, and click on test.
shows error:
error message 'chromedriver' executable needs to be in path
can I upload chrome driver in S3 and show the path.
Having just fought this exact issue for a few hours, I think I can help.
Within your Lambda Layer, you need to include the chromedriver binary under /bin. Os it will look something like:
layerZip/
|- bin/
|- chromedriver
Within your lambda function's infrastructure, it will exist at /opt/bin/chromedriver. As such, we need to point our Python towards that as the executable. To get it to work, I had to add the following:
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.binary_location = "/opt/bin/chromedriver"
driver = webdriver.Chrome(executable_path="opt/bin/chromedriver", options=chrome_options)
Related
Using the python + selenium chromedriver with a specific user folder to keep my cookies and settings saved with the following code:
chrome_options = Options()
chrome_options.add_argument("user-data-dir=cookies")
driver = webdriver.Chrome("C:\Python\chromedriver\chromedriver.exe", options=chrome_options)
Everything used to work well, but had to reinstall Windows 10 machine and now everytime I run the script, there is an error "Google Chrome is not able to reach the data folder for reading nor writing" (not these words exactly - translated).
I noticed that all windows folders are set to read only and am not able to change it (using admin account).The controlled folder access in windows settings is turned off.
Has anyone been dealing with the same problem?
How are you running Selenium? From Powershell/CMD?
These sometimes needs to be run as admin.. Right click => Run as Administrator.
You can create a .bat file or similiar and change the properties to always run as admin.
For a project of mine I decided to go with Selenium. It's a "scanner" for an web based online game, hence interaction with the page is needed. All things set and done I'm worrying for the security of my application and hence I took a look at well-known questions on SO. I found bunch of suggestions which may help securing the detectability of said programme.
What is left is something related to Chrome Plugins. In my Project File I got the chromedriver.exe and a dedicated resource folder for stuff like the cookies and settings it's supposed to run with. Now I want to run the Chromedriver.exe with Chrome Plugins.
So is there any way to download Chrome Plugins in the Chromedriver.exe and run them every time with the Selenium process?
First method is to create a chrome profile and load it with youre chrome plugins
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("user-data-dir=C:\\Path") #Path to your chrome profile
w = webdriver.Chrome(executable_path="C:\\Users\\chromedriver.exe",
chrome_options=options)
Read more here How to load default profile in chrome using Python Selenium Webdriver?
another method is to add it in the chrome options:
options.add_extension('./exampleOfExtensionDownloadedToFolder.crx')
To download the chrome extension, follow below steps.
add chrome extension.
Go to http://chrome://extensions/]
Enable developer mode.
Click on 'Details' of desired extension.
Click on 'Pack Extension'.
Browse extension root directory. Eg. C:\Users\Admin\AppData\Local\Google\Chrome\User Data\Default\Extensions\mooikfkahbdckldjjndioackbalphokd\3.17.0_0 ( Don't forget to sort the folders by date and select latest folder.)
Click on 'Pack Extension'.( If any error get encountered related to key browse manifest.json file from step 6 path)
Browse to extension path (Eg. C:\Users\Admin\AppData\Local\Google\Chrome\User Data\Default\Extensions\mooikfkahbdckldjjndioackbalphokd ) you will find .crx and .pem file.
Copy and paste both the files in resources folder of your project.
Below java code will help you to access the extension.
System.setProperty("webdriver.chrome.driver",
System.getProperty("user.dir") + "\\src\\test\\resources\\executables\\chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.addExtensions(new File(System.getProperty("user.dir") + "\\src\\test\\resources\\executables\\5.6_0.crx"));
options.addArguments("start-maximized");
WebDriver driver = new ChromeDriver(options);
So when I run my python selenium script through Jenkins, how should I write the driver = webdriver.Chrome()
How should I put the chrome webdriver EXE in jenkins?
Where should I put it?
If you have added your repository path in jenkins during job configuration, Jenkins will create a virtual copy of your workspace. So, as long as the webdriver file is somewhere in your project folder structure and as long as you are using relative path to reference it in your code, there shouldn't be any issues with respect to driver in invocation.
You question also depends on several params like:
1. Whether you are using Maven to run the test
2. Whether you are running tests on Jenkins locally or on a remote machine using Selenium Grid Architecture.
I'm trying to run a python script with Selenium inside docker. Since the script is in a docker container, I keep getting this error:
selenium.common.exceptions.WebDriverException: Message: 'chromedriver'
executable needs to be in PATH.
The code works just fine, when I run it on my computer and use the PATH to the chromedriver file, however a docker container can't see the PATH as it is isolated. I would use a pre-made docker image, however I need other dependencies for my code to run such as smtplib, imapclient, and beautifulsoup to name a few.
Is there a command that will tell Selenium that the chromedriver is in the same folder as the script, or some other way to make it visible to docker?
Docker container is a like a VM. Inside the image you have scripts. But not chromedriver. I would say do not copy the chromedriver inside the image. Just copying the chromedriver alone will not work.
Instead use selenium/node-chrome image and give the container name as the host to your container.
Take a look at this example.
http://www.testautomationguru.com/selenium-webdriver-how-to-run-multiple-test-suites-using-docker-compose/
You could use the following code snippet to set the path to the chrome driver within your script:
import os
path_to_chrome_driver = os.path.join(os.getcwd(), 'chromedriver.exe')
browser = webdriver.Chrome(executable_path=path_to_chrome_driver )
I am running a python script for a webpage. The python script connects to localhost:5000. In a html I have some images that are in local storage I want to load.
I have set the file to src to file:///path/to/file/some_image.jpg
However I keep getting the error Not allowed to load local resource
I have tried to allow local access using:
"C:\Program Files (x86)\Google\Chrome\Application\chrome.exe" --allow-file-access-from-files
However I get the same error. I have tried restarting my computer, chrome and ran everything from admin but still no luck.
How can I run local files in chrome with localhost?
Well, have you checked whether you have permission to access those files?
Maybe you can try to use selenium to control the chrome through python, below is a simple python code that you can check it out(Replace the google.com to your html file).
http://selenium-python.readthedocs.io/installation.html#introduction
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("--allow-file-access-from-files")
driver = webdriver.Chrome("C:\Program Files (x86)\Google\Chrome\Application\chrome.exe", chrome_options=options)
driver.get("http://www.google.com")