I have a python script that uses Chromedriver and selenium to scrape a handful of websites. What resources are currently used to run these type of python scripts in the cloud?
For reference, I am using version 84.0.4147.30
One possible solution would be to setup an EC2 instance and install chrome, chromedriver and python...
Related
You may know that heroku will stop their free dyno, free postgres etc from November. So I was finding some alternative to run my python web apps. I have almost 10 regular web apps which I visit regularly, like: url shortener, keyword research, google drive direct link generator site and many more. All of these are hosted on heroku. But I'm moving to vercel now. I setup all projects on vercel but the last one is complicated. My last project is python selenium bot. This one is my keyword research web app. I used some buildpack eg: Headless Chrome (https://github.com/heroku/heroku-buildpack-google-chrome) and Chromedriver (https://github.com/heroku/heroku-buildpack-chromedriver) to make this project run properly. But the problem is I could not find anything like buildpack in vercel to add Chrome and Chromedriver.
Anyone know about this?
Edit:
That was a kind of story and many people didn’t understand what I was asking.
So, My project is about selenium (python). Selenium needs google chrome browser installed and a chromedriver to run itself. There is another option without installing chrome is to set chrome binary location in webdriver.ChromeOptions(). I want to host this selenium project on vercel.com which is linux based.
So my question is how can I install Chrome Browser and ChromeDriver in vercel?
I have created a website that scrapes multiple hockey websites for game scores. It runs perfectly on my local server and I am now in the process of trying to deploy. I have tried using pythonanywhere.com but selenium does not seem to be working on it. For any one who has deployed a website that uses selenium/webdriver, what is the easiest/best platform to deploy a website like this (it does not have to be free like pythonanywhere, as long as it is not too expensive, lol!). Thanks in advance
Selenium does work on PythonAnywhere. If you use a free account, you'd have restricted internet access though. Also it's recommended to scrape outside of the web app, since it would slow the views down -- you should rather use a Schedule/Always-on task for that instead. You can also refer to those PythonAnywhere help pages:
Using Selenium
Async work in web apps
You can use the AWS, GCP, or Digitalocean Linux servers. In this case, you first have to install chrome in Linux and then put the relevant version of the chrome driver in your project directory. Make sure to check the chrome version first and then put the relevant Chrome driver on your machine.
I tried to make an automation script using Selenium which automates my timelog that I've to fill everyday. It's working fine locally, but only I can use it in my organization.
I can share the script, but others will have to configure their system for Python, pip, Selenium and also the environment variables which is not easy for everyone.
Is there a way I can upload my script on web and create a virtual environment with all the required packages, and other can run that script from there?
I am developing a python script to take screenshots from many websites. for this I am using below tools,
phantomjs with selenium
python
windows PC
I have used pyside(instead of phantomjs) for that job but I faced many issues on pyside..
now I have found phantomjs tool from Google.com, I have used phantomjs with selenium for python in windows machine it is working flawless. but it has only one issue phantomjs doesn't support flash player, so am not able to process youtube and some flash websites.. please give me the some quick fix for this
PhantomJS does not and probably will not support Flash and other plugins (see here).
But you can use SlimerJS in your Selenium tests, which is a headless browser based on the Gecko engine. It does support the WebDriver protocol, so use it.
There is also a fork of PhantomJS with Flash support, but it didn't merge changes in PhantomJS back into it, so it is standing still at version 1.9.0.
Phantomjs now don't rely on xwindow enviroment since 1.5, also it has remove plugin support at that time. So there is no officially support for running flash player in current phantomjs version.
Howerver, there are so many project fork from the old phantomjs that has flash player enabled and keep update. You can try r3b phantomjs. Recently I had build a perfect service upon this project under ubuntu os.
How to Integrate jenkins with Selenium-webdriver. I am using Python language for Selenium TestCases. I have tried 'pyvirtualdisplay' Python module to hide the display on Jenkins CLI. Currently I have used ant build scripts to execute the Python Test Suit.
I want to use Jenkins "Selenium" plugin to integrate my Python Testcases. But "Selenium" plugin allows only "HTMLSuite" files.
How to run my Python TestCase using Jenkins "Selenium" Plugin?
I have tried downloading the Selenium Plugin (the one that says Grid on the description) and what it does is creates a Selenium Server on Jenkins. Then, if you read the specifications of the plugin (https://wiki.jenkins-ci.org/display/JENKINS/Selenium+Plugin) you will see that you need to indicate your script to target Jenkin's Selenium Server that is running.
I get to run my test outside Jenkins but does not start a broswer (is headless) and is not doing some of the visual commands (as easy as take as screenshoot).
Then I saw this video that helps to build a task with a Selenium testcase:
http://www.dailymotion.com/video/xkw0zb_jenkins-running-a-selenium-job_tech
Hudson is an older version of Jenkins
He creates a build.xml that basically does everything.
I am still investigating on how to create that build.xml file
Hope all this helps you somehow.
Best of lucks,