I’m relatively knew to the selenium package and have been using it for a couple weeks. My current script uses selenium to scrape data, I analyze the data by running a few tests, and if there is a datastring that passes said tests python texts me using Twilio. I’m currently using my mac to run all of this but I was looking to run this script every 5 minutes, headless, and on a platform such that I dont need to keep my computer on. I have been looking at some potential solutions and it seems as though running this on a headless raspberry pie is the right option. I was wondering if anyone see’s any potential problems with doing so as I haven’t seen a thread with someone using Twilio? And, I’ve encountered problems trying to set up a cron task to automate it on my mac because of selenium and was wondering if this will be possible on the pi (looking at the raspberry pi 4)? Sorry, if this is a little long winded, appreciate the help.
Run the script though any CI CD tools like Jenkins ,GoCD,Gitlab in a scheduled job so that the script would run in every 5 minutes in Agent node specified and you don't have to keep your computer on.
Related
Ever since I started using Python Selenium and windows task manager to automate a python script to scrape the web using Google chrome I am having this issue where even though I haven't opened Chrome it still shows in the task manager and uses 60-70% CPU. This is slowing my laptop significantly and I can't seem to solve it. If I end task Chrome then it fixes but randomly after some time chrome again pops up in task manager slowing my laptop. If anyone has a solution to this it would be a great help if you can help me.
I have a python script that checks the temperature every 24 hours, is there a way to leave it running if I shut the computer down/log off.
Shutdown - no.
Logoff - potentially, yes.
If you want to the script to automatically start when you turn the computer back on, then you can add the script to your startup folder (Windows) or schedule the script (Windows tasks, cron job, systemd timer).
If you really want a temperature tracker that is permanently available, you can use a low-power solution like the Raspberry Pi rather than leaving your pc on.
The best way to accomplish this is to have your program run on some type of server that your computer can connect to. A server could be anything from a raspberry pi to an old disused computer or a web server or cloud server. You would have to build a program that can be accessed from your computer, and depending on the server and you would access it in a lot of different ways depending the way you build your program and your server.
Doing things this way means your script will always be able to check the temperature because it will be running on a system that stays on.
Scripts are unable to run while your computer is powered off. What operating system are you running? How are you collecting the temperature? It is hard to give much more help without this information.
One thing I might suggest is powering on the system remotely at a scheduled time, using another networked machine.
You can take a look at the following pages
http://www.wikihow.com/Automatically-Turn-on-a-Computer-at-a-Specified-Time
http://lifehacker.com/5831504/how-can-i-start-and-shut-down-my-computer-automatically-every-morning
Additionally once it turn on, you can perform a cronjob, for execute your python code by a console command >> python yourfile.py . What is the Windows version of cron?
I am using selenium to download data from various websites but one site in particular is giving me trouble. I narrowed it down to the fact that it will not run when the computer is locked. Everything works fine if I run it manually or if I remote desktop, but if I schedule the task to run and then lock the computer it will not run properly.
Is there a way to add something to my script which will:
Unlock my computer
Run the selenium script
Lock my computer
Or maybe schedule a remote desktop session to create an active desktop? Anyone have some advice on how to accomplish this correctly?
I'm running a selenium test using Firefox-17, and it will randomly "freeze" - the window is visible but completely useless. The mouse cursor that usually shows up when you hover over a link is active over the entire firefox window, but cannot actually interact/click on the page or firefox's menus. This only happens on the Debian machine, and only with selenium. I use Firefox-28 for daily browsing, and I've never experienced any issues like this.
The code runs fine for several minutes, but then it always randomly freezes in the middle of requesting a new page. The process must then be force killed.
Things I've tried:
Using firefox-28 - still freezes at random
Running the same code on my Windows machine - this runs for hours with no problem
Hypothesis:
I'm running the tests with python's multiprocessing. (For debugging purposes I've only been using one master queue that feeds to a single driver instance.)
Could this freezing problem be related to the forking mechanism used by Linux for multiprocessing?
Maybe somehow related to the http://shallowsky.com/blog/linux/firefox-freeze-and-dbus.html - although I have no problems accessing the bbc podcast referenced in that link
I have other code that runs firefox with javascript disabled, and it hasn't had any issues on this Debian machine. Could this be something to do with Linux's javascript engine?
I finally figured out that this only occurs with javascript enabled in firefox. Fortunately, chromium doesn't have this javascript problem on the same debian system.
Here's a short description of what I have:
I have to raspberry pi's in a local net work. I one of them I have a .py script named watchdog.py that starts a stream and then uses a sshpass command to the other pi to display the video stream.It also has some signaling LEDs a some push buttons for control
the problem is:
If I open a terminal and run the watchdog.py script in the GUI everything runs as it should be. So I thought of running it as a service as boot and installed upstart and made it run as a service (successfully I think). The thing is. If I boot the pi and then press the button to start the streams,they wont play on the other Pi, the LEDs ligh up and all the buttons work. And even the CPU load behaves the same way, but i still don't video nor audio. I have thought of trying automatically open a terminal (LXterminal) widow and run the python scrip on that window. but I didn't want the streaming raspberry pi also booting into gui (tough I guess I would mind if that makes the whole thing work).This little thing i making the whole project useless.
What are you using to play the streams? Depending on how you boot up the second Raspberry it might not have started some daemons for audio/video playback?!
You should (if you're not already doing) write a log (import logging ;)) and write a logfile which you can track for errors.
answer moved from OP's question itself:
I found a way that seems to work so far. instead of running the python script as a service I tried running it as cron job at reboot, and it worked. now it all works straight from reboot and I have Audio and video.