Develop Reference

Python Chromedriver End Task Annoying Flicker

I am building a Tkinter app with python that initializes multiple selenium webdrivers. The initial problem was that lots of chromedriver.exe instances were filling up user's memory, even after using driver.quit() (sometimes). So to get rid of this issue, when closing the tkinter app, I wrote this line os.system("taskkill /f /im chromedriver.exe /T"), that solves my problem, but, by using this, a command prompt instance is initiated that self kills almost instantly. The problem is that the user can see it and I find it kinda disturbing. Is there any way I could hide it? Or is there a workaround for my initial problem, that is user friendly?

Use both driver.close() and driver.quit() in your code in order to free memory.
driver.close()
driver.quit()

To reduce the memory footprint you should use ChromeDriver-Service:
First you start the service, then use it when creating new drivers, and finally stop the service before program exit.
Put the code below in chrome_factory.py and then:
on program start call chrome_factory.start_service()
to create new driver call chrome_factory.create_driver()
the service and drivers will be automatically stopped/quit at program exit.
Using this approach will result in only ever having single chromedriver.exe process.
# chrome_factory.py
import atexit
from os.path import expanduser
from typing import Optional
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
EXECUTABLE = expanduser("~/bin/chromedriver")
_chrome_service: Optional[Service] = None
def start_service():
global _chrome_service
_chrome_service = Service(EXECUTABLE)
_chrome_service.start()
atexit.register(_chrome_service.stop)
def create_driver() -> webdriver.Remote:
global _chrome_service
opts = webdriver.ChromeOptions()
opts.add_argument("--headless")
driver = webdriver.Remote(_chrome_service.service_url,
desired_capabilities=opts.to_capabilities())
atexit.register(driver.quit)
return driver
def main():
start_service()
for _ in range(20):
create_driver()
if __name__ == '__main__':
main()

Function running in background all the time (and startup itself) in Django app

I create simple Django app. Inside this app I have single checkbox. I save this checkbox state to database if it's checked in database I have True value if checkbox is uncecked I have False value. There is no problem with this part. Now I created function that prints for me every 10 second all the time this checkbox state value from database.
Function I put into views.py file and it looks like:
def get_value():
while True:
value_change = TurnOnOff.objects.first()
if value_change.turnOnOff:
print("true")
else:
print("false")
time.sleep(10)
The point is that function should work all the time. For example If I in models.py code checkbox = models.BooleanField(default=False) after I run command python manage.py runserver it should give me output like:
Performing system checks...
System check identified no issues (0 silenced).
January 04, 2019 - 09:19:47
Django version 2.1.3, using settings 'CMS.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.
true
true
true
true
then if I visit website and change state is should print false this is obvious. But as you notice problem is how start this method. It should work all the time even if I don't visit the website yet. And this part confuse me. How to do this properly ?
I need to admit that I tried some solutions
put this function at the end of manage.py file,
put this function into def ready(self),
create middleware class and put method here (example code below).
But this solutions doesn't work.
middleware class :
class SimpleMiddleware:
def __init__(self, get_response):
self.get_response = get_response
get_value()

You can achieve this by using the AppConfig.ready() hook and combining it with a sub-process/thread.
Here is an example apps.py file (based on the tutorial Polls app):
import time
from multiprocessing import Process
from django.apps import AppConfig
from django import db
class TurnOnOffMonitor(Process):
def __init__(self):
super().__init__()
self.daemon = True
def run(self):
# This import needs to be delayed. It needs to happen after apps are
# loaded so we put it into the method here (it won't work as top-level
# import)
from .models import TurnOnOff
# Because this is a subprocess, we must ensure that we get new
# connections dedicated to this process to avoid interfering with the
# main connections. Closing any existing connection *should* ensure
# this.
db.connections.close_all()
# We can do an endless loop here because we flagged the process as
# being a "daemon". This ensures it will exit when the parent exists
while True:
value_change = TurnOnOff.objects.first()
if value_change.turnOnOff:
print("true")
else:
print("false")
time.sleep(10)
class PollsConfig(AppConfig):
name = 'polls'
def ready(self):
monitor = TurnOnOffMonitor()
monitor.start()

Celery is the thing that best suits your needs from what you've described.
Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.
The execution units, called tasks, are executed concurrently on a single or more worker servers using multiprocessing, Eventlet, or gevent. Tasks can execute asynchronously (in the background) or synchronously (wait until ready).
You need to create task, run it periodically, call if you want to manually trigger is (in some view/controller).
NOTE: do not use time.sleep(10)

Python 3.5 logger.info in #classmethod

New to python, never user #classmethod before.
The problem - for some reason logger methods are not executed within the shutdown_webdriver function.
import time
import logging
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.webdriver.common.proxy import Proxy
class WebBrowserSettings(object):
logger = logging.getLogger(__name__)
def __init__(self, proxy):
self.proxy = proxy
def setup_remote_chromedriver(self):
**irrelevant code**
return browser
#classmethod
def shutdown_webdriver(cls, browser):
print('here')
cls.logger.info("Shutting down 1")
for index in range(0, 20):
error_check = 0
try:
time.sleep(5)
browser.quit()
except Exception:
error_check = 1
if error_check == 0:
break
cls.logger.info("Browser is down")
So i see only the print('here') message in my console output.
P.S. the logging config is setup and stable, working in other classes.

I see no problem with the code shown here. I suspect that you will see the logging output when you replace cls.logger.info with cls.logger.error. This means that something is going wrong with the configuration of the logging system. This is also the part that is not shown here, so I can't tell what is going wrong.
In any case you have to make sure that the logging system is configured to display level INFO before the instantiation of the logger instance.
Your logger gets instantiated at the time the code is being parsed, probably at the time some other code imports this module.
This in turn means that you have to configure the logging system before you import this module.

Returning a Selenium Webdriver from a Thread

Trying to deal with the creation of a webdriver timing out (which happens once in a while covered here). I can't use a signal based timeout because my server is running on Windows so I've been trying to find an alternative.
I looked at the timeout from eventlet but I don't think that will cut it. A time.sleep(10000) doesn't trigger the timeout so I don't think the timeout itself would.
What I'm thinking is calling a thread to create and return the browser and then setting a join timeout. So something like:
def SpawnPhantomJS(dcap, service_args):
browser = webdriver.PhantomJS('C:\phantomjs.exe',desired_capabilities=dcap, service_args=service_args)
print "browser made!"
return browser
proxywrite = '--proxy=',nextproxy
service_args = [
proxywrite,
'--proxy-type=http',
'--ignore-ssl-errors=true',
]
dcap = dict(DesiredCapabilities.PHANTOMJS)
dcap["phantomjs.page.settings.userAgent"] = (nextuseragent)
newDriver = Thread(target=SpawnPhantomJS, args=[dcap, service_args]).start().join(20)
So I'm having some issues with the syntax on how to do this properly in theory this should work. If the creation stalls the SpawnPhamtomJS thread will stall not the main one so the timeout join should help it move on.
Is this possible though? Can I create a webdriver in a thread and return it? Any points appreciated.
Updates:
Just calling a function returned a webcontrol so that bodes well for what I'm trying to do.
newDriver = SpawnPhantomJS(dcap, service_args)
So I'm hoping it's just a syntax issue I have running this as a thread with a timeout.
This didn't do it however:
spawnthread = Thread(target=SpawnPhantomJS, args=[dcap, service_args])
spawnthread.start()
newDriver = spawnthread.join()
Wishful thinking there.

Thread pooling.
from multiprocessing.pool import ThreadPool
pool = ThreadPool(processes=1)
async_result = pool.apply_async(SpawnPhantomJS, (dcap, service_args))
newDriver = async_result.get(10)

Why does flask-testing spawn two test instances?

I want to use the LiveServerTestCase class which is provided by flask-testing to test my flask application in combination with Selenium.
I tried implementing the tests the way described in the flask-testing documentation. But documentation on the LiveServerTestCase is very sparse and I always end up getting two instances of my testcases which are executed at the same time.
I ran my tests through Eclipse and PyCharm with the same behaviour.
How do I have to run/configure my tests to only get one testing instance?
This is how I setup my tests:
import unittest
import urllib2
from selenium import webdriver
from CodeLoad import app
from flask_testing import LiveServerTestCase
class flask_tests(LiveServerTestCase):
def create_app(self):
return app
def setUp(self):
self.driver = webdriver.Firefox()
def tearDown(self):
self.driver.close()
def test_0_server_is_up_and_running(self):
response = urllib2.urlopen(self.get_server_url())
self.assertEqual(response.code, 200)
if __name__ == '__main__':
unittest.main()

Because of a bug.
https://github.com/jarus/flask-testing/issues/33
Try turning DEBUG off