Develop Reference

Gtk.Spinner in Python GTK while importing large library

I have a GTK application in C that will spawn a Python GTK process to embed a matplotlib figure into a window in the C process using GtkSocket/GtkPlug (uses XEmbed Protocol). The problem I am having is that the import of the matplotlib library takes about 2.5 seconds and during that time the socket widget is simply transparent. I would like to place a Gtk.Spinner in the plug (so the Python side) before the matplotlib import and have the spinner animate asynchronously during the process of importing the matplotlib library. The problem is that in order for the widget to be placed in the plug, and subsequently, for the Gtk.Spinner to animate, it requires iterations of the GTK main loop. I have approached this from a ton of different angles:
(1) Using a thread. The first attempt was trying to run Gtk.main_iteration() via the thread, however, GTK can only be run on the main thread and this does not work. It stalls the program.
(2) Then I tried to use GObject.idle_add from the thread, where the main loop iterations would run from the idle function (apparently the function called via idle is done on the main thread?), but this didn't work either.
(3) Then I tried to import the modules on the thread, while the main thread runs the Gtk.main_iteration()'s to allow the spinner to spin while the imports are taking place. The idea was once the imports are complete, a boolean flag would change to trigger a break from the loop of main iterations. In this case the spinner appears and spins but the plot never shows up. I get an X Server error:
Gdk-WARNING **: master: Fatal IO error 104 (Connection reset by peer) on X server :0.
(4) In lieu of threading, I tried to use GObject.timeout_add to call a function regularly that would perform the Gtk.main_iteration()'s, but doing that results in the original behavior where the socket/plug is transparent until the plot shows up (i.e. no spinner appears nor spins).
I have run out of ideas and now I am coming here hoping for an assist. They key idea is to get the Gtk.Spinner spinning while the Python script is loading the matplotlib library, and once that is done, replace the spinner widget with the figure (while all of this is taking place in a GtkSocket/Plug). I have not created a minimal reproducible example for this since it would be rather complex given the circumstances, but if anyone that is willing to help requests one I could come up with it. However, the relevant code section is below (with previous attempts commented out):
import sys
import gi
import time
import threading
gi.require_version('Gtk', '3.0')
from gi.repository import Gtk, Gdk, GObject
from gi.repository import Pango as pango
if sys.platform != "win32":
GObject.threads_init()
Gdk.threads_init()
# Open socket ID file, read Socket ID into variable, close file
socketFile = open('resources/com/gtkSocket', 'r+')
gtkSock = socketFile.read()
print("The ID of the sockets window in Python is: ", int(gtkSock))
socketFile.close()
# Create plug, create GTK box, add box to plug, add spinner to box
spin_plug = Gtk.Plug.new(int(gtkSock))
socketbox = Gtk.Box()
spin_plug.add(socketbox)
spinner = Gtk.Spinner()
socketbox.pack_start(spinner, expand=True, fill=True, padding=False)
spinner.start()
finished = False
def thread_run():
time.sleep(4)
'''
# Loop for four seconds checking if gtk events are pending, and if so, main loop iterate
t_end = time.time() + 4
while time.time() < t_end:
if (Gtk.events_pending()):
Gtk.main_iteration()
print("Events Pending...")
'''
'''
import argparse
import collections
import csv
import matplotlib
import matplotlib.pyplot as plt
import matplotlib.patches as patches
import matplotlib.lines as mlines
from collections import defaultdict
from enum import Enum
from matplotlib.backend_bases import MouseEvent
from matplotlib.pyplot import draw
from matplotlib.widgets import SpanSelector
from matplotlib.backends.backend_gtk3agg import FigureCanvasGTK3Agg as FC
from matplotlib.backends.backend_gtk3 import NavigationToolbar2GTK3
'''
# You cannot run GTK Code on a separate thread from the one running the main loop
# Idle add allows scheduling code to be executed on the main thread
GObject.idle_add(cleanup)
def cleanup():
# Note: Trying to add the Gtk Main Iterations to the idle add function did not work...
print("Closing Spinner Thread...")
spinner.stop()
finished = True
thread.join()
# start a separate thread and immediately return to main loop
#thread = threading.Thread(target=thread_run)
#thread.start()
spin_plug.show_all()
def spin():
busy_wait = 0
while (Gtk.events_pending() or busy_wait < 10):
Gtk.main_iteration()
if (not Gtk.events_pending()):
busy_wait = busy_wait + 1
print("Spin Call complete.")
return True
GObject.timeout_add(50, spin)
'''
# We cannot simply run an infinite Gtk.main() loop, so iterate until the plug has been filled
busy_wait = 0
while (Gtk.events_pending() or busy_wait < 10):
if (finished):
break
print("Busy Wait: %d" % busy_wait)
Gtk.main_iteration()
if (not Gtk.events_pending()):
busy_wait = busy_wait + 1
print("Gtk Main Loop iterations complete.")
'''
Any pointers or ideas would be greatly appreciated.

The solution was performing the imports on a thread while allowing the main thread to do the main loop iterations. Simply doing "import " did not work. Some previous Stack Overflow posts that were useful are here:
Python thread for pre-importing modules
import a module from a thread does not work
Import python modules in the background in REPL
The solution looks like this:
GObject.threads_init()
Gdk.threads_init()
# Open socket ID file, read Socket ID into variable, close file
socketFile = open('resources/com/gtkSocket', 'r+')
gtkSock = socketFile.read()
print("The ID of the sockets window in Python is: ", int(gtkSock))
socketFile.close()
# Create plug, create GTK box, add box to plug, add figure to box
spin_plug = Gtk.Plug.new(int(gtkSock))
socketbox = Gtk.Box()
spin_plug.add(socketbox)
# Create a spinner, pack it, and start it spinning
spinner = Gtk.Spinner()
socketbox.pack_start(spinner, expand=True, fill=True, padding=False)
spinner.start()
spinner.show()
# Flag to break from the Gtk.events_pending() loop
finished = False
# This will load modules on a thread. A simple "import module" does not work
def do_import(module_name):
thismodule = sys.modules[__name__]
module = importlib.import_module(module_name)
setattr(thismodule, module_name, module)
print(module_name, 'imported')
# Use the last module being imported so we know when to break from the Gtk.events_pending()
if (module_name == "matplotlib.pyplot"):
global finished
finished = True
spin_plug.show_all()
modules_to_load = ['argparse', 'collections', 'csv', 'matplotlib', 'matplotlib.pyplot']
# Loop through and create a thread for each module to import from the list
for module_name in modules_to_load:
thread = threading.Thread(target=do_import, args=(module_name,))
thread.start()
# We cannot simply run an infinite Gtk.main() loop, so iterate until the plug has been filled
# Busy wait continues to allow the spinner to spin until the computer loads the modules. Since
# each computer will have a different loading speed, a busy wait of 300 should cover slower
# machines. We can break out of the loop early once the last module is loaded.
busy_wait = 0
while (Gtk.events_pending() or busy_wait < 300):
#print("Busy Wait: %d" % busy_wait)
#print ("finished: %d" % finished)
if (finished):
break
Gtk.main_iteration()
if (not Gtk.events_pending()):
busy_wait = busy_wait + 1

How to make my code stopable? (Not killing/interrupting)

I'm asking this question in a more broad spectrum because I'm not facing this specific issue right now, but I'm wondering how to do it in the future.
If I have a long running python script, that is supposed to do something all the time (could be a infine loop, if that helps). The code is started by running python main.py command on a terminal.
The code doesn't have an ending, so there will be no sys.exit().
I don't want to use KeyboardInterrupt and I don't want to kill the task. Because those options are abrupt, and you can't predict precisely at what point you are stoping the code.
Is there a way to 'softly' terminate the code when I eventually decide to fo it? For example using another command, preparing a class or running another script?
What would be the best practice for this?
PS.: Please, bear in mind that I'm a novice coder.
EDIT:
I'm adding some generic code, in order to make my question clearer.
import time,csv
import GenericAPI
class GenericDataCollector:
def __init__(self):
self.generic_api = GenericAPI()
def collect_data(self):
while True: #Maybe this could be a var that is changed from outside of the class?
data = self.generic_api.fetch_data() #Returns a JSON with some data
self.write_on_csv(data)
time.sleep(1)
def write_on_csv(self, data):
with open('file.csv','wt') as f:
writer = csv.writer(f)
writer.writerow(data)
def run():
obj = GenericDataCollector()
obj.collect_data()
if __name__ == "__main__":
run()
In this particular case, the class is collecting data from some generic API (that comes in JSON) and writing it in a csv file, in a infinite loop. How could I code a way (method?) to stop it (when called uppon, so unexpected), without abruptly interrupting (Ctrl+C or killing task).

I would recommend use the signal module. This allows you to handle signal interrupts (SIGINT) and clean up the program before your exit. Take the following code for example:
import signal
running = True
def handle(a, b):
global running
running = False
# catch the SIGINT signal and call handle() when the process
# receives it
signal.signal(signal.SIGINT, handle)
# your code here
while running:
pass
You can still exit with a Ctrl+C, but what you put in the while loop will not be cut off half way.

Based on #Calder White, how about this (not tested):
import signal
import time,csv
import GenericAPI
class GenericDataCollector:
def __init__(self):
self.generic_api = GenericAPI()
self.cont = True
def collect_data(self):
while self.cont:
signal.signal(signal.SIGINT, self.handle)
data = self.generic_api.fetch_data() #Returns a JSON with some data
self.write_on_csv(data)
time.sleep(1)
def handle(self):
self.cont = False
def write_on_csv(self, data):
with open('file.csv','wt') as f:
writer = csv.writer(f)
writer.writerow(data)
def run():
obj = GenericDataCollector()
obj.collect_data()
if __name__ == "__main__":
run()

Threading with Bottle.py Server

I'm having an issue with threading that I can't solve in any way I've tried. I searched in StackOverflow too, but all I could find was cases that didn't apply to me, or explanations that I didn't understand.
I'm trying to build an app with BottlePy, and one of the features I want requires a function to run in background. For this, I'm trying to make it run in a thread. However, when I start the thread, it runs twice.
I've read in some places that it would be possible to check if the function was in the main script or in a module using if __name__ == '__main__':, however I'm not able to do this, since __name__ is always returning the name of the module.
Below is an example of what I'm doing right now.
The main script:
# main.py
from MyClass import *
from bottle import *
arg = something
myObject = Myclass(arg1)
app = Bottle()
app.run('''bottle args''')
The class:
# MyClass.py
import threading
import time
class MyClass:
def check_list(self, theList, arg1):
a_list = something()
time.sleep(5)
self.check_list(a_list, arg1)
def __init__(self, arg1):
if __name__ == '__main__':
self.a_list = arg.returnAList()
t = threading.Thread(target=self.check_list, args=(a_list, arg1))
So what I intend here is to have check_list running in a thread all the time, doing something and waiting some seconds to run again. All this so I can have the list updated, and be able to read it with the main script.
Can you explain to me what I'm doing wrong, why the thread is running twice, and how can I avoid this?

This works fine:
import threading
import time
class MyClass:
def check_list(self, theList, arg1):
keep_going=True
while keep_going:
print("check list")
#do stuff
time.sleep(1)
def __init__(self, arg1):
self.a_list = ["1","2"]
t = threading.Thread(target=self.check_list, args=(self.a_list, arg1))
t.start()
myObject = MyClass("something")

Figured out what was wrong thanks to the user Weeble's comment. When he said 'something is causing your main.py to run twice' I remembered that Bottle has an argument that is called 'reloader'. When set to True, this will make the application load twice, and thus the thread creation is run twice as well.

wrpcap not creating pcap file, when running in separate thread

I'm trying to capture network traffic using scapy , by running sniff in a thread rather than running it in the main thread itself. So as to avoid blocking of the app . But the problem that i'm facing is that that the wrpcap is not creating a file, if it does it creates a 0 Kb file.
The other reason why i'm using a separate thread is, because i want to be able to close the thread when the user wishes to end the capture.
from scapy.all import *
from time import gmtime , strftime
import threading
import time
def bomber(stop_event):
data=[]
pkts=[]
while not stop_event.isSet() :
pkt100=sniff(count=100)
data.append(pkt100)
for pkt in data:
for x in range(100):
pkts.append(pkt[x])
pktsRoll=PacketList(pkts)
savename="F:\\%s.pcap" % strftime("%Y-%m-%d,%H:%M", gmtime())
wrpcap(savename,pktsRoll)
print " its done "
def main():
stop_event = threading.Event()
c_thread = threading.Thread(target=bomber, args=(stop_event,))
c_thread.start()
time.sleep(20)
stop_event.set()
This is the code that i'm trying out . I'm using Python 2.5
Thanks in advance.

libvlc and dbus interface

I'm trying a to create a basic media player using libvlc which will be controlled through dbus. I'm using the gtk and libvlc bindings for python. The code is based on the official example from the vlc website
The only thing I modified is to add the dbus interface to the vlc instance
# Create a single vlc.Instance() to be shared by (possible) multiple players.
instance = vlc.Instance()
print vlc.libvlc_add_intf(instance, "dbus"); // this is what i added. // returns 0 which is ok
All is well, the demo works and plays any video files. but for some reason the dbus control module doesn't work (I can't believe I just said the dreaded "doesn't work" words):
I already have the working client dbus code which binds to the MPRIS 2 interface. I can control a normal instance of a VLC media player - that works just fine, but with the above example nothing happens. The dbus control module is loaded properly, since libvlc_add_intf doesn't return an error and i can see the MPRIS 2 service in D-Feet (org.mpris.MediaPlayer2.vlc).
Even in D-Feet, trying to call any of the methods of the dbus vlc object returns no error but nothing happens.
Do I need to configure something else in order to make the dbus module control the libvlc player?
Thanks
UPDATE
It seems that creating the vlc Instance and setting a higher verbosity, shows that the DBus calls are received but they have no effect whatsoever on the player itself.
Also, adding the RC interface to the instance instead of DBus, has some problems too: When I run the example from the command line it drops me to the RC interface console where i can type the control commands, but it has the same behaviour as DBus - nothing happens, no error, nada, absolutely nothing. It ignores the commands completely.
Any thoughts?
UPDATE 2
Here is the code that uses libvlc to create a basic player:
from dbus.mainloop.glib import DBusGMainLoop
import gtk
import gobject
import sys
import vlc
from gettext import gettext as _
# Create a single vlc.Instance() to be shared by (possible) multiple players.
instance = vlc.Instance("--one-instance --verbose 2")
class VLCWidget(gtk.DrawingArea):
"""Simple VLC widget.
Its player can be controlled through the 'player' attribute, which
is a vlc.MediaPlayer() instance.
"""
def __init__(self, *p):
gtk.DrawingArea.__init__(self)
self.player = instance.media_player_new()
def handle_embed(*args):
if sys.platform == 'win32':
self.player.set_hwnd(self.window.handle)
else:
self.player.set_xwindow(self.window.xid)
return True
self.connect("map", handle_embed)
self.set_size_request(640, 480)
class VideoPlayer:
"""Example simple video player.
"""
def __init__(self):
self.vlc = VLCWidget()
def main(self, fname):
self.vlc.player.set_media(instance.media_new(fname))
w = gtk.Window()
w.add(self.vlc)
w.show_all()
w.connect("destroy", gtk.main_quit)
self.vlc.player.play()
DBusGMainLoop(set_as_default = True)
gtk.gdk.threads_init()
gobject.MainLoop().run()
if __name__ == '__main__':
if not sys.argv[1:]:
print "You must provide at least 1 movie filename"
sys.exit(1)
if len(sys.argv[1:]) == 1:
# Only 1 file. Simple interface
p=VideoPlayer()
p.main(sys.argv[1])
the script can be run from the command line like:
python example_vlc.py file.avi
The client code which connects to the vlc dbus object is too long to post so instead pretend that i'm using D-Feet to get the bus connection and post messages to it.
Once the example is running, i can see the players dbus interface in d-feet, but i am unable to control it. Is there anything else that i should add to the code above to make it work?

I can't see your implementation of your event loop, so it's hard to tell what might be causing commands to not be recognized or to be dropped. Is it possible your threads are losing the stacktrace information and are actually throwing exceptions?
You might get more responses if you added either a psuedo-code version of your event loop and DBus command parsing or a simplified version?

The working programs found on nullege.com use ctypes. One which acted as a server used rpyc. Ignoring that one.
The advantages of ctypes over dbus is a huge speed advantage (calling the C library code, not interacting using python) as well as not requiring the library to implement the dbus interface.
Didn't find any examples using gtk or dbus ;-(
Notable examples
PyNuvo vlc.py
Milonga Tango DJing program
Using dbus / gtk
dbus uses gobject mainloop, not gtk mainloop. Totally different beasts. Don't cross the streams! Some fixes:
Don't need this. Threads are evil.
gtk.gdk.threads_init()
gtk.main_quit() shouldn't work when using gobject Mainloop. gobject mainloop can't live within ur class.
if __name__ == '__main__':
loop = gobject.MainLoop()
loop.run()
Pass in loop into ur class. Then call to quit the app
loop.quit()
dbus (notify) / gtk working example
Not going to write ur vlc app for u. But here is a working example of using dbus / gtk. Just adapt to vlc. Assumed u took my advise on gtk above. As u know any instance of DesktopNotify must be called while using gobject.Mainloop . But u can place it anywhere within ur main class.
desktop_notify.py
from __future__ import print_function
import gobject
import time, dbus
from dbus.exceptions import DBusException
from dbus.mainloop.glib import DBusGMainLoop
class DesktopNotify(object):
""" Notify-OSD ubuntu's implementation has a 20 message limit. U've been warned. When queue is full, delete old message before adding new messages."""
#Static variables
dbus_loop = None
dbus_proxy = None
dbus_interface = None
loop = None
#property
def dbus_name(self):
return ("org.freedesktop.Notifications")
#property
def dbus_path(self):
return ("/org/freedesktop/Notifications")
#property
def dbus_interface(self):
return self.dbus_name
def __init__(self, strInit="initializing passive notification messaging")
strProxyInterface = "<class 'dbus.proxies.Interface'>"
""" Reinitializing dbus when making a 2nd class instance would be bad"""
if str(type(DesktopNotify.dbus_interface)) != strProxyInterface:
DesktopNotify.dbus_loop = DBusGMainLoop(set_as_default=True)
bus = dbus.SessionBus(mainloop=DesktopNotify.dbus_loop)
DesktopNotify.dbus_proxy = bus.get_object(self.dbus_name, self.dbus_path)
DesktopNotify.dbus_interface = dbus.Interface(DesktopNotify.dbus_proxy, self.dbus_interface )
DesktopNotify.dbus_proxy.connect_to_signal("NotificationClosed", self.handle_closed)
def handle_closed(self, *arg, **kwargs):
""" Notification closed by user or by code. Print message or not"""
lngNotificationId = int(arg[0])
lngReason = int(arg[1])
def pop(self, lngID):
""" ID stored in database, but i'm going to skip this and keep it simple"""
try:
DesktopNotify.dbus_interface.CloseNotification(lngID)
except DBusException as why:
print(self.__class__.__name__ + ".pop probably no message with id, lngID, why)
finally:
pass
def push(self, strMsgTitle, strMsg, dictField):
""" Create a new passive notification (took out retrying and handling full queues)"""
now = time.localtime( time.time() )
strMsgTime = strMsg + " " + time.asctime(now)
del now
strMsgTime = strMsgTime % dictField
app_name="[your app name]"
app_icon = ''
actions = ''
hint = ''
expire_timeout = 10000 #Use seconds * 1000
summary = strMsgTitle
body = strMsgTime
lngNotificationID = None
try:
lngNotificationID = DesktopNotify.dbus_interfacec.Notify(app_name, 0, app_icon, summary, body, actions, hint, expire_timeout)
except DBusException as why:
#Excellent spot to delete oldest notification and then retry
print(self.__class__.__name__ + ".push Being lazy. Posting passive notification was unsuccessful.", why)
finally:
#Excellent spot to add to database upon success
pass