Develop Reference

Python multiprocesing pool opens new window

Good evening all,
I have created a CPU heavy function which I need to iterate 50 times with input from a list and then I am using the multiprocessing pool to synchronously compute the 50 functions. It works but creates a new window for each new process and only after I close these windows I do get the result. Is there a way to not open the window each time? I have tried multiple multiprocessing methods, but all do the same.
import sqlite3
import random
import time
import concurrent.futures
from multiprocessing import Pool
from tkinter import *
import time
window = Tk()
window.title("APP")
def bigF(list):
n=list[0] # Extract multiple var from list
n2=list[1]
new = f(n) # The CPU heavy computation.
if n2 == new:
return True
else:
return False
def Screen1():
window.geometry("350x250")
# Lables and entry windows.
btn = Button(window, text='S2', command=Screen2)
btn.pack(pady=10)
def Screen2():
window.geometry("350x220")
# Lables and entry windows.
def getP():
user = user.get()
cursor.execute(database)
try:
return cursor.fetchall()[0][0]
except IndexError:
raise ValueError('')
def getS():
user = user.get()
cursor.execute(database)
return cursor.fetchall()[0][0]
def getS2():
user = user.get()
cursor.execute(database)
return cursor.fetchall()[0][0]
def function():
try:
var = getP()
except ValueError:
return False
s = getS()
s2 = getS2()
t = t.get()
if __name__ == '__main__':
list1=[]
listt=[]
list3=[]
list1[:0]=somestring
random.shuffle(list1)
for i in range(len(list1)):
listt.append(list1[i])
listt.append(var)
listt.append(s)
listt.append(s2)
listt.append(t)
list3.append(listt)
listt=[]
with Pool(50) as p:
values = p.map(bigF, list3)
if True in values:
someF()
else:
# Lables
btn = Button(window, text='function', command=function)
btn.pack(pady=10)
sgn = Button(window, text='S1', command=Screen1)
sgn.pack(padx=10, side=RIGHT)
def someF():
window.geometry("350x350")
# Lables and entry windows.
Screen1()
window.mainloop()
I don't know if the problem is in bigF(list) or the multiprocessing way. I aim to shorten the processing time to less than 2 seconds where 1 bigF(list) takes around 0.5 seconds.
Thank you for any suggestions and comments.

You need to read the cautions in the multiprocessing doc. When your secondary processes start, it starts a new copy of the interpreter and re-executes your main code. That includes starting a new main window. Anything that should be run in the main process only needs to be inside if __name__=='__main__':. So you need:
if __name__ == '__main__':
windows = Tk()
window.title("APP")
Screen1()
window.mainloop()

How can I run a function forever?

Im trying to run the listener function forever, for example I want any time there is new information available in the stream it updates on the list automatically. Any idea in how to do it would be appreciated.
class Notifications(Screen):
notificationslist = ObjectProperty(None)
def listener(self, event = None):
notifications_screen = self.manager.get_screen('notif')
print(event.event_type) # can be 'put' or 'patch'
print(event.path) # relative to the reference, it seems
print(event.data) # new data at /reference/event.path. None if deleted
notifications = event.data
if notifications.items() == None:
return
else:
for key, value in notifications.items():
thevalue = value
notifications_screen.notificationslist.adapter.data.extend([value[0:17] + '\n' + value[18:]])
print(thevalue)
id = (thevalue[thevalue.index("(") + 1:thevalue.rindex(")")])
print(id)

If you want a function to run forever but that does not block you from doing other functions, then you can use threads.
Here is an example with example_of_process that runs forever, and then the main program with time.sleep(3)
import threading
import time
def example_of_process():
count = 0
while True:
print("count", count)
count += 1
time.sleep(1)
thread_1 = threading.Thread(target=example_of_process)
thread_1.daemon = True # without the daemon parameter, the function in parallel will continue even if your main program ends
thread_1.start()
# Now you can do anything else. I made a time sleep of 3s, otherwise the program ends instantly
time.sleep(3)

tkinter change stringvar in loop but no effect on window

I want to change label every sec when i press the button in tk:
# --coding:utf-8 -----
from Tkinter import *
import time
import random
def test(a):
begin =time.time()
end =time.time()
while True:
ran = random.random()
after = time.time()
if(after-begin >1):
a.set(str(ran))
print a.get()
begin =after
if(after-end>10):
a.set('over')
break
t = Tk()
a = StringVar()
a.set('0')
b = Label(t,textvariable = a)
b.pack()
Button(t,text ='test',command = lambda x=a:test(a)).pack()
t.mainloop()
My console output is right,but it doesnot effect on windows.WHY?

You can start the test() function in a separate thread, like this:
# --coding:utf-8 -----
from Tkinter import *
import time
import random
import threading
def startThread(a):
threading.Thread(target=test, args=(a,)).start()
def test(a):
begin =time.time()
end =time.time()
while True:
ran = random.random()
after = time.time()
if(after-begin >1):
a.set(str(ran))
print a.get()
begin =after
if(after-end>10):
a.set('over')
break
t = Tk()
a = StringVar()
a.set('0')
b = Label(t,textvariable = a)
b.pack()
Button(t,text ='test',command = lambda x=a:startThread(a)).pack()
t.mainloop()
However, the thread won't end until the end of the 10 seconds, even if you close the window. You'll need to find some code for the while loop to check if the application is still running.

Creating a behavior that must execute for a given time period is a frequent problem; you can tackle it with a dedicated Waiter class; this avoids the use of threads that are poorly supported by tkinter.
The following example opens a window with a label and a button. When the button is pressed, the label will update every seconds for 10 seconds, then stop.
Pressing the button again within 10 seconds has no effect; after that, it restarts the process for 10 more seconds.
import Tkinter as tk # tkinter if python >= 3
import time
import random
class Waiter(object):
def __init__(self, waiting_time):
"""
:param waiting_time: int, in seconds
"""
self.waiting_time = waiting_time
self.expiring_time = time.time() + self.waiting_time
self.waiting = True
# print('waiter started')
def stop(self):
# print('waiter stopping')
self.expiring_time = None
self.waiting = False
def is_waiting(self):
"""returns True while waiting, false otherwise"""
if time.time() > self.expiring_time:
self.stop()
return self.waiting
def atest():
global waiter
if waiter is None:
waiter = Waiter(10)
_atest()
def _atest():
""" equivalent to:
while the waiter is waiting,
change the label every second),
then destroy the waiter
"""
global waiter
if waiter.is_waiting():
a.set(random.random())
# print(time.time())
t.after(1000, _atest)
else:
waiter = None
if __name__ == '__main__':
waiter = None
t = tk.Tk()
a = tk.StringVar()
a.set('0')
tk.Label(t, textvariable=a).pack()
tk.Button(t, text='test', command=atest).pack()
t.mainloop()
Note:
You could make _atest an inner function of atest, but maybe it is easier to understand as it is?
using import Tkinter as tk instead of from Tkinter import * prevents cluttering the namespace, and arguably makes the code clearer.
You should probably consider using python 3.

How to run a function/thread in a different terminal window in python?

I have a program like this:
from threading import Thread
def foo1(arg):
print("foo1 >>> Something")
input("foo1 >>> Enter Something")
...
def foo2(arg):
print("foo2 >>> Something")
input("foo2 >>> Enter Something")
...
def main():
th1 = Thread(target= foo1)
th1.start()
th2 = Thread(target= foo2)
th2.start()
This program runs both the functions(foo1 and foo2) in the same terminal window. Can I in some way run them in a different terminal window. What I don't wish is to re-run the program. The reason is that they print and take input at the same place and same time. I don't want. Any method?

What you are trying to accomplish isn't possible with just threads, when you create a new Thread it shares all the variables with other threads in your program, including sys.stdout / sys.stdin.
Normally you don't have to worry about PIPES in python programs because it takes care of it for you. print sends the text to sys.stdout and input grabs text from sys.stdin (and error messages are sent to sys.stderr)
So running one program in two terminal windows would mean you would have to have more then one input/output streams, to which there are two solutions:
run a completely separate program with subprocess.Popen like the other fellow described and figure out how to bridge information across the two which is a real pain.
or 2. create your own terminal window with something like tkinter, which is difficult from scratch but luckily IDLE has the majority of the code available in the standard library.
Here is an adapted version of PyShell from idlelib.PyShell to run a Thread instead of the interactive interpretive:
from idlelib import PyShell,EditorWindow
import threading,sys
try:
import tkinter as tk #python 3.X
except ImportError:
import Tkinter as tk #python 2
import tkMessageBox as messagebox
tk.messagebox = messagebox
class ThreadShell(PyShell.PyShell):
"""mostly copied from idlelib.PyShell module but adapted to work with threads"""
#__adapted_by__ = "Tadhg McDonald-Jensen"
def __init__(self, tk_root,target=None):
#not sure exactly what the FileList object is for but it is required by the shell
flist = PyShell.PyShellFileList(tk_root)
super(ThreadShell,self).__init__(flist)
#internal event flag for input, allows thread waiting for input to wait until a tk event handles it
self.__input_flag = threading.Event()
#target is stored and called in .run_command() which also deals with finishing the shell
self.target = target
self.thread = threading.Thread(target=self.run_command)
#tk_root.after makes the .start method call when the program starts (after 0 miliseconds)
tk_root.after(0,self.start)
def start(self):
"""starts executing the Thread"""
super(ThreadShell,self).beginexecuting()
try:
self.thread.start()
except RuntimeError:
self.executing = 0
self.canceled = 0
#self.top.quit() #this causes double deletion warnings with better Implementation of mainloop
beginexecuting = start
def run_command(self):
"""calls target from constructor with self as argument then cleans up shell"""
if self.target:
self.target(self)
self.prompt_exit()
self.executing = 0
self.canceled = 0
try:
self.text.after(1,self.close)
except RuntimeError:
pass #tkinter has issues with changing threads so often after closing one shell others will throw this error
def printf(self,*stuff,**kw):
"""works just like python 3.x print function but writes to shell's .stdout file"""
if self.executing:
## if USING_OLD_METHOD: #Pretty sure this would do exact same thing
## kw.setdefault("file",self.stdout)
## print(*stuff,**kw), self.resetoutput()
## return
sep = kw.get("sep"," ")
end = kw.get("end","\n")
text = sep.join(stuff) + end
self.stdout.write(text)
self.resetoutput()
def input(self,prompt="",timeout=None):
"""python 2 equivelent to raw_input or py 3+ input
Prompts user for input and freezes thread until input is given
Will return "" if .executing is False or it timed out from optional timeout argument"""
if self.executing or self.closing:
if prompt:
self.stdout.write(prompt)
self.__in_buffer = ""
self.__input_flag.clear()
self.reading=True
self.__input_flag.wait(timeout)
#input is inserted into .__in_buffer by other events
#then set __input_flag so that it can be delivered to thread
self.reading = False
return self.__in_buffer.strip("\n")
else:
raise RuntimeError("cannot take input after finished")
def prompt_exit(self):
"""writes press enter to quit" to the console colour then waits for input"""
self.executing = False
self.closing = True
self.console.write("\n press enter to quit")
self.input()
def join_thread(self,timeout=None):
"""sets .executing label to False then waits to join thead,
returns True if thread finished or False if timeout activated"""
self.executing = False
self.closing = True
if self.thread:
self.thread.join(timeout)
return not self.thread.is_alive()
def _close(self):
"Extend EditorWindow._close(), joins thread to close it"
# Restore std streams
sys.stdout = self.save_stdout
sys.stderr = self.save_stderr
sys.stdin = self.save_stdin
# Break cycles
self.interp = None
self.console = None
self.flist.pyshell = None
self.history = None
EditorWindow.EditorWindow._close(self)
self.join_thread()
def stop_readline(self):
self.__in_buffer = ""
self.__input_flag.set()
def update_in(self):
"""updates input from user, I think some of the labels are probably unnecessary but it is easier to leave it alone"""
line = self.text.get("iomark", "end-1c")
if len(line) == 0: # may be EOF if we quit our mainloop with Ctrl-C
line = "\n"
self.resetoutput()
if self.canceled:
self.canceled = 0
if self.endoffile:
self.endoffile = 0
line = ""
self.__in_buffer = line
self.__input_flag.set()
def cancel_callback(self, event=None):
try:
if self.text.compare("sel.first", "!=", "sel.last"):
return # Active selection -- always use default binding
except:
pass
if not (self.executing or self.reading):
return "break"
self.endoffile = 0
self.canceled = 1
if self.reading:
self.update_in()
return "break"
def eof_callback(self, event):
if self.executing and not self.reading:
return # Let the default binding (delete next char) take over
if not (self.text.compare("iomark", "==", "insert") and
self.text.compare("insert", "==", "end-1c")):
return # Let the default binding (delete next char) take over
if not self.executing:
self.resetoutput()
self.close()
else:
self.canceled = 0
self.endoffile = 1
self.update_in()
return "break"
def enter_callback(self, event):
"""called when the enter/return key is pressed,
only the recursive self.top.mainloop() / self.top.quit() had to be changed for support"""
# it is very long to copy/paste for the one line change, so I override the method temporarily
save = self.top.quit
self.top.quit = self.update_in
super(ThreadShell,self).enter_callback(event)
self.top.quit = save
#stupid module depends on this being set from the main function, so it needs to be done manually
PyShell.use_subprocess = True
#this defines the root tkinter window and sets it up
root = tk.Tk()
EditorWindow.fixwordbreaks(root)
root.withdraw()
#I need this to work on my mac, not sure if there are other OS specific stuff that should be included
try:
from idlelib import macosxSupport
macosxSupport.setupApp(root, None)
except (ImportError,AttributeError):
pass
##!!!!!!!!!!!!!!!!!!!! And This Is The Part You Need To Worry About !!!!!!!!!!!!!!!!!!!!##
switch = threading.Event()
switch.clear()
def foo(shell):
global x
x = shell.input("enter a message: ")
switch.set()
shell.printf("message sent")
def foo2(shell):
shell.printf("waiting for message...")
while shell.executing and not switch.is_set():
switch.wait(2) # by using shell.executing in the loop it will occasionally check
# if the program should quit because the window was closed
if shell.executing:
shell.printf("message recieved: ",x)
shell1 = ThreadShell(root,foo)
shell2 = ThreadShell(root,foo2)
first_time = True
while shell1.executing or shell2.executing or first_time:
first_time = False
root.mainloop()
root.destroy()

#!/usr/bin/env python
"""Show messages in two new console windows simultaneously."""
import sys
import platform
from subprocess import Popen
messages = 'This is Console1', 'This is Console2'
def randomFunction():
return "import sys; print(sys.argv[1]); input('Press Enter..')"
# define a command that starts new terminal
if platform.system() == "Windows":
new_window_command = "cmd.exe /c start".split()
else: #XXX this can be made more portable
new_window_command = "x-terminal-emulator -e".split()
# open new consoles, display messages
echo = [sys.executable, "-c",randomFunction()
]
processes = [Popen(new_window_command + echo + [msg]) for msg in messages]
# wait for the windows to be closed
for proc in processes:
proc.wait()

Find working solution for your problem, I haven't used thread, but can be done. And this solution is motivated from solution provided by "Miodrag Novakovic"
You have to change few paths as per your env. Below code is tested on windows
test_code.py -
import sys
import platform
from subprocess import Popen
messages = 'This is Console1', 'This is Console2'
def foo1():
print "In foo1"
i = input("Enter Something - ")
print i
input("Enter to exit")
def foo2():
print "In foo2"
i = input("Enter Something - ")
print i
input("Enter to exit")
def run_foo1():
print("foo1 >>> Something")
return "import sys; sys.path.append('path_to_your_program_folder'); from test_code import foo1; foo1()"
def run_foo2():
print("foo2 >>> Something")
return "import sys; sys.path.append('path_to_your_program_folder'); from test_code import foo2; foo2()"
# define a command that starts new terminal
if platform.system() == "Windows":
new_window_command = "cmd.exe /c start".split()
else: #XXX this can be made more portable
new_window_command = "x-terminal-emulator -e".split()
if __name__ == '__main__':
# open new consoles, display messages
echos = [[sys.executable, "-c",run_foo1()],
[sys.executable, "-c",run_foo2()]
]
processes = [Popen(new_window_command + echo) for echo in echos]
# wait for the windows to be closed
for proc in processes:
proc.wait()

Clear Tkinter Entry text - so new text looks new?

I want to modify the code below to clear the Entry text prior to new text being written. Basically I want to delete text, wait one second, then write new text. This should give the appearance of "NEW" text being written.
Any ideas? TIA - Brad
import thread, Queue, time, random, poster
from Tkinter import *
dataQueue = Queue.Queue()
def status(t):
try:
data = dataQueue.get(block=False)
except Queue.Empty:
pass
else:
t.delete(0, END)
time.sleep(1)
t.insert(0, '%s\n' % str(data))
t.after(2, lambda: status(t))
def makethread():
thread.start_new_thread(poster.poster, (1,dataQueue))
if __name__ == '__main__':
root = Tk()
root.geometry("240x45")
t = Entry(root)
t.pack(side=TOP, fill=X)
Button(root, text='Start Epoch Display',
command=makethread).pack(side=BOTTOM, fill=X)
status(t)
root.mainloop()
In another file called poster
import random, time
def poster(id,que):
while True:
delay=random.uniform(5, 10)
time.sleep(delay)
que.put(' epoch=%f, delay=%f' % (time.time(), delay))

Since there are potentially many threads writing to the queue (one for every time the button is pressed) it is a little unclear when text should be deleted and new text should be inserted. For example, if text has just been written and new text arrives, should the new text be written immediately or should it be added to a queue for later display as time permits?
You can setup the status handler to process delete commands as well as insert commands. This version of the handler starts a thread after every insert that sends back a delete command. If the ID of the delete command matches the ID of the text currently being displayed, then the status handler erases the display:
def status(t, current_id, queue):
try:
data = queue.get(block = False)
# Insert text for ID command:
if type(data) == tuple:
(id, str) = data
t.delete(0, END)
t.insert(0, str)
current_id = id
# Thread that sends a delete command
# after a fixed delay.
make_delete_thread(id, queue)
# Delete text for ID command:
elif data == current_id:
t.delete(0, END)
except Queue.Empty:
pass
t.after(10, lambda: status(t, current_id, queue))
def make_delete_thread(id, queue):
thread.start_new_thread(delete_thread, (id, queue))
def delete_thread(id, queue):
time.sleep(1)
queue.put(id)

Made these changes and it works... Thanks to #anonakos. See my comments to his answer.
Main code:
else:
t.delete(0, END)
time.sleep(1)
t.insert(0, '%s\n' % str(data))
t.after(2, lambda: status(t))
Poster code:
def poster(id,que):
while True:
delay=random.uniform(5, 10)
time.sleep(delay-0.5)
que.put(' ')
time.sleep(.5)
que.put(' epoch=%f, delay=%f' % (time.time(), delay))