I am trying to run a simple function that checks whether a folder has a single .txt file in it every 5 seconds. I am using threading to run the function every 5 seconds. When the function is to be called a second time i get an error "'bool' object is not callable". I am not sure what I dm doing wrong. It was able to run the first time
program code
import threading
import glob
DIR = r'C:\test_folder\dl_file'
def download_wait5(path_to_downloads):
dl_wait2 = True
dlCounter = len(glob.glob1(path_to_downloads,"*.txt"))
if dlCounter == 1:
dl_wait2 = True
print("True")
else:
dl_wait2 = False
print("False")
return dl_wait2
t = threading.Timer(5.0, download_wait5(DIR))
t.start()
Output
True
Exception in thread Thread-6:
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\threading.py", line 917, in _bootstrap_inner
self.run()
File "C:\ProgramData\Anaconda3\lib\threading.py", line 1166, in run
self.function(*self.args, **self.kwargs)
TypeError: 'bool' object is not callable
I am not sure wha t I am doing wrong. All I want the function to do is return True or False depending if the folder has a .txt file or not.
I also tried this code below after reading the correct syntax for Threading https://docs.python.org/2/library/threading.html but this time it only runs once and then stops:
t = threading.Timer(1.0, download_wait5, args=[DIR])
t.start()
Related
I am having a hard time mocking out threading.Event.wait in one module while leaving threading fully in-tact in another module.
This question is similar to Patch method only in one module, except it has the added complexity of threading having both Event and Thread classes in the same module, with Thread actually using Event internally.
foo.py:
from threading import Event
def use_event_wait() -> None:
Event().wait(5.0) # I want to patch this wait call
bar.py:
import time
from threading import Thread
def use_thread() -> None:
t = Thread(target=time.sleep, args=(0.1,)) # I need Thread to work here
t.start() # NOTE: this uses Event.wait
t.join()
main.py:
from unittest.mock import patch
from bar import use_thread
def swapped_in_wait(_, arg: float) -> bool:
assert arg == 5.0, f"{arg} wasn't 5.0"
return False
with patch("foo.Event.wait", side_effect=swapped_in_wait, autospec=True):
use_thread()
Running this code it blows up:
Traceback (most recent call last):
File "/path/to/main.py", line 9, in <module>
use_thread()
File "/path/to/bar.py", line 6, in use_thread
t.start()
File "/path/to/.pyenv/versions/3.10.3/lib/python3.10/threading.py", line 933, in start
self._started.wait()
File "<string>", line 3, in wait
File "/path/to/.pyenv/versions/3.10.3/lib/python3.10/unittest/mock.py", line 1104, in __call__
return self._mock_call(*args, **kwargs)
File "/path/to/.pyenv/versions/3.10.3/lib/python3.10/unittest/mock.py", line 1108, in _mock_call
return self._execute_mock_call(*args, **kwargs)
File "/Users/james.braza/.pyenv/versions/3.10.3/lib/python3.10/unittest/mock.py", line 1169, in _execute_mock_call
result = effect(*args, **kwargs)
TypeError: swapped_in_wait() missing 1 required positional argument: 'arg'
How can one successfully patch out threading.Event.wait in one module, while leaving it in-tact inside another module?
https://nats-io.github.io/nats.py/modules.html
Traceback (most recent call last):
File "c:\Users\lb\nats.py", line 22, in <module>
asyncio.run(main())
File "D:\anaconda\lib\asyncio\runners.py", line 44, in run
return loop.run_until_complete(main)
File "D:\anaconda\lib\asyncio\base_events.py", line 616, in run_until_complete
return future.result()
File "c:\Users\lb\nats.py", line 6, in main
nc = await nats.connect(servers=["nats://216.48.189.5:4222"])
AttributeError: module 'nats' has no attribute 'connect'
An Attribute error: I am not able to figure out what is the issue with 'connect'.
import asyncio
import nats
async def main():
# Connect to NATS!
nc = await nats.connect(servers=["nats://216.48.189.5:4222"])
# Receive messages on 'foo'
sub = await nc.subscribe("foo")
# Publish a message to 'foo'
await nc.publish("foo", b'Hello from Python!')
# Process a message
msg = await sub.next_msg()
print("Received:", msg)
# Close NATS connection
await nc.close()
if __name__ == '__main__':
asyncio.run(main())
Help me with this connect issue, please.
The solution is to rename your file to anything else.
When you type import nats python recursively tries to find a file or folder named nats. In this case the first file if will find is nats.py.
You never define a function named connect in your nats.py so that fails. You want to instead let the recursive import continue upto site_packages where the actual nats folder is that contains the connect function.
For example, if you name this file hello.py:
import hello
print(hello.hello)
print(hello)
print(hello.hello.hello)
You will see that all 3 print statements print the same thing, since hello will be the file itself.
Simply renaming your file anything else will prevent python from finding the module too soon and will keep searching till it find the correct module.
I just ran your code and got no error.
Try uninstalling and reinstalling nats:
pip install nats-py
I wrote this function that handles the "rate limit error" of a Tweepy's cursor in order to keep downloading from Twitter APIs.
def limit_handled(cursor, user):
over = False
while True:
try:
if (over == True):
print "Routine Riattivata, Serviamo il numero:", user
over = False
yield cursor.next()
except tweepy.RateLimitError:
print "Raggiunto Limite, Routine in Pausa"
threading.Event.wait(15*60 + 15)
over = True
except tweepy.TweepError:
print "TweepError"
threading.Event.wait(5)
Since I am using serveral threads to connect I would like to stop each one of them when the RateLimitError error raises and restart them after 15 minutes.
I previously used the function:
time.sleep(x)
But I understood that doesn't work well for threads (the counter do not increase if the thread is not active) so I tried to use:
threading.Event.wait(x)
But it this error raises:
Exception in thread Thread-15:
Traceback (most recent call last):
File "/home/xor/anaconda/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/home/xor/anaconda/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/home/xor/spyder/algo/HW2/hw2.py", line 117, in work
storeFollowersOnMDB(ids, api, k)
File "/home/xor/spyder/algo/HW2/hw2.py", line 111, in storeFollowersOnMDB
for followersPag in limit_handled(tweepy.Cursor(api.followers_ids, id = user, count=5000).pages(), user):
File "/home/xor/spyder/algo/HW2/hw2.py", line 52, in limit_handled
threading.Event.wait(15*60 + 15)
AttributeError: 'function' object has no attribute 'wait'
How can I "sleep/wait" my threads being sure that they will wake up at the right moment?
Try doing it like this instead:
import threading
dummy_event = threading.Event()
dummy_event.wait(timeout=1)
also try google-ing next time first: Issues with time.sleep and Multithreading in Python
I'm using threading.Timer on python 2.74. This is the relevant piece of code:
__lock = threading.Lock()
def RegeneratePopulationData():
__lock.acquire()
print 'I feel regenerated!'
__lock.release()
def __setRegenerationTimer(firstTime = False):
global __regenerationTimer
now = _getNow().date()
nextRun = datetime(now.year, now.month, now.day + 1, 2, 0)
__regenerationTimer = threading.Timer((nextRun - _getNow()).total_seconds(), __setRegenerationTimer)
print 'Regeneration timer set to run in %s' % (nextRun - _getNow())
print __regenerationTimer.interval
__regenerationTimer.start()
if not firstTime:
RegeneratePopulationData()
def _getNow():
return datetime.now() + timedelta(hours = TIME_DIF)
This cause a TypeError exception to be thrown from threading.py. I've searched the net for hours, and couldn't find a solution.
After changing the code at threading.py (for debugging purposes), around line 349, to:
# than 20 times per second (or the timeout time remaining).
print _time()
print timeout
endtime = _time() + timeout
delay = 0.0005 # 500 us -> initial delay of 1 ms
This is the exception I'm getting:
Regeneration timer set to run in 2:52:12.337000
10332.337
1377634067.66
10332.337
1377634067.66
2013-08-27 21:07:47.663000
Exception in thread Thread-2:
Traceback (most recent call last):
File "C:\Python27\lib\threading.py", line 812, in __bootstrap_inner
self.run()
File "C:\Python27\lib\threading.py", line 1082, in run
self.finished.wait(self.interval)
File "C:\Python27\lib\threading.py", line 622, in wait
self.__cond.wait(timeout)
File "C:\Python27\lib\threading.py", line 350, in wait
endtime = _time() + timeout
TypeError: unsupported operand type(s) for +: 'float' and 'datetime.datetime'
I can't figure out why the interval is suddenly turning into datetime!
Can anyone please help?
Your code doesn't show where timeout is coming from. It's not "suddenly becoming" anything. Whatever your code, at some point previously, a datetime.datetime was assigned to timeout. Either eliminate that condition, or handle it appropriately (perhaps by catching the exception).
Update: You're modifying standard library code, which is an awful idea. Directly or indirectly, you're passing in a datetime.datetime. Use pdb to figure out where. Run your code in the python console. When you trigger the exception do import pdb; pdb.pm(), and you'll drop into the debugger at the site of the exception. You can explore from there.
Sorry in advance, this is going to be long ...
Possibly related:
Python Multiprocessing atexit Error "Error in atexit._run_exitfuncs"
Definitely related:
python parallel map (multiprocessing.Pool.map) with global data
Keyboard Interrupts with python's multiprocessing Pool
Here's a "simple" script I hacked together to illustrate my problem...
import time
import multiprocessing as multi
import atexit
cleanup_stuff=multi.Manager().list([])
##################################################
# Some code to allow keyboard interrupts
##################################################
was_interrupted=multi.Manager().list([])
class _interrupt(object):
"""
Toy class to allow retrieval of the interrupt that triggered it's execution
"""
def __init__(self,interrupt):
self.interrupt=interrupt
def interrupt():
was_interrupted.append(1)
def interruptable(func):
"""
decorator to allow functions to be "interruptable" by
a keyboard interrupt when in python's multiprocessing.Pool.map
**Note**, this won't actually cause the Map to be interrupted,
It will merely cause the following functions to be not executed.
"""
def newfunc(*args,**kwargs):
try:
if(not was_interrupted):
return func(*args,**kwargs)
else:
return False
except KeyboardInterrupt as e:
interrupt()
return _interrupt(e) #If we really want to know about the interrupt...
return newfunc
#atexit.register
def cleanup():
for i in cleanup_stuff:
print(i)
return
#interruptable
def func(i):
print(i)
cleanup_stuff.append(i)
time.sleep(float(i)/10.)
return i
#Must wrap func here, otherwise it won't be found in __main__'s dict
#Maybe because it was created dynamically using the decorator?
def wrapper(*args):
return func(*args)
if __name__ == "__main__":
#This is an attempt to use signals -- I also attempted something similar where
#The signals were only caught in the child processes...Or only on the main process...
#
#import signal
#def onSigInt(*args): interrupt()
#signal.signal(signal.SIGINT,onSigInt)
#Try 2 with signals (only catch signal on main process)
#import signal
#def onSigInt(*args): interrupt()
#signal.signal(signal.SIGINT,onSigInt)
#def startup(): signal.signal(signal.SIGINT,signal.SIG_IGN)
#p=multi.Pool(processes=4,initializer=startup)
#Try 3 with signals (only catch signal on child processes)
#import signal
#def onSigInt(*args): interrupt()
#signal.signal(signal.SIGINT,signal.SIG_IGN)
#def startup(): signal.signal(signal.SIGINT,onSigInt)
#p=multi.Pool(processes=4,initializer=startup)
p=multi.Pool(4)
try:
out=p.map(wrapper,range(30))
#out=p.map_async(wrapper,range(30)).get() #This doesn't work either...
#The following lines don't work either
#Effectively trying to roll my own p.map() with p.apply_async
# results=[p.apply_async(wrapper,args=(i,)) for i in range(30)]
# out = [ r.get() for r in results() ]
except KeyboardInterrupt:
print ("Hello!")
out=None
finally:
p.terminate()
p.join()
print (out)
This works just fine if no KeyboardInterrupt is raised. However, if I raise one, the following exception occurs:
10
7
9
12
^CHello!
None
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python2.6/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "test.py", line 58, in cleanup
for i in cleanup_stuff:
File "<string>", line 2, in __getitem__
File "/usr/lib/python2.6/multiprocessing/managers.py", line 722, in _callmethod
self._connect()
File "/usr/lib/python2.6/multiprocessing/managers.py", line 709, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
File "/usr/lib/python2.6/multiprocessing/connection.py", line 143, in Client
c = SocketClient(address)
File "/usr/lib/python2.6/multiprocessing/connection.py", line 263, in SocketClient
s.connect(address)
File "<string>", line 1, in connect
error: [Errno 2] No such file or directory
Error in sys.exitfunc:
Traceback (most recent call last):
File "/usr/lib/python2.6/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "test.py", line 58, in cleanup
for i in cleanup_stuff:
File "<string>", line 2, in __getitem__
File "/usr/lib/python2.6/multiprocessing/managers.py", line 722, in _callmethod
self._connect()
File "/usr/lib/python2.6/multiprocessing/managers.py", line 709, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
File "/usr/lib/python2.6/multiprocessing/connection.py", line 143, in Client
c = SocketClient(address)
File "/usr/lib/python2.6/multiprocessing/connection.py", line 263, in SocketClient
s.connect(address)
File "<string>", line 1, in connect
socket.error: [Errno 2] No such file or directory
Interestingly enough, the code does exit the Pool.map function without calling any of the additional functions ... The problem seems to be that the KeyboardInterrupt isn't handled properly at some point, but it is a little confusing where that is, and why it isn't handled in interruptable. Thanks.
Note, the same problem happens if I use out=p.map_async(wrapper,range(30)).get()
EDIT 1
A little closer ... If I enclose the out=p.map(...) in a try,except,finally clause, it gets rid of the first exception ... the other ones are still raised in atexit however. The code and traceback above have been updated.
EDIT 2
Something else that does not work has been added to the code above as a comment. (Same error). This attempt was inspired by:
http://jessenoller.com/2009/01/08/multiprocessingpool-and-keyboardinterrupt/
EDIT 3
Another failed attempt using signals added to the code above.
EDIT 4
I have figured out how to restructure my code so that the above is no longer necessary. In the (unlikely) event that someone stumbles upon this thread with the same use-case that I had, I will describe my solution ...
Use Case
I have a function which generates temporary files using the tempfile module. I would like those temporary files to be cleaned up when the program exits. My initial attempt was to pack each temporary file name into a list and then delete all the elements of the list with a function registered via atexit.register. The problem is that the updated list was not being updated across multiple processes. This is where I got the idea of using multiprocessing.Manager to manage the list data. Unfortunately, this fails on a KeyboardInterrupt no matter how hard I tried because the communication sockets between processes were broken for some reason. The solution to this problem is simple. Prior to using multiprocessing, set the temporary file directory ... something like tempfile.tempdir=tempfile.mkdtemp() and then register a function to delete the temporary directory. Each of the processes writes to the same temporary directory, so it works. Of course, this solution only works where the shared data is a list of files that needs to be deleted at the end of the program's life.