Develop Reference

Trying to make a one hour loop in python

from notifypy import Notify
import schedule
def remember_water():
notification = Notify()
notification.title = "XXX"
notification.message = "XXX"
notification.send()
schedule.every().hour.do(remember_water())
while True:
schedule.run_pending()
time.sleep(1)
Tried to make a notification to drink water every hour... getting a type error when execute, maybe there are some other modules that are better.
Did´nt get in touch with these modules ever, pls help :)

Running your code produces:
Traceback (most recent call last):
File "/home/lars/tmp/python/drink.py", line 11, in <module>
schedule.every().hour.do(remember_water())
File "/home/lars/.local/share/virtualenvs/lars-rUjSNCQn/lib/python3.10/site-packages/schedule/__init__.py", line 625, in do
self.job_func = functools.partial(job_func, *args, **kwargs)
TypeError: the first argument must be callable
Your problem is here:
schedule.every().hour.do(remember_water())
The argument to the .do method must be a callable (like a function), but you are calling your function here and passing the result. You want to pass the function itself:
schedule.every().hour.do(remember_water)

Python 'module' object has no attribute '_strptime'

This error has been arising for quite sometime but doesn't like to appear very frequently, it's time I squashed it.
I see that it appears whenever I have more than a single thread. The applications quite extensive so I'll be posting the code snippet down below.
I'm using the datetime.datetime strptime to format my message into a datetime object. When I use this within a multithreaded function, that error arises on the one thread but works perfectly fine on the other.
The error
Exception in thread Thread-7:
Traceback (most recent call last):
File "C:\Users\kdarling\AppData\Local\Continuum\anaconda3\envs\lobsandbox\lib\threading.py", line 801, in __bootstrap_inner
self.run()
File "C:\Users\kdarling\AppData\Local\Continuum\anaconda3\envs\lobsandbox\lib\threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "d:\kdarling_lob\gryphon_eagle\proj_0012_s2_lobcorr\main\lobx\src\correlator\handlers\inputhandler.py", line 175, in _read_socket
parsed_submit_time = datetime.datetime.strptime(message['msg']['submit_time'], '%Y-%m-%dT%H:%M:%S.%fZ')
AttributeError: 'module' object has no attribute '_strptime'
Alright so this is a bit strange as this was called on the first thread but the second thread works completely fine by using datetime.datetime.
Thoughts
Am I overwriting anything? It is Python afterall. Nope, I don't use datetime anywhere.
I am using inheritance through ABC, how about the parent? Nope, this bug was happening long before and I don't overwrite anything in the parent.
My next go to was thinking "Is is this Python blackmagic with the Datetime module?" in which I decided to time.strptime.
The error
Exception in thread Thread-6:
Traceback (most recent call last):
File "C:\Users\kdarling\AppData\Local\Continuum\anaconda3\envs\lobsandbox\lib\threading.py", line 801, in __bootstrap_inner
self.run()
File "C:\Users\kdarling\AppData\Local\Continuum\anaconda3\envs\lobsandbox\lib\threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "d:\kdarling_lob\gryphon_eagle\proj_0012_s2_lobcorr\main\lobx\src\correlator\handlers\inputhandler.py", line 176, in _read_socket
parsed_submit_time = time.strptime(message['msg']['submit_time'], '%Y-%m-%dT%H:%M:%S.%fZ')
AttributeError: 'module' object has no attribute '_strptime_time'
The code
import datetime
import heapq
import json
import os
import socket
import sys
import time
from io import BytesIO
from threading import Thread
from handler import Handler
class InputStreamHandler(Handler):
def __init__(self, configuration, input_message_heap):
"""
Initialization function for InputStreamHandler.
:param configuration: Configuration object that stores specific information.
:type configuration: Configuration
:param input_message_heap: Message heap that consumers thread will populate.
:type input_message_heap: Heap
"""
super(InputStreamHandler, self).__init__()
self.release_size = configuration.get_release_size()
self.input_src = configuration.get_input_source()
self.input_message_heap = input_message_heap
self.root_path = os.path.join(configuration.get_root_log_directory(), 'input', 'sensor_data')
self.logging = configuration.get_logger()
self.Status = configuration.Status
self.get_input_status_fn = configuration.get_input_functioning_status
self.update_input_status = configuration.set_input_functioning_status
if configuration.get_input_state() == self.Status.ONLINE:
self._input_stream = Thread(target=self._spinup_sockets)
elif configuration.get_input_state() == self.Status.OFFLINE:
self._input_stream = Thread(target=self._read_files)
def start(self):
"""
Starts the input stream thread to begin consuming data from the sensors connected.
:return: True if thread hasn't been started, else False on multiple start fail.
"""
try:
self.update_input_status(self.Status.ONLINE)
self._input_stream.start()
self.logging.info('Successfully started Input Handler.')
except RuntimeError:
return False
return True
def status(self):
"""
Displays the status of the thread, useful for offline reporting.
"""
return self.get_input_status_fn()
def stop(self):
"""
Stops the input stream thread by ending the looping process.
"""
if self.get_input_status_fn() == self.Status.ONLINE:
self.logging.info('Closing Input Handler execution thread.')
self.update_input_status(self.Status.OFFLINE)
self._input_stream.join()
def _read_files(self):
pass
def _spinup_sockets(self):
"""
Enacts sockets onto their own thread to collect messages.
Ensures that blocking doesn't occur on the main thread.
"""
active_threads = {}
while self.get_input_status_fn() == self.Status.ONLINE:
# Check if any are online
if all([value['state'] == self.Status.OFFLINE for value in self.input_src.values()]):
self.update_input_status(self.Status.OFFLINE)
for active_thread in active_threads.values():
active_thread.join()
break
for key in self.input_src.keys():
# Check if key exists, if not, spin up call
if (key not in active_threads or not active_threads[key].isAlive()) and self.input_src[key]['state'] == self.Status.ONLINE:
active_threads[key] = Thread(target=self._read_socket, args=(key, active_threads,))
active_threads[key].start()
print(self.input_src)
def _read_socket(self, key, cache):
"""
Reads data from a socket, places message into the queue, and pop the key.
:param key: Key corresponding to socket.
:type key: UUID String
:param cache: Key cache that corresponds the key and various others.
:type cache: Dictionary
"""
message = None
try:
sensor_socket = self.input_src[key]['sensor']
...
message = json.loads(stream.getvalue().decode('utf-8'))
if 'submit_time' in message['msg'].keys():
# Inherited function
self.write_to_log_file(self.root_path + key, message, self.release_size)
message['key'] = key
parsed_submit_time = time.strptime(message['msg']['submit_time'], '%Y-%m-%dT%H:%M:%S.%fZ')
heapq.heappush(self.input_message_heap, (parsed_submit_time, message))
cache.pop(key)
except:
pass
Additional thought
When this error wasn't being thrown, the two threads share a common function as seen called write_to_log_file. Sometimes, an error would occur where when the write_to_log_file was checking if the OS has a specific directory, it would return False even though it was there. Could this be something with Python and accessing the same function at the same time, even for outside modules? The error was never consistent as well.
Overall, this error won't arise when only running a single thread/connection.

How to share string between processes

I have the hardest time to share a string between processes. I looked at the following Q1, Q2, Q3 but still fail in an actual example. If I understood the docs correctly, looking should be done automatically therefore there should be no race condition with setting/reading. Here's what I've got:
#!/usr/bin/env python3
import multiprocessing
import ctypes
import time
SLEEP = 0.1
CYCLES = 20
def child_process_fun(share):
for i in range(CYCLES):
time.sleep(SLEEP)
share.value = str(time.time())
if __name__ == '__main__':
share = multiprocessing.Value(ctypes.c_wchar_p, '')
process = multiprocessing.Process(target=child_process_fun, args=(share,))
process.start()
for i in range(CYCLES):
time.sleep(SLEEP)
print(share.value)
which produces:
Traceback (most recent call last):
File "test2.py", line 23, in <module>
print(share.value)
File "<string>", line 5, in getvalue
ValueError: character U+e479b7b0 is not in range [U+0000; U+10ffff]
Edit: ´id(share.value)´ is different for each process. However if I try a double as shared variable instead they are the same and it works like a charm. Could this be a python bug?

Chaining Celery Task Methods Error

I'm trying to chain together two task_methods using celery, but I get the following error:
error:
>>> from proj.tasks import A
>>> a = A()
>>> s = (a.add.s(1,2) | a.show.s()).delay().get()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.6/site-packages/celery/result.py", line 175, in get
raise meta['result']
TypeError: show() takes exactly 2 arguments (1 given)
Note, I don't get this error when chaining regular (standalone functions) celery tasks together, only task_methods (class functions). I can't tell if the self object isn't being passed or if the result from the first task isn't being passed.
Here is my project layout:
proj/__init__.py
celery_app.py
tasks.py
tasks.py:
from __future__ import absolute_import
from celery import current_app
from celery.contrib.methods import task_method
from proj.celery_app import app
class A:
def __init__(self):
self.something = 'something'
#current_app.task(filter=task_method)
def add(self,x, y):
return x + y
#current_app.task(filter=task_method)
def show(self,s):
print s
return s
celery_app.py:
from __future__ import absolute_import
from celery import Celery
app = Celery('proj',
broker='amqp://',
backend='amqp://',
include=['proj.tasks'])
app.conf.update(
CELERY_TASK_RESULT_EXPIRES=3600,
)
if __name__ == '__main__':
app.start()
Here's the error from the celery worker:
[2015-04-15 19:57:52,338: ERROR/MainProcess] Task proj.tasks.show[e1e5bc12-6d36-46cd-beb7-fd92a0a5f5c2] raised unexpected: TypeError('show() takes exactly 2 arguments (1 given)',)
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
TypeError: show() takes exactly 2 arguments (1 given)
Has anyone successfully chained task_methods with celery? Thanks!
edit: It's also worth noting that the following code is successful:
>>> from proj.tasks import A
>>> a = A()
>>> sum = a.add.s(1,2).delay().get()
>>> show = a.show.s(sum).delay().get()
Also, I know it could be argued that these functions don't need to be in a class, but pretend they do. I used simple functions to help illustrate the question.
EDIT
I've found a workaround, although a better solution is still desired:
First, revise tasks.py:
...
def show(s,self):
print s
return s
...
Then you can call s = (a.add.s(1,1) | a.show.s(a) ).delay().get(), setting s to 2.
Interestingly enough, calling s = (a.add.s(1,1) | a.show.s(self=a) ).delay().get() spits back the following error:
TypeError: s() got multiple values for keyword argument 'self'
This is not ideal, since the show function can not be called unless in a chain.
For example, the following issues:
>>> a.show(s='test')
TypeError: show() got multiple values for keyword argument 's'
and
>>> a.show(s='test',self=a)
TypeError: __call__() got multiple values for keyword argument 'self'

According to Ask Solem Hoel, the creator of Celery, task methods were a "failed experiment", and will no longer be supported. I guess that answers my question - It can't be done, currently.
Source

Learning the Python Thread Module

I am trying to learn more about the thread module. I've come up with a quick script but am getting an error when I run it. The docs show the format as:
thread.start_new_thread ( function, args[, kwargs] )
My method only has one argument.
#!/usr/bin/python
import ftplib
import thread
sites = ["ftp.openbsd.org","ftp.ucsb.edu","ubuntu.osuosl.org"]
def ftpconnect(target):
ftp = ftplib.FTP(target)
ftp.login()
print "File list from: %s" % target
files = ftp.dir()
print files
for i in sites:
thread.start_new_thread(ftpconnect(i))
The error I am seeing occurs after one iteration of the for loop:
Traceback (most recent call last): File "./ftpthread.py", line 16,
in
thread.start_new_thread(ftpconnect(i)) TypeError: start_new_thread expected at least 2 arguments, got 1
Any suggestions for this learning process would be appreciated. I also looked into using threading, but I am unable to import threading since its not install apparently and I haven't found any documentation for installing that module yet.
Thank You!
There error I get when trying to import threading on my Mac is:
>>> import threading
# threading.pyc matches threading.py
import threading # precompiled from threading.pyc
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "threading.py", line 7, in <module>
class WorkerThread(threading.Thread) :
AttributeError: 'module' object has no attribute 'Thread'

The thread.start_new_thread function is really low-level and doesn't give you a lot of control. Take a look at the threading module, more specifically the Thread class: http://docs.python.org/2/library/threading.html#thread-objects
You then want to replace the last 2 lines of your script with:
# This line should be at the top of your file, obviously :p
from threading import Thread
threads = []
for i in sites:
t = Thread(target=ftpconnect, args=[i])
threads.append(t)
t.start()
# Wait for all the threads to complete before exiting the program.
for t in threads:
t.join()
Your code was failing, by the way, because in your for loop, you were calling ftpconnect(i), waiting for it to complete, and then trying to use its return value (that is, None) to start a new thread, which obviously doesn't work.
In general, starting a thread is done by giving it a callable object (function/method -- you want the callable object, not the result of a call -- my_function, not my_function()), and optional arguments to give the callable object (in our case, [i] because ftpconnect takes one positional argument and you want it to be i), and then calling the Thread object's start method.

Now that you can import threading, start with best practices at once ;-)
import threading
threads = [threading.Thread(target=ftpconnect, args=(s,))
for s in sites]
for t in threads:
t.start()
for t in threads: # shut down cleanly
t.join()

What you want is to pass the function object and arguments to the function to thread.start_new_thread, not execute the function.
Like this:
for i in sites:
thread.start_new_thread(ftpconnect, (i,))