I'm trying to chain together two task_methods using celery, but I get the following error:
error:
>>> from proj.tasks import A
>>> a = A()
>>> s = (a.add.s(1,2) | a.show.s()).delay().get()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.6/site-packages/celery/result.py", line 175, in get
raise meta['result']
TypeError: show() takes exactly 2 arguments (1 given)
Note, I don't get this error when chaining regular (standalone functions) celery tasks together, only task_methods (class functions). I can't tell if the self object isn't being passed or if the result from the first task isn't being passed.
Here is my project layout:
proj/__init__.py
celery_app.py
tasks.py
tasks.py:
from __future__ import absolute_import
from celery import current_app
from celery.contrib.methods import task_method
from proj.celery_app import app
class A:
def __init__(self):
self.something = 'something'
#current_app.task(filter=task_method)
def add(self,x, y):
return x + y
#current_app.task(filter=task_method)
def show(self,s):
print s
return s
celery_app.py:
from __future__ import absolute_import
from celery import Celery
app = Celery('proj',
broker='amqp://',
backend='amqp://',
include=['proj.tasks'])
app.conf.update(
CELERY_TASK_RESULT_EXPIRES=3600,
)
if __name__ == '__main__':
app.start()
Here's the error from the celery worker:
[2015-04-15 19:57:52,338: ERROR/MainProcess] Task proj.tasks.show[e1e5bc12-6d36-46cd-beb7-fd92a0a5f5c2] raised unexpected: TypeError('show() takes exactly 2 arguments (1 given)',)
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
TypeError: show() takes exactly 2 arguments (1 given)
Has anyone successfully chained task_methods with celery? Thanks!
edit: It's also worth noting that the following code is successful:
>>> from proj.tasks import A
>>> a = A()
>>> sum = a.add.s(1,2).delay().get()
>>> show = a.show.s(sum).delay().get()
Also, I know it could be argued that these functions don't need to be in a class, but pretend they do. I used simple functions to help illustrate the question.
EDIT
I've found a workaround, although a better solution is still desired:
First, revise tasks.py:
...
def show(s,self):
print s
return s
...
Then you can call s = (a.add.s(1,1) | a.show.s(a) ).delay().get(), setting s to 2.
Interestingly enough, calling s = (a.add.s(1,1) | a.show.s(self=a) ).delay().get() spits back the following error:
TypeError: s() got multiple values for keyword argument 'self'
This is not ideal, since the show function can not be called unless in a chain.
For example, the following issues:
>>> a.show(s='test')
TypeError: show() got multiple values for keyword argument 's'
and
>>> a.show(s='test',self=a)
TypeError: __call__() got multiple values for keyword argument 'self'
According to Ask Solem Hoel, the creator of Celery, task methods were a "failed experiment", and will no longer be supported. I guess that answers my question - It can't be done, currently.
Source
Related
from notifypy import Notify
import schedule
def remember_water():
notification = Notify()
notification.title = "XXX"
notification.message = "XXX"
notification.send()
schedule.every().hour.do(remember_water())
while True:
schedule.run_pending()
time.sleep(1)
Tried to make a notification to drink water every hour... getting a type error when execute, maybe there are some other modules that are better.
Did´nt get in touch with these modules ever, pls help :)
Running your code produces:
Traceback (most recent call last):
File "/home/lars/tmp/python/drink.py", line 11, in <module>
schedule.every().hour.do(remember_water())
File "/home/lars/.local/share/virtualenvs/lars-rUjSNCQn/lib/python3.10/site-packages/schedule/__init__.py", line 625, in do
self.job_func = functools.partial(job_func, *args, **kwargs)
TypeError: the first argument must be callable
Your problem is here:
schedule.every().hour.do(remember_water())
The argument to the .do method must be a callable (like a function), but you are calling your function here and passing the result. You want to pass the function itself:
schedule.every().hour.do(remember_water)
I am trying to use concurrent.futures.ThreadPoolExecutor module to run a class method in parallel, the simplified version of my code is pretty much the following:
class TestClass:
def __init__(self, secondsToSleepFor):
self.secondsToSleepFor = secondsToSleepFor
def testMethodToExecInParallel(self):
print("ThreadName: " + threading.currentThread().getName())
print(threading.currentThread().getName() + " is sleeping for " + str(self.secondsToSleepFor) + " seconds")
time.sleep(self.secondsToSleepFor)
print(threading.currentThread().getName() + " has finished!!")
with concurrent.futures.ThreadPoolExecutor(max_workers = 2) as executor:
futuresList = []
print("before try")
try:
testClass = TestClass(3)
future = executor.submit(testClass.testMethodToExecInParallel)
futuresList.append(future)
except Exception as exc:
print('Exception generated: %s' % exc)
If I execute this code it seems to behave like it is intended to.
But if I make a mistake like specifying a wrong number of parameters in "testMethodToExecInParallel" like:
def testMethodToExecInParallel(self, secondsToSleepFor):
and then still submitting the function as:
future = executor.submit(testClass.testMethodToExecInParallel)
or trying to concatenate a string object with an integer object (without using str(.) ) inside a print statement in "testMethodToExecInParallel" method:
def testMethodToExecInParallel(self):
print("ThreadName: " + threading.currentThread().getName())
print("self.secondsToSleepFor: " + self.secondsToSleepFor) <-- Should report an Error here
the program doesn't return any error; just prints "before try" and ends execution...
Is trivial to understand that this makes the program nearly undebuggable... Could someone explain me why such behaviour happens?
(for the first case of mistake) concurrent.futures.ThreadPoolExecutor doesn't check for a function with the specified signature to submit and, eventually, throw some sort of "noSuchFunction" exception?
Maybe there is some sort of problem in submitting to ThreadPoolExecutor class methods instead of simple standalone functions and, so, such behaviour could be expected?
Or maybe the error is thrown inside the thread and for some reason I can't read it?
-- EDIT --
Akshay.N suggestion of inserting future.result() after submitting functions to ThreadPoolExecutor makes the program behave as expected: goes nice if the code is correct, prints the error if something in the code is wrong.
I thing users must be warned about this very strange behaviour of ThreadPoolExecutor:
if you only submit functions to ThreadPoolExecutor WITHOUT THEN CALLING future.result():
- if the code is correct, the program goes on and behaves as expected
- if something in the code is wrong seems the program doesn't call the submitted function, whatever it does: it doesn't report the errors in the code
As far as my knowledge goes which is "not so far", you have to call "e.results()" after "executor.submit(testClass.testMethodToExecInParallel)" in order to execute the threadpool .
I have tried what you said and it is giving me error, below is the code
>>> import concurrent.futures as cf
>>> executor = cf.ThreadPoolExecutor(1)
>>> def a(x,y):
... print(x+y)
...
>>> future = executor.submit(a, 2, 35, 45)
>>> future.result()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\username
\AppData\Local\Programs\Python\Python37\lib\concurrent\futures\_base.py", line
425, in result
return self.__get_result()
File "C:\Users\username
\AppData\Local\Programs\Python\Python37\lib\concurrent\futures\_base.py", line
384, in __get_result
raise self._exception
File "C:\Users\username
\AppData\Local\Programs\Python\Python37\lib\concurrent\futures\thread.py", line
57, in run
result = self.fn(*self.args, **self.kwargs)
TypeError: a() takes 2 positional arguments but 3 were given
Let me know if it still doesn't work
I'm trying to do post automatic schedule in Telegram. It runs first run but after its try to looping I get an error:
TypeError: the first argument must be callable
My code :
import time
import schedule
from pyrogram import Client, ReplyKeyboardMarkup, InlineKeyboardMarkup, InlineKeyboardButton
from pyrogram.api import functions, types
from pyrogram.api.errors import FloodWait
person1 = Client(
session_name="*cenzored*",
api_id=*cenzored*,
api_hash="*cenzored*"
)
person2 = Client(
session_name="*cenzored*",
api_id=*cenzored*,
api_hash="*cenzored*"
)
wick.start()
def Publish(session,dat,msgid,session_name):
try:
session.start()
print("[%s]Posting..." % (session_name))
session.send(
functions.messages.GetBotCallbackAnswer(
peer=session.resolve_peer("*cenzored*"),
msg_id=msgid,
data=b'publish %d' % (dat)
)
)
session.idle()
except:
print("Crashed,starting over")
schedule.every(0.3).minutes.do(Publish(person1,142129,12758, 'Dani')) // Here is the line is crashing.
schedule.every(0.3).minutes.do(Publish(person2,137351,13177, 'Wick'))
while 1:
schedule.run_pending()
time.sleep(1)
Traceback :
Pyrogram v0.7.4, Copyright (C) 2017-2018 Dan Tès <https://github.com/delivrance>
Licensed under the terms of the GNU Lesser General Public License v3 or later (LGPLv3+)
[person1]Posting...
3: 2018-06-16 12:07:26.529829 Retrying <class 'pyrogram.api.functions.messages.get_bot_callback_answer.GetBotCallbackAnswer'>
4: 2018-06-16 12:07:42.041309 Retrying <class 'pyrogram.api.functions.messages.get_bot_callback_answer.GetBotCallbackAnswer'>
Crashed,starting over
Traceback (most recent call last):
File "C:\Users\343df\OneDrive\Desktop\Maor\python\tele\tele.py", line 35, in <module>
schedule.every(0.3).minutes.do(Publish('ss',dani,140129,12758, 'Dani'))
File "C:\Program Files (x86)\Python36-32\lib\site-packages\schedule\__init__.py", line 385, in do
self.job_func = functools.partial(job_func, *args, **kwargs)
TypeError: the first argument must be callable
Basically my problem is that is not running (TypeError: the first argument must be callable) after first time and its scheduled to run every 0.3 seconds.
According to [ReadTheDocs]: do(job_func, *args, **kwargs), you don't have to call Publish, but just pass it, followed by its argument list (the call will be performed by schedule framework):
schedule.every(0.3).minutes.do(Publish, person1, 142129, 12758, "Dani")
schedule.every(0.3).minutes.do(Publish, person2, 137351, 13177, "Wick")
.do() expects a callable as the first argument (and additional arguments to .do will be passed to that callable).
So instead of:
schedule.every(0.3).minutes.do(Publish(person1,142129,12758, 'Dani'))
schedule.every(0.3).minutes.do(Publish(person2,137351,13177, 'Wick'))
you likely want:
schedule.every(0.3).minutes.do(Publish, person1, 142129, 12758, 'Dani')
schedule.every(0.3).minutes.do(Publish, person2, 137351, 13177, 'Wick')
I'm currently trying out the IronPython interpreter. While doing the Tutorial i came across delegates and event handlers. The tutorial does something like this:
from System.IO import FileSystemWatcher
w = FileSystemWatcher()
def handle(*args):
print args
w.Changed += handle
So i tried to be smart and do this:
from System.IO import FileSystemWatcher
from __future__ import print_function
from functools import partial
w = FileSystemWatcher()
w.Changed += partial(print, "Changed: ")
Which failed with:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Object is not callable.
Where line 1 refers to the last line in the (interactive session)
So IronPython thinks a partial object is not callable although callable(partial(print, "Changed: ")) returns True
With this workaround the handler is accepted:
w.Changed += partial(print, "Changed: ").__call__
My question:
Why is a partial object not accepted as an event handler. Is this a bug?
This is probably not solution, one could expect, but there is an issue, opened for couple years now - https://github.com/IronLanguages/main/issues/808
Doesn't work in 2.6.2 and 2.7b1 on .NET Version: 4.0.30319.1 ipy26
testcase-26482.py
Object is not callable.
ipy27 testcase-26482.py
Object is not callable.py
testcase-26482.py
Object is not callable.
I just tried to learn the mock and nosetests by running simple examples, but got no luck:
john$ nosetests test_mylib.py
E
======================================================================
ERROR: test_mylib.test_mylib_foo
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/wjq/py-virtenv-2.7.5/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/Users/wjq/py-virtenv-2.7.5/lib/python2.7/site-packages/mock.py", line 1201, in patched
return func(*args, **keywargs)
TypeError: test_mylib_foo() takes exactly 2 arguments (1 given)
However if I run the test directly, it's ok:
john$ python test_mylib.py
john$
I think I must miss some key understanding on the tow libraries since I'm new to them. Really appreciate if someone can point them out.
The followings are my example codes.
test_mylib.py
import mock
import mylib
#mock.patch('mylib.incr')
def test_mylib_foo(aa, incr):
incr.return_value=5
assert mylib.foo(1) == 6
if __name__ == '__main__':
test_mylib_foo(123)
mylib.py
from depen import incr
def foo(aa):
return incr(aa) +1
depen.py
def incr(aa):
return aa+1
Remove the aa argument and it'll work just fine:
#mock.patch('mylib.incr')
def test_mylib_foo(incr):
incr.return_value=5
assert mylib.foo(1) == 6
if __name__ == '__main__':
test_mylib_foo()
A better __main__ execution would call nose.runmodule:
if __name__ == '__main__':
import nose
nose.runmodule()