Develop Reference

Python RPC Time module function not found

I am trying to get started with Python + gRCP and so I checked out their repository as mentioned in the gRPC guide (https://grpc.io/docs/quickstart/python/).
Now I could execute the Hello World-Script (Client + Server), and so I tried to modify it. To ensure I did not missconfigure anything I just extended the Hello World-function (that use to work out before). I added the following lines:
import time
def SayHello(self, request, context):
currentTime = time.clock_gettime(time.CLOCK_REALTIME)
return helloworld_pb2.HelloReply(message='Time is, %s!' % currentTime)
Now what I inmagined it would do is to simply pass the currentTime-object back in this message I am returning upon that function is called - yet, what happens is the following error:
ERROR:grpc._server:Exception calling application: 'module' object has
no attribute 'clock_gettime' Traceback (most recent call last): File
"/home/user/.local/lib/python2.7/site-packages/grpc/_server.py", line
435, in _call_behavior
response_or_iterator = behavior(argument, context) File "greeter_server.py", line 29, in SayHello
currentTime = time.clock_gettime(time.CLOCK_REALTIME) AttributeError: 'module' object has no attribute 'clock_gettime'
I tried to Google around and I found that this might occur if you have a file named time in the same directory (so Python confuses the file in the current directory with the time-file. Yet there is no such file and he seems to find the correct time-file (since I can see the documentation when I hover the import and the function). What did I do wrong here?
The "full" Server Code (up to the serve() function):
from concurrent import futures
import logging
import time
import grpc
import helloworld_pb2
import helloworld_pb2_grpc
class Greeter(helloworld_pb2_grpc.GreeterServicer):
def SayHello(self, request, context):
currentTime = time.clock_gettime(time.CLOCK_REALTIME)
return helloworld_pb2.HelloReply(message='Time is, %s!' % currentTime)
Edit: I am using Ubuntu if that is important.

time.clock_gettime is a 3.3+ API and you are using 2.7.

Chaining Celery Task Methods Error

I'm trying to chain together two task_methods using celery, but I get the following error:
error:
>>> from proj.tasks import A
>>> a = A()
>>> s = (a.add.s(1,2) | a.show.s()).delay().get()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.6/site-packages/celery/result.py", line 175, in get
raise meta['result']
TypeError: show() takes exactly 2 arguments (1 given)
Note, I don't get this error when chaining regular (standalone functions) celery tasks together, only task_methods (class functions). I can't tell if the self object isn't being passed or if the result from the first task isn't being passed.
Here is my project layout:
proj/__init__.py
celery_app.py
tasks.py
tasks.py:
from __future__ import absolute_import
from celery import current_app
from celery.contrib.methods import task_method
from proj.celery_app import app
class A:
def __init__(self):
self.something = 'something'
#current_app.task(filter=task_method)
def add(self,x, y):
return x + y
#current_app.task(filter=task_method)
def show(self,s):
print s
return s
celery_app.py:
from __future__ import absolute_import
from celery import Celery
app = Celery('proj',
broker='amqp://',
backend='amqp://',
include=['proj.tasks'])
app.conf.update(
CELERY_TASK_RESULT_EXPIRES=3600,
)
if __name__ == '__main__':
app.start()
Here's the error from the celery worker:
[2015-04-15 19:57:52,338: ERROR/MainProcess] Task proj.tasks.show[e1e5bc12-6d36-46cd-beb7-fd92a0a5f5c2] raised unexpected: TypeError('show() takes exactly 2 arguments (1 given)',)
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
TypeError: show() takes exactly 2 arguments (1 given)
Has anyone successfully chained task_methods with celery? Thanks!
edit: It's also worth noting that the following code is successful:
>>> from proj.tasks import A
>>> a = A()
>>> sum = a.add.s(1,2).delay().get()
>>> show = a.show.s(sum).delay().get()
Also, I know it could be argued that these functions don't need to be in a class, but pretend they do. I used simple functions to help illustrate the question.
EDIT
I've found a workaround, although a better solution is still desired:
First, revise tasks.py:
...
def show(s,self):
print s
return s
...
Then you can call s = (a.add.s(1,1) | a.show.s(a) ).delay().get(), setting s to 2.
Interestingly enough, calling s = (a.add.s(1,1) | a.show.s(self=a) ).delay().get() spits back the following error:
TypeError: s() got multiple values for keyword argument 'self'
This is not ideal, since the show function can not be called unless in a chain.
For example, the following issues:
>>> a.show(s='test')
TypeError: show() got multiple values for keyword argument 's'
and
>>> a.show(s='test',self=a)
TypeError: __call__() got multiple values for keyword argument 'self'

According to Ask Solem Hoel, the creator of Celery, task methods were a "failed experiment", and will no longer be supported. I guess that answers my question - It can't be done, currently.
Source

When can you dynamically add a field to an object?

I'm learning curses in python, and wanted to add an attribute to the curses window object.
my minimal program is:
import curses
try:
stdscr = curses.initscr()
stdscr.cur_line = 0
finally:
#clean-up so your terminal isn't wrecked by above error
curses.nocbreak()
stdscr.keypad(False)
curses.echo()
curses.endwin()
The error is:
$ python3 tmp
Traceback (most recent call last):
File "tmp", line 4, in <module>
stdscr.cur_line = 0
AttributeError: '_curses.curses window' object has no attribute 'cur_line'
However, this works:
class Temp:
def __init__(self):
pass
t = Temp()
t.cur_line = 0 #does not fail
My questions is: When does dynamically adding a field to an instance fail? How is python discerning between an instance of my user defined class, and an instance of the class from the curses library?

Most often, this happens because you're trying to add attributes to libraries written in C, and not pure python objects.
>>> import pickle, cPickle
>>> pickle.dump.a=1
>>> cPickle.dump.a=1
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'builtin_function_or_method' object has no attribute 'a'

Learning the Python Thread Module

I am trying to learn more about the thread module. I've come up with a quick script but am getting an error when I run it. The docs show the format as:
thread.start_new_thread ( function, args[, kwargs] )
My method only has one argument.
#!/usr/bin/python
import ftplib
import thread
sites = ["ftp.openbsd.org","ftp.ucsb.edu","ubuntu.osuosl.org"]
def ftpconnect(target):
ftp = ftplib.FTP(target)
ftp.login()
print "File list from: %s" % target
files = ftp.dir()
print files
for i in sites:
thread.start_new_thread(ftpconnect(i))
The error I am seeing occurs after one iteration of the for loop:
Traceback (most recent call last): File "./ftpthread.py", line 16,
in
thread.start_new_thread(ftpconnect(i)) TypeError: start_new_thread expected at least 2 arguments, got 1
Any suggestions for this learning process would be appreciated. I also looked into using threading, but I am unable to import threading since its not install apparently and I haven't found any documentation for installing that module yet.
Thank You!
There error I get when trying to import threading on my Mac is:
>>> import threading
# threading.pyc matches threading.py
import threading # precompiled from threading.pyc
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "threading.py", line 7, in <module>
class WorkerThread(threading.Thread) :
AttributeError: 'module' object has no attribute 'Thread'

The thread.start_new_thread function is really low-level and doesn't give you a lot of control. Take a look at the threading module, more specifically the Thread class: http://docs.python.org/2/library/threading.html#thread-objects
You then want to replace the last 2 lines of your script with:
# This line should be at the top of your file, obviously :p
from threading import Thread
threads = []
for i in sites:
t = Thread(target=ftpconnect, args=[i])
threads.append(t)
t.start()
# Wait for all the threads to complete before exiting the program.
for t in threads:
t.join()
Your code was failing, by the way, because in your for loop, you were calling ftpconnect(i), waiting for it to complete, and then trying to use its return value (that is, None) to start a new thread, which obviously doesn't work.
In general, starting a thread is done by giving it a callable object (function/method -- you want the callable object, not the result of a call -- my_function, not my_function()), and optional arguments to give the callable object (in our case, [i] because ftpconnect takes one positional argument and you want it to be i), and then calling the Thread object's start method.

Now that you can import threading, start with best practices at once ;-)
import threading
threads = [threading.Thread(target=ftpconnect, args=(s,))
for s in sites]
for t in threads:
t.start()
for t in threads: # shut down cleanly
t.join()

What you want is to pass the function object and arguments to the function to thread.start_new_thread, not execute the function.
Like this:
for i in sites:
thread.start_new_thread(ftpconnect, (i,))

How do I disable and then re-enable a warning?

I'm writing some unit tests for a Python library and would like certain warnings to be raised as exceptions, which I can easily do with the simplefilter function. However, for one test I'd like to disable the warning, run the test, then re-enable the warning.
I'm using Python 2.6, so I'm supposed to be able to do that with the catch_warnings context manager, but it doesn't seem to work for me. Even failing that, I should also be able to call resetwarnings and then re-set my filter.
Here's a simple example which illustrates the problem:
>>> import warnings
>>> warnings.simplefilter("error", UserWarning)
>>>
>>> def f():
... warnings.warn("Boo!", UserWarning)
...
>>>
>>> f() # raises UserWarning as an exception
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in f
UserWarning: Boo!
>>>
>>> f() # still raises the exception
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in f
UserWarning: Boo!
>>>
>>> with warnings.catch_warnings():
... warnings.simplefilter("ignore")
... f() # no warning is raised or printed
...
>>>
>>> f() # this should raise the warning as an exception, but doesn't
>>>
>>> warnings.resetwarnings()
>>> warnings.simplefilter("error", UserWarning)
>>>
>>> f() # even after resetting, I'm still getting nothing
>>>
Can someone explain how I can accomplish this?
EDIT: Apparently this is a known bug: http://bugs.python.org/issue4180

Reading through the docs and few times and poking around the source and shell I think I've figured it out. The docs could probably improve to make clearer what the behavior is.
The warnings module keeps a registry at __warningsregistry__ to keep track of which warnings have been shown. If a warning (message) is not listed in the registry before the 'error' filter is set, any calls to warn() will not result in the message being added to the registry. Also, the warning registry does not appear to be created until the first call to warn:
>>> import warnings
>>> __warningregistry__
------------------------------------------------------------
Traceback (most recent call last):
File "<ipython console>", line 1, in <module>
NameError: name '__warningregistry__' is not defined
>>> warnings.simplefilter('error')
>>> __warningregistry__
------------------------------------------------------------
Traceback (most recent call last):
File "<ipython console>", line 1, in <module>
NameError: name '__warningregistry__' is not defined
>>> warnings.warn('asdf')
------------------------------------------------------------
Traceback (most recent call last):
File "<ipython console>", line 1, in <module>
UserWarning: asdf
>>> __warningregistry__
{}
Now if we ignore warnings, they will get added to the warnings registry:
>>> warnings.simplefilter("ignore")
>>> warnings.warn('asdf')
>>> __warningregistry__
{('asdf', <type 'exceptions.UserWarning'>, 1): True}
>>> warnings.simplefilter("error")
>>> warnings.warn('asdf')
>>> warnings.warn('qwerty')
------------------------------------------------------------
Traceback (most recent call last):
File "<ipython console>", line 1, in <module>
UserWarning: qwerty
So the error filter will only apply to warnings that aren't already in the warnings registry. To make your code work you'll need to clear the appropriate entries out of the warnings registry when you're done with the context manager (or in general any time after you've used the ignore filter and want a prev. used message to be picked up the error filter). Seems a bit unintuitive...

Brian Luft is correct about __warningregistry__ being the cause of the problem. But I wanted to clarify one thing: the way the warnings module appears to work is that it sets module.__warningregistry__ for each module where warn() is called. Complicating things even more, the stacklevel option to warnings causes the attribute to be set for the module the warning was issued "in the name of", not necessarily the one where warn() was called... and that's dependent on the call stack at the time the warning was issued.
This means you may have a lot of different modules where the __warningregistry__ attribute is present, and depending on your application, they may all need clearing before you'll see the warnings again. I've been relying on the following snippet of code to accomplish this... it clears the warnings registry for all modules whose name matches the regexp (which defaults to everything):
def reset_warning_registry(pattern=".*"):
"clear warning registry for all match modules"
import re
import sys
key = "__warningregistry__"
for mod in sys.modules.values():
if hasattr(mod, key) and re.match(pattern, mod.__name__):
getattr(mod, key).clear()
Update: CPython issue 21724 addresses issue that resetwarnings() doesn't clear warning state. I attached an expanded "context manager" version to this issue, it can be downloaded from reset_warning_registry.py.

Brian is spot on about the __warningregistry__. So you need to extend catch_warnings to save/restore the global __warningregistry__ too
Something like this may work
class catch_warnings_plus(warnings.catch_warnings):
def __enter__(self):
super(catch_warnings_plus,self).__enter__()
self._warningregistry=dict(globals.get('__warningregistry__',{}))
def __exit__(self, *exc_info):
super(catch_warnings_plus,self).__exit__(*exc_info)
__warningregistry__.clear()
__warningregistry__.update(self._warningregistry)

Following on from Eli Collins' helpful clarification, here is a modified version of the catch_warnings context manager that clears the warnings registry in a given sequence of modules when entering the context manager, and restores the registry on exit:
from warnings import catch_warnings
class catch_warn_reset(catch_warnings):
""" Version of ``catch_warnings`` class that resets warning registry
"""
def __init__(self, *args, **kwargs):
self.modules = kwargs.pop('modules', [])
self._warnreg_copies = {}
super(catch_warn_reset, self).__init__(*args, **kwargs)
def __enter__(self):
for mod in self.modules:
if hasattr(mod, '__warningregistry__'):
mod_reg = mod.__warningregistry__
self._warnreg_copies[mod] = mod_reg.copy()
mod_reg.clear()
return super(catch_warn_reset, self).__enter__()
def __exit__(self, *exc_info):
super(catch_warn_reset, self).__exit__(*exc_info)
for mod in self.modules:
if hasattr(mod, '__warningregistry__'):
mod.__warningregistry__.clear()
if mod in self._warnreg_copies:
mod.__warningregistry__.update(self._warnreg_copies[mod])
Use with something like:
import my_module_raising_warnings
with catch_warn_reset(modules=[my_module_raising_warnings]):
# Whatever you'd normally do inside ``catch_warnings``

I've run into the same issues, and while all of the other answers are valid I choose a different route. I don't want to test the warnings module, nor know about it's inner workings. So I just mocked it instead:
import warnings
import unittest
from unittest.mock import patch
from unittest.mock import call
class WarningTest(unittest.TestCase):
#patch('warnings.warn')
def test_warnings(self, fake_warn):
warn_once()
warn_twice()
fake_warn.assert_has_calls(
[call("You've been warned."),
call("This is your second warning.")])
def warn_once():
warnings.warn("You've been warned.")
def warn_twice():
warnings.warn("This is your second warning.")
if __name__ == '__main__':
__main__=unittest.main()
This code is Python 3, for 2.6 you need the use an external mocking library as unittest.mock was only added in 2.7.