Develop Reference

python bottle boddle - patch(... side-effect="Exception) not working. What am I doing wrong

With: Python 3.7, Bottle, Boddle, Requests
I have one module to be tested (server), and a second module with the test code (test).
The tested code (server.py):
#bt_app.post("/")
def server_post():
try:
bt.request.forms.get("msg", None)
except bt.BottleException:
print("BottleException")
Testing code (test.py):
def test_test():
with boddle(), \
patch(
"server.bt.request.forms.get",
side_effect=server.bt.BottleException,
), \
pytest.raises(server.bt.BottleException):
server.server_post()
I first run 'server', which successfully sets up a server.
I then run 'test', which successfully accesses and tests the server.
I expect the test to raise "BottleException" and PASS, but it does not. It DOES properly fail stating: "Failed: DID NOT RAISE <class 'bottle.BottleException'>"
Can anyone give me a pointer of what I'm doing wrong?
Thanks

Try it without the alias.
...
with pytest.raises(bottle.BottleException):
...

teradatasql: runtime/cgo: could not obtain pthread_keys

When I'm trying to read data from sqlalchemy df=pd.read_sql_table(table, con, schema) getting runtime error :
runtime/cgo: could not obtain pthread_keys
tried 0x115 0x116 0x117 0x118 0x119 0x11a 0x11b 0x11c 0x11d 0x11e 0x11f 0x120 0x121 0x122 0x123 0x124 0x125 0x126 0x127 0x128 0x129 0x12a 0x12b 0x12c 0x12d 0x12e 0x12f 0x130 0x131 0x132 0x133 0x134 0x135 0x136 0x137 0x138 0x139 0x13a 0x13b 0x13c 0x13d 0x13e 0x13f 0x140 0x141 0x142 0x143 0x144 0x145 0x146 0x147 0x148 0x149 0x14a 0x14b 0x14c 0x14d 0x14e 0x14f 0x150 0x151 0x152 0x153 0x154 0x155 0x156 0x157 0x158 0x159 0x15a 0x15b 0x15c 0x15d 0x15e 0x15f 0x160 0x161 0x162 0x163 0x164 0x165 0x166 0x167 0x168 0x169 0x16a 0x16b 0x16c 0x16d 0x16e 0x16f 0x170 0x171 0x172 0x173 0x174 0x175 0x176 0x177 0x178 0x179 0x17a 0x17b 0x17c 0x17d 0x17e 0x17f 0x180 0x181 0x182 0x183 0x184 0x185 0x186 0x187 0x188 0x189 0x18a 0x18b 0x18c 0x18d 0x18e 0x18f 0x190 0x191 0x192 0x193 0x194
Below is the code:
class TeradataWriter:
def __init__(self):
print("in init")
def read_data_from_teradata(self):
try:
print('Create main')
import pdb;pdb.set_trace()
eng = self.create_connection_engine()
df = pd.read_sql_table("table_name", eng, schema="schema")
print(df)
except Exception as ex:
print('Exception: %s', ex.with_traceback())
def create_connection_engine(self):
try:
return create_engine('teradatasql://' + constants.TERADATA_HOST + '/?user='+ constants.TERADATA_USER_NAME + '&password=' + constants.TERADATA_PWD, echo=False)
except Exception as ex:
LOGGER.error('Exception: %s', ex)
raise Exception(message_constants.ERROR_WHILE_CREATING_CONNECTION_WITH_TERADATA)
if __name__ == "__main__":
p = TeradataWriter()
p.write_dataframe_to_teradata()

Edit: This is fixed. I was finally able to get their support and engineering team to reproduce the issue. They now build the driver with a newer version of go. Upgrade to >= 17.0.3, and you shouldn't see anymore segfaults.
I think I finally figured out why this happens. According to this Go issue, it happens if "If the host process spawns threads prior to loading the shared library, the offset will have changed."
In my case, I was importing matplotlib.pyplot in IPython before calling code that loads the shared library. This starts an event loop and causes the conditions that lead to the segfault.
I changed my code to import matplotlib.pyplot after configuring the teradata driver, and it went away.
According to the Go issue, they just need to recompile the library with a newer version of Go, which I've asked them to do. We'll see what they say.

I have run in to same issue -
So to fix the problem, I moved connect statement to main and it kind of fixed. Its worth trying in your case.

Python 3.6: "AttributeError: 'SimpleQueue' object has no attribute '_poll'"

When I use a standard Queue to send samples to a process, everything works fine. However, since my needs are simple, I tried to use a SimpleQueue and for some reason the 'empty' method doesn't work. Here's the details:
Error comes from the consumer process (when sample_queue is Queue, everything works, when sample_queue is SimpleQueue, things break):
def frame_update(i):
while not self.sample_queue.empty():
sample = self.sample_queue.get()
for line in lines:
While executing sample_queue.empty() -- SimpleQueue.empty(), from Python 3.6 on windows (queues.py) we get:
def empty(self):
return not self._poll()
Where self._poll() has been set in init by:
def __init__(self, *, ctx):
self._reader, self._writer = connection.Pipe(duplex=False)
self._rlock = ctx.Lock()
self._poll = self._reader.poll
if sys.platform == 'win32':
self._wlock = None
else:
self._wlock = ctx.Lock()
So I follow the self._reader which is set from connection.Pipe (connection.py):
...
c1 = PipeConnection(h1, writable=duplex)
c2 = PipeConnection(h2, readable=duplex)
Ok, great. The _reader is going to be a PipeConnection and pipe connection has this method:
def _poll(self, timeout):
if (self._got_empty_message or
_winapi.PeekNamedPipe(self._handle)[0] != 0):
return True
return bool(wait([self], timeout))
Alright -- So a couple of questions:
1) Shouldn't the init of SimpleQueue be assigning self.poll to self._reader._poll instead of self._reader.poll? Or am I missing something in the inheritance hierarchy?
2) The PipeConnection _poll routine takes a timeout parameter, so #1 shouldn't work...
*) -- Is there some other binding of PipeConnection _poll that I'm missing?
Am I missing something? I am using Python3.6, Windows, debugging in PyCharm and I follow all the paths and they're in the standard multiprocessing paths. I'd appreciate any help or advice. Thanks!
EDIT: After further review, I can see that PipeConnection is a subclass of _ConnectionBase which does indeed have a 'poll' method and it is bound with a default timeout parameter.
So the question is: When SimpleQueue is initializing and sets
self._poll = self._reader.poll
Why doesn't it go up the class hierarchy to grab that from _ConnectionBase?

After looking at why the Queue type works and why the SimpleQueue doesn't, I found that Queue sets the _poll method 'after_fork' as well as before. SimpleQueue doesn't. By changing the setstate method to add self._poll = self._reader.poll as follows (queues.py, line 338), SimpleQueue works
def __setstate__(self, state):
(self._reader, self._writer, self._rlock, self._wlock) = state
self._poll = self._reader.poll
Seems like a bug to me unless I'm really misunderstanding something. I'll submit a bug report and reference this post. Hope this helps someone!
http://bugs.python.org/issue30301

Factory instance has no attribute 'startedConnecting'

I've looked around quite a bit, but haven't found an error quite like this. When I execute my code (below), I get the exception ControllerFactory instance has no attribute 'startedConnecting'. I tried adding the method with the body just being pass, but that simply causes it to stall without transmitting anything, leading me to believe that the problem lies within the way I've set up the classes.
This code is based on code from the twisted website. It's meant to be able to transmit python files, which the server saves, then later transmit arguments for running the python file.
#!/usr/bin/env python
from twisted.internet import reactor, protocol
import argparse
file_header = "pfile:"
run_header = "runwith:"
class Controller(protocol.Protocol):
def sendMessage(self,message):
self.transport.write(message)
class ControllerFactory(protocol.Factory):
def buildProtocol(self, addr):
cont = Controller()
cont.factory = self
return cont
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument("--address")
parser.add_argument("--file")
parser.add_argument("--args")
args = parser.parse_args()
if(args.file and args.args):
raise Exception("Can't send file and args at same time.")
reactor.connectTCP(args.address, 1337, ControllerFactory())
reactor.run()
if(args.file):
print(args.file)
a = open(args.file)
factory.connectedProtocol.sendMessage(file_header + a.read())
a.close()
if(args.args):
print(args.args)
factory.connectedProtocol.sendMessage(run_header + args.args)

Use twisted.internet.protocol.ClientFactory for clients (instead of twisted.internet.protocol.Factory). Or use something from twisted.internet.endpoints instead of twisted.internet.reactor.connectTCP.
Also, note that reactor.run() blocks. All of your code that follows that line will not run in any useful way.

Python hashlib.md5 and ejabberd

I am using a python script as an external auth option in ejabberd 2.1.6.
I wanted to start encrypting the clear text passwords that come across in the auth verification, so that they are not being stored in plain text in the backend database. When I add the following code to my python script and restart ejabberd, it hangs:
import hashlib
clear = "barfoo"
salt = "foobar"
hash = hashlib.md5( salt + clear ).hexdigest()
Does hashlib require specific priviledges to run?
When I run it as a normal user (ejabberd) it works without issue. When the python script is run within the external auth of ejabberd it hangs.
I've attempted to have it write out the 'hash' to a file and it never gets there ... if i run it as the 'ejabberd' user, it writes out to file fine.
I've tried to find information about restrictions for using this library on ubuntu without any success. Any ideas?
-sd
** 22.02.2011: Here is the full script adapted from https://git.process-one.net/ejabberd/mainline/blobs/raw/2.1.x/doc/dev.html#htoc8 :
#!/usr/bin/python
import sys
from struct import *
import hashlib
def from_ejabberd():
input_length = sys.stdin.read(2)
(size,) = unpack('>h', input_length)
return sys.stdin.read(size).split(':')
def to_ejabberd(bool):
answer = 0
if bool:
answer = 1
token = pack('>hh', 2, answer)
sys.stdout.write(token)
sys.stdout.flush()
def auth(username, server, password):
clear = "barfoo"
salt = "foobar"
hash = hashlib.md5( salt + clear ).hexdigest()
if (password == hash): return True
else: return False
def isuser(username, server):
return True
def setpass(username, server, password):
return True
while True:
data = from_ejabberd()
success = False
if data[0] == "auth":
success = auth(data[1], data[2], data[3])
elif data[0] == "isuser":
success = isuser(data[1], data[2])
elif data[0] == "setpass":
success = setpass(data[1], data[2], data[3])
to_ejabberd(success)

I have been messing with the same problem. I Can't really track down the problem with openssl binings in _hashlib. Whatever the problem is, i will have to patch python distribution sources. Not really an feasible solution. So i ended up using a pycrypto wrapper for the crypto functions which don't block in this case.
pip install pycrypto
from Crypto.Hash import MD5
m = MD5.new()
m.update("%s%s" % (salt ,clear))
h.hexdigest()

I looked into the hashlib source and while it does not seem to require too much, it does import .so files as modules and one of them hits openssl. It all looks pretty safe but if ejabberd tries to fence itself against calls to 3rd party code (or if you have SELinux or something else to that effect running), stuff can conceivably get weird. I got this in a REPL:
>>> import _md5
>>> _md5.__file__
'/usr/lib/python2.7/lib-dynload/_md5module.so'
Try this on your box and then try putting
_md5 = imp.load_dynamic('_md5', '/usr/lib/python2.7/lib-dynload/_md5module.so')
Or just
import _md5
(with the appropriate path updated to yours) in your code before the offending line and with some trace statement afterwards. Try the same with _hashlib instead of _md5 (hashlib defaults to _hashlib which wraps openssl, but if it doesn't load or doesn't have the needed hash it falls back to _md5, _sha etc.). If it's not the imports that fail/hang then you can try calling _md5.new(salt + clear) and _hashlib.openssl_md5(salt + clear) and see if it's one of them.
If it is the import at fault, then possibly a similar problem was tackled here
I don't know ejabberd, so I can't relate their solution to your problem, unfortunately.
I do have to say it, though: in all python implementations I know, = instead of == in a condition would raise a SyntaxError and that's that - the program would never even enter the main while loop.

hashlib does not require anything special. What means hangs? Where does it hang? Use pdb.set_trace() to step trough the code or use 'strace' or 'ltrace' to investigate API calls.

Try to use logging module, it can help you to watch input and output data, also check scripts permissions, let executing by user which is ejabberd, or for debug just set chmod 777 external.py.