Develop Reference

Retrieving the netnsid of a network namespace in Python

When trying to find the correct peer network interface of a veth pair that lives in a different namespace, that end is not only indicated by its iflink property, but also by a link-netnsid. This link-netnsid is a network namespace ID which is only meaningful within the current network namespace.
The Linux kernel doesn't offer to map a netnsid to a network namespace inode number, which is the only unique identification. However, Linux offers the RTM_GETNSID request that maps a network namespace identified either by fd (NETNSA_FD) or by PID (NETNSA_PID) to the local netnsid.
How do I make such a RTM_GETNSID request in Python, preferably using the pyroute2 library? So far, I could not successfully request the netnsid for the namespace identified by PID, but only get back an invalid argument error 22, using the following script:
from pyroute2 import IPRoute
from pyroute2.netlink import NLM_F_REQUEST
import pyroute2.netlink.rtnl as rtnl
import pyroute2.netlink.rtnl.nsidmsg as nsidmsg
netstack = IPRoute()
req = nsidmsg.nsidmsg()
req['rtgen_family'] = 0
# 12345 is PID of a process inside another network namespace
req['attrs'] = [('NETNSA_PID', 12345)]
ret = netstack.nlm_request(req, rtnl.RTM_GETNSID, NLM_F_REQUEST)

It turns out that my code actually is correct, but there's a bug in pyroute2 causing the RTNETLINK message to be a few octets too short (visible in strace). As a temporary hack around this library bug it is sufficient to set two attributes, so that the kernel accepts the RTNETLINK packet and works on it, even if it isn't fully correct.
from pyroute2 import IPRoute
from pyroute2.netlink import NLM_F_REQUEST
import pyroute2.netlink.rtnl as rtnl
import pyroute2.netlink.rtnl.nsidmsg as nsidmsg
netstack = IPRoute()
req = nsidmsg.nsidmsg()
req['rtgen_family'] = 0
# 12345 is PID of a process inside another network namespace
req['attrs'] = [('NETNSA_PID', 12345), ('NETNSA_PID', 0] # hack around pyroute 0.5.0 bug
ret = netstack.nlm_request(req, rtnl.RTM_GETNSID, NLM_F_REQUEST)

Python 3.6: "AttributeError: 'SimpleQueue' object has no attribute '_poll'"

When I use a standard Queue to send samples to a process, everything works fine. However, since my needs are simple, I tried to use a SimpleQueue and for some reason the 'empty' method doesn't work. Here's the details:
Error comes from the consumer process (when sample_queue is Queue, everything works, when sample_queue is SimpleQueue, things break):
def frame_update(i):
while not self.sample_queue.empty():
sample = self.sample_queue.get()
for line in lines:
While executing sample_queue.empty() -- SimpleQueue.empty(), from Python 3.6 on windows (queues.py) we get:
def empty(self):
return not self._poll()
Where self._poll() has been set in init by:
def __init__(self, *, ctx):
self._reader, self._writer = connection.Pipe(duplex=False)
self._rlock = ctx.Lock()
self._poll = self._reader.poll
if sys.platform == 'win32':
self._wlock = None
else:
self._wlock = ctx.Lock()
So I follow the self._reader which is set from connection.Pipe (connection.py):
...
c1 = PipeConnection(h1, writable=duplex)
c2 = PipeConnection(h2, readable=duplex)
Ok, great. The _reader is going to be a PipeConnection and pipe connection has this method:
def _poll(self, timeout):
if (self._got_empty_message or
_winapi.PeekNamedPipe(self._handle)[0] != 0):
return True
return bool(wait([self], timeout))
Alright -- So a couple of questions:
1) Shouldn't the init of SimpleQueue be assigning self.poll to self._reader._poll instead of self._reader.poll? Or am I missing something in the inheritance hierarchy?
2) The PipeConnection _poll routine takes a timeout parameter, so #1 shouldn't work...
*) -- Is there some other binding of PipeConnection _poll that I'm missing?
Am I missing something? I am using Python3.6, Windows, debugging in PyCharm and I follow all the paths and they're in the standard multiprocessing paths. I'd appreciate any help or advice. Thanks!
EDIT: After further review, I can see that PipeConnection is a subclass of _ConnectionBase which does indeed have a 'poll' method and it is bound with a default timeout parameter.
So the question is: When SimpleQueue is initializing and sets
self._poll = self._reader.poll
Why doesn't it go up the class hierarchy to grab that from _ConnectionBase?

After looking at why the Queue type works and why the SimpleQueue doesn't, I found that Queue sets the _poll method 'after_fork' as well as before. SimpleQueue doesn't. By changing the setstate method to add self._poll = self._reader.poll as follows (queues.py, line 338), SimpleQueue works
def __setstate__(self, state):
(self._reader, self._writer, self._rlock, self._wlock) = state
self._poll = self._reader.poll
Seems like a bug to me unless I'm really misunderstanding something. I'll submit a bug report and reference this post. Hope this helps someone!
http://bugs.python.org/issue30301

udisks FilesystemUnmount appears to not exist when calling from python

I'm trying to unmount a filesystem that I mounted using FilesystemMount, but I keep getting UnknownMethod exceptions. I've verified that I can call the method on the Device interface via D-Feet, but trying to do it via dbus directly doesn't appear to work at all. I've tried using the following arguments:
''
None
[]
['']
The following code demonstrates the problem:
import dbus
bus = dbus.SystemBus()
proxy = bus.get_object('org.freedesktop.UDisks', '/dev/fd0')
dev = dbus.Interface(proxy, 'org.freedesktop.UDisks.Device')
dev.FilesystemUnmount(['force'])
Exception:
dbus.exceptions.DBusException: org.freedesktop.DBus.Error.UnknownMethod: Method "FilesystemUmount" with signature "as" on interface "org.freedesktop.UDisks.Device" doesn't exist

Turns out that the problem is that FilesystemUnmount will only take an ObjectPath that udisks handed out. So by adding a check for that and then looking it up I got it to work. See the code below.
import dbus
path = '/dev/fd0'
bus = dbus.SystemBus()
if not isinstance(path, dbus.ObjectPath):
manager_obj = bus.get_object('org.freedesktop.UDisks',
'/org/freedesktop/UDisks')
manager = dbus.Interface(manager_obj, 'org.freedesktop.UDisks')
path = manager.FindDeviceByDeviceFile(path)
proxy = bus.get_object('org.freedesktop.UDisks', path)
dev = dbus.Interface(proxy, 'org.freedesktop.UDisks.Device')
dev.FilesystemUnmount('')

How to use startTLS with ldaptor?

I'm trying to use ldaptor to connect via startTLS to a LDAP server. Searching on internet and trying myself I arrived to this snippet of code:
from ldaptor.protocols.ldap import ldapclient, ldapsyntax, ldapconnector, distinguishedname
[...]
def main(base, serviceLocationOverrides):
c=ldapconnector.LDAPClientCreator(reactor, ldapclient.LDAPClient)
d = c.connect(base, serviceLocationOverrides)
d.addCallbacks(lambda proto: proto.startTLS(), error)
[...]
d.addErrback(error)
d.addBoth(lambda dummy: reactor.stop())
reactor.run()
but the code exits with an AssertionError:
[Failure instance: Traceback: <type 'exceptions.AssertionError'>:
/usr/lib/python2.7/dist-packages/twisted/internet/base.py:1167:mainLoop
/usr/lib/python2.7/dist-packages/twisted/internet/base.py:789:runUntilCurrent
/usr/lib/python2.7/dist-packages/twisted/internet/defer.py:361:callback
/usr/lib/python2.7/dist-packages/twisted/internet/defer.py:455:_startRunCallbacks
--- <exception caught here> ---
/usr/lib/python2.7/dist-packages/twisted/internet/defer.py:542:_runCallbacks
/usr/lib/pymodules/python2.7/ldaptor/protocols/ldap/ldapclient.py:239:_startTLS
/usr/lib/pymodules/python2.7/ldaptor/protocols/pureldap.py:1278:__init__
/usr/lib/pymodules/python2.7/ldaptor/protocols/pureldap.py:1144:__init__
]
I have tried to look in ldaptor code for the incriminated assertion but seems all ok.
Is there someone who succeded in using ldaptorClient startTLS?
A code snippet?
Thank you very much
Bye

I'm pretty certain that your problem is one I ran into a while back. In ldaptor/protocols/pureldap.py, line 1144 asserts that the LDAPExtendedRequest requestValue must be a string. But according to RFC 2251, that value is optional, and specifically should NOT be present in startTLS requests.
So your approach is correct; this is just a major bug in ldaptor. As far as I can tell, the author only tested using simple bind without TLS. You need to comment out that line in pureldap.py. If you're deploying this with the expectation that users will download or easy-install ldaptor, then you'll need to create a fixed copy of the LDAPExtendedRequest class in your own code, and sub it in at run-time.
Having had to maintain a project using ldaptor for several years, I would strongly urge you to switch to python-ldap if at all possible. Since it wraps the OpenLDAP libs, it can be much more difficult to build, especially with full support for SSL/SASL. But it's well worth it, because ldaptor has a lot more problems than just the one you ran across.

Using ldaptor 0.0.54 from https://github.com/twisted/ldaptor, I had no problems using StartTLS.
Here is the code:
#! /usr/bin/env python
from twisted.internet import reactor, defer
from ldaptor.protocols.ldap import ldapclient, ldapsyntax, ldapconnector
#defer.inlineCallbacks
def example():
serverip = 'your.server.name.or.ip'
basedn = 'o=Organization'
binddn = 'cn=admin,o=Organization'
bindpw = 'Sekret'
query = '(uid=jetsong)'
c = ldapconnector.LDAPClientCreator(reactor, ldapclient.LDAPClient)
overrides = {basedn: (serverip, 389)}
client = yield c.connect(basedn, overrides=overrides)
client = yield client.startTLS()
yield client.bind(binddn, bindpw)
o = ldapsyntax.LDAPEntry(client, basedn)
results = yield o.search(filterText=query)
for entry in results:
print entry
if __name__ == '__main__':
df = example()
df.addErrback(lambda err: err.printTraceback())
df.addCallback(lambda _: reactor.stop())
reactor.run()

Python hashlib.md5 and ejabberd

I am using a python script as an external auth option in ejabberd 2.1.6.
I wanted to start encrypting the clear text passwords that come across in the auth verification, so that they are not being stored in plain text in the backend database. When I add the following code to my python script and restart ejabberd, it hangs:
import hashlib
clear = "barfoo"
salt = "foobar"
hash = hashlib.md5( salt + clear ).hexdigest()
Does hashlib require specific priviledges to run?
When I run it as a normal user (ejabberd) it works without issue. When the python script is run within the external auth of ejabberd it hangs.
I've attempted to have it write out the 'hash' to a file and it never gets there ... if i run it as the 'ejabberd' user, it writes out to file fine.
I've tried to find information about restrictions for using this library on ubuntu without any success. Any ideas?
-sd
** 22.02.2011: Here is the full script adapted from https://git.process-one.net/ejabberd/mainline/blobs/raw/2.1.x/doc/dev.html#htoc8 :
#!/usr/bin/python
import sys
from struct import *
import hashlib
def from_ejabberd():
input_length = sys.stdin.read(2)
(size,) = unpack('>h', input_length)
return sys.stdin.read(size).split(':')
def to_ejabberd(bool):
answer = 0
if bool:
answer = 1
token = pack('>hh', 2, answer)
sys.stdout.write(token)
sys.stdout.flush()
def auth(username, server, password):
clear = "barfoo"
salt = "foobar"
hash = hashlib.md5( salt + clear ).hexdigest()
if (password == hash): return True
else: return False
def isuser(username, server):
return True
def setpass(username, server, password):
return True
while True:
data = from_ejabberd()
success = False
if data[0] == "auth":
success = auth(data[1], data[2], data[3])
elif data[0] == "isuser":
success = isuser(data[1], data[2])
elif data[0] == "setpass":
success = setpass(data[1], data[2], data[3])
to_ejabberd(success)

I have been messing with the same problem. I Can't really track down the problem with openssl binings in _hashlib. Whatever the problem is, i will have to patch python distribution sources. Not really an feasible solution. So i ended up using a pycrypto wrapper for the crypto functions which don't block in this case.
pip install pycrypto
from Crypto.Hash import MD5
m = MD5.new()
m.update("%s%s" % (salt ,clear))
h.hexdigest()

I looked into the hashlib source and while it does not seem to require too much, it does import .so files as modules and one of them hits openssl. It all looks pretty safe but if ejabberd tries to fence itself against calls to 3rd party code (or if you have SELinux or something else to that effect running), stuff can conceivably get weird. I got this in a REPL:
>>> import _md5
>>> _md5.__file__
'/usr/lib/python2.7/lib-dynload/_md5module.so'
Try this on your box and then try putting
_md5 = imp.load_dynamic('_md5', '/usr/lib/python2.7/lib-dynload/_md5module.so')
Or just
import _md5
(with the appropriate path updated to yours) in your code before the offending line and with some trace statement afterwards. Try the same with _hashlib instead of _md5 (hashlib defaults to _hashlib which wraps openssl, but if it doesn't load or doesn't have the needed hash it falls back to _md5, _sha etc.). If it's not the imports that fail/hang then you can try calling _md5.new(salt + clear) and _hashlib.openssl_md5(salt + clear) and see if it's one of them.
If it is the import at fault, then possibly a similar problem was tackled here
I don't know ejabberd, so I can't relate their solution to your problem, unfortunately.
I do have to say it, though: in all python implementations I know, = instead of == in a condition would raise a SyntaxError and that's that - the program would never even enter the main while loop.

hashlib does not require anything special. What means hangs? Where does it hang? Use pdb.set_trace() to step trough the code or use 'strace' or 'ltrace' to investigate API calls.

Try to use logging module, it can help you to watch input and output data, also check scripts permissions, let executing by user which is ejabberd, or for debug just set chmod 777 external.py.