Develop Reference

What data types should I use for arguments to redis-py functions?

For example, when I use xack, do I need to convert each argument to bytes, or is this done automatically by the client?
xack(name, groupname, *ids)
It seems that using either a str or bytes id works, but I can't find an explanation for this in the documentation. Links to the relevant source code would be appreciated.

The source comments are a good reference to see the expected types. str will do most of the time, but some commands expect some other types in some arguments.
For example, see XCLAIM in client.py:
def xclaim(self, name, groupname, consumername, min_idle_time, message_ids,
idle=None, time=None, retrycount=None, force=False,
justid=False):
"""
Changes the ownership of a pending message.
name: name of the stream.
groupname: name of the consumer group.
consumername: name of a consumer that claims the message.
min_idle_time: filter messages that were idle less than this amount of
milliseconds
message_ids: non-empty list or tuple of message IDs to claim
idle: optional. Set the idle time (last time it was delivered) of the
message in ms
time: optional integer. This is the same as idle but instead of a
relative amount of milliseconds, it sets the idle time to a specific
Unix time (in milliseconds).
retrycount: optional integer. set the retry counter to the specified
value. This counter is incremented every time a message is delivered
again.
force: optional boolean, false by default. Creates the pending message
entry in the PEL even if certain specified IDs are not already in the
PEL assigned to a different client.
justid: optional boolean, false by default. Return just an array of IDs
of messages successfully claimed, without returning the actual message
You'll find in that file the other commands.
Redis strings are binary-safe. You can use bytearray as string. Here an example on the tests.
r.set(key, b'\xff\xf0\x00')
You can use bytearray on the key as well.

Python Serial readline doesn't work when called from within Object

So I'm writing a simple serial driver for an agilent power supply with pyserial. Everything works fine until I package it up as a class and try to run some simple serial write/read from within the object that I create.
A code snippet of the class:
class agilent:
"""
Class for controlling the chroma class power supplies
Basically a wrapper around the serial interface to the chromas
"""
def __init__(self, port_name="/dev/ttyUSB0", baud_rate=9600):
self.ser = serial.Serial(port=port_name, baudrate=9600, bytesize=8, parity=serial.PARITY_NONE, stopbits=2, dsrdtr=1, timeout=5)
self.reset()
self.identify() #Calls the IDN function to see if this is a multi-service device
print("Connected to: " + self.idn)
if self.multi_service:
print("This is a multi-service device, please make sure to specify one of: P25V, P25N, or P6V when setting limits")
def reset(self):
"""
Reset the agilent power supply to a reasonable(safe) settings
"""
self.ser.write("*CLS\n") # Clear any errors on the chroma
self.ser.write("system:remote\n") # set the controls back to remote
def identify(self):
self.ser.write('*IDN?\n')
self.idn = self.ser.readline()
if 'E3631' in self.idn:
self.multi_service=True
else:
self.multi_service=False
return self.idn
When I call the identify() function from the __init__() function the readline() times out and returns an empty string.
When running in ipython and import the class and create an object, I can call the identify function manually on the object and it works just fine....so either I'm doing something wrong with class methods or some funny business is going on with the serial class.
I know the right function is being called with the right context because if I put a
print(self.ser.port)
in the identify function it returns the right instance information.
Any ideas?
A typical run looks likes this:
from agilent_ps_controller import agilent
remote_ps = agilent() # << This hangs until the readline in __init__() times-out
remote_ps.reset() # Serial message is correctly written on wire
remote_ps.identify() # << this works just fine! Why didn't it work when the object constructor calls it?
remote_ps.__init__() # << this also times-out the readline.

I had the same problem exactly. It turns out that pyserial's readline() uses \n as the default eol character. If your device uses something else, such as \r\n as eol (as Arduino's println() does, for instance), this might be the problem.
So, to make a long story short, try calling readline() this way:
self.ser.readline(eol=b'\r\n')
or any other eol character(s) your device uses.

Detect the end of a raw data

Twisted has two data reception modes: a Line Mode and a Raw Mode, and we can switch between them using setRawMode() and setLineMode() functions.
the line mode detects and end of line and then calls the lineReceived() function.
From Twisted doc:
def rawDataReceived(self, data):
Override this for when raw data is received.
How can Twisted detect the end of a raw data and then call rawDataReceived() ?
EDIT:
I'll add this to to complete my question.
I'm using this Qt function to send data to the Twisted server
qint64 QIODevice::write(const QByteArray & byteArray)
I thought that using write() two times means that the Twisted server will trigger the rawDataReceived() functions two times too.
write( "raw1" );
write( "raw2" );
but data are received in one time.

You asked:
How can Twisted detect the end of a raw data and then call rawDataReceived() ?
In short, when you turn on raw your asking Twisted not to detect.
... but let me explain
When you talk about 'detecting the end of data' inside of a connection (I.E. if your not closing the connection at the end of data), your talking about an idea that is normally referred to as framing.
Framing is one of the primary issues you have to keep in mind when your doing application level networking programming, because most (networking) protocols don't guarantee data framing to the application.
Confusingly many networking protocols (of which TCP is one of the most notorious) will often but not always present data to the receiver in the same way as it is transmitted (I.E. As though it had framing, each write will cause one read to happen - but only in cases of slow-use and low-load). Because of this maybe-it-will-work-maybe-it-won't behavior the best practice is to always explicitly add/build-in some sort of framing.
The most common method to add application-level framing in TCP/Serial/Keyboard style interfaces is to use line-breaks as end-of-frame makers, which is what LineMode is about.
Turning on raw mode in Twisted is like saying 'I want to write my own framing', but I doubt thats really what your after.
Instead you probably want to look at some of the other helper protocols (netstring, prefixed-message-length) that Twisted offers that will do binary framing for you (also see SO: Fragmented data in Twisted dataRecivied by Twisted's author Glyph)

Twisted does not detect the end of the raw data. It just calls rawDataReceived as it receive data.
Following is relevant part from Twisted code. (protocols/basic.py)
def dataReceived(self, data):
"""
Protocol.dataReceived.
Translates bytes into lines, and calls lineReceived (or
rawDataReceived, depending on mode.)
"""
if self._busyReceiving:
self._buffer += data
return
try:
self._busyReceiving = True
self._buffer += data
while self._buffer and not self.paused:
if self.line_mode:
....
else:
data = self._buffer
self._buffer = b''
why = self.rawDataReceived(data) # <--------
if why:
return why
finally:
self._busyReceiving = False

Non-blocking socket in Python?

Is it me, or can I not find a good tutorial on non-blocking sockets in python?
I'm not sure how to exactly work the .recv and the .send in it. According to the python docs, (my understanding of it, at least) the recv'ed or send'ed data might be only partial data. So does that mean I have to somehow concatenate the data while recv and make sure all data sends through in send. If so, how? An example would be much appreciated.

It doesn't really matter if your socket is in non-blocking mode or not, recv/send work pretty much the same; the only difference is that non-blocking socket throws 'Resource temporarily unavailable' error instead of waiting for data/socket.
recv method returns numbers of bytes received, which is told to be less or equal to the passed bufsize. If you want to receive exactly size bytes, you should do something similar to the following code:
def recvall(sock, size):
data = ''
while len(data) < size:
d = sock.recv(size - len(data))
if not d:
# Connection closed by remote host, do what best for you
return None
data += d
return data
This is important to remember, that in blocking mode you have to do exactly the same. (The number of bytes passed to application layer is for example limited by recv buffer size in the OS.)
send method returns number of bytes sent, which is told to be less or equal to the length of passed string. If you want to ensure the whole message was sent, you should do something similar to the following code:
def sendall(sock, data):
while data:
sent = sock.send(data)
data = data[sent:]
You can use sock.sendall directly, but (according to the documentation) on error, an exception is raised, and there is no way to determine how much data, if any, was successfully sent.
The sockets in Python follow the BSD socket API and behave in the similar way to c-style sockets (the difference is, for example, they throw exception instead of returning error code). You should be happy with any socket tutorial on the web and manpages.

Keep bytes you want to send in a buffer. (A list of byte-strings would be best, since you don't have to concatenate them.) Use the fcntl.fcntl function to set the socket in non-blocking mode:
import fcntl, os
fcntl.fcntl(mysocket, fcntl.F_SETFL, os.O_NONBLOCK)
Then select.select will tell you when it is OK to read and write to the socket. (Writing when it is not OK will give you the EAGAIN error in non-blocking mode.) When you write, check the return value to see how many bytes were actually written. Eliminate that many bytes from your buffer. If you use the list-of-strings approach, you only need to try writing the first string each time.
If you read the empty string, your socket has closed.

Read from socket: Is it guaranteed to at least get x bytes?

I have a rare bug that seems to occur reading a socket.
It seems, that during reading of data sometimes I get only 1-3 bytes of a data package that is bigger than this.
As I learned from pipe-programming, there I always get at least 512 bytes as long as the sender provides enough data.
Also my sender does at least transmit >= 4 Bytes anytime it does transmit anything -- so I was thinking that at least 4 bytes will be received at once in the beginning (!!) of the transmission.
In 99.9% of all cases, my assumption seems to hold ... but there are really rare cases, when less than 4 bytes are received. It seems to me ridiculous, why the networking system should do this?
Does anybody know more?
Here is the reading-code I use:
mySock, addr = masterSock.accept()
mySock.settimeout(10.0)
result = mySock.recv(BUFSIZE)
# 4 bytes are needed here ...
...
# read remainder of datagram
...
The sender sends the complete datagram with one call of send.
Edit: the whole thing is working on localhost -- so no complicated network applications (routers etc.) are involved. BUFSIZE is at least 512 and the sender sends at least 4 bytes.

I assume you're using TCP. TCP is a stream based protocol with no idea of packets or message boundaries.
This means when you do a read you may get less bytes than you request. If your data is 128k for example you may only get 24k on your first read requiring you to read again to get the rest of the data.
For an example in C:
int read_data(int sock, int size, unsigned char *buf) {
int bytes_read = 0, len = 0;
while (bytes_read < size &&
((len = recv(sock, buf + bytes_read,size-bytes_read, 0)) > 0)) {
bytes_read += len;
}
if (len == 0 || len < 0) doerror();
return bytes_read;
}

As far as I know, this behaviour is perfectly reasonable. Sockets may, and probably will fragment your data as they transmit it. You should be prepared to handle such cases by applying appropriate buffering techniques.
On other hand, if you are transmitting the data on the localhost and you are indeed getting only 4 bytes it probably means you have a bug somewhere else in your code.
EDIT: An idea - try to fire up a packet sniffer and see whenever the packet transmitted will be full or not; this might give you some insight whenever your bug is in your client or in your server.

The simple answer to your question, "Read from socket: Is it guaranteed to at least get x bytes?", is no. Look at the doc strings for these socket methods:
>>> import socket
>>> s = socket.socket()
>>> print s.recv.__doc__
recv(buffersize[, flags]) -> data
Receive up to buffersize bytes from the socket. For the optional flags
argument, see the Unix manual. When no data is available, block until
at least one byte is available or until the remote end is closed. When
the remote end is closed and all data is read, return the empty string.
>>>
>>> print s.settimeout.__doc__
settimeout(timeout)
Set a timeout on socket operations. 'timeout' can be a float,
giving in seconds, or None. Setting a timeout of None disables
the timeout feature and is equivalent to setblocking(1).
Setting a timeout of zero is the same as setblocking(0).
>>>
>>> print s.setblocking.__doc__
setblocking(flag)
Set the socket to blocking (flag is true) or non-blocking (false).
setblocking(True) is equivalent to settimeout(None);
setblocking(False) is equivalent to settimeout(0.0).
From this it is clear that recv() is not required to return as many bytes as you asked for. Also, because you are calling settimeout(10.0), it is possible that some, but not all, data is received near the expiration time for the recv(). In that case recv() will return what it has read - which will be less than you asked for (but consistenty < 4 bytes does seem unlikely).
You mention datagram in your question which implies that you are using (connectionless) UDP sockets (not TCP). The distinction is described here. The posted code does not show socket creation so we can only guess here, however, this detail can be important. It may help if you could post a more complete sample of your code.
If the problem is reproducible you could disable the timeout (which incidentally you do not seem to be handling) and see if that fixes the problem.

This is just the way TCP works. You aren't going to get all of your data at once. There are just too many timing issues between sender and receiver including the senders operating system, NIC, routers, switches, the wires themselves, the receivers NIC, OS, etc. There are buffers in the hardware, and in the OS.
You can't assume that the TCP network is the same as a OS pipe. With the pipe, it's all software so there's no cost in delivering the whole message at once for most messages. With the network, you have to assume there will be timing issues, even in a simple network.
That's why recv() can't give you all the data at once, it may just not be available, even if everything is working right. Normally, you will call recv() and catch the output. That should tell you how many bytes you've received. If it's less than you expect, you need to keep calling recv() (as has been suggested) until you get the correct number of bytes. Be aware that in most cases, recv() returns -1 on error, so check for that and check your documentation for ERRNO values. EAGAIN in particular seems to cause people problems. You can read about it on the internet for details, but if I recall, it means that no data is available at the moment and you should try again.
Also, it sounds like from your post that you're sure the sender is sending the data you need sent, but just to be complete, check this:
http://beej.us/guide/bgnet/output/html/multipage/advanced.html#sendall
You should be doing something similar on the recv() end to handle partial receives. If you have a fixed packet size, you should read until you get the amount of data you expect. If you have a variable packet size, you should read until you have the header that tells you how much data you send(), then read that much more data.

From the Linux man page of recv http://linux.about.com/library/cmd/blcmdl2_recv.htm:
The receive calls normally return any
data available, up to the requested
amount, rather than waiting for
receipt of the full amount requested.
So, if your sender is still transmitting bytes, the call will only give what has been transmitted so far.

If the sender sends 515 bytes, and your BUFSIZE is 512, then the first recv will return 512 bytes, and the next will return 3 bytes... Could this be what's happening?
(This is just one case amongst many which will result in a 3-byte recv from a larger send...)

If you are still interested, patterns like this :
# 4 bytes are needed here ......
# read remainder of datagram...
may create the silly window thing.
Check this out

Use recv_into(...) method from the socket module.
Robert S. Barnes written the example in C.
But you can use Python 2.x with standard python-libraries:
def readReliably(s,n):
buf = bytearray(n)
view = memoryview(buf)
sz = s.recv_into(view,n)
return sz,buf
while True:
sk,skfrom = s.accept()
sz,buf = io.readReliably(sk,4)
a = struct.unpack("4B",buf)
print repr(a)
...
Notice, that sz returned by readReliably() function may be greater than n.