I am working with a library which returns a "byte string" (bytes) and I need to convert this to a string.
Is there actually a difference between those two things? How are they related, and how can I do the conversion?
The only thing that a computer can store is bytes.
To store anything in a computer, you must first encode it, i.e. convert it to bytes. For example:
If you want to store music, you must first encode it using MP3, WAV, etc.
If you want to store a picture, you must first encode it using PNG, JPEG, etc.
If you want to store text, you must first encode it using ASCII, UTF-8, etc.
MP3, WAV, PNG, JPEG, ASCII and UTF-8 are examples of encodings. An encoding is a format to represent audio, images, text, etc. in bytes.
In Python, a byte string is just that: a sequence of bytes. It isn't human-readable. Under the hood, everything must be converted to a byte string before it can be stored in a computer.
On the other hand, a character string, often just called a "string", is a sequence of characters. It is human-readable. A character string can't be directly stored in a computer, it has to be encoded first (converted into a byte string). There are multiple encodings through which a character string can be converted into a byte string, such as ASCII and UTF-8.
'I am a string'.encode('ASCII')
The above Python code will encode the string 'I am a string' using the encoding ASCII. The result of the above code will be a byte string. If you print it, Python will represent it as b'I am a string'. Remember, however, that byte strings aren't human-readable, it's just that Python decodes them from ASCII when you print them. In Python, a byte string is represented by a b, followed by the byte string's ASCII representation.
A byte string can be decoded back into a character string, if you know the encoding that was used to encode it.
b'I am a string'.decode('ASCII')
The above code will return the original string 'I am a string'.
Encoding and decoding are inverse operations. Everything must be encoded before it can be written to disk, and it must be decoded before it can be read by a human.
Assuming Python 3 (in Python 2, this difference is a little less well-defined) - a string is a sequence of characters, ie unicode codepoints; these are an abstract concept, and can't be directly stored on disk. A byte string is a sequence of, unsurprisingly, bytes - things that can be stored on disk. The mapping between them is an encoding - there are quite a lot of these (and infinitely many are possible) - and you need to know which applies in the particular case in order to do the conversion, since a different encoding may map the same bytes to a different string:
>>> b'\xcf\x84o\xcf\x81\xce\xbdo\xcf\x82'.decode('utf-16')
'蓏콯캁澽苏'
>>> b'\xcf\x84o\xcf\x81\xce\xbdo\xcf\x82'.decode('utf-8')
'τoρνoς'
Once you know which one to use, you can use the .decode() method of the byte string to get the right character string from it as above. For completeness, the .encode() method of a character string goes the opposite way:
>>> 'τoρνoς'.encode('utf-8')
b'\xcf\x84o\xcf\x81\xce\xbdo\xcf\x82'
Note: I will elaborate more my answer for Python 3 since the end of life of Python 2 is very close.
In Python 3
bytes consists of sequences of 8-bit unsigned values, while str consists of sequences of Unicode code points that represent textual characters from human languages.
>>> # bytes
>>> b = b'h\x65llo'
>>> type(b)
<class 'bytes'>
>>> list(b)
[104, 101, 108, 108, 111]
>>> print(b)
b'hello'
>>>
>>> # str
>>> s = 'nai\u0308ve'
>>> type(s)
<class 'str'>
>>> list(s)
['n', 'a', 'i', '̈', 'v', 'e']
>>> print(s)
naïve
Even though bytes and str seem to work the same way, their instances are not compatible with each other, i.e, bytes and str instances can't be used together with operators like > and +. In addition, keep in mind that comparing bytes and str instances for equality, i.e. using ==, will always evaluate to False even when they contain exactly the same characters.
>>> # concatenation
>>> b'hi' + b'bye' # this is possible
b'hibye'
>>> 'hi' + 'bye' # this is also possible
'hibye'
>>> b'hi' + 'bye' # this will fail
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: can't concat str to bytes
>>> 'hi' + b'bye' # this will also fail
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: can only concatenate str (not "bytes") to str
>>>
>>> # comparison
>>> b'red' > b'blue' # this is possible
True
>>> 'red'> 'blue' # this is also possible
True
>>> b'red' > 'blue' # you can't compare bytes with str
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: '>' not supported between instances of 'bytes' and 'str'
>>> 'red' > b'blue' # you can't compare str with bytes
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: '>' not supported between instances of 'str' and 'bytes'
>>> b'blue' == 'red' # equality between str and bytes always evaluates to False
False
>>> b'blue' == 'blue' # equality between str and bytes always evaluates to False
False
Another issue when dealing with bytes and str is present when working with files that are returned using the open built-in function. On one hand, if you want ot read or write binary data to/from a file, always open the file using a binary mode like 'rb' or 'wb'. On the other hand, if you want to read or write Unicode data to/from a file, be aware of the default encoding of your computer, so if necessary pass the encoding parameter to avoid surprises.
In Python 2
str consists of sequences of 8-bit values, while unicode consists of sequences of Unicode characters. One thing to keep in mind is that str and unicode can be used together with operators if str only consists of 7-bit ASCI characters.
It might be useful to use helper functions to convert between str and unicode in Python 2, and between bytes and str in Python 3.
Let's have a simple one-character string 'š' and encode it into a sequence of bytes:
>>> 'š'.encode('utf-8')
b'\xc5\xa1'
For the purpose of this example, let's display the sequence of bytes in its binary form:
>>> bin(int(b'\xc5\xa1'.hex(), 16))
'0b1100010110100001'
Now it is generally not possible to decode the information back without knowing how it was encoded. Only if you know that the UTF-8 text encoding was used, you can follow the algorithm for decoding UTF-8 and acquire the original string:
11000101 10100001
^^^^^ ^^^^^^
00101 100001
You can display the binary number 101100001 back as a string:
>>> chr(int('101100001', 2))
'š'
From What is Unicode?:
Fundamentally, computers just deal with numbers. They store letters and other characters by assigning a number for each one.
......
Unicode provides a unique number for every character, no matter what the platform, no matter what the program, no matter what the language.
So when a computer represents a string, it finds characters stored in the computer of the string through their unique Unicode number and these figures are stored in memory. But you can't directly write the string to disk or transmit the string on network through their unique Unicode number because these figures are just simple decimal number. You should encode the string to byte string, such as UTF-8. UTF-8 is a character encoding capable of encoding all possible characters and it stores characters as bytes (it looks like this). So the encoded string can be used everywhere because UTF-8 is nearly supported everywhere. When you open a text file encoded in UTF-8 from other systems, your computer will decode it and display characters in it through their unique Unicode number.
When a browser receive string data encoded UTF-8 from the network, it will decode the data to string (assume the browser in UTF-8 encoding) and display the string.
In Python 3, you can transform string and byte string to each other:
>>> print('中文'.encode('utf-8'))
b'\xe4\xb8\xad\xe6\x96\x87'
>>> print(b'\xe4\xb8\xad\xe6\x96\x87'.decode('utf-8'))
中文
In a word, string is for displaying to humans to read on a computer and byte string is for storing to disk and data transmission.
Unicode is an agreed-upon format for the binary representation of characters and various kinds of formatting (e.g., lower case/upper case, new line, and carriage return), and other "things" (e.g., emojis). A computer is no less capable of storing a Unicode representation (a series of bits), whether in memory or in a file, than it is of storing an ASCII representation (a different series of bits), or any other representation (series of bits).
For communication to take place, the parties to the communication must agree on what representation will be used.
Because Unicode seeks to represent all the possible characters (and other "things") used in inter-human and inter-computer communication, it requires a greater number of bits for the representation of many characters (or things) than other systems of representation that seek to represent a more limited set of characters/things. To "simplify," and perhaps to accommodate historical usage, Unicode representation is almost exclusively converted to some other system of representation (e.g., ASCII) for the purpose of storing characters in files.
It is not the case that Unicode cannot be used for storing characters in files, or transmitting them through any communications channel. It is simply that it is not.
The term "string," is not precisely defined. "String," in its common usage, refers to a set of characters/things. In a computer, those characters may be stored in any one of many different bit-by-bit representations. A "byte string" is a set of characters stored using a representation that uses eight bits (eight bits being referred to as a byte). Since, these days, computers use the Unicode system (characters represented by a variable number of bytes) to store characters in memory, and byte strings (characters represented by single bytes) to store characters to files, a conversion must be used before characters represented in memory will be moved into storage in files.
Putting it simple, think of our natural languages like - English, Bengali, Chinese, etc. While talking, all of these languages make sound. But do we understand all of them even if we hear them? -
The answer is generally no. So, if I say I understand English, it means that I know how those sounds are encoded to some meaningful English words and I just decode these sounds in the same way to understand them. So, the same goes for any other language. If you know it, you have the encoder-decoder pack for that language in your mind, and again if you don't know it, you just don't have this.
The same goes for digital systems. Just like ourselves, as we can only listen sounds with our ears and make sound with mouth, computers can only store bytes and read bytes. So, the certain application knows how to read bytes and interpret them (like how many bytes to consider to understand any information) and also write in the same way such that its fellow applications also understand it. But without the understanding (encoder-decoder) all data written to a disk are just strings of bytes.
A string is a bunch of items strung together. A byte string is a sequence of bytes, like b'\xce\xb1\xce\xac' which represents "αά". A character string is a bunch of characters, like "αά". Synonymous to a sequence.
A byte string can be directly stored to the disk directly, while a string (character string) cannot be directly stored on the disk. The mapping between them is an encoding.
The Python languages includes str and bytes as standard "built-in types". In other words, they are both classes. I don't think it's worthwhile trying to rationalize why Python has been implemented this way.
Having said that, str and bytes are very similar to one another. Both share most of the same methods. The following methods are unique to the str class:
casefold
encode
format
format_map
isdecimal
isidentifier
isnumeric
isprintable
The following methods are unique to the bytes class:
decode
fromhex
hex
Related
I know that using this code can remove the b prefix
>>> b'Hello'
b'Hello'
>>> b'Hello'.decode() # decodes bytes type
'Hello'
But if I use a unicode escape (with unicode_escape codec because the utf-8 codec struggles), it works fine... [Python 3.10, Windows 10, AMD64]
>>> b'Hello\xeb'.decode() # utf-8 codec does not work that well
Traceback (most recent call last):
File "<pyshell#25>", line 1, in <module>
b'Hello\xeb'.decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xeb in position 5: unexpected end of data
>>> b'Hello\xeb'.decode('unicode_escape')
'Helloë'
...but for example if I use a .exe file it does not work (and the b prefix is still there??)
>>> # some thing that reads file, code: with open('autoclicker.exe', 'rb') ...
b'MZ\x90\x00\x03\x00\x00\x00\x04\x00\x00\x00\xff\xff\x00\x00\xb8\x00\x00\x00\x00\x00\x00\x00#\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x10\x01\x00\x00\x0e\x1f\xba\x0e\x00\xb4\t\xcd!\xb8\x01L\xcd!This program cannot be run in DOS mode.\r\r\n$\x00\x00\x00\x00\x00\x00\x00-\x82\xc1\xedi\xe3\xaf\xbei\xe3\xaf\xbei\xe3\xaf\xbe\xd4\xac9\xbek\xe3\xaf\xbe`\x9b:\xbew\xe3\xaf\xbe`\x9b,\xbe\xdb\xe3\xaf\xbe`\x9b+\xbeP\xe3\xaf\xbeN%\xc2\xbec\xe3\xaf\xbeN%\xd4\xbeH\xe3\xaf\xbei\xe3\xae\xbed\xe1\xaf\xbe`\x9b \xbe/\xe3\xaf\xbew\xb1:\xbek\xe3\xaf\xbew\xb1;\xbeh\xe3\xaf\xbei\xe38\xbeh\xe3\xaf\xbe`\x9b>\xbeh\xe3\xaf\xbeRichi\xe3\xaf\xbe\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00PE\x00\x00L\x01\x04\x00\x14\xb3\x8a\\\x00\x00\x00\x00\x00\x00\x00\x00\xe0\x00#\x01\x0b\x01\t\x00\x00\x02\x08\x00\x00\xf0\x01\x00\x00\x00\x00\x00\x10c\x01\x00\x00\x10\x00\x00\x00 \x08\x00\x00\x00#\x00\x00\x10\x00\x00\x00\x02\x00\x00\x05\x00\x00\x00\x00\x00\x00\x00\x05\x00\x00\x00\x00\x00\x00\x00\x00`\x0b\x00\x00\x04\x00\x00\x03\t\x0e\x00\x02\x00\x00\x80\x00\x00#\x00\x00\x10\x00\x00\x00\x00#\x00\x00\x10\x00\x00\x00\x00\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00<\xcd\x08\x00T\x01\x00\x00\x00\xb0\n\x00\xfc\xac\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 \x08\x00#\x08\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00.text\x00\x00\x00\x17\x00\x08\x00\x00\x10\x00\x00\x00\x02\x08\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 \x00\x00`.rdata\x00\x00\\\xd9\x00\x00\x00 \x08\x00\x00\xda\x00\x00\x00\x06\x08\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00#\x00\x00#.data\x00\x00\x00\x18\xa5\x01\x00\x00\x00\t\x00\x00h\x00\x00\x00\xe0\x08\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00#\x00\x00\xc0.rsrc\x00\x00\x00\xfc\xac\x00\x00\x00\xb0\n\x00\x00\xae\x00\x00\x00H\t\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00#\x00\x00#\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x003\xc0\x81\xec\xac\x03\x00\x008\x05\x92rI\x00tCh\xa4\x03\x00\x00P\x8dT$\x0cR\xa2\x92rI\x00\x89\x81\x8c\x01\x00\x00\xe8\xc5!\x01\x00\xa1\xc0rI\x00\x83\xc4\x0c\x8d\x0c$Qj\x02\xc7D$\x08\xa8\x03\x00\x00\x89D$\x0c\xc7D$\x10\x01\x00\x00\x00\xff\x15\x8c$H\x00\x81\xc4\xac\x03\x00\x00\xc3\xcc\xcc\xcc\xcc\xcc\xcc\x8bF$S3\xdb;\xc3\x0f\x85J\x85\x02\x00\x8bF,\x89^$;\xc3\x0f\x85J\x85\x02\x00\x89^,\x89^0\x89^4\x89^8\x88^\x10[\xc3\xcc\xcc\xcc\x80~\t\x00\x0f\x85\x95\x82\x02\x00j\x08\xe8y\x06\x01\x00\x83\xc4\x04\x85\xc0t\x10\x8b\x17\x89\x10\x8bN\x04\x89H\x04\xff\x06\x89F\x04\xc3\x8bN\x043\xc0\x89H\x04\xff\x06\x89F\x04\xc3\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\x8bD$\x10\x8bL$\x0cS\x8b\\$\x08W\x8b|$\x10PQ\xe8\x07\x00\x00\x00_[\xc2\x10\x00\xcc\xccU\x8b\xec\x83\xe4\xf8QV;\x1d\xc0rI\x00uP\x81\xff\x11\x01\x00\x00s(\x83\xff\x12r#;=\x80\x93J\x00\x0f\x84r\xd6\x02\x00\x8bU\x0c\x8bE\x08RPWS\xff\x15\x84&H\x00^\x8b\xe5]\xc2\x08\x00\x81\xff\x13\x01\x00\x00u#\x8bE\x08h\xb8\x84J\x00\x8b\xf3\xe8\xe9\x00\x00\x003\xc0^\x8b\xe5]\xc2\x08\x00\xa1\xc0rI\x00\x85\xc0t\xa7\xeb\xbe\x83\xff\x10w9\x0f\x84\xa5\xd5\x02\x00\x8dG\xff\x83\xf8\x06w\x9f\xff$\x85\x14\x12#\x00j\x01S\xff\x15\xd0&H\x00\xb9\xb8\x84J\x00\xe8x\xfe\xff\xffj\x00\xff\x15\xcc&H\x003\xc0^\x8b\xe5]\xc2\x08\x00\x81\xff\x12\x03\x00\x00wa\x0f\x84\xc9\xd5\x02\x00\x83\xff\x11\x0f\x84\x90\xd5\x02\x00\x81\xff\x11\x01\x00\x00\x0f\x85Q\xff\xff\xff\xe9`\xd5\x02\x00j\x00h\xee\x02\x00\x00j\x01S\xff\x15\xdc&H\x00hhHH\x00\xff\x15\xd8&H\x00\x83=\xb8\x84J\x00\x00\xa3\x80\x93J\x00\x0f\x85Y\xff\xff\xff\xff\x15\xd4&H\x00\xa3\xb8\x84J\x003\xc0^\x8b\xe5]\xc2\x08\x00\x81\xff\x01\x04\x00\x00\x0f\x85\xff\xfe\xff\xff\xe9\x9e\xd5\x02\x00\x90\xc1\x11#\x00u\x11#\x00\r\x11#\x00\r\x11#\x00\xe7\xe6B\x00\r\x11#\x00\xd6\xe6B\x00\x81\xec\xa8\x03\x00\x00\x83\xe8\x01SW\x0f\x85\x84\x00\x00\x00h\xa4\x03\x00\x00P\x8dX\x01\x8dD$\x14P\xc7D$\x14\xa8\x03\x00\x00\xe8\x94\x1f\x01\x00\x8b\x84$\xc0\x03\x00\x00\x83\xc4\x0c\xe8\x05\x0c\x00\x00\x80=\x92rI\x00\x00t:\x80=\x94rI\x00\x00\x8b\xbc$\xb4\x03\x00\x00\x89t$\x0c\x89\\$\x10\xc7D$\x14\x02\x00\x00\x00\x0f\x85\xc9\x97\x02\x00\x80\x7f\t\x00\x0f\x85\n\x98\x02\x008\x9f\x84\x01\x00\x00\x0f\x84J\x98\x02\x00SV\xff\x15\xd0&H\x00j\x00h\xee\x02\x00\x00SV\xff\x15\xdc&H\x00_[\x81\xc4\xa8\x03\x00\x00\xc2\x04\x00\x8b\x03\x85\xc0t\tP\xe8\x13\x00\x01\x00\x83\xc4\x04V\x8d\xb3\xec\x00\x00\x00W\xc7\x06p\xa0H\x00\xe8\x1e\xf5\x00\x00\x8bF\x04P\xe8\xf4\xff\x00\x00\x83\xc4\x04\x8d\x8b\xbc\x00\x00\x00\xe8\x17\x13\x00\x00\x8d{x\xe8O\x00\x00\x00\x8d{4\xe8G\x00\x00\x00\x8dK$\xe8\xff\x12\x00\x00_\x8dK\x14^\xe9\xf5\x12\x00\x00\xcc\xcc\xcc\xcc\xcc\x8bF\x0c\xff\x08\x8bF\x0c\x838\x00u\x14\x8b\x0eQ\xe8\xaa\xff\x00\x00\x8bV\x0cR\xe8\xa1\xff\x00\x00\x83\xc4\x08\xc3\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xccV\x8b\xf7\xe8\xf8\xfc\xff\xff\x8dw\x14\xe8\xc0\xff\xff\xff\x8b\xf7\xe8\xb9\xff\xff\xff^\xc3\xcc\xcc\xcc\xcc\xcc\xcc\xcc\x8b\x06\x85\xc0t\tP\xe8c\xff\x00\x00\x83\xc4\x04W\x8d\xbe\xec\x00\x00\x00\xe8\xc5\xff\xff\xff\x8d\x8e\xbc\x00\x00\x00\xe8z\x12\x00\x00\x8d\x8e\xac\x00\x00\x00\xe8o\x12\x00\x00\x8d\x8e\x9c\x00\x00\x00\xe8d\x12\x00\x00\x8d\x8e\x8c\x00\x00\x00\xe8Y\x12\x00\x00\x8d~\x08\xe8\xd1\xf0\x00\x00_\xc3\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xccj\x04\xe83\x03\x01\x00\x83\xc4\x04\x85\xc0\x0f\x84\xc2\x82\x02\x00\xc7\x00\x01\x00\x00\x00\x89F\x0c\xc3\xcc\xcc\xcc\xccV\x8b\xf1#3\xc9\x89F\x08\xba\x02\x00\x00\x00\xf7\xe2\x0f\x90\xc1\xc7F\x04\x00\x00\x00\x00\xf7\xd9\x0b\xc8Q\xe8\xf6\x02\x01\x003\xc9\x89\x06\x83\xc4\x04f\x89\x08\xe8\xad\xff\xff\xff\x8b\xc6^\xc3\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\x81\xec\x14\x02\x00\x00\xe8\x85\x13\x00\x00\x84\xc0\x0f\x84n\xae\x02\x00\xe8\x98\x00\x00\x00\x85\xc0\x0f\x85a\xae\x02\x00\xe8k\xc1\x00\x00\x85\xc0\x0f\x85T\xae\x02\x00\x8b\x94$\x18\x02\x00\x00\x8d\x04$P\x8dL$\x08Qh\x04\x01\x00\x00R\xff\x15 #H\x00\x8dD$\x04P\xb8\xe8\x7fJ\x00\xe8H\r\x00\x00\x8b\x0c$Q\xb8\xd8\x7fJ\x00\xe8:\r\x00\x00\x8b\x04$3\xd2f\x89P\xfef9T$\x08\x0f\x84\x11\xae\x02\x00\x8dT$\x04R\xb8\xf8\x7fJ\x00\xe8\x17\r\x00\x00\x8b\x84$\x1c\x02\x00\x00\xa3\xd4\x7fJ\x003\xc0\x81\xc4\x14\x02\x00\x00\xc2\x08\x00\x8b#\x04\x85\xc0\x0f\x85\xb4\x81\x02\x00\xc3\xcc\xcc\xcc\xcc\x83\xect3\xc0SUVW\xbd\x01\x00\x00\x00\x89D$\x18\x89D$,\x89D$0\x89D$$\x89D$ \x89D$\x1c\x89D ...
>>> b'hi\xbd'.decode('unicode_escape') # using some (unicode) escape methods work separately
'hi½'
>>> b'abc\'
The b'' isn't a "string prefix", instead it indicates that you are dealing with a sequence of bytes. Bytes can represent anything, including a text which is just a series of characters in some encoding, like UTF-8, ASCII, etc.
That's what .decode() does, it takes the sequence of bytes and interprets it as if it were a string of characters in that encoding and returns a string of those characters. Conversely, you could then encode the resulting string of characters into some other encoding by calling .encode() on the string and you'd get the sequence of bytes that represents that string in that encoding.
However, you can't just take any sequence of bytes and 'decode' it as any decoding - the bytes will have a certain encoding if they represent some string, but the example you give (of an executable) doesn't represent a string of characters at all and thus won't successfully decode into a string if you just call .decode() on it.
If you're lucky, the decoding works on the parts of the executable that are strings in that encoding, but even that's not guaranteed to work, as the strings will be surrounded by bytes that don't represent that encoding.
If you want to extract strings from an executable, you need to correctly identify what parts of the executable represent strings, extract those sequences of bytes and decode them with the correct encoding. How to do that will depend on the operating system the executable is for, whether it's 32-bit or 64-bit, etc.
Note: many programmers new to Python or coding in general get confused by the fact that Python (for the sake of convenience) shows you a bytes object as very similar to a string (it looks just like string with a b before it), this is even more confusing if it happens to be an encoding that's UTF or very similar, as the contents of the bytes object will even be readable then. But that doesn't mean the bytes objects actually is a string.
This question already has answers here:
Process escape sequences in a string in Python
(8 answers)
Closed 7 months ago.
My problem is as follows:
I'm reading a .csv generated by some software and to read it I'm using Pandas. Pandas read the .csv properly but one of the columns stores bytes sequences representing vectors and Pandas stores them as a string.
So I have data (string) and I want to use np.frombuffer() to get the proper vector. The problem is, data is a string so its already encoded so when I use .encode() to turn it into bytes, the sequence is not the original one.
Example: The .csv contains \x00\x00 representing the vector [0,0] with dtype=np.uint8. Pandas stores it as a string and when I try to process it something like this happens:
data = df.data[x] # With x any row.
type(data)
<class 'str'>
print(data)
\x00\x00
e_data = data.encode("latin1")
print(e_data)
b'\\x00\\x00'
v = np.frombuffer(e_data, np.uint8)
print(v)
array([ 92 120 48 48 92 120 48 48], dtype=uint8)
I just want to get b'\x00\x00' from data instead of b'\\x00\\x00' which I understand is a little encoding mess I have not been able to fix yet.
Any way to do this?
Thanks!
Issue: you (apparently) have a string that contains literal backslash escape sequences, such as:
>>> x = r'\x00' # note the use of a raw string literal
>>> x # Python's representation of the string escapes the backslash
'\\x00'
>>> print(x) # but it looks right when printing
\x00
From this, you wish to create a corresponding bytes object, wherein the backslash-escape sequences are translated into the corresponding byte.
Handling these kinds of escape sequences is done using the unicode-escape string encoding. As you may be aware, string encodings convert between bytes and str objects, specifying the rules for which byte sequences correspond to what Unicode code points.
However, the unicode-escape codec assumes that the escape sequences are on the bytes side of the equation and that the str side will have the corresponding Unicode characters:
>>> rb'\x00'.decode('unicode-escape') # create a string with a NUL char
'\x00'
Applying .encode to the string will reverse that process; so if you start with the backslash-escape sequence, it will re-escape the backslash:
>>> r'\x00'.encode('unicode-escape') # the result contains two backslashes, represented as four
b'\\\\x00'
>>> list(r'\x00'.encode('unicode-escape')) # let's look at the numeric values of the bytes
[92, 92, 120, 48, 48]
As you can see, that is clearly not what we want.
We want to convert from bytes to str to do the backslash-escaping. But we have a str to start, so we need to change that to bytes; and we want bytes at the end, so we need to change the str that we get from the backslash-escaping. In both cases, we need to make each Unicode code point from 0-255 inclusive, correspond to a single byte with the same value.
The encoding we need for that task is called latin-1, also known as iso-8859-1.
For example:
>>> r'\x00'.encode('latin-1')
b'\\x00'
Thus, we can reason out the overall conversion:
>>> r'\x00'.encode('latin-1').decode('unicode-escape').encode('latin-1')
b'\x00'
As desired: our str with a literal backslash, lowercase x and two zeros, is converted to a bytes object containing a single zero byte.
Alternately: we can request that backslash-escapes are processed while decoding, by using escape_decode from the codecs standard library module. However, this isn't documented and isn't really meant to be used that way - it's internal stuff used to implement the unicode-escape codec and possibly some other things.
If you want to expose yourself to the risk of that breaking in the future, it looks like:
>>> import codecs
>>> codecs.escape_decode(r'\x00\x00')
(b'\x00\x00', 8)
We get a 2-tuple, with the desired bytes and what I assume is the number of Unicode code points that were decoded (i.e. the length of the string). From my testing, it appears that it can only use UTF-8 encoding for the non-backslash sequences (but this could be specific to how Python is configured), and you can't change this; there is no actual parameter to specify the encoding, for a decode method. Like I said - not meant for general use.
Yes, all of that is as awkward as it seems. The reason you don't get easy support for this kind of thing is that it isn't really how you're intended to design your system. Fundamentally, all data is bytes; text is an abstraction that is encoded by that byte data. Using a single byte (with value 0) to represent four characters of text (the symbols \, x, 0 and 0) is not a normal encoding, and not a reversible one (how do I know whether to decode the byte as those four characters, or as a single NUL character?). Instead, you should strongly consider using some other friendly string representation of your data (perhaps a plain hex dump) and a non-text-encoding-related way to parse it. For example:
>>> data = '41 42' # a string in a simple hex dump format
>>> bytes.fromhex(data) # support is built-in, and works simply
b'AB'
>>> list(bytes.fromhex(data))
[65, 66]
Apparently, the following is the valid syntax:
b'The string'
I would like to know:
What does this b character in front of the string mean?
What are the effects of using it?
What are appropriate situations to use it?
I found a related question right here on SO, but that question is about PHP though, and it states the b is used to indicate the string is binary, as opposed to Unicode, which was needed for code to be compatible from version of PHP < 6, when migrating to PHP 6. I don't think this applies to Python.
I did find this documentation on the Python site about using a u character in the same syntax to specify a string as Unicode. Unfortunately, it doesn't mention the b character anywhere in that document.
Also, just out of curiosity, are there more symbols than the b and u that do other things?
Python 3.x makes a clear distinction between the types:
str = '...' literals = a sequence of Unicode characters (Latin-1, UCS-2 or UCS-4, depending on the widest character in the string)
bytes = b'...' literals = a sequence of octets (integers between 0 and 255)
If you're familiar with:
Java or C#, think of str as String and bytes as byte[];
SQL, think of str as NVARCHAR and bytes as BINARY or BLOB;
Windows registry, think of str as REG_SZ and bytes as REG_BINARY.
If you're familiar with C(++), then forget everything you've learned about char and strings, because a character is not a byte. That idea is long obsolete.
You use str when you want to represent text.
print('שלום עולם')
You use bytes when you want to represent low-level binary data like structs.
NaN = struct.unpack('>d', b'\xff\xf8\x00\x00\x00\x00\x00\x00')[0]
You can encode a str to a bytes object.
>>> '\uFEFF'.encode('UTF-8')
b'\xef\xbb\xbf'
And you can decode a bytes into a str.
>>> b'\xE2\x82\xAC'.decode('UTF-8')
'€'
But you can't freely mix the two types.
>>> b'\xEF\xBB\xBF' + 'Text with a UTF-8 BOM'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: can't concat bytes to str
The b'...' notation is somewhat confusing in that it allows the bytes 0x01-0x7F to be specified with ASCII characters instead of hex numbers.
>>> b'A' == b'\x41'
True
But I must emphasize, a character is not a byte.
>>> 'A' == b'A'
False
In Python 2.x
Pre-3.0 versions of Python lacked this kind of distinction between text and binary data. Instead, there was:
unicode = u'...' literals = sequence of Unicode characters = 3.x str
str = '...' literals = sequences of confounded bytes/characters
Usually text, encoded in some unspecified encoding.
But also used to represent binary data like struct.pack output.
In order to ease the 2.x-to-3.x transition, the b'...' literal syntax was backported to Python 2.6, in order to allow distinguishing binary strings (which should be bytes in 3.x) from text strings (which should be str in 3.x). The b prefix does nothing in 2.x, but tells the 2to3 script not to convert it to a Unicode string in 3.x.
So yes, b'...' literals in Python have the same purpose that they do in PHP.
Also, just out of curiosity, are there
more symbols than the b and u that do
other things?
The r prefix creates a raw string (e.g., r'\t' is a backslash + t instead of a tab), and triple quotes '''...''' or """...""" allow multi-line string literals.
To quote the Python 2.x documentation:
A prefix of 'b' or 'B' is ignored in
Python 2; it indicates that the
literal should become a bytes literal
in Python 3 (e.g. when code is
automatically converted with 2to3). A
'u' or 'b' prefix may be followed by
an 'r' prefix.
The Python 3 documentation states:
Bytes literals are always prefixed with 'b' or 'B'; they produce an instance of the bytes type instead of the str type. They may only contain ASCII characters; bytes with a numeric value of 128 or greater must be expressed with escapes.
The b denotes a byte string.
Bytes are the actual data. Strings are an abstraction.
If you had multi-character string object and you took a single character, it would be a string, and it might be more than 1 byte in size depending on encoding.
If took 1 byte with a byte string, you'd get a single 8-bit value from 0-255 and it might not represent a complete character if those characters due to encoding were > 1 byte.
TBH I'd use strings unless I had some specific low level reason to use bytes.
From server side, if we send any response, it will be sent in the form of byte type, so it will appear in the client as b'Response from server'
In order get rid of b'....' simply use below code:
Server file:
stri="Response from server"
c.send(stri.encode())
Client file:
print(s.recv(1024).decode())
then it will print Response from server
The answer to the question is that, it does:
data.encode()
and in order to decode it(remove the b, because sometimes you don't need it)
use:
data.decode()
Here's an example where the absence of b would throw a TypeError exception in Python 3.x
>>> f=open("new", "wb")
>>> f.write("Hello Python!")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'str' does not support the buffer interface
Adding a b prefix would fix the problem.
It turns it into a bytes literal (or str in 2.x), and is valid for 2.6+.
The r prefix causes backslashes to be "uninterpreted" (not ignored, and the difference does matter).
In addition to what others have said, note that a single character in unicode can consist of multiple bytes.
The way unicode works is that it took the old ASCII format (7-bit code that looks like 0xxx xxxx) and added multi-bytes sequences where all bytes start with 1 (1xxx xxxx) to represent characters beyond ASCII so that Unicode would be backwards-compatible with ASCII.
>>> len('Öl') # German word for 'oil' with 2 characters
2
>>> 'Öl'.encode('UTF-8') # convert str to bytes
b'\xc3\x96l'
>>> len('Öl'.encode('UTF-8')) # 3 bytes encode 2 characters !
3
You can use JSON to convert it to dictionary
import json
data = b'{"key":"value"}'
print(json.loads(data))
{"key":"value"}
FLASK:
This is an example from flask. Run this on terminal line:
import requests
requests.post(url='http://localhost(example)/',json={'key':'value'})
In flask/routes.py
#app.route('/', methods=['POST'])
def api_script_add():
print(request.data) # --> b'{"hi":"Hello"}'
print(json.loads(request.data))
return json.loads(request.data)
{'key':'value'}
b"hello" is not a string (even though it looks like one), but a byte sequence. It is a sequence of 5 numbers, which, if you mapped them to a character table, would look like h e l l o. However the value itself is not a string, Python just has a convenient syntax for defining byte sequences using text characters rather than the numbers itself. This saves you some typing, and also often byte sequences are meant to be interpreted as characters. However, this is not always the case - for example, reading a JPG file will produce a sequence of nonsense letters inside b"..." because JPGs have a non-text structure.
.encode() and .decode() convert between strings and bytes.
bytes(somestring.encode()) is the solution that worked for me in python 3.
def compare_types():
output = b'sometext'
print(output)
print(type(output))
somestring = 'sometext'
encoded_string = somestring.encode()
output = bytes(encoded_string)
print(output)
print(type(output))
compare_types()
In a python source code I stumbled upon I've seen a small b before a string like in:
b"abcdef"
I know about the u prefix signifying a unicode string, and the r prefix for a raw string literal.
What does the b stand for and in which kind of source code is it useful as it seems to be exactly like a plain string without any prefix?
The b prefix signifies a bytes string literal.
If you see it used in Python 3 source code, the expression creates a bytes object, not a regular Unicode str object. If you see it echoed in your Python shell or as part of a list, dict or other container contents, then you see a bytes object represented using this notation.
bytes objects basically contain a sequence of integers in the range 0-255, but when represented, Python displays these bytes as ASCII codepoints to make it easier to read their contents. Any bytes outside the printable range of ASCII characters are shown as escape sequences (e.g. \n, \x82, etc.). Inversely, you can use both ASCII characters and escape sequences to define byte values; for ASCII values their numeric value is used (e.g. b'A' == b'\x41')
Because a bytes object consist of a sequence of integers, you can construct a bytes object from any other sequence of integers with values in the 0-255 range, like a list:
bytes([72, 101, 108, 108, 111])
and indexing gives you back the integers (but slicing produces a new bytes value; for the above example, value[0] gives you 72, but value[:1] is b'H' as 72 is the ASCII code point for the capital letter H).
bytes model binary data, including encoded text. If your bytes value does contain text, you need to first decode it, using the correct codec. If the data is encoded as UTF-8, for example, you can obtain a Unicode str value with:
strvalue = bytesvalue.decode('utf-8')
Conversely, to go from text in a str object to bytes you need to encode. You need to decide on an encoding to use; the default is to use UTF-8, but what you will need is highly dependent on your use case:
bytesvalue = strvalue.encode('utf-8')
You can also use the constructor, bytes(strvalue, encoding) to do the same.
Both the decoding and encoding methods take an extra argument to specify how errors should be handled.
Python 2, versions 2.6 and 2.7 also support creating string literals using b'..' string literal syntax, to ease code that works on both Python 2 and 3.
bytes objects are immutable, just like str strings are. Use a bytearray() object if you need to have a mutable bytes value.
This is Python3 bytes literal. This prefix is absent in Python 2.5 and older (it is equivalent to a plain string of 2.x, while plain string of 3.x is equivalent to a literal with u prefix in 2.x). In Python 2.6+ it is equivalent to a plain string, for compatibility with 3.x.
I am working with Russian words written in the Cyrillic orthography. Everything is working fine except for how many (but not all) of the Cyrillic characters are encoded as two characters when in an str. For instance:
>>>print ["ё"]
['\xd1\x91']
This wouldn't be a problem if I didn't want to index string positions or identify where a character is and replace it with another (say "e", without the diaeresis). Obviously, the 2 "characters" are treated as one when prefixed with u, as in u"ё":
>>>print [u"ё"]
[u'\u0451']
But the strs are being passed around as variables, and so can't be prefixed with u, and unicode() gives a UnicodeDecodeError (ascii codec can't decode...).
So... how do I get around this? If it helps, I am using python 2.7
There are two possible situations here.
Either your str represents valid UTF-8 encoded data, or it does not.
If it represents valid UTF-8 data, you can convert it to a Unicode object by using mystring.decode('utf-8'). After it's a unicode instance, it will be indexed by character instead of by byte, as you have already noticed.
If it has invalid byte sequences in it... You're in trouble. This is because the question of "which character does this byte represent?" no longer has a clear answer. You're going to have to decide exactly what you mean when you say "the third character" in the presence of byte sequences that don't actually represent a particular Unicode character in UTF-8 at all...
Perhaps the easiest way to work around the issue would be to use the ignore_errors flag to decode(). This will entirely discard invalid byte sequences and only give you the "correct" portions of the string.
These are actually different encodings:
>>>print ["ё"]
['\xd1\x91']
>>>print [u"ё"]
[u'\u0451']
What you're seeing is the __repr__'s for the elements in the lists. Not the __str__ versions of the unicode objects.
But the strs are being passed around as variables, and so can't be
prefixed with u
You mean the data are strings, and need to be converted into the unicode type:
>>> for c in ["ё"]: print repr(c)
...
'\xd1\x91'
You need to coerce the two-byte strings into double-byte width unicode:
>>> for c in ["ё"]: print repr(unicode(c, 'utf-8'))
...
u'\u0451'
And you'll see with this transform they're perfectly fine.
To convert bytes into Unicode, you need to know the corresponding character encoding and call bytes.decode:
>>> b'\xd1\x91'.decode('utf-8')
u'\u0451'
The encoding depends on the data source. It can be anything e.g., if the data comes from a web page; see A good way to get the charset/encoding of an HTTP response in Python
Don't use non-ascii characters in a bytes literal (it is explicitly forbidden in Python 3). Add from __future__ import unicode_literals to treat all "abc" literals as Unicode literals.
Note: a single user-perceived character may span several Unicode codepoints e.g.:
>>> print(u'\u0435\u0308')
ё