I am trying to use Mxnet-js library to visualize my Mxnet trained model in browser. I am following Mxnet-js git readme file.
They provided a python script. ./tool/model2json, to convert model to json file.
When i am running this script with my model i am getting error:
TypeError: write() argument must be str, not bytes
Getting this error make sense because, how can i write byte to a file that is opened in string mode. At line
model = base64.b64encode(bytes(open(sys.argv[3], 'rb').read()))
they are reading it in bytes but in line
with open(sys.argv1, 'w') as fo:
they are opening file in string mode and in line
fo.write(model)
they are writing bytes to string.
Am i missing something here ? Why they are trying to write bytes to string?
#!/usr/bin/env python
"""Simple util to convert mxnet model to json format."""
import sys
import json
import base64
if len(sys.argv) < 4:
print('Usage: <output.json> <symbol.json> <model.param>
[mean_image.nd] [synset]')
exit(0)
symbol_json = open(sys.argv[2]).read()
model = base64.b64encode(bytes(open(sys.argv[3], 'rb').read()))
mean_image = None
synset = None
if len(sys.argv) > 4:
mean_image = base64.b64encode(bytes(open(sys.argv[4],
'rb').read()))
if len(sys.argv) > 5:
synset = [l.strip() for l in open(sys.argv[5]).readlines()]
with open(sys.argv[1], 'w') as fo:
fo.write('{\n\"symbol\":\n')
fo.write(symbol_json)
if synset:
fo.write(',\n\"synset\": ')
fo.write(json.dumps(synset))
fo.write(',\n\"parambase64\": \"')
fo.write(model)
fo.write('\"\n')
if mean_image is not None:
fo.write(',\n\"meanimgbase64\": \"')
fo.write(mean_image)
fo.write('\"\n')
fo.write('}\n')
TL;DR
Are you using Python3? If so - use Python2 and things should work!
More details:
The code opens the model's binary weight file, reads the binary data, constructs a Bytes sequence (Python builtin type), and converts it into String.
Now, while Python 2 implicitly converts Bytes to String, Python 3 does not do it. So I suspect you are using Python 3, and then your conversion is incorrect.
To check your version run python --version
If you are indeed using Python 3, you can try and update line 12 of model2json.py to have explicit conversion:
model = str(base64.b64encode(bytes(open(sys.argv[3], 'rb').read())))
Note that for Python 3 you will also need to launch the local web server using a different command than the one noted on the readme.md: $ python3 -m http.server
My recommendation is that you use Python 2 since this entire repo is written for it, and using Python3 you might encounter other issues.
Related
I am trying to pipe the output of xz to a custom python script:
xz -cd file.pcap.xz | myscripy.py
However, I get an error when the script attempts to run this line:
#!/usr/bin/env python2.7
from __future__ import print_function
import pcap
import io
STDIN_ALIAS = '/proc/self/fd/0'
pcap.pcap(io.open(STDIN_ALIAS, 'r'))
and received an error
pcap.pcap(io.open(STDIN_ALIAS, 'r'))
File "pcap.pyx", line 196, in pcap.pcap.__init__
TypeError: expected string or Unicode object, _io.TextIOWrapper found
I am on Ubuntu 18.04 and running under python 2.7.
You can't use Python to pass in packets from a file to pcap.pcap(). The pypcap library you are using is a thin wrapper around the pcap_open_offline() and pcap_create() C functions, and offers no facilities for passing in a Python file object. This wrapper only accepts a filename or a network interface name, nothing else.
The pcap_open_offline() function does accept - as an alias for stdin, so just pass that in directly:
import pcap
sniffer = pcap.pcap('-')
The error message already tell you what happened. You need a string to the pcap() function, not a file object. To fix this, try
pcap.pcap(io.open(STDIN_ALIAS, 'r').read())
But I am not sure this will work as your file might be binary instead of text. In such case, you may want to open with 'rb' instead of 'r' flag, and do some conversion afterwards (especially if you use Python 3 instead of Python 2.7).
I see another issue: your code is not portable as it depends on this:
STDIN_ALIAS = '/proc/self/fd/0'
A pythonic way to read the stdin is the follows (see Reading binary data from stdin)
import sys
string = sys.stdin.read()
Have you tried upgrading PyPcap to work on Python 3? This could help, since Unicode handling is a lot cleaner and less prone to surprises on Python 3. The appropriate package is available, at least on Debian (and probably derived distros as well). Look for: python3-pypcap.
Beginner here, want to read in data with the file ending p.
My code looks like this :
import pickle
training_file = "/home/sk/CarND-Traffic-Sign-Classifier-Project/train.p"
testing_file = "/home/sk/CarND-Traffic-Sign-Classifier-Project/test.p"
with open(training_file, mode='rb') as f:
train = pickle.load(f)
with open(testing_file, mode='rb') as f:
test = pickle.load(f)
I get the following error:
ValueError: unsupported pickle protocol: 3
Can someone point out how i can fix it, either changing protocol or reading in the data some other way ?
Had the same issue when i created a pickle file using python3 and then tried loading it in python2. Try running your program with python3 or try creating a pickle file using python2.
Pickle uses different protocols to convert your data to a binary stream.
In python 2 there are 3 different protocols (0, 1, 2) and the default is 0.
In python 3 there are 5 different protocols (0, 1, 2, 3, 4) and the default is 3.
You must specify in python 3 a protocol lower than 3 in order to be able to load the data in python 2. You can specify the protocol parameter when invoking pickle.dump.
It seems as those files was created with a protocol >=3 (probably 3). So the only option you get is to load it into python 3 and then dump it with a lower protocol.
Evidently pickle protocol 3 was used in whatever python 3 code pickled the object. You can't unpickle with protocol 3 in python 2. You could however write a short python 3 program that loads it and then dumps it with protocol = 2. Then you can load them in python 2.
https://docs.python.org/2/library/pickle.html#usage
https://github.com/zopefoundation/zodbpickle
Under Python2, this package forks both Python 2.7's pickle and cPickle modules, adding support for the protocol 3 opcodes.
I converted a huge file which I wrote it at python 2.7.3 and then now I wanted to upgrade to python 3+ (i have 3.5).
what I have done so far:
installed the python interpreter 3.5+
updated the environment path to read from python3+ folder
upgraded the numpy, pandas,
I used >python 2to3.py -w viterbi.py to convert to version 3+
the section that I have error
import sys
import numpy as np
import pandas as pd
# Counting number of lines in the text file
lines = 0
buffer = bytearray(2048)
with open(inputFilePatheName) as f:
while f.readinto(buffer) > 0:
lines += buffer.count('\n')
My error is:
AttributeError: '_io.TextIOWrapper' object has no attribute 'readinto'
This is the first error and I cannot proceed to see if there is any other error. I dont know what is the equivalent command for readinto
In 3.x, the readinto method is only available on binary I/O streams. Thus: with open(inputFilePatheName, 'rb') as f:.
Separately, buffer.count('\n') will not work any more, because Python 3.x handles text properly, as something distinct from a raw sequence of bytes. buffer, being a bytearray, stores bytes; it still has a .count method, but it has to be given either an integer (representing the numeric value of a byte to look for) or a "bytes-like object" (representing a subsequence of bytes to look for). So we also have to update that, as buffer.count(b'\n') (using a bytes literal).
Finally, we need to be aware that processing the file this way means we don't get universal newline translation by default any more.
Open the file as binary.
As long as you can guarantee it's utf-8 or CP encoded, all \ns will necessarily be newlines:
with open(inputFilePatheName, "rb") as f:
while f.readinto(buffer) > 0:
lines += buffer.count(b'\n')
That way you also save the time of decoding the file, and use your buffer in the most efficient way possible.
A better approach to what you're trying to achieve is using memory mapped files.
In case of Windows:
file_handle = os.open(r"yourpath", os.O_RDONLY|os.O_BINARY|os.O_SEQUENTIAL)
try:
with mmap.mmap(file_handle, 0, access=mmap.ACCESS_READ) as f:
pos = -1
total = 0
while (pos := f.find(b"\n", pos+1)) != -1:
total +=1
finally:
os.close(file_handle)
Again, make sure you are not encoding the text as UTF-16 which is the default for Windows.
I'm trying to read a binary file into a buffer and then transmit it using pyserial/xmodem.
my test code:
send_buf = open('test.py', 'rb')
xmInst = XMODEM(self.getc, self.putc)
xmInst.send (send_buf)
xmodem send code:
...
data = stream.read(packet_size)
if not data:
break/
total_packets += 1
data = data.ljust(packet_size, self.pad)
However, when it executes that last line it says
'must be a byte string of length 1, not str'. Presumably a byte / string / unicode issue?
The xmodem package was written for python 2.7, so how do I read/pass the file in Python 3.4 such that xmodem can work with it?
I'm using PyOBEX to exchange binary files (e.g. images etc.) between my computer (Windows 7) and my phone (Android). However, when I use get() to get a file from my phone, it arrives on my computer as a str. I tried using the chardet module to find out what encoding to use to decode it and eventually turn it into a binary file, but it returned None. type() says that it's a str.
The code is the following:
import bluetooth
import BTDeviceFinder
import PyOBEX.client
name = "myDevice"
address = BTDeviceFinder.find_by_name(name)
port = BTDeviceFinder.find_port(address)
client = PyOBEX.client.BrowserClient(address, port)
client.connect()
a, b = client.get("pic.jpg")
where a is the header (that comes with a file sent via OBEX) and b is the actual file object. b looks something like this: https://drive.google.com/file/d/0By0ywTLTjb3LaFJaM2hWVEdBakE/view?usp=sharing
The PyOBEX documentation or Python forums say nothing about what encoding is used with get().
Do you know how to turn this string into binary data that can be used with write() and then saved in the original file format (i.e. .jpg)?
In python 2.7 strings represent raw bytes (this changes in python 3)
You simply need to save the data to a binary type file:
with open('file.jpg', 'wb') as handle:
handle.write(data_string)
Here is a link to the python doc on open:
https://docs.python.org/2/library/functions.html#open
Note that the "b" represents binary.
Again, this is assuming Python 2.7