I'm making a program using the website http://placekitten.com, but I've run into a bit of a problem. Using this:
im = urllib2.urlopen(url).read()
f = open('kitten.jpeg', 'w')
f.write(im)
f.close()
The image turns out distorted with mismatched colors, like this:
http://imgur.com/zVg64Kn.jpeg
I was wondering if there was an alternative to extracting images with urllib2. If anyone could help, that would be great!
You need to open the file in binary mode:
f = open('kitten.jpeg', 'wb')
Python will otherwise translate line endings to the native platform form, a transformation that breaks binary data, as documented for the open() function:
The default is to use text mode, which may convert '\n' characters to a platform-specific representation on writing and back on reading. Thus, when opening a binary file, you should append 'b' to the mode value to open the file in binary mode, which will improve portability.
When copying data from a URL to a file, you could use shutil.copyfileob() to handle streaming efficiently:
from shutil import copyfileobj
im = urllib2.urlopen(url)
with open('kitten.jpeg', 'wb') as out:
copyfileobj(im, out)
This will read data in chunks, avoiding filling memory with large blobs of binary data. The with statement handles closing the file object for you.
Change
f = open('kitten.jpeg', 'w')
to read
f = open('kitten.jpeg', 'wb')
See http://docs.python.org/2/library/functions.html#open for more information. What's happening is that the newlines in the jpeg are getting modified in the process of saving, and opening as a binary file will prevent this.
If you're using Windows, you have to open the file in binary mode:
f = open('kitten.jpeg', 'wb')
Or more Pythonically:
import urllib2
url = 'http://placekitten.com.s3.amazonaws.com/homepage-samples/200/140.jpg'
image = urllib2.urlopen(url).read()
with open('kitten.jpg', 'wb') as handle:
handle.write(image)
Related
I have a file and want to convert it into BytesIO object so that it can be stored in database's varbinary column.
Please can anyone help me convert it using python.
Below is my code:
f = open(filepath, "rb")
print(f.read())
myBytesIO = io.BytesIO(f)
myBytesIO.seek(0)
print(type(myBytesIO))
Opening a file with open and mode read-binary already gives you a Binary I/O object.
Documentation:
The easiest way to create a binary stream is with open() with 'b' in the mode string:
f = open("myfile.jpg", "rb")
So in normal circumstances, you'd be fine just passing the file handle wherever you need to supply it. If you really want/need to get a BytesIO instance, just pass the bytes you've read from the file when creating your BytesIO instance like so:
from io import BytesIO
with open(filepath, "rb") as fh:
buf = BytesIO(fh.read())
This has the disadvantage of loading the entire file into memory, which might be avoidable if the code you're passing the instance to is smart enough to stream the file without keeping it in memory. Note that the example uses open as a context manager that will reliably close the file, even in case of errors.
I try to download images, but they become corrupted for some reason? For example: This is an image I want to get.
And the result is this
My test code is:
import urllib2
def download_web_image(url):
request = urllib2.Request(url)
img = urllib2.urlopen(request).read()
with open ('test.jpg', 'w') as f: f.write(img)
download_web_image("http://upload.wikimedia.org/wikipedia/commons/8/8c/JPEG_example_JPG_RIP_025.jpg")
Why is this and how do I fix this?
You are opening 'test.jpg' file in the default (text) mode, which causes Python to use the "correct" newlines on Windows:
In text mode, the default when reading is to convert platform-specific
line endings (\n on Unix, \r\n on Windows) to just \n. When writing in
text mode, the default is to convert occurrences of \n back to
platform-specific line endings.
Of course, JPEG files are not text files, and 'fixing' the newlines will only corrupt the image. Instead, open the file in binary mode:
with open('test.jpg', 'wb') as f:
f.write(img)
For more details, see the documentation.
everyone!
I need a little bit of help in working with Binary Files (.dat) in python.
I am using Pickle Library to input in file which succeeds but when it comes to reading from file, my program does not work.
I need help in;
calculating the file size of the specific file, in bytes.
moving the pointer to specific bytes in file. (using .seek would be better.)
reading a specific byte from file. (using pickle.load would be better.)
looping over the specific file to print all bytes. (I get EOFError using while True: )
Any help would be appreciated.
This is my testing code so far and it has a lot of issues.
import pickle
with open ("BinaryFile.dat" , mode = "ab") as MyFile:
pickle.dump("New" , MyFile)
with open("BinaryFile.dat" , mode = "rb") as MyReadFile:
MyReadFile.seek(3)
NewLine = pickle.load(MyReadFile)
print (NewLine)
input("-> ")
pickle will do everything for you, just check its API
import pickle
with open ('BinaryFile.dat', mode='ab') as MyFile:
pickle.dump('New', MyFile)
pickle.dump([1, 2], MyFile)
pickle.dump(pickle.dump, MyFile)
# etc.
with open('BinaryFile.dat', mode='rb') as MyReadFile:
try:
while 1:
print pickle.load(MyReadFile)
except EOFError:
pass
I'm having problems getting images to convert out of bytes/strings/etc. I can turn an image into a string, or a byte array, or use b64encode on it, but when I try decode/revert it back to an image, it never works. I've tried a lot of things, locally converting an image and then reconverting it, saving it under a different name. However, the resulting files will never actually show anything. (black on Linux, "can't display image" on windows)
My most basic b64encoding script is as follows:
import base64
def convert(image):
f = open(image)
data = f.read()
f.close()
string = base64.b64encode(data)
convertit = base64.b64decode(string)
t = open("Puppy2.jpg", "w+")
t.write(convertit)
t.close()
if __name__ == "__main__":
convert("Puppy.jpg")
I've been stuck on this for a while. I'm sure it's a simple solution, but being new to Python, it's been a bit difficult trying to sort things out.
If it helps with any insight, the end goal here is to transfer images over a network, possibly MQTT.
Any help is much appreciated. Thanks!
Edit** This is in Python 2.7.
Edit 2** Wow, you guys move fast. What a great intro to the community - thanks a lot for the quick responses and super fast results!
For python3, you need to open and write in binary mode:
def convert(image):
f = open(image,"rb")
data = f.read()
f.close()
string = base64.b64encode(data)
convert = base64.b64decode(string)
t = open("Puppy2.jpg", "wb")
t.write(convert)
t.close()
Using python 2 on linux, simply r and w should work fine. On windows you need to do the same as above.
from the docs:
On Windows, 'b' appended to the mode opens the file in binary mode, so there are also modes like 'rb', 'wb', and 'r+b'. Python on Windows makes a distinction between text and binary files; the end-of-line characters in text files are automatically altered slightly when data is read or written. This behind-the-scenes modification to file data is fine for ASCII text files, but it’ll corrupt binary data like that in JPEG or EXE files. Be very careful to use binary mode when reading and writing such files. On Unix, it doesn’t hurt to append a 'b' to the mode, so you can use it platform-independently for all binary files.
You can also write your code a little more succinctly by using with to open your files which will automatically close them for you:
from base64 import b64encode, b64decode
def convert(image):
with open(image, "rb") as f, open("Puppy2.jpg", "wb") as t:
conv = b64decode(b64encode(f.read()))
t.write(conv)
import base64
def convert(image):
f = open(image)
data = f.read()
f.close()
return data
if __name__ == "__main__":
data = convert("Puppy2.jpg")
string = base64.b64encode(data)
convert = base64.b64decode(string)
t = open("Puppy2.jpg", "w+")
t.write(convert)
t.close()
I'm slightly new to Python and have a question as to why the following code doesn't produce any output in the csv file. The code is as follows:
import csv
import urllib2
url = 'http://www.rba.gov.au/statistics/tables/csv/f17-yields.csv'
response = urllib2.urlopen(url)
cr = csv.reader(response)
for row in cr:
with open("AusCentralbank.csv", "wb") as f:
writer = csv.writer(f)
writer.writerows(row)
Cheers.
Edit:
Brien and Albert solved the initial issue I had. However, I now have one further question. When I download the CSV File which I have listed above which is in "http://www.rba.gov.au/statistics/tables/#interest-rates" under Zero-coupon "Interest Rates - Analytical Series - 2009 to Current - F17" and is the F-17 Yields CSV I see that it has 5 workbooks and I actually just want to gather the data in the 5th Workbook. Is there a way I could do this? Cheers.
I could only test my code using Python 3. However, the only diffence should be urllib2, hence I am using urllib.respose for opening the desired url.
The variable html is type bytes and can generally be written to a file in binary mode. Additionally, your source is a csv-file already, so there should be no need to convert it somehow:
#!/usr/bin/env python3
# coding: utf-8
import urllib
url = 'http://www.rba.gov.au/statistics/tables/csv/f17-yields.csv'
response = urllib.request.urlopen(url)
html = response.read()
with open('output.csv', 'wb') as f:
f.write(html)
It is probably because of your opening mode.
According to documentation:
'w' for only writing (an existing file with the same name will be
erased)
You should use append(a) mode to append it to the end of the file.
'a' opens the file for appending; any data written to the file is
automatically added to the end.
Also, since the file you are trying to download is csv file, you don't need to convert it.
#albert had a great answer. I've gone ahead and converted it to the equivalent Python 2.x code. You were doing a bit too much work in your original program; since the file was already a csv you didn't need to do any special work to turn it into a csv.
import urllib2
url = 'http://www.rba.gov.au/statistics/tables/csv/f17-yields.csv'
response = urllib2.urlopen(url)
html = response.read()
with open('AusCentralbank.csv', 'wb') as f:
f.write(html)