Python urlopen return value - python

I'm trying to pass existing URLs as parameter to load it's HTML in a single txt file:
for line in open('C:\Users\me\Desktop\URLS-HERE.txt'):
if line.startswith('http') and line.endswith('html\n') :
fichier = open("C:\Users\me\Desktop\other.txt", "a")
allhtml = urllib.urlopen(line)
fichier.write(allhtml)
fichier.close()
but i get the following error:
TypeError: expected a character buffer object

The value returned by urllib.urlopen() is a file like object, once you have opened it, you should read it with the read() method, as showed in the following snippet:
for line in open('C:\Users\me\Desktop\URLS-HERE.txt'):
if line.startswith('http') and line.endswith('html\n') :
fichier = open("C:\Users\me\Desktop\other.txt", "a")
allhtml = urllib.urlopen(line)
fichier.write(allhtml.read())
fichier.close()
Hope this helps!

The problem here is that urlopen returns a reference to a file object from which you should retrieve HTML.
for line in open(r"C:\Users\me\Desktop\URLS-HERE.txt"):
if line.startswith('http') and line.endswith('html\n') :
fichier = open(r"C:\Users\me\Desktop\other.txt", "a")
allhtml = urllib2.urlopen(line)
fichier.write(allhtml.read())
fichier.close()
Please note that urllib.urlopen function is marked as deprecated since python 2.6. It's recommended to use urllib2.urlopen instead.
Additionally, you have to be careful working with paths in your code. You should either escape each \
"C:\\Users\\me\\Desktop\\other.txt"
or use r prefix before a string. When an 'r' or 'R' prefix is present, a character following a backslash is included in the string without change.
r"C:\Users\me\Desktop\other.txt"

Related

Python writing to file and json returns None/null instead of value

I'm trying to write data to a file with the following code
#!/usr/bin/python37all
print('Content-type: text/html\n\n')
import cgi
from Alarm import *
import json
htmldata = cgi.FieldStorage()
alarm_time = htmldata.getvalue('alarm_time')
alarm_date = htmldata.getvalue('alarm_date')
print(alarm_time,alarm_date)
data = {'time':alarm_time,'date':alarm_date}
# print(data['time'],data['date'])
with open('alarm_data.txt','w') as f:
json.dump(data,f)
...
but when opening the the file, I get the following output:
{'time':null,'date':null}
The print statement returns what I except it to: 14:26 2020-12-12.
I've tried this same method with f.write() but it returns both values as None. This is being run on a raspberry pi. Why aren't the correct values being written?
--EDIT--
The json string I expect to see is the following:{'time':'14:26','date':'2020-12-12'}
Perhaps you meant:
data = {'time':str(alarm_time), 'date':str(alarm_date)}
I would expect to see your file contents like this:
{"time":"14:26","date":"2020-12-12"}
Note the double quotes: ". json is very strict about these things, so don't fool yourself into having single quotes ' in a file and expecting json to parse it.

Python: returning 'None' from a sub and printing return value

I have the python script below which returns a variable which can either be None or a blob of data read from a file. I thought this would be an easy way to determine one of three states:
any error returns None
empty data returns an empty variable
successful read returns data in variable
This (also below) is the error I am getting. I could just assign "" to data instead of None and test for an empty variable but that seems a little cheesy. Printing the variable wrapped in str() might work but that seems like a lot of extra typing (or special function to print the variable) every time I want to print the variable. I must be waaayyy off base here.
$ ./foo.py
Traceback (most recent call last):
File "./foo.py", line 20, in <module>
print " data read was " + line_read
TypeError: cannot concatenate 'str' and 'builtin_function_or_method' objects
code:
#!/usr/bin/python
import sys, os
def get_data (src):
data = None
if os.path.isfile (src):
with open(src, 'rt') as f:
data = f.read
return data
line_read = get_data('foo.txt')
if line_read is None:
print "no data in file"
sys.exit(1)
print " data read was " + line_read
It seems the problem is you are using f.read. Python supports high-order programming, which in essence allows you to assign functions/method to variables. Then, what you are returning from get_data is basically a function which cannot be concatenated with a string which is the error you are getting, so to fix this call the method read as follows:
#!/usr/bin/python
import sys, os
def get_data (src):
data = None
if os.path.isfile (src):
with open(src, 'rt') as f:
data = f.read() # this will solve your problem
return data
line_read = get_data('foo.txt')
if line_read is None:
print "no data in file"
sys.exit(1)
print " data read was " + line_read
You have just assigned variable data to a method with that data = f.read call, you should actually call that method with:
data = f.read()
You should perhaps add parenthesis from wherever you are calling a function. In your program,f.read() is a function but you typed it as f.read which causes the error. The solution is to replace f.read with f.read().

Can't convert a string to JSON using python 3? [duplicate]

This question already has answers here:
How can I parse (read) and use JSON?
(5 answers)
Closed 25 days ago.
In Python I'm getting an error:
Exception: (<type 'exceptions.AttributeError'>,
AttributeError("'str' object has no attribute 'read'",), <traceback object at 0x1543ab8>)
Given python code:
def getEntries (self, sub):
url = 'http://www.reddit.com/'
if (sub != ''):
url += 'r/' + sub
request = urllib2.Request (url +
'.json', None, {'User-Agent' : 'Reddit desktop client by /user/RobinJ1995/'})
response = urllib2.urlopen (request)
jsonStr = response.read()
return json.load(jsonStr)['data']['children']
What does this error mean and what did I do to cause it?
The problem is that for json.load you should pass a file like object with a read function defined. So either you use json.load(response) or json.loads(response.read()).
Ok, this is an old thread but.
I had a same issue, my problem was I used json.load instead of json.loads
This way, json has no problem with loading any kind of dictionary.
Official documentation
json.load - Deserialize fp (a .read()-supporting text file or binary file containing a JSON document) to a Python object using this conversion table.
json.loads - Deserialize s (a str, bytes or bytearray instance containing a JSON document) to a Python object using this conversion table.
You need to open the file first. This doesn't work:
json_file = json.load('test.json')
But this works:
f = open('test.json')
json_file = json.load(f)
If you get a python error like this:
AttributeError: 'str' object has no attribute 'some_method'
You probably poisoned your object accidentally by overwriting your object with a string.
How to reproduce this error in python with a few lines of code:
#!/usr/bin/env python
import json
def foobar(json):
msg = json.loads(json)
foobar('{"batman": "yes"}')
Run it, which prints:
AttributeError: 'str' object has no attribute 'loads'
But change the name of the variablename, and it works fine:
#!/usr/bin/env python
import json
def foobar(jsonstring):
msg = json.loads(jsonstring)
foobar('{"batman": "yes"}')
This error is caused when you tried to run a method within a string. String has a few methods, but not the one you are invoking. So stop trying to invoke a method which String does not define and start looking for where you poisoned your object.
AttributeError("'str' object has no attribute 'read'",)
This means exactly what it says: something tried to find a .read attribute on the object that you gave it, and you gave it an object of type str (i.e., you gave it a string).
The error occurred here:
json.load(jsonStr)['data']['children']
Well, you aren't looking for read anywhere, so it must happen in the json.load function that you called (as indicated by the full traceback). That is because json.load is trying to .read the thing that you gave it, but you gave it jsonStr, which currently names a string (which you created by calling .read on the response).
Solution: don't call .read yourself; the function will do this, and is expecting you to give it the response directly so that it can do so.
You could also have figured this out by reading the built-in Python documentation for the function (try help(json.load), or for the entire module (try help(json)), or by checking the documentation for those functions on http://docs.python.org .
Instead of json.load() use json.loads() and it would work:
ex:
import json
from json import dumps
strinjJson = '{"event_type": "affected_element_added"}'
data = json.loads(strinjJson)
print(data)
So, don't use json.load(data.read()) use json.loads(data.read()):
def findMailOfDev(fileName):
file=open(fileName,'r')
data=file.read();
data=json.loads(data)
return data['mail']
use json.loads() function , put the s after that ... just a mistake btw i just realized after i searched error
def getEntries (self, sub):
url = 'http://www.reddit.com/'
if (sub != ''):
url += 'r/' + sub
request = urllib2.Request (url +
'.json', None, {'User-Agent' : 'Reddit desktop client by /user/RobinJ1995/'})
response = urllib2.urlopen (request)
jsonStr = response.read()
return json.loads(jsonStr)['data']['children']
try this
Open the file as a text file first
json_data = open("data.json", "r")
Now load it to dict
dict_data = json.load(json_data)
If you need to convert string to json. Then use loads() method instead of load(). load() function uses to load data from a file so used loads() to convert string to json object.
j_obj = json.loads('["label" : "data"]')

I need to use the format method to substitute an unknown number of parameters into a text file

The pagesubs() function reads a text file into a string,then uses Python's format() to substitute for arguments given in the subs parameter.
Here are examples:
My attempt was as follows:
def pagesubs(N,*subs):
assert type(N)==str
F= open(N,'r')
return F.format(subs)
I get an error that F is type(file) but I thought the open() reads a text file into a string. Any help is greatly appreciated
EDIT: Example
pagesubs('index.html', 'December', 18, 2012)
This will return the content of the file index.html, but
with "December" substituted for {0}, and 18 substituted
for {1}, and 2012 substituted for {2}.
You need to call the read() function on your file in order to get it's content into a string. Try the following code instead:
def pagesubs(N, *subs):
assert type(N)==str
with open(N,'r') as F:
content = F.read()
return content.format(subs)

About encode/decode in ftplib in python3

I have a request to handle with filename list on ftp server. But filename includes Asian character and other unknown characters. So I need to judge which filename can be decoded by gb2312, which can be decoded by iso-8859-1. That means if the filename list cannot be gotten using gb2312, then use iso-88591-1 to get. So I don't know how to write code in the following function which is in ftplib
def retrlines(self, cmd, callback = None):
"""Retrieve data in line mode. A new port is created for you.
Args:
cmd: A RETR, LIST, NLST, or MLSD command.
callback: An optional single parameter callable that is called
for each line with the trailing CRLF stripped.
[default: print_line()]
Returns:
The response code.
"""
if callback is None: callback = print_line
resp = self.sendcmd('TYPE A')
##################I need to update here############################
with self.transfercmd(cmd) as conn, \
conn.makefile('r', encoding='iso-8859-1') as fp:
###################################################################
while 1:
line = fp.readline()
print(line)
if self.debugging > 2: print('*retr*', repr(line))
if not line:
break
if line[-2:] == CRLF:
line = line[:-2]
elif line[-1:] == '\n':
line = line[:-1]
callback(line)
return self.voidresp()
You aren't including much of the code, so it's hard to tell exactly what is going on. But as a general rule, if the data you are interacting with isn't consistent in it's usage of encodings, you will have to interact with it in binary mode.
So try not passing in an encoding at all. Hopefully that will give you bytes data back, and you can then encode/decode according to the needs of each file.

Categories

Resources