I am having problems writing to a file where I am using pylast. Following a template given in pylast, I added a regular expression to extract what I need (which is doing ok), but when I tried to print to a file, I get an error, and don't know how to fix it (I am teaching myself python and some of its libraries).
I suspect there is an encoding specification I need to make somewhere (some of the output to screen also shows non-standard characters). I don't know how to solve my problem.
Can anybody help?
Thanks
import re
import pylast
RawArtistList = []
ArtistList = []
# You have to have your own unique two values for API_KEY and API_SECRET
# Obtain yours from http://www.last.fm/api/account for Last.fm
API_KEY = "XXX"
API_SECRET = "YYY"
###### In order to perform a write operation you need to authenticate yourself
username = "username"
password_hash = pylast.md5("password")
network = pylast.LastFMNetwork(api_key = API_KEY, api_secret = API_SECRET, username = username, password_hash = password_hash)
## _________INIT__________
COUNTRY = "Germany"
#---------------------- Get Geo Country --------------------
geo_country = network.get_country(COUNTRY)
#---------------------- Get artist --------------------
top_artists_of_country = str(geo_country.get_top_artists())
RawArtistList = re.findall(r"u'(.*?)'", top_artists_of_country)
top_artists_file = open("C:\artist.txt", "w")
for artist in RawArtistList:
print artist
top_artists_file.write(artist + "\n")
top_artists_file.close()
The name of the file I am trying to create "artist.txt" changes to "x07rtist.txt" and the error kicks in. I get this:
Traceback (most recent call last):
File "C:\music4A.py", line 32, in <module>
top_artists_file = open("C:\artist.txt", "w")
IOError: [Errno 22] invalid mode ('w') or filename:'C:\x07rtist.txt'
Thank you very much for any help! Cheers.
The Python docs say:
The backslash () character is used to escape characters that
otherwise have a special meaning, such as newline, backslash itself,
or the quote character.
...so when you say
top_artists_file = open("C:\artist.txt", "w")
that string literal is being interpreted as
C: \a rtist.txt
...where \a is a single character that has a value of 0x07.
...that line should instead be:
# doubling the backslash prevents misinterpreting the 'a'
top_artists_file = open("C:\\artist.txt", "w")
or
# define the string literal as a raw string to prevent the escape behavior
top_artists_file = open(r"C:\artist.txt", "w")
or
# forward slashes work just fine as path separators on Windows.
top_artists_file = open("C:/artist.txt", "w")
Related
How to replace string after specific character using python
I have a file with below contents
The Test file contents are as below "Test1"{
Serial = 12345
IP = 12.10.23.10
User = user1
}
how do i replace the contents after the = symbol using python ?
for example i want to replace with below contents.
The Test file contents are as below "Test1"{
Serial = 22330011
IP = 1.1.1.1
User = User_11
}
The contents after = symbols are not pre defined, hence substituting 12345 with 22330011 is not required here.
need a logic to find what is there after = symbol and replace it with the user defined value.
Lets say i have above data in temp.txt
file=open('temp.txt','r')
data=file.readlines()
outdata=[]
for line in data:
try:
lhs,rhs=line.split('=')
rhs=input()
outdata.append('='.join([lhs,' '+rhs,'\n']))
except:
outdata.append(line)
file.close()
file=open('temp.txt','w')
for line in outdata:
file.write(line)
This code read from the file and ask the input from user for rhs and updates in the file again
I'm building a Python program to parse some calls to a social media API into CSV and I'm running into an issue with a key that has two keys above it in the hierarchy. I get this error when I run the code with PyDev in Eclipse.
Traceback (most recent call last):
line 413, in <module>
main()
line 390, in main
postAgeDemos(monitorID)
line 171, in postAgeDemos
age0To17 = str(i["ageCount"]["sortedAgeCounts"]["ZERO_TO_SEVENTEEN"])
KeyError: 'ZERO_TO_SEVENTEEN'
Here's the section of the code I'm using for it. I have a few other functions built already that work with two layers of keys.
import urllib.request
import json
def postAgeDemos(monitorID):
print("Enter the date you'd like the data to start on")
startDate = input('The date must be in the format YYYY-MM-DD. ')
print("Enter the date you'd like the data to end on")
endDate = input('The date must be in the format YYYY-MM-DD. ')
dates = "&start="+startDate+"&end="+endDate
urlStart = getURL()
authToken = getAuthToken()
endpoint = "/monitor/demographics/age?id=";
urlData = urlStart+endpoint+monitorID+authToken+dates
webURL = urllib.request.urlopen(urlData)
fPath = getFilePath()+"AgeDemographics"+startDate+"&"+endDate+".csv"
print("Connecting...")
if (webURL.getcode() == 200):
print("Connected to "+urlData)
print("This query returns information in a CSV file.")
csvFile = open(fPath, "w+")
csvFile.write("postDate,totalPosts,totalPostsWithIdentifiableAge,0-17,18-24,25-34,35+\n")
data = webURL.read().decode('utf8')
theJSON = json.loads(data)
for i in theJSON["ageCounts"]:
postDate = i["startDate"]
totalDocs = str(i["numberOfDocuments"])
totalAged = str(i["ageCount"]["totalAgeCount"])
age0To17 = str(i["ageCount"]["sortedAgeCounts"]["ZERO_TO_SEVENTEEN"])
age18To24 = str(i["ageCount"]["sortedAgeCounts"]["EIGHTEEN_TO_TWENTYFOUR"])
age25To34 = str(i["ageCount"]["sortedAgeCounts"]["TWENTYFIVE_TO_THIRTYFOUR"])
age35Over = str(i["ageCount"]["sortedAgeCounts"]["THIRTYFIVE_AND_OVER"])
csvFile.write(postDate+","+totalDocs+","+totalAged+","+age0To17+","+age18To24+","+age25To34+","+age35Over+"\n")
print("File printed to "+fPath)
csvFile.close()
else:
print("Server Error, No Data" + str(webURL.getcode()))
Here's a sample of the JSON I'm trying to parse.
{"ageCounts":[{"startDate":"2016-01-01T00:00:00","endDate":"2016-01-02T00:00:00","numberOfDocuments":520813,"ageCount":{"sortedAgeCounts":{"ZERO_TO_SEVENTEEN":3245,"EIGHTEEN_TO_TWENTYFOUR":4289,"TWENTYFIVE_TO_THIRTYFOUR":2318,"THIRTYFIVE_AND_OVER":70249},"totalAgeCount":80101}},{"startDate":"2016-01-02T00:00:00","endDate":"2016-01-03T00:00:00","numberOfDocuments":633709,"ageCount":{"sortedAgeCounts":{"ZERO_TO_SEVENTEEN":3560,"EIGHTEEN_TO_TWENTYFOUR":1702,"TWENTYFIVE_TO_THIRTYFOUR":2786,"THIRTYFIVE_AND_OVER":119657},"totalAgeCount":127705}}],"status":"success"}
Here it is again with line breaks so it's a little easier to read.
{"ageCounts":[{"startDate":"2016-01-01T00:00:00","endDate":"2016-01-02T00:00:00","numberOfDocuments":520813,"ageCount":
{"sortedAgeCounts":{"ZERO_TO_SEVENTEEN":3245,"EIGHTEEN_TO_TWENTYFOUR":4289,"TWENTYFIVE_TO_THIRTYFOUR":2318,"THIRTYFIVE_AND_OVER":70249},"totalAgeCount":80101}},
{"startDate":"2016-01-02T00:00:00","endDate":"2016-01-03T00:00:00","numberOfDocuments":633709,"ageCount":
{"sortedAgeCounts":{"ZERO_TO_SEVENTEEN":3560,"EIGHTEEN_TO_TWENTYFOUR":1702,"TWENTYFIVE_TO_THIRTYFOUR":2786,"THIRTYFIVE_AND_OVER":119657},"totalAgeCount":127705}}],"status":"success"}
I've tried removing the ["sortedAgeCounts"] from in the middle of
age0To17 = str(i["ageCount"]["sortedAgeCounts"]["ZERO_TO_SEVENTEEN"])
but I still get the same error. I've remove the 0-17 section to test the other age ranges and I get the same error for them as well. I tried removing all the underscores from the JSON and then using keys without the underscores.
I've also tried moving the str() to convert to string from the call to where the output is printed but the error persists.
Any ideas? Is this section not actually a JSON key, maybe a problem with the all caps or am I just doing something dumb? Any other code improvements are welcome as well but I'm stuck on this one.
Let me know if you need to see anything else. Thanks in advance for your help.
Edited(This works):
JSON=json.loads(s)
for i in JSON:
print str(JSON[i][0]["ageCount"]["sortedAgeCounts"]["ZERO_TO_SEVENTEEN"])
s is a string which contains the your JSON.
I'm trying to get authenticated by an API I'm attempting to access. I'm using urllib.parse.urlencode to encode the parameters which go in my URL. I'm using urllib.request.urlopen to fetch the content.
This should return 3 values from the server, such as:
SID=AAAAAAAAAAA
LSID=BBBBBBBBBBB
AUTH=CCCCCCCCCCC
The problem is it only returns the first value, and the trailing new line character.
import urllib.request
import urllib.parse
Emailparamx = 'Email'
Emailparam = Emailparamx.encode('utf-8')
email = 'myemail#stackoverflow.com'
email = email.encode('utf-8')
Passwdparam = 'Passwd'
Passwdparam = Passwdparam.encode('utf-8')
password = 'hidden'
password = password.encode('utf-8')
Accounttypeparam = 'accountType'
Accounttypeparam = Accounttypeparam.encode('utf-8')
accounttype = 'GOOGLE'
accounttype = accounttype.encode('utf-8')
Serviceparam = 'service'
Serviceparam = Serviceparam.encode('utf-8')
service = 'adwords'
service = service.encode('utf-8')
url = 'https://accounts.google.com/ClientLogin?'
urlen = url.encode('utf-8')
data = [(Emailparamx, email), (Passwdparam, password),
(Accounttypeparam, accounttype), (Serviceparam, service)]
auth = ''
dataurl = urllib.parse.urlencode(data)
accessurl = (url + "%s" % dataurl)
fh = urllib.request.urlopen(accessurl)
equals = '='
eqenc = equals.encode('utf-8')
try:
msg = fh.readline().split(eqenc)
print (msg)
And then msg prints
[b'SID', b'AAAAAAAAAAAAAAAAA\n']
I know that's some seriously ugly code, I'm about a week old in Python. Any help would be greatly appreciated.
The problem is that you're only calling readline once, so it only reads one line. If you want to read the lines one by one, you have to keep calling readline in a loop until done:
while True:
msg = fh.readline()
if not msg:
break
msg = msg.split(eqenc)
print(msg)
However, there's really no good reason to call readline here, because any file-like object (including a urlopen object) is already an iterable full of lines, so you can just do this:
for msg in fh:
print(msg)
Meanwhile, your original code has a try without an except or a finally, which will just raise a SyntaxError. Presumably you wanted something like this:
try:
for msg in fh:
print(msg)
except Exception as e:
print('Exception: {}'.format(e))
While we're at it, we can simplify your code a bit.
If you look at the examples:
Here is an example session that uses the GET method to retrieve a URL containing parameters:
That's exactly what you want to do here (except for the last line). All the extra stuff you're doing with encoding the strings is not only unnecessary, but incorrect. UTF-8 is the wrong encoding is the wrong encoding to use for URLs (you get away with it because all of your strings are pure ASCII); urlopen requires a string rather than an encoded byte string (although, at least in CPython 3.0-3.3, it happens to work if you give it byte strings that happen to be encoded properly); urlencode can take byte strings but may not do the right thing (you want to give it the original Unicode so it can quote things properly); etc.
Also, you probably want to decode the result (which is sent as ASCII—for more complicated examples, you'll have to either parse the fh.getheader('Content-Type'), or read the documentation for the API), and strip the newlines.
You also may want to build a structure you can use in your code instead of just printing it out. For example, if you store the results in login_info, and you need the SID in a later request, it's just login_info['SID'].
So, let's wrap things up in a function, then call that function:
import urllib.request
import urllib.parse
def client_login(email, passwd, account_type, service):
params = {'Email': email,
'Passwd': passwd,
'accountType': account_type,
'service': service}
qs = urllib.parse.urlencode(params)
url = 'https://accounts.google.com/ClientLogin?'
with urllib.request.urlopen(url + qs) as fh:
return dict(line.strip().decode('ascii').split('=', 1) for line in fh)
email = 'myemail#stackoverflow.com'
password = 'hidden'
accounttype = 'GOOGLE'
service = 'adwords'
try:
results = client_login(email, password, accounttype, service)
for key, value in results.items():
print('key "{}" is "{}".format(key, value))
except Exception as e:
print('Exception: {}'.format(e))
I'm processing a large (120mb) text file from my thunderbird imap directory and attempting to extract to/from info from the headers using mbox and regex. the process runs for a while until I eventually get an exception: "TypeError: expected string or buffer".
The exception references the fifth line of this code:
PAT_EMAIL = re.compile(r"[0-9A-Za-z._-]+\#[0-9A-Za-z._-]+")
temp_list = []
mymbox = mbox("data.txt")
for email in mymbox.values():
from_address = PAT_EMAIL.findall(email["from"])
to_address = PAT_EMAIL.findall(email["to"])
for item in from_address:
temp_list.append(item) #items are added to a temporary list where they are sorted then written to file
I've run the code on other (smaller) files, so I'm guessing the issue is my file. The file appears to be just a bunch of text. Can someone point me in the write direction for debugging this?
There can only be one from address (I think!):
In the following:
from_address = PAT_EMAIL.findall(email["from"])
I have a feeling you're trying to duplicate the work of email.message_from_file and email.utils.parseaddr
from email.utils import parseaddr
>>> s = "Jon Clements <jon#example.com>"
>>> from email.utils import parseaddr
>>> parseaddr(s)
('Jon Clements', 'jon#example.com')
So you can use parseaddr(email['from'])[1] to get the email address and use that.
Similarly, you may wish to look at email.utils.getaddresses to handle to and cc addresses...
Well, I didn't solve the issue but have worked around it for my own purposes. I inserted a try statement so that the iteration just continues past any TypeError. For every thousand email addresses I'm getting about 8 failures, which will suffice. Thanks for your input!
PAT_EMAIL = re.compile(r"[0-9A-Za-z._-]+\#[0-9A-Za-z._-]+")
temp_list = []
mymbox = mbox("data.txt")
for email in mymbox.values():
try:
from_address = PAT_EMAIL.findall(email["from"])
except(TypeError):
print "TypeError!"
try:
to_address = PAT_EMAIL.findall(email["to"])
except(TypeError):
print "TypeError!"
for item in from_address:
temp_list.append(item) #items are added to a temporary list where they are sorted then written to file
I have a request to handle with filename list on ftp server. But filename includes Asian character and other unknown characters. So I need to judge which filename can be decoded by gb2312, which can be decoded by iso-8859-1. That means if the filename list cannot be gotten using gb2312, then use iso-88591-1 to get. So I don't know how to write code in the following function which is in ftplib
def retrlines(self, cmd, callback = None):
"""Retrieve data in line mode. A new port is created for you.
Args:
cmd: A RETR, LIST, NLST, or MLSD command.
callback: An optional single parameter callable that is called
for each line with the trailing CRLF stripped.
[default: print_line()]
Returns:
The response code.
"""
if callback is None: callback = print_line
resp = self.sendcmd('TYPE A')
##################I need to update here############################
with self.transfercmd(cmd) as conn, \
conn.makefile('r', encoding='iso-8859-1') as fp:
###################################################################
while 1:
line = fp.readline()
print(line)
if self.debugging > 2: print('*retr*', repr(line))
if not line:
break
if line[-2:] == CRLF:
line = line[:-2]
elif line[-1:] == '\n':
line = line[:-1]
callback(line)
return self.voidresp()
You aren't including much of the code, so it's hard to tell exactly what is going on. But as a general rule, if the data you are interacting with isn't consistent in it's usage of encodings, you will have to interact with it in binary mode.
So try not passing in an encoding at all. Hopefully that will give you bytes data back, and you can then encode/decode according to the needs of each file.