I'm creating a discord bot in python and I want to have a log system for the warn command. To set up this, I'm using a .csv file in which I write all the information I want about a user.
It's the first time I use .csv, so to manipulate the data, I take the content of the file and convert it into a list of list. One sublist = one line of the table.
After treatment, I put all the elements of each sublists side by side, separated by a ";" to store it properly in the .csv file.
The problem I have comes from this :
csvwriter = csv.writer(wfile, delimiter=";")
For each caracter in the final file, a ";" is put right after so instead of this : (abc;def;ghi), I have this : (a;b;c;d;...).
All of the code :
#commands.command()
#commands.has_any_role(765280296781742090, 765609632482721852, 882222368976670772)
async def test(self, ctx, member:discord.Member):
messages = await ctx.channel.history(limit = 1).flatten()
for each_message in messages:
await each_message.delete()
with open("ressources/logs/warn_logs.csv", 'r', newline='') as csvfile:
content = []
new_registrated = []
is_registrated = False
for line in csvfile:
line = line.split(";")
content.append(line)
for cell in content:
if cell == content[0]:
pass
else:
if int(cell[0]) == int(member.id):
is_registrated = True
cell[3] = int(cell[3])+1
cell[3] = str(cell[3])
if is_registrated == False:
new_registrated.append(str(member.id))
new_registrated.append(str(member._user))
new_registrated.append(member.nick)
new_registrated.append("1\r\n")
content.append(new_registrated)
with open("ressources/logs/warn_logs.csv", 'w', newline='') as wfile:
csvwriter = csv.writer(wfile, delimiter=";")
# for line in content:
# print(line)
# line = ";".join(line)
# print(line)
# csvwriter.writerow(line)
csvwriter.writerow("abc")
wfile.close()
csvfile.close()
What I have :
What I want :
I'm working on it for a while now so if someone could help me, I would be very greatful.
NB: I'm French, and my English isn't perfect so don't hesitate to correct me
I tried to:
Not put the delimiter : the ";" became ","
Put delimiter="smth" / =False / =None : it doesn't work
Any other character : ";" became any other character
The writerow() function expects an iterable object to write a whole row.
csvwriter.writerow("abc")
will write to the file:
a;b;c
and
csvwriter.writerow(["abc", "def"])
will write:
abc;def
If line is a list of strings then line = ";".join(line) is wrong, line should go directly to writerow(). The whole point of using a csv module in not to think about delimiters.
Also if you are using a with open(...) as f: you don't have to close the file.
Related
Gotta write a function that receives a file path and a "new_song" string(aka def my_mp4_playlist(file_path, new_song). The function does 2 things: open the file and write the name of the new song in the 3rd line of the file (in the correct position) and then prints out the new file content.
Starting file content:
Tudo Bom;Static and Ben El Tavori;5:13;
I Gotta Feeling;The Black Eyed Peas;4:05;
Instrumental;Unknown;4:15;
Paradise;Coldplay;4:23;
Where is the love?;The Black Eyed Peas;4:13;
Required print:
Tudo Bom;Static and Ben El Tavori;5:13;
I Gotta Feeling;The Black Eyed Peas;4:05;
**new_song**;Unknown;4:15;
Paradise;Coldplay;4:23;
Where is the love?;The Black Eyed Peas;4:13;
What I have so far:
def my_mp4_playlist(file_path, new_song):
with open(file_path, "r") as reading_file:
data = reading_file.read()
data = data.split("\n")
individual = []
for element in data:
individual.append(element.split(";"))
for song in individual:
song.remove(song[3]) #this removes a space leftover as the last value in each list
individual = individual[3][0].replace(individual[3][0], new_song)
with open(file_path, "w") as writing_file:
writing_file.write(individual)
with open(file_path, "r") as complete_file:
print(complete_file.read())
Unfortunately the replace() method overwrites the entire file content and only new_song appears there. How should I go about this?
This line here is your issue:
individual = individual[3][0].replace(individual[3][0], new_song)
A simple assignment should do the trick:
individual[2][0] = new_song
Please note that I am using index 2 for the 3rd line.
Also, you cannot pass a list as parameter to write(), you can only pass a string.
You can use join() and list comprehension to build your string. Your code would then look like this:
def my_mp4_playlist(file_path, new_song):
with open(file_path, "r") as reading_file:
data = reading_file.read()
data = data.split("\n")
individual = []
for element in data:
individual.append(element.split(";"))
individual[2][0] = new_song
data = "\n".join([";".join(el) for el in individual])
with open(file_path, "w") as writing_file:
writing_file.write(data)
with open(file_path, "r") as complete_file:
print(complete_file.read())
Try this one. I did not know if your replacement is always at the same place or has always the same name so I added another attribute where you have to put the name of the part which has to be replaced
path = "YOUR/PATH/FILE.txt"
old_song= "Instrumental"
new_song = "ABCDEFG"
def my_mp4_playlist(file_path, new_song, old_song):
with open(file_path, "r+") as f:
content = f.read().replace(old_song, new_song)
f.truncate(0)
f.seek(0, 0)
f.write(content)
my_mp4_playlist(path, new_song, old_song)
I will just tell you the mistake in your code.
individual = individual[3][0].replace(individual[3][0], new_song)
You have mentioned it as 3rd element so individual[3][0] should be individual[2][0]
This one replaces individual with new value of individual[2][0].
But you just need to update that line:
individual[2][0] = individual[2][0].replace(individual[2][0], new_song)
which can be further simplified to:
individual[2][0] = new_song
First of all, as somebody said, you don't need to split the data, you can use reading_file.readlines().
Secondly, you said you want to replace the third line, not the forth, but in your code you are doing
individual = individual[3][0].replace(individual[3][0], new_song)
Remember that lists starts with the index 0.
With that being said, here's what I think would work:
def my_mp4_playlist(file_path, new_song):
with open(file_path, "r") as reading_file:
data = reading_file.readlines()
song_to_be_changed = data[2][:data[2].index(";")]
with open(file_path, "w") as writing_file:
writing_file.write("".join(data).replace(song_to_be_changed, new_song))
with open(file_path, "r") as complete_file:
print(complete_file.read())
def generate_daily_totals(input_filename, output_filename):
"""result in the creation of a file blahout.txt containing the two lines"""
with open(input_filename, 'r') as reader, open(output_filename, 'w') as writer: #updated
for line in reader: #updated
pieces = line.split(',')
date = pieces[0]
rainfall = pieces[1:] #each data in a line
total_rainfall = 0
for data in rainfall:
pure_data = data.rstrip()
total_rainfall = total_rainfall + float(pure_data)
writer.write(date + "=" + '{:.2f}'.format(total_rainfall) + '\n') #updated
#print(date, "=", '{:.2f}'.format(total_rainfall)) #two decimal point format,
generate_daily_totals('data60.txt', 'totals60.txt')
checker = open('totals60.txt')
print(checker.read())
checker.close()
By reading a file, the original program runs well but I was required to convert it by writing the file. I am confused as the write method applies to string only so does that mean only the print section can be replaced by write method? This is the first time I am trying to use the write method. Thanks!
EDIT: the above codes have been updated based on the blhsing instruction which helped a lot! But still not running well as the for loop which gets skipped for some reason. Proper suggestions would be appreciated!
expected output:
2006-04-10 = 1399.46
2006-04-11 = 2822.36
2006-04-12 = 2803.81
2006-04-13 = 1622.71
2006-04-14 = 3119.60
2006-04-15 = 2256.14
2006-04-16 = 3120.05
2006-04-20 = 1488.00
You should open both the input file for reading, and the output file for writing, so change:
with open(input_filename, 'w') as writer:
for line in writer: # error not readable
to:
with open(input_filename, 'r') as reader, open(output_filename, 'w') as writer:
for line in reader:
Also, unlike the print function, the write method of a file object does not automatically add a trailing newline character to the output, so you would have to add it on your own.
Change:
writer.write(date + "=" + '{:.2f}'.format(total_rainfall))
to:
writer.write(date + "=" + '{:.2f}'.format(total_rainfall) + '\n')
or you can use print with the outputting file object specified as the file argument:
print(date, "=", '{:.2f}'.format(total_rainfall), file=writer)
I have this csv file:
89,Network activity,ip-dst,80.179.42.44,,1,20160929
89,Payload delivery,md5,4ad2924ced722ab65ff978f83a40448e,,1,20160929
89,Network activity,domain,alkamaihd.net,,1,20160929
90,Payload delivery,md5,197c018922237828683783654d3c632a,,1,20160929
90,Network activity,domain,dnsrecordsolver.tk,,1,20160929
90,Network activity,ip-dst,178.33.94.47,,1,20160929
90,Payload delivery,filename,Airline.xls,,1,20160929
91,Payload delivery,md5,23a9bbf8d64ae893db17777bedccdc05,,1,20160929
91,Payload delivery,md5,07e47f06c5ed05a062e674f8d11b01d8,,1,20160929
91,Payload delivery,md5,bd75af219f417413a4e0fae8cd89febd,,1,20160929
91,Payload delivery,md5,9f4023f2aefc8c4c261bfdd4bd911952,,1,20160929
91,Network activity,domain,mailsinfo.net,,1,20160929
91,Payload delivery,md5,1e4653631feebf507faeb9406664792f,,1,20160929
92,Payload delivery,md5,6fa869f17b703a1282b8f386d0d87bd4,,1,20160929
92,Payload delivery,md5,24befa319fd96dea587f82eb945f5d2a,,1,20160929
I need to divide this csv file to 4 csv files where as the condition is the event number at the beginning of every row. so far I created a set that includes al the event numbers {89,90,91,92}, and I know that I need to make loop in a loop and copy each row to its dedicated csv file.
data = {
'89': [],
'90': [],
'91': [],
'92': []
}
with open('yourfile.csv') as infile:
for line in infile:
prefix = line[:2]
data[prefix].append(line)
for prefix in data.keys():
with open('csv' + prefix + '.csv', 'w') as csv:
csv.writelines(''.join(data[prefix]))
However if your are open to solutions other than python then this can be easily accomplished by running four commands
grep ^89 file.csv > 89.csv
grep ^90 file.csv > 90.csv
Similarly for other values.
It would be best to not hardcode the event numbers in your code so it's not dependent on the values of the data. I also prefer to use the csv module which has been optimized to read and write .csv files.
Here's a way to do that:
import csv
prefix = 'events' # of output csv file names
data = {}
with open('conditions.csv', 'rb') as conditions:
reader = csv.reader(conditions)
for row in reader:
data.setdefault(row[0], []).append(row)
for event in sorted(data):
csv_filename = '{}_{}.csv'.format(prefix, event)
print(csv_filename)
with open(csv_filename, 'wb') as csvfile:
writer = csv.writer(csvfile)
writer.writerows(data[event])
Update
The approach implemented above first reads the entire csv file into memory, and then writes the all the rows associated with each event value into a separate output file, one at a time.
A more memory-efficient approach would be to open multiple output files simultaneously and write each row immediately after it has been read out to the proper destination file. Doing this requires keeping track of what files are already open. Something else the file managing code needs to do is make sure all the files are closed when processing is complete.
In the code below all of this has been accomplished by defining and using a Python Context Manager type to centralize the handling of all the csv output files that might be generated depending on how many different event values there are in the input file.
import csv
import sys
PY3 = sys.version_info.major > 2
class MultiCSVOutputFileManager(object):
"""Context manager to open and close multiple csv files and csv writers.
"""
def __enter__(self):
self.files = {}
return self
def __exit__(self, exc_type, exc_value, traceback):
for file, csv_writer in self.files.values():
print('closing file: {}'.format(file.name))
file.close()
self.files.clear()
return None
def get_csv_writer(self, filename):
if filename not in self.files: # new file?
open_kwargs = dict(mode='w', newline='') if PY3 else dict(mode='wb')
print('opening file: {}'.format(filename))
file = open(filename, **open_kwargs)
self.files[filename] = file, csv.writer(file)
return self.files[filename][1] # return associated csv.writer object
And here's how to use it:
prefix = 'events' # to name of each csv output file
with open('conditions.csv', 'rb') as conditions:
reader = csv.reader(conditions)
with MultiCSVOutputFileManager() as file_manager:
for row in reader:
csv_filename = '{}_{}.csv'.format(prefix, row[0]) # row[0] is event
writer = file_manager.get_csv_writer(csv_filename)
writer.writerow(row)
You can even dynamically create the resulting files if the first field has not been encountered by keeping a mapping of that id and the associated file:
files = {}
with open('file.csv') as fd:
for line in fd:
if 0 == len(line.strip()): continue # skip empty lines
try:
id_field = line.split(',', 1)[0] # extract first field
if not id in files.keys(): # if not encountered open a new result file
files[id] = open(id + '.csv')
files[id].write(line) # write the line in proper file
except Exception as e:
print('ERR', line, e) # catchall in case of problems...
Forewarning: I am very new to Python and programming in general. I am trying to use Python 3 to get some CSV data and making some changes to it before writing it to a file. My problem lies in accessing the CSV data from a variable, like so:
import csv
import requests
csvfile = session.get(url)
reader = csv.reader(csvfile.content)
for row in reader:
do(something)
This returns:
_csv.Error: iterator should return strings, not int (did you open the file in text mode?)
Googling revealed that I should be feeding the reader text instead of bytes, so I also attempted:
reader = csv.reader(csvfile.text)
This also does not work as the loop works through it letter by letter instead of line by line. I also experimented with TextIOWrapper and similar options with no success. The only way I have managed to get this to work is by writing the data to a file, reading it, and then making changes, like so:
csvfile = session.get(url)
with open("temp.txt", 'wb') as f:
f.write(csvfile.content)
with open("temp.txt", 'rU', encoding="utf8") as data:
reader = csv.reader(data)
for row in reader:
do(something)
I feel like this is far from the most optimal way of doing this, even if it works. What is the proper way to read and edit the CSV data directly from memory, without having to save it to a temporary file?
you don't have to write to a temp file, here is what I would do, using the "csv" and "requests" modules:
import csv
import requests
__csvfilepathname__ = r'c:\test\test.csv'
__url__ = 'https://server.domain.com/test.csv'
def csv_reader(filename, enc = 'utf_8'):
with open(filename, 'r', encoding = enc) as openfileobject:
reader = csv.reader(openfileobject)
for row in reader:
#do something
print(row)
return
def csv_from_url(url):
line = ''
datalist = []
s = requests.Session()
r = s.get(url)
for x in r.text.replace('\r',''):
if not x[0] == '\n':
line = line + str(x[0])
else:
datalist.append(line)
line = ''
datalist.append(line)
# at this point you already have a data list 'datalist'
# no need really to use the csv.reader object, but here goes:
reader = csv.reader(datalist)
for row in reader:
#do something
print(row)
return
def main():
csv_reader(__csvfilepathname__)
csv_from_url(__url__)
return
if __name__ == '__main__':
main ()
not very pretty, and probably not very good in regards to memory/performance, depending on how "big" your csv/data is
HTH, Edwin.
I am trying to append values to a json file. How can i append the data? I have been trying so many ways but none are working ?
Code:
def all(title,author,body,type):
title = "hello"
author = "njas"
body = "vgbhn"
data = {
"id" : id,
"author": author,
"body" : body,
"title" : title,
"type" : type
}
data_json = json.dumps(data)
#data = ast.literal_eval(data)
#print data_json
if(os.path.isfile("offline_post.json")):
with open('offline_post.json','a') as f:
new = json.loads(f)
new.update(a_dict)
json.dump(new,f)
else:
open('offline_post.json', 'a')
with open('offline_post.json','a') as f:
new = json.loads(f)
new.update(a_dict)
json.dump(new,f)
How can I append data to json file when this function is called?
I suspect you left out that you're getting a TypeError in the blocks where you're trying to write the file. Here's where you're trying to write:
with open('offline_post.json','a') as f:
new = json.loads(f)
new.update(a_dict)
json.dump(new,f)
There's a couple of problems here. First, you're passing a file object to the json.loads command, which expects a string. You probably meant to use json.load.
Second, you're opening the file in append mode, which places the pointer at the end of the file. When you run the json.load, you're not going to get anything because it's reading at the end of the file. You would need to seek to 0 before loading (edit: this would fail anyway, as append mode is not readable).
Third, when you json.dump the new data to the file, it's going to append it to the file in addition to the old data. From the structure, it appears you want to replace the contents of the file (as the new data contains the old data already).
You probably want to use r+ mode, seeking back to the start of the file between the read and write, and truncateing at the end just in case the size of the data structure ever shrinks.
with open('offline_post.json', 'r+') as f:
new = json.load(f)
new.update(a_dict)
f.seek(0)
json.dump(new, f)
f.truncate()
Alternatively, you can open the file twice:
with open('offline_post.json', 'r') as f:
new = json.load(f)
new.update(a_dict)
with open('offline_post.json', 'w') as f:
json.dump(new, f)
This is a different approach, I just wanted to append without reloading all the data. Running on a raspberry pi so want to look after memory. The test code -
import os
json_file_exists = 0
filename = "/home/pi/scratch_pad/test.json"
# remove the last run json data
try:
os.remove(filename)
except OSError:
pass
count = 0
boiler = 90
tower = 78
while count<10:
if json_file_exists==0:
# create the json file
with open(filename, mode = 'w') as fw:
json_string = "[\n\t{'boiler':"+str(boiler)+",'tower':"+str(tower)+"}\n]"
fw.write(json_string)
json_file_exists=1
else:
# append to the json file
char = ""
boiler = boiler + .01
tower = tower + .02
while(char<>"}"):
with open(filename, mode = 'rb+') as f:
f.seek(-1,2)
size=f.tell()
char = f.read()
if char == "}":
break
f.truncate(size-1)
with open(filename, mode = 'a') as fw:
json_string = "\n\t,{'boiler':"+str(boiler)+",'tower':"+str(tower)+"}\n]"
fw.seek(-1, os.SEEK_END)
fw.write(json_string)
count = count + 1