Develop Reference

csv.DictWriter writeheader() and writerow do not write properly Greek characters in CSV file when displayed in Microsoft Excel

Using python v3.7.8, I'm trying to read data from JSON file and write some data in CSV. JSON contains Greek and Latin Characters.
Data from JSON are read properly (I print them). However, when I'm writing data to CSV, Greek characters and not shown properly.
This is my code:
import json
import csv
# Opening JSON file to read data
f = open('test_2021-11-22-Andrias_lecture.json', 'r',encoding= 'utf-8')
# returns JSON object as a dictionary
data = json.load(f)
namesRowList=[]
namesColumnList=[]
connectionCountList=[]
connectionCountListTemp=[]
connectionsRow = {'':''}
# Iterating through the json list
for i in data['playerArray']:
namesRowList.append(i['score'])
namesColumnList.append(i['score'])
connectionsRowTemp = {i['score']:''}
connectionsRow.update(connectionsRowTemp)
print(connectionsRow)
# Open CSV file to store data
with open('matrix_10_Jan_2022.csv', 'w', newline='', encoding='utf-8') as file:
headerList = [''] + namesRowList.copy()
dw = csv.DictWriter(file, delimiter=';', fieldnames=headerList)
dw.writeheader()
for i in data['playerArray']:
name = i['score']
connections = i['connections']
connectionsRow['']=name
index = 0
for name in namesRowList:
for con in connections:
if (name == con):
connectionsRow[name] = 1
else:
connectionsRow[name] = 0
index = index + 1
dw.writerow(connectionsRow)
file.close()

Excel requires the byte order mark (BOM) signature or it will interpret the file in the local ANSI encoding. There is a codec for that, utf-8-sig.

Pandas cannot load data, csv encoding mystery

I am trying to load a dataset into pandas and cannot get seem to get past step 1. I am new so please forgive if this is obvious, I have searched previous topics and not found an answer. The data is mostly in Chinese characters, which may be the issue.
The .csv is very large, and can be found here: http://weiboscope.jmsc.hku.hk/datazip/
I am trying on week 1.
In my code below, I identify 3 types of decoding I attempted, including an attempt to see what encoding was used
import pandas
import chardet
import os
#this is what I tried to start
data = pandas.read_csv('week1.csv', encoding="utf-8")
#spits out error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9a in position 69: invalid start byte
#Code to check encoding -- this spits out ascii
bytes = min(32, os.path.getsize('week1.csv'))
raw = open('week1.csv', 'rb').read(bytes)
chardet.detect(raw)
#so i tried this! it also fails, which isn't that surprising since i don't know how you'd do chinese chars in ascii anyway
data = pandas.read_csv('week1.csv', encoding="ascii")
#spits out error: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe6 in position 0: ordinal not in range(128)
#for god knows what reason this allows me to load data into pandas, but definitely not correct encoding because when I print out first 5 lines its gibberish instead of Chinese chars
data = pandas.read_csv('week1.csv', encoding="latin1")
Any help would be greatly appreciated!
EDIT: The answer provided by #Kristof does in fact work, as does the program a colleague of mine put together yesterday:
import csv
import pandas as pd
def clean_weiboscope(file, nrows=0):
res = []
with open(file, 'r', encoding='utf-8', errors='ignore') as f:
reader = csv.reader(f)
for i, row in enumerate(f):
row = row.replace('\n', '')
if nrows > 0 and i > nrows:
break
if i == 0:
headers = row.split(',')
else:
res.append(tuple(row.split(',')))
df = pd.DataFrame(res)
return df
my_df = clean_weiboscope('week1.csv', nrows=0)
I also wanted to add for future searchers that this is the Weiboscope open data for 2012.

It seems that there's something very wrong with the input file. There are encoding errors throughout.
One thing you could do, is to read the CSV file as a binary, decode the binary string and replace the erroneous characters.
Example (source for the chunk-reading code):
in_filename = 'week1.csv'
out_filename = 'repaired.csv'
from functools import partial
chunksize = 100*1024*1024 # read 100MB at a time
# Decode with UTF-8 and replace errors with "?"
with open(in_filename, 'rb') as in_file:
with open(out_filename, 'w') as out_file:
for byte_fragment in iter(partial(in_file.read, chunksize), b''):
out_file.write(byte_fragment.decode(encoding='utf_8', errors='replace'))
# Now read the repaired file into a dataframe
import pandas as pd
df = pd.read_csv(out_filename)
df.shape
>> (4790108, 11)
df.head()

how to remove non utf 8 code and save as a csv file python

I have some amazon review data and I have converted from the text format to CSV format successfully, now the problem is when I trying to read it into a dataframe using pandas, i got error msg:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf8 in position 13: invalid start byte
I understand there must be some non utf-8 in the review raw data, how can I remove the non UTF-8 and save to another CSV file?
thank you!
EDIT1:
Here is the code i convert to text to csv:
import csv
import string
INPUT_FILE_NAME = "small-movies.txt"
OUTPUT_FILE_NAME = "small-movies1.csv"
header = [
"product/productId",
"review/userId",
"review/profileName",
"review/helpfulness",
"review/score",
"review/time",
"review/summary",
"review/text"]
f = open(INPUT_FILE_NAME,encoding="utf-8")
outfile = open(OUTPUT_FILE_NAME,"w")
outfile.write(",".join(header) + "\n")
currentLine = []
for line in f:
line = line.strip()
#need to reomve the , so that the comment review text won't be in many columns
line = line.replace(',','')
if line == "":
outfile.write(",".join(currentLine))
outfile.write("\n")
currentLine = []
continue
parts = line.split(":",1)
currentLine.append(parts[1])
if currentLine != []:
outfile.write(",".join(currentLine))
f.close()
outfile.close()
EDIT2:
Thanks to all of you trying to helping me out.
So I have solved it by modify the output format in my code:
outfile = open(OUTPUT_FILE_NAME,"w",encoding="utf-8")

If the input file in not utf-8 encoded, it it probably not a good idea to try to read it in utf-8...
You have basically 2 ways to deal with decode errors:
use a charset that will accept any byte such as iso-8859-15 also known as latin9
if output should be utf-8 but contains errors, use errors=ignore -> silently removes non utf-8 characters, or errors=replace -> replaces non utf-8 characters with a replacement marker (usually ?)
For example:
f = open(INPUT_FILE_NAME,encoding="latin9")
or
f = open(INPUT_FILE_NAME,encoding="utf-8", errors='replace')

If you are using python3, it provides inbuilt support for unicode content -
f = open('file.csv', encoding="utf-8")
If you still want to remove all unicode data from it, you can read it as a normal text file and remove the unicode content
def remove_unicode(string_data):
""" (str|unicode) -> (str|unicode)
recovers ascii content from string_data
"""
if string_data is None:
return string_data
if isinstance(string_data, bytes):
string_data = bytes(string_data.decode('ascii', 'ignore'))
else:
string_data = string_data.encode('ascii', 'ignore')
remove_ctrl_chars_regex = re.compile(r'[^\x20-\x7e]')
return remove_ctrl_chars_regex.sub('', string_data)
with open('file.csv', 'r+', encoding="utf-8") as csv_file:
content = remove_unicode(csv_file.read())
csv_file.seek(0)
csv_file.write(content)
Now you can read it without any unicode data issues.

Ascii Code error while converting from xlsx to csv

I have referred some post related to unicode error but didn't get any solution for my problem. I am converting xlsx to csv fom a workbook of 6 sheets.
Use the following code
def csv_from_excel(file_loc):
#file_acess check
print os.access(file_loc, os.R_OK)
wb = xlrd.open_workbook(file_loc)
print wb.nsheets
sheet_names = wb.sheet_names()
print sheet_names
counter = 0
while counter < wb.nsheets:
try:
sh = wb.sheet_by_name(sheet_names[counter])
file_name = str(sheet_names[counter]) + '.csv'
print file_name
fh = open(file_name, 'wb')
wr = csv.writer(fh, quoting=csv.QUOTE_ALL)
for rownum in xrange(sh.nrows):
wr.writerow(sh.row_values(rownum))
except Exception as e:
print str(e)
finally:
fh.close()
counter += 1
I get an error in 4th sheet
'ascii' codec can't encode character u'\u2018' in position 0: ordinal not in range(128)"
but position 0 is blank and it has converted to csv till 33rd row.
I am unable to figure out. CSV was easy way to read content and put in my data structure .

You'll need to manually encode Unicode values to bytes; for CSV usually UTF-8 is fine:
for rownum in xrange(sh.nrows):
wr.writerow([unicode(c).encode('utf8') for c in sh.row_values(rownum)])
Here I use unicode() for column data that is not text.
The character you encountered is the U+2018 LEFT SINGLE QUOTATION MARK, which is just a fancy form of the ' single quote. Office software (spreadsheets, word processors, etc.) often auto-replace single and double quotes with the 'fancy' versions. You could also just replace those with ASCII equivalents. You can do that with the Unidecode package:
from unidecode import unidecode
for rownum in xrange(sh.nrows):
wr.writerow([unidecode(unicode(c)) for c in sh.row_values(rownum)])
Use this when non-ASCII codepoints are only used for quotes and dashes and other punctuation.

Python error "Ordinal not in range" with accents

I'm scraping a table from the Internet and saving as a CSV file. There are characters with French accents in the text, resulting in a unicode error on save:
UnicodeEncodeError: 'ascii' codec can't encode characters in position 5-6: ordinal not in range(128)
I'd like to find an elegant solution for saving accented characters that I can apply to any situation. I've sometimes used the following:
encode('ascii','ignore')
but it doesn't work this time, for reasons unknown. I'm also trying to replace the <sup> tags in a cell, so I'm converting it using str() first.
Here's the pertinent part of my code:
data = [
str(td[0]).split('<sup')[0].split('>')[1].split('<')[0],
td[1].getText()
]
output.append(data)
csv_file = csv.writer(open('savedFile.csv', 'w'), delimiter=',')
for line in output:
csv_file.writerow(line)

If td[0] is u"a<sup>b</sup>c" :
td[0].split('<sup') is u"a".
td[0].partition('>')[2].split('<')[0] is u"b".
td[0][td[0].rindex('>') + 1:] is u"c".
If this kind of string indexing and matching is too simple you might consider createing a regular expression and matching it against the text in the html tag:
import re
r = re.compile("[^<]*<sup>([^<]*)</sup>")
m = r.match("some<sup>text</sup>")
print(m.groups()[0])

The csv.reader() and csv.writer() require the files opened in binary mode. You should also close the file at the end. Therefore, you should write it like:
f = open('output.csv', 'wb')
writer = csv.writer(f, delimiter=',')
for row in output:
writer.writerow(row)
f.close()
Or you can use the with construct when using newer versions of Python:
with open('output.csv', 'wb') as f:
writer = csv.writer(f, delimiter=',')
for row in output:
writer.writerow(row)
... and the file will be closed automatically.
Anyway, the csv.writer() expects the row composed of the byte sequences (not Unicode strings). If you have Unicode strings, convert them using .encode('utf-8'):
for row in output:
encoded_row = [s.encode('utf-8') for s in row]
writer.writerow(encoded_row)