Develop Reference

Encoding foreign alphabet characters

I am getting data from an XML provided by an API that for some reason lists Czechslovak characters in a different encoding (e.g. instead of correct Czechoslovak "ý" it uses "ý"). Therefore, instead of providing the
correct output to the user -> "Zelený"
the output is -> "Zelený"
I went through multiple StackOverflow posts, other fora and tutorials, but I still cannot figure out how to make it turn "Zelený" into "Zelený" (this is just one of the weird characters used by the XML so I cannot use str.replace).
I figured out, that the correct encoding for the Czech/Slovak language is "windows-1250"
My code:
def change_encoding(what):
what = what.encode("windows-1250")
return what
clean_xml_input = change_encoding(xml_input)
This produces error:
'charmap' codec can't encode characters in position 5-6: character
maps to <undefined>

"Zelený".encode("Windows-1252").decode("utf-8") #'Zelený'
"Zelený".encode("windows-1254").decode("utf-8") #'Zelený'
"Zelený".encode("iso-8859-1").decode("utf-8") #'Zelený'
"Zelený".encode("iso-8859-9").decode("utf-8") #'Zelený'
If it is helpful
from itertools import permutations
all_encoding = ['ASMO-708',
'big5',
'cp1025',
'cp866',
'cp875',
'csISO2022JP',
'DOS-720',
'DOS-862',
'EUC-CN',
'EUC-JP',
'euc-jp',
'euc-kr',
'GB18030',
'gb2312',
'hz-gb-2312',
'IBM00858',
'IBM00924',
'IBM01047',
'IBM01140',
'IBM01141',
'IBM01142',
'IBM01143',
'IBM01144',
'IBM01145',
'IBM01146',
'IBM01147',
'IBM01148',
'IBM01149',
'IBM037',
'IBM1026',
'IBM273',
'IBM277',
'IBM278',
'IBM280',
'IBM284',
'IBM285',
'IBM290',
'IBM297',
'IBM420',
'IBM423',
'IBM424',
'IBM437',
'IBM500',
'ibm737',
'ibm775',
'ibm850',
'ibm852',
'IBM855',
'ibm857',
'IBM860',
'ibm861',
'IBM863',
'IBM864',
'IBM865',
'ibm869',
'IBM870',
'IBM871',
'IBM880',
'IBM905',
'IBM-Thai',
'iso-2022-jp',
'iso-2022-jp',
'iso-2022-kr',
'iso-8859-1',
'iso-8859-13',
'iso-8859-15',
'iso-8859-2',
'iso-8859-3',
'iso-8859-4',
'iso-8859-5',
'iso-8859-6',
'iso-8859-7',
'iso-8859-8',
'iso-8859-8-i',
'iso-8859-9',
'Johab',
'koi8-r',
'koi8-u',
'ks_c_5601-1987',
'macintosh',
'shift_jis',
'us-ascii',
'utf-16',
'utf-16BE',
'utf-32',
'utf-32BE',
'utf-7',
'utf-8',
'windows-1250',
'windows-1251',
'Windows-1252',
'windows-1253',
'windows-1254',
'windows-1255',
'windows-1256',
'windows-1257',
'windows-1258',
'windows-874',
'x-Chinese-CNS',
'x-Chinese-Eten',
'x-cp20001',
'x-cp20003',
'x-cp20004',
'x-cp20005',
'x-cp20261',
'x-cp20269',
'x-cp20936',
'x-cp20949',
'x-cp50227',
'x-EBCDIC-KoreanExtended',
'x-Europa',
'x-IA5',
'x-IA5-German',
'x-IA5-Norwegian',
'x-IA5-Swedish',
'x-iscii-as',
'x-iscii-be',
'x-iscii-de',
'x-iscii-gu',
'x-iscii-ka',
'x-iscii-ma',
'x-iscii-or',
'x-iscii-pa',
'x-iscii-ta',
'x-iscii-te',
'x-mac-arabic',
'x-mac-ce',
'x-mac-chinesesimp',
'x-mac-chinesetrad',
'x-mac-croatian',
'x-mac-cyrillic',
'x-mac-greek',
'x-mac-hebrew',
'x-mac-icelandic',
'x-mac-japanese',
'x-mac-korean',
'x-mac-romanian',
'x-mac-thai',
'x-mac-turkish',
'x-mac-ukrainian']
for i,j in permutations(all_encoding, 2):
try:
if("Zelený".encode(i).decode(j) == 'Zelený'):
print(f'encode with `{i}` and decode with `{j}`')
except:
pass

Output not displaying full list of elements appended

from csv import reader
def func(sku_list):
values = []
with open(sku_list, 'r', encoding = 'utf-8') as pr:
rows = reader(pr)
for sku in rows:
values.append(sku[1])
return(values)
if __name__ == '__main__':
dir_path = "C:/Users/XXXX/Downloads/"
vendors = dir_path + 'file.csv'
new_prices = func(vendors)
print(new_prices)
sku_list is a csv file filled with pairs of brand names and their skus that I have downloaded from my db, for some reason as it iters through the rows and grabs just the sku value, hence sku[1], it stops well short of the actual length I expect the list to be
sku_list is 85,892 tuples long but when I print out the values appended to the list values it simply returns this:
['SKU', 'MWGB4896', 'MWGB4872', 'MWGB4848', 'MWGB3648', 'WGB4896', 'WGB4872', 'WGB4848', 'WGB3648', 'WGB2436', 'WGB1824', 'BKGB4896NT', 'BKGB4872NT', 'BKGB4848NT', 'BKGB3648NT', 'BKGB2436NT', 'BKGB1824NT', 'WFC2418G', 'WFC2418', 'WFC3624', 'WFC2418LB', 'WFC3648LB', 'WFC3624LB', 'WFC3624G', 'WFC3648G', 'WFC3648', 'LOWFC3624LB', 'LOWFC3624G', 'LOWFC3624', 'LOWFC2418LB', 'LOWFC2418G', 'LOWFC2418', 'LOWFC3648LB', 'LOWFC3648', 'LOWFC3648G', 'WM-7-B', 'WM-7-G', 'WM-7-BK', 'WMC-7', 'WM-7-R', 'APS-50', 'APS-70', 'APS-60', 'APS-84', 'SS15W', 'SC15W', 'SB15W', 'MFL-2W', 'WP-48', 'WP-40', 'WP-36', 'MP-48', 'MP-40', 'MP-36', 'OP-40', 'OP-36', 'OP-48', 'FFVSU96-2', 'FFVSU144-2', 'FFVSU192-2', '1-WA-1B', '1-WA-1BP', 'WCS-12', 'WCS-144', 'OPLD3416LSPP-2', 'OPLD3416LSPP-4', 'OPLD3416LSPP-5', 'OPLD3416LSPP-7', 'OPLD3416LSPP-8', 'OPLD1818LSPP-2', 'OPLD1818LSPP-4', 'OPLD1818LSPP-5', 'OPLD1818LSPP-7', 'OPLD1818LSPP-8', 'OPLD1818L-2', 'OPLD1818L-5', 'OPLD1818L-4', 'OPLD3416L-2', 'OPLD3416L-4', 'OPLD3416L-5', 'OPLD3416L-7', 'OPLD3416L-8', 'OPLD1818L-7', 'OPLD1818L-8', 'OPLD3416SPP-8-892', 'OPLD3416SPP-8-897', 'OPLD3416SPP-8-878', 'OPLD3416SPP-8-885', 'OPLD3416SPP-8-887', 'OPLD3416SPP-8-890', 'OPLD3416SPP-8-845', 'OPLD3416SPP-8-854', 'OPLD3416SPP-8-856', 'OPLD3416SPP-8-876', 'OPLD3416SPP-8-802', 'OPLD3416SPP-8-706', 'OPLD3416SPP-8-705', 'OPLD3416SPP-8-704', 'OPLD3416SPP-8-837', 'OPLD3416SPP-8-831', 'OPLD3416SPP-8-819', 'OPLD3416SPP-8-812', 'OPLD3416SPP-8-685', 'OPLD3416SPP-8-683', 'OPLD3416SPP-8-679', 'OPLD3416SPP-8-531', 'OPLD3416SPP-8-703', 'OPLD3416SPP-8-702', 'OPLD3416SPP-8-701', 'OPLD3416SPP-8-700', 'OPLD3416SPP-7-892', 'OPLD3416SPP-7-897', 'OPLD3416SPP-7-887', 'OPLD3416SPP-7-890', 'OPLD3416SPP-8-530', 'OPLD3416SPP-7-845', 'OPLD3416SPP-7-854', 'OPLD3416SPP-7-831', 'OPLD3416SPP-7-837', 'OPLD3416SPP-7-878', 'OPLD3416SPP-7-885', 'OPLD3416SPP-7-856', 'OPLD3416SPP-7-876', 'OPLD3416SPP-7-703', 'OPLD3416SPP-7-702', 'OPLD3416SPP-7-705', 'OPLD3416SPP-7-704', 'OPLD3416SPP-7-802', 'OPLD3416SPP-7-706', 'OPLD3416SPP-7-819', 'OPLD3416SPP-7-812', 'OPLD3416SPP-7-530', 'OPLD3416SPP-7-679', 'OPLD3416SPP-7-531', 'OPLD3416SPP-7-685', 'OPLD3416SPP-7-683', 'OPLD3416SPP-7-701', 'OPLD3416SPP-7-700', 'OPLD3416SPP-5-878', 'OPLD3416SPP-5-885', 'OPLD3416SPP-5-887', 'OPLD3416SPP-5-890', 'OPLD3416SPP-5-892', 'OPLD3416SPP-5-897', 'OPLD3416SPP-5-812', 'OPLD3416SPP-5-819', 'OPLD3416SPP-5-831', 'OPLD3416SPP-5-837', 'OPLD3416SPP-5-845', 'OPLD3416SPP-5-854', 'OPLD3416SPP-5-856', 'OPLD3416SPP-5-876', 'OPLD1818SPP-8-819', 'OPLD1818SPP-8-831', 'OPLD1818SPP-8-802', 'OPLD1818SPP-8-812', 'OPLD1818SPP-8-854', 'OPLD1818SPP-8-856', 'OPLD1818SPP-8-837', 'OPLD1818SPP-8-845', 'OPLD1818SPP-8-701', 'OPLD1818SPP-8-702', 'OPLD1818SPP-8-685', 'OPLD1818SPP-8-700', 'OPLD1818SPP-8-705', 'OPLD1818SPP-8-706', 'OPLD1818SPP-8-703', 'OPLD1818SPP-8-704', 'OPLD1818SPP-8-887', 'OPLD1818SPP-8-885', 'OPLD1818SPP-8-878', 'OPLD1818SPP-8-876', 'OPLD1818SPP-8-897', 'OPLD1818SPP-8-892', 'OPLD1818SPP-8-890', 'OPLD3416SPP-4-837', 'OPLD3416SPP-4-831', 'OPLD3416SPP-4-854', 'OPLD3416SPP-4-845', 'OPLD3416SPP-4-802', 'OPLD3416SPP-4-706',
'OPLD3416SPP-4-819', 'OPLD3416SPP-4-812', 'OPLD3416SPP-4-890', 'OPLD3416SPP-4-887', 'OPLD3416SPP-4-897', 'OPLD3416SPP-4-892', 'OPLD3416SPP-4-876', 'OPLD3416SPP-4-856', 'OPLD3416SPP-4-885', 'OPLD3416SPP-4-878', 'OPLD3416SPP-5-531', 'OPLD3416SPP-5-679', 'OPLD3416SPP-5-683', 'OPLD3416SPP-5-685', 'OPLD3416SPP-5-530', 'OPLD3416SPP-5-704', 'OPLD3416SPP-5-705', 'OPLD3416SPP-5-706', 'OPLD3416SPP-5-802', 'OPLD3416SPP-5-700', 'OPLD3416SPP-5-701', 'OPLD3416SPP-5-702', 'OPLD3416SPP-5-703', 'OPLD3416SPP-2-837', 'OPLD3416SPP-2-831', 'OPLD3416SPP-2-819', 'OPLD3416SPP-2-812', 'OPLD3416SPP-2-802', 'OPLD3416SPP-2-706', 'OPLD3416SPP-2-705', 'OPLD3416SPP-2-704', 'OPLD3416SPP-2-890', 'OPLD3416SPP-2-887', 'OPLD3416SPP-2-885', 'OPLD3416SPP-2-878', 'OPLD3416SPP-2-876', 'OPLD3416SPP-2-856', 'OPLD3416SPP-2-854', 'OPLD3416SPP-2-845', 'OPLD3416SPP-4-531', 'OPLD3416SPP-4-679', 'OPLD3416SPP-4-530', 'OPLD3416SPP-2-892', 'OPLD3416SPP-2-897', 'OPLD3416SPP-4-704', 'OPLD3416SPP-4-705', 'OPLD3416SPP-4-702', 'OPLD3416SPP-4-703', 'OPLD3416SPP-4-700', 'OPLD3416SPP-4-701', 'OPLD3416SPP-4-683', 'OPLD3416SPP-4-685', 'OPLD3416SPP-2-530', 'OPLD3416SPP-2-531', 'OPLD3416SPP-2-679', 'OPLD3416SPP-2-683', 'OPLD3416SPP-2-685', 'OPLD3416SPP-2-700', 'OPLD3416SPP-2-701', 'OPLD3416SPP-2-702', 'OPLD3416SPP-2-703', 'OPLD1818SPP-7-819', 'OPLD1818SPP-7-831', 'OPLD1818SPP-7-837', 'OPLD1818SPP-7-845', 'OPLD1818SPP-7-705', 'OPLD1818SPP-7-706', 'OPLD1818SPP-7-802', 'OPLD1818SPP-7-812', 'OPLD1818SPP-7-701', 'OPLD1818SPP-7-702', 'OPLD1818SPP-7-703', 'OPLD1818SPP-7-704', 'OPLD1818SPP-7-679', 'OPLD1818SPP-7-683', 'OPLD1818SPP-7-685', 'OPLD1818SPP-7-700', 'OPLD1818SPP-8-531', 'OPLD1818SPP-8-530', 'OPLD1818SPP-8-683', 'OPLD1818SPP-8-679', 'OPLD1818SPP-7-897', 'OPLD1818SPP-7-887', 'OPLD1818SPP-7-885', 'OPLD1818SPP-7-892', 'OPLD1818SPP-7-890', 'OPLD1818SPP-7-856', 'OPLD1818SPP-7-854', 'OPLD1818SPP-7-878', 'OPLD1818SPP-7-876', 'OPLD1818SPP-5-819', 'OPLD1818SPP-5-831', 'OPLD1818SPP-5-802', 'OPLD1818SPP-5-812', 'OPLD1818SPP-5-705', 'OPLD1818SPP-5-706', 'OPLD1818SPP-5-703', 'OPLD1818SPP-5-704', 'OPLD1818SPP-5-701', 'OPLD1818SPP-5-702', 'OPLD1818SPP-5-685', 'OPLD1818SPP-5-700', 'OPLD1818SPP-5-679', 'OPLD1818SPP-5-683', 'OPLD1818SPP-5-530', 'OPLD1818SPP-5-531', 'OPLD1818SPP-7-531', 'OPLD1818SPP-7-530', 'OPLD1818SPP-5-897', 'OPLD1818SPP-5-892', 'OPLD1818SPP-5-890', 'OPLD1818SPP-5-887', 'OPLD1818SPP-5-885', 'OPLD1818SPP-5-878', 'OPLD1818SPP-5-876', 'OPLD1818SPP-5-856', 'OPLD1818SPP-5-854', 'OPLD1818SPP-5-845', 'OPLD1818SPP-5-837', 'OPLD1818SPP-4-701', 'OPLD1818SPP-4-702', 'OPLD1818SPP-4-703', 'OPLD1818SPP-4-704', 'OPLD1818SPP-4-705', 'OPLD1818SPP-4-706', 'OPLD1818SPP-4-802', 'OPLD1818SPP-4-812', 'OPLD1818SPP-4-530', 'OPLD1818SPP-4-531', 'OPLD1818SPP-4-679', 'OPLD1818SPP-4-683', 'OPLD1818SPP-4-685', 'OPLD1818SPP-4-700', 'OPLD1818SPP-4-887', 'OPLD1818SPP-2-837', 'OPLD1818SPP-2-845', 'OPLD1818SPP-2-854', 'OPLD1818SPP-2-856', 'OPLD1818SPP-2-802', 'OPLD1818SPP-2-812', 'OPLD1818SPP-2-819', 'OPLD1818SPP-2-831', 'OPLD1818SPP-2-890', 'OPLD1818SPP-2-892', 'OPLD1818SPP-2-897', 'OPLD1818SPP-2-876', 'OPLD1818SPP-2-878', 'OPLD1818SPP-2-885', 'OPLD1818SPP-2-887', 'OPLD1818SPP-2-531', 'OPLD1818SPP-2-530', 'OPLD1818SPP-2-683', 'OPLD1818SPP-2-679', 'OPLD1818SPP-2-704', 'OPLD1818SPP-2-703',
'OPLD1818SPP-2-706', 'OPLD1818SPP-2-705', 'OPLD1818SPP-2-700', 'OPLD1818SPP-2-685', 'OPLD1818SPP-2-702', 'OPLD1818SPP-2-701', 'OPLD1818SPP-4-876', 'OPLD1818SPP-4-878', 'OPLD1818SPP-4-854', 'OPLD1818SPP-4-856', 'OPLD1818SPP-4-837', 'OPLD1818SPP-4-845', 'OPLD1818SPP-4-819', 'OPLD1818SPP-4-831', 'OPLD1818SPP-4-897', 'OPLD1818SPP-4-890', 'OPLD1818SPP-4-892', 'OPLD1818SPP-4-885', 'PLD4832DPP-2-845', 'PLD4832DPP-2-837', 'PLD4832DPP-2-856', 'PLD4832DPP-2-854', 'PLD4832DPP-2-878', 'PLD4832DPP-2-876', 'PLD4832DPP-2-887', 'PLD4832DPP-2-885', 'PLD4832DPP-2-892', 'PLD4832DPP-2-890', 'PLD4832DPP-2-897', 'PLD4832DPP-4-531', 'PLD4832DPP-4-530', 'PLD4832DPP-4-679', 'PLD4832DPP-4-683', 'PLD4832DPP-4-685', 'PLD4832DPP-4-700', 'PLD4832DPP-4-701', 'PLD4832DPP-4-702', 'PLD4832DPP-4-703', 'PLD4832DPP-4-704', 'PLD4832DPP-4-705', 'PLD4832DPP-4-706', 'PLD4832DPP-4-802', 'PLD4832DPP-4-812', 'PLD4832DPP-4-819', 'PLD4832DPP-4-831', 'PLD4832DPP-4-837', 'PLD4832DPP-4-845', 'PLD4832DPP-4-878', 'PLD4832DPP-4-876', 'PLD4832DPP-4-856', 'PLD4832DPP-4-854', 'PLD4832DPP-4-892', 'PLD4832DPP-4-890', 'PLD4832DPP-4-887', 'PLD4832DPP-4-885', 'PLD4832DPP-4-897', 'PLD4832DPP-5-683', 'PLD4832DPP-5-679', 'PLD4832DPP-5-531', 'PLD4832DPP-5-530', 'PLD4832DPP-5-701', 'PLD4832DPP-5-702', 'PLD4832DPP-5-685', 'PLD4832DPP-5-700', 'PLD4832DPP-5-705', 'PLD4832DPP-5-706', 'PLD4832DPP-5-703', 'PLD4832DPP-5-704', 'PLD4832DPP-5-819', 'PLD4832DPP-5-831', 'PLD4832DPP-5-802', 'PLD4832DPP-5-812', 'PLD4832DPP-5-854', 'PLD4832DPP-5-856', 'PLD4832DPP-5-837', 'PLD4832DPP-5-845', 'PLD4832DPP-2-701', 'PLD4832DPP-2-702', 'PLD4832DPP-2-685', 'PLD4832DPP-2-700', 'PLD4832DPP-2-679', 'PLD4832DPP-2-683', 'PLD4832DPP-2-530', 'PLD4832DPP-2-531', 'PLD4832DPP-2-819', 'PLD4832DPP-2-831', 'PLD4832DPP-2-802', 'PLD4832DPP-2-812', 'PLD4832DPP-2-705', 'PLD4832DPP-2-706', 'PLD4832DPP-2-703', 'PLD4832DPP-2-704', 'PLD4226DPP-8-887', 'PLD4226DPP-8-890', 'PLD4226DPP-8-892', 'PLD4226DPP-8-897', 'PLD4226DPP-8-856', 'PLD4226DPP-8-876', 'PLD4226DPP-8-878', 'PLD4226DPP-8-885', 'PLD4226DPP-8-831', 'PLD4226DPP-8-837', 'PLD4226DPP-8-845', 'PLD4226DPP-8-854', 'PLD4226DPP-8-706', 'PLD4226DPP-8-802', 'PLD4226DPP-8-812', 'PLD4226DPP-8-819', 'PLD4226DPP-5-892', 'PLD4226DPP-5-897', 'PLD4226DPP-5-887', 'PLD4226DPP-5-890', 'PLD4226DPP-7-530', 'PLD4226DPP-7-683', 'PLD4226DPP-7-685', 'PLD4226DPP-7-531', 'PLD4226DPP-7-679', 'PLD4226DPP-7-702', 'PLD4226DPP-7-703', 'PLD4226DPP-7-700', 'PLD4226DPP-7-701', 'PLD4226DPP-5-705', 'PLD4226DPP-5-704', 'PLD4226DPP-5-703', 'PLD4226DPP-5-702', 'PLD4226DPP-5-819', 'PLD4226DPP-5-812', 'PLD4226DPP-5-802', 'PLD4226DPP-5-706', 'PLD4226DPP-5-854', 'PLD4226DPP-5-845', 'PLD4226DPP-5-837', 'PLD4226DPP-5-831', 'PLD4226DPP-5-885', 'PLD4226DPP-5-878', 'PLD4226DPP-5-876', 'PLD4226DPP-5-856', 'PLD4226DPP-7-892', 'PLD4226DPP-7-897', 'PLD4226DPP-8-530', 'PLD4226DPP-8-531', 'PLD4226DPP-8-679', 'PLD4226DPP-8-683', 'PLD4226DPP-8-685', 'PLD4226DPP-8-700', 'PLD4226DPP-8-701', 'PLD4226DPP-8-702', 'PLD4226DPP-8-703', 'PLD4226DPP-8-704', 'PLD4226DPP-8-705', 'PLD4226DPP-7-705', 'PLD4226DPP-7-704', 'PLD4226DPP-7-802', 'PLD4226DPP-7-706', 'PLD4226DPP-7-819', 'PLD4226DPP-7-812', 'PLD4226DPP-7-837', 'PLD4226DPP-7-831', 'PLD4226DPP-7-854', 'PLD4226DPP-7-845', 'PLD4226DPP-7-876', 'PLD4226DPP-7-856', 'PLD4226DPP-7-885', 'PLD4226DPP-7-878', 'PLD4226DPP-7-890', 'PLD4226DPP-7-887', 'PLD4226DPP-2-892', 'PLD4226DPP-2-897', 'PLD4226DPP-2-887', 'PLD4226DPP-2-890', 'PLD4226DPP-2-878', 'PLD4226DPP-2-885', 'PLD4226DPP-2-856', 'PLD4226DPP-2-876', 'PLD4226DPP-4-683', 'PLD4226DPP-4-685', 'PLD4226DPP-4-531', 'PLD4226DPP-4-679', 'PLD4226DPP-4-530', 'PLD4226DPP-2-705', 'PLD4226DPP-2-704', 'PLD4226DPP-2-703', 'PLD4226DPP-2-702', 'PLD4226DPP-2-701', 'PLD4226DPP-2-700', 'PLD4226DPP-2-685', 'PLD4226DPP-2-683', 'PLD4226DPP-2-854', 'PLD4226DPP-2-845', 'PLD4226DPP-2-837', 'PLD4226DPP-2-831', 'PLD4226DPP-2-819', 'PLD4226DPP-2-812', 'PLD4226DPP-2-802', 'PLD4226DPP-2-706', 'PLD4226DPP-4-892', 'PLD4226DPP-4-897', 'PLD4226DPP-4-878', 'PLD4226DPP-4-885', 'PLD4226DPP-4-887', 'PLD4226DPP-4-890', 'PLD4226DPP-5-683', 'PLD4226DPP-5-685', 'PLD4226DPP-5-700', 'PLD4226DPP-5-701', 'PLD4226DPP-5-530', 'PLD4226DPP-5-531', 'PLD4226DPP-5-679', 'PLD4226DPP-4-705', 'PLD4226DPP-4-704', 'PLD4226DPP-4-802', 'PLD4226DPP-4-706', 'PLD4226DPP-4-701', 'PLD4226DPP-4-700', 'PLD4226DPP-4-703', 'PLD4226DPP-4-702', 'PLD4226DPP-4-854', 'PLD4226DPP-4-845', 'PLD4226DPP-4-876', 'PLD4226DPP-4-856', 'PLD4226DPP-4-819', 'PLD4226DPP-4-812', 'PLD4226DPP-4-837', 'PLD4226DPP-4-831', 'PLD4226DPP-2-530', 'PLD4226DPP-2-679', 'PLD4226DPP-2-531', 'PLD5438DPP-5-683', 'PLD5438DPP-5-685', 'PLD5438DPP-5-531', 'PLD5438DPP-5-679', 'PLD5438DPP-5-702', 'PLD5438DPP-5-703', 'PLD5438DPP-5-700', 'PLD5438DPP-5-701', 'PLD5438DPP-5-706', 'PLD5438DPP-5-802', 'PLD5438DPP-5-704', 'PLD5438DPP-5-705', 'PLD5438DPP-5-831', 'PLD5438DPP-5-837', 'PLD5438DPP-5-812', 'PLD5438DPP-5-819', 'PLD5438DPP-5-876', 'PLD5438DPP-5-856', 'PLD5438DPP-5-854', 'PLD5438DPP-5-845', 'PLD5438DPP-5-890', 'PLD5438DPP-5-887', 'PLD5438DPP-5-885', 'PLD5438DPP-5-878', 'PLD5438DPP-5-897', 'PLD5438DPP-5-892', 'PLD5438DPP-7-679', 'PLD5438DPP-7-531', 'PLD5438DPP-7-530', 'PLD5438DPP-4-530', 'PLD5438DPP-4-531', 'PLD5438DPP-4-679', 'PLD5438DPP-4-683', 'PLD5438DPP-4-685', 'PLD5438DPP-4-700', 'PLD5438DPP-4-701', 'PLD5438DPP-4-702', 'PLD5438DPP-4-703', 'PLD5438DPP-4-704', 'PLD5438DPP-4-705', 'PLD5438DPP-4-706', 'PLD5438DPP-4-802', 'PLD5438DPP-4-812', 'PLD5438DPP-4-819', 'PLD5438DPP-4-837', 'PLD5438DPP-4-831', 'PLD5438DPP-4-854', 'PLD5438DPP-4-845', 'PLD5438DPP-4-876', 'PLD5438DPP-4-856', 'PLD5438DPP-4-885', 'PLD5438DPP-4-878', 'PLD5438DPP-4-890', 'PLD5438DPP-4-887', 'PLD5438DPP-4-897', 'PLD5438DPP-4-892', 'PLD5438DPP-5-530',
'PLD5438DPP
None
the final sku in there PLD5438DPP should be PLD5438DPP-5-683 and for some reason the list cuts off there, which is only element 564/85,892, and the program terminates without an error code
I cannot attach the file of skus, this is for my job, just hoping someone can shed light on what I am doing to cause the list to cut short like that
this may or may not also be relevant but when I call .append(sku) as opposed to sku[1] and grab the whole tuple the same issue occurs but at element 292, exactly half of the amount of element appended when only doing half the tuple

The issue seems to be one on my local machine, where it was unable to print such a long list, this one was of size 85,892
for those with similar issues see if any of our specs are overlapping and that may determine the cause of this issue:
VSCode:
Version: 1.52.0 (user setup)
Commit: 940b5f4bb5fa47866a54529ed759d95d09ee80be
Date: 2020-12-10T22:45:11.850Z
Electron: 9.3.5
Chrome: 83.0.4103.122
Node.js: 12.14.1
V8: 8.3.110.13-electron.0
OS: Windows_NT x64 10.0.18363
Python:
3.9.0
see discussion in comments for other details

invert regex pattern in python

I'm trying to filter from string only the Arabic character but the next function doesn't work for me:
import re
def remove_any_non_arabic_char(text):
non_arabic_char = re.compile('^[\u0627-\u064a]')
text = re.sub(non_arabic_char, "", text)
print(text)
for example:
s = "Kühn xvii, 346] قال جالينوس: [1] قد اتفق جل من فسر هذا الكتا"
The desired output of remove_any_non_arabic_char(s) should be قال جالينوس قد اتفق جل من فسر هذا الكتا but the input stays without changes.
What should I do?

First, you need to fix your regex as suggested in the comments, then for a more efficient solution, you will need to expand your Unicode character selection to include all Arabic character mappings. Finally, you need to keep at least one space between Arabic words to keep the Arabic text legible:
import re
def remove_any_non_arabic_char(text):
non_arabic_char = re.compile('[^\s\\u0600-\u06FF]')
text_with_no_spaces = re.sub(non_arabic_char, "", text)
text_with_single_spaces = " ".join(re.split("\s+", text_with_no_spaces))
return text_with_single_spaces
text_1 = "Kühn xvii, 346] قال جالينوس: [1] قد اتفق جل من فسر هذا الكتا"
text_2 = '''
تغيّر مفهوم كلمة (أدب) من العصر الجاهلي jahili (pre-Islamic) era إلى الآن عبر
مراحل periods التاريخ المتعددة. ففي الجاهلية، كانت كلمة أدب تعني (الدعوة إلى
الطعام). وبعدها، استخدم الرسول محمد (عليه السلام) الكلمة بمعنى "التهذيب والتربية"
education and mannerism. وفي العصر الأموي، اتصلت had to do كلمة أدب
بالتاريخ والفقه والقرآن والحديث. أما في العصرالعباسي، فأصبحت تعني تعلّم الشعر
والنثر prose واتسع الأدب ليشمل أنواع المعرفة وألوانها وخصوصاً علم البلاغة واللغة.
أما في الوقت الحالي، فأصبحت كلمة أدب ذات صلة pertinent بالكلام البليغ
الجميل المؤثر that impacts في أحاسيس القاريء أو السامع.
'''
# Isleem, N. M., & Abuhakema, G. M. (2020). Kalima wa Nagham: A Textbook for
# Teaching Arabic, Volume 2 (Vol. 3). University of Texas Press. (page 5)
print('text_1: \n', remove_any_non_arabic_char(text_1))
print('\ntext_2: \n\n', remove_any_non_arabic_char(text_2))
Running the code on the two texts above in Jupyter, you get:
Notice that punctuation marks shared between Arabic and English (like periods and brackets) have also been removed. To keep those, you would need to introduce more complex conditionals.

Tokenize tweet based on Regex

I have the following example text / tweet:
RT #trader $AAPL 2012 is o´o´o´o´o´pen to ‘Talk’ about patents with GOOG definitely not the treatment #samsung got:-) heh url_that_cannot_be_posted_on_SO
I want to follow the procedure of Table 1 in Li, T, van Dalen, J, & van Rees, P.J. (Pieter Jan). (2017). More than just noise? Examining the information content of stock microblogs on financial markets. Journal of Information Technology. doi:10.1057/s41265-016-0034-2 in order to clean up the tweet.
They clean the tweet up in such a way that the final result is:
{RT|123456} {USER|56789} {TICKER|AAPL} {NUMBER|2012} notooopen nottalk patent {COMPANY|GOOG} notdefinetli treatment {HASH|samsung} {EMOTICON|POS} haha {URL}
I use the following script to tokenize the tweet based on the regex:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import re
emoticon_string = r"""
(?:
[<>]?
[:;=8] # eyes
[\-o\*\']? # optional nose
[\)\]\(\[dDpP/\:\}\{#\|\\] # mouth
|
[\)\]\(\[dDpP/\:\}\{#\|\\] # mouth
[\-o\*\']? # optional nose
[:;=8] # eyes
[<>]?
)"""
regex_strings = (
# URL:
r"""http[s]?://(?:[a-zA-Z]|[0-9]|[$-_#.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+"""
,
# Twitter username:
r"""(?:#[\w_]+)"""
,
# Hashtags:
r"""(?:\#+[\w_]+[\w\'_\-]*[\w_]+)"""
,
# Cashtags:
r"""(?:\$+[\w_]+[\w\'_\-]*[\w_]+)"""
,
# Remaining word types:
r"""
(?:[+\-]?\d+[,/.:-]\d+[+\-]?) # Numbers, including fractions, decimals.
|
(?:[\w_]+) # Words without apostrophes or dashes.
|
(?:\.(?:\s*\.){1,}) # Ellipsis dots.
|
(?:\S) # Everything else that isn't whitespace.
"""
)
word_re = re.compile(r"""(%s)""" % "|".join(regex_strings), re.VERBOSE | re.I | re.UNICODE)
emoticon_re = re.compile(regex_strings[1], re.VERBOSE | re.I | re.UNICODE)
######################################################################
class Tokenizer:
def __init__(self, preserve_case=False):
self.preserve_case = preserve_case
def tokenize(self, s):
try:
s = str(s)
except UnicodeDecodeError:
s = str(s).encode('string_escape')
s = unicode(s)
# Tokenize:
words = word_re.findall(s)
if not self.preserve_case:
words = map((lambda x: x if emoticon_re.search(x) else x.lower()), words)
return words
if __name__ == '__main__':
tok = Tokenizer(preserve_case=False)
test = ' RT #trader $AAPL 2012 is oooopen to ‘Talk’ about patents with GOOG definitely not the treatment #samsung got:-) heh url_that_cannot_be_posted_on_SO'
tokenized = tok.tokenize(test)
print("\n".join(tokenized))
This yields the following output:
rt
#trader
$aapl
2012
is
oooopen
to
‘
talk
’
about
patents
with
goog
definitely
not
the
treatment
#samsung
got
:-)
heh
url_that_cannot_be_posted_on_SO
How can I adjust this script to get:
rt
{USER|trader}
{CASHTAG|aapl}
{NUMBER|2012}
is
oooopen
to
‘
talk
’
about
patents
with
goog
definitely
not
the
treatment
{HASHTAG|samsung}
got
{EMOTICON|:-)}
heh
{URL|url_that_cannot_be_posted_on_SO}
Thanks in advance for helping me out big time!

You really need to use named capturing groups (mentioned by thebjorn), and use groupdict() to get name-value pairs upon each match. It requires some post-processing though:
All pairs where the value is None must be discarded
If the self.preserve_case is false the value can be turned to lower case at once
If the group name is WORD, ELLIPSIS or ELSE the values are added to words as is
If the group name is HASHTAG, CASHTAG, USER or URL the values are added first stripped of $, # and # chars at the start and then added to words as {<GROUP_NAME>|<VALUE>} item
All other matches are added to words as {<GROUP_NAME>|<VALUE>} item.
Note that \w matches underscores by default, so [\w_] = \w. I optimized the patterns a little bit.
Here is a fixed code snippet:
import re
emoticon_string = r"""
(?P<EMOTICON>
[<>]?
[:;=8] # eyes
[-o*']? # optional nose
[][()dDpP/:{}#|\\] # mouth
|
[][()dDpP/:}{#|\\] # mouth
[-o*']? # optional nose
[:;=8] # eyes
[<>]?
)"""
regex_strings = (
# URL:
r"""(?P<URL>https?://(?:[-a-zA-Z0-9_$#.&+!*(),]|%[0-9a-fA-F][0-9a-fA-F])+)"""
,
# Twitter username:
r"""(?P<USER>#\w+)"""
,
# Hashtags:
r"""(?P<HASHTAG>\#+\w+[\w'-]*\w+)"""
,
# Cashtags:
r"""(?P<CASHTAG>\$+\w+[\w'-]*\w+)"""
,
# Remaining word types:
r"""
(?P<NUMBER>[+-]?\d+(?:[,/.:-]\d+[+-]?)?) # Numbers, including fractions, decimals.
|
(?P<WORD>\w+) # Words without apostrophes or dashes.
|
(?P<ELLIPSIS>\.(?:\s*\.)+) # Ellipsis dots.
|
(?P<ELSE>\S) # Everything else that isn't whitespace.
"""
)
word_re = re.compile(r"""({}|{})""".format(emoticon_string, "|".join(regex_strings)), re.VERBOSE | re.I | re.UNICODE)
#print(word_re.pattern)
emoticon_re = re.compile(regex_strings[1], re.VERBOSE | re.I | re.UNICODE)
######################################################################
class Tokenizer:
def __init__(self, preserve_case=False):
self.preserve_case = preserve_case
def tokenize(self, s):
try:
s = str(s)
except UnicodeDecodeError:
s = str(s).encode('string_escape')
s = unicode(s)
# Tokenize:
words = []
for x in word_re.finditer(s):
for key, val in x.groupdict().items():
if val:
if not self.preserve_case:
val = val.lower()
if key in ['WORD','ELLIPSIS','ELSE']:
words.append(val)
elif key in ['HASHTAG','CASHTAG','USER','URL']: # Add more here if needed
words.append("{{{}|{}}}".format(key, re.sub(r'^[##$]+', '', val)))
else:
words.append("{{{}|{}}}".format(key, val))
return words
if __name__ == '__main__':
tok = Tokenizer(preserve_case=False)
test = ' RT #trader $AAPL 2012 is oooopen to ‘Talk’ about patents with GOOG definitely not the treatment #samsung got:-) heh http://some.site.here.com'
tokenized = tok.tokenize(test)
print("\n".join(tokenized))
With test = ' RT #trader $AAPL 2012 is oooopen to ‘Talk’ about patents with GOOG definitely not the treatment #samsung got:-) heh http://some.site.here.com', it outputs
rt
{USER|trader}
{CASHTAG|aapl}
{NUMBER|2012}
is
oooopen
to
‘
talk
’
about
patents
with
goog
definitely
not
the
treatment
{HASHTAG|samsung}
got
{EMOTICON|:-)}
heh
{URL|http://some.site.here.com}
See the regex demo online.

How to train a sense2vec model

The documentation of sense2vec mentions 3 primary files - the first of them being merge_text.py. I have tried several types of inputs- txt,csv,bzipped file since merge_text.py tries to open files compressed by bzip2.
The file can be found at:
https://github.com/spacy-io/sense2vec/blob/master/bin/merge_text.py
What type of input format does this script require?
Further, if anyone could please suggest how to train the model.

I extended and adjusted the code samples from sense2vec.
You go from this input text:
"As far as Saudi Arabia and its motives, that is very simple also. The Saudis are
good at money and arithmetic. Faced with the painful choice of losing money
maintaining current production at US$60 per barrel or taking two million barrels
per day off the market and losing much more money - it's an easy choice: take
the path that is less painful. If there are secondary reasons like hurting US
tight oil producers or hurting Iran and Russia, that's great, but it's really
just about the money."
To this:
as|ADV far|ADV as|ADP saudi_arabia|ENT and|CCONJ its|ADJ motif|NOUN that|ADJ is|VERB very|ADV simple|ADJ also|ADV saudis|ENT are|VERB good|ADJ at|ADP money|NOUN and|CCONJ arithmetic|NOUN faced|VERB with|ADP painful_choice|NOUN of|ADP losing|VERB money|NOUN maintaining|VERB current_production|NOUN at|ADP us$|SYM 60|MONEY per|ADP barrel|NOUN or|CCONJ taking|VERB two_million|CARDINAL barrel|NOUN per|ADP day|NOUN off|ADP market|NOUN and|CCONJ losing|VERB much_more_money|NOUN it|PRON 's|VERB easy_choice|NOUN take|VERB path|NOUN that|ADJ is|VERB less|ADV painful|ADJ if|ADP there|ADV are|VERB secondary_reason|NOUN like|ADP hurting|VERB us|ENT tight_oil_producer|NOUN or|CCONJ hurting|VERB iran|ENT and|CCONJ russia|ENT 's|VERB great|ADJ but|CCONJ it|PRON 's|VERB really|ADV just|ADV about|ADP money|NOUN
Double line breaks are interpreted as separate documents.
Urls are recognized as such, stripped down to domain.tld and marked as |URL
Nouns (also noun being part of noun phrases) are lemmatized (as motives become motifs)
Words with POS-tags like DET (determinate article) and PUNCT (for punctuation) are dropped
Here's the code. Let me know if you have questions.
I'll probably publish it on github.com/woltob soon.
import spacy
import re
nlp = spacy.load('en')
nlp.matcher = None
LABELS = {
'ENT': 'ENT',
'PERSON': 'PERSON',
'NORP': 'ENT',
'FAC': 'ENT',
'ORG': 'ENT',
'GPE': 'ENT',
'LOC': 'ENT',
'LAW': 'ENT',
'PRODUCT': 'ENT',
'EVENT': 'ENT',
'WORK_OF_ART': 'ENT',
'LANGUAGE': 'ENT',
'DATE': 'DATE',
'TIME': 'TIME',
'PERCENT': 'PERCENT',
'MONEY': 'MONEY',
'QUANTITY': 'QUANTITY',
'ORDINAL': 'ORDINAL',
'CARDINAL': 'CARDINAL'
}
pre_format_re = re.compile(r'^[\`\*\~]')
post_format_re = re.compile(r'[\`\*\~]$')
url_re = re.compile(r'(https?:\/\/)?([a-z0-9-]+\.)?([\d\w]+?\.[^\/]{2,63})')
single_linebreak_re = re.compile('\n')
double_linebreak_re = re.compile('\n{2,}')
whitespace_re = re.compile(r'[ \t]+')
quote_re = re.compile(r'"|`|´')
def strip_meta(text):
text = text.replace('per cent', 'percent')
text = text.replace('>', '>').replace('<', '<')
text = pre_format_re.sub('', text)
text = post_format_re.sub('', text)
text = double_linebreak_re.sub('{2break}', text)
text = single_linebreak_re.sub(' ', text)
text = text.replace('{2break}', '\n')
text = whitespace_re.sub(' ', text)
text = quote_re.sub('', text)
return text
def transform_doc(doc):
for ent in doc.ents:
ent.merge(ent.root.tag_, ent.text, LABELS[ent.label_])
for np in doc.noun_chunks:
while len(np) > 1 and np[0].dep_ not in ('advmod', 'amod', 'compound'):
np = np[1:]
np.merge(np.root.tag_, np.text, np.root.ent_type_)
strings = []
for sent in doc.sents:
sentence = []
if sent.text.strip():
for w in sent:
if w.is_space:
continue
w_ = represent_word(w)
if w_:
sentence.append(w_)
strings.append(' '.join(sentence))
if strings:
return '\n'.join(strings) + '\n'
else:
return ''
def represent_word(word):
if word.like_url:
x = url_re.search(word.text.strip().lower())
if x:
return x.group(3)+'|URL'
else:
return word.text.lower().strip()+'|URL?'
text = re.sub(r'\s', '_', word.text.strip().lower())
tag = LABELS.get(word.ent_type_)
# Dropping PUNCTUATION such as commas and DET like the
if tag is None and word.pos_ not in ['PUNCT', 'DET']:
tag = word.pos_
elif tag is None:
return None
# if not word.pos_:
# tag = '?'
return text + '|' + tag
corpus = '''
As far as Saudi Arabia and its motives, that is very simple also. The Saudis are
good at money and arithmetic. Faced with the painful choice of losing money
maintaining current production at US$60 per barrel or taking two million barrels
per day off the market and losing much more money - it's an easy choice: take
the path that is less painful. If there are secondary reasons like hurting US
tight oil producers or hurting Iran and Russia, that's great, but it's really
just about the money.
'''
corpus_stripped = strip_meta(corpus)
doc = nlp(corpus_stripped)
corpus_ = []
for word in doc:
# only lemmatize NOUN and PROPN
if word.pos_ in ['NOUN', 'PROPN'] and len(word.text) > 3 and len(word.text) != len(word.lemma_):
# Keep the original word with the length of the lemma, then add the white space, if it was there.:
lemma_ = str(word.text[:1]+word.lemma_[1:]+word.text_with_ws[len(word.text):])
# print(word.text, lemma_)
corpus_.append(lemma_)
# print(word.text, word.text[:len(word.lemma_)]+word.text_with_ws[len(word.text):])
# All other words are added normally.
else:
corpus_.append(word.text_with_ws)
result = transform_doc(nlp(''.join(corpus_)))
sense2vec_filename = 'text.txt'
file = open(sense2vec_filename,'w')
file.write(result)
file.close()
print(result)
You could visualise your model using Gensim in Tensorboard using this approach:
https://github.com/ArdalanM/gensim2tensorboard
I'll also adjust this code to work with the sense2vec approach (e.g. the words become lowercase in the preprocessing step, just comment it out in the code).
Happy coding,
woltob

The input file should be a bzipped json. To use a plain text file just edit the merge_text.py as follow:
def iter_comments(loc):
with bz2.BZ2File(loc) as file_:
for i, line in enumerate(file_):
yield line.decode('utf-8', errors='ignore')
# yield ujson.loads(line)['body']