Develop Reference

Generate ID Batch for each xml´s files import

hope I can make it clear to what I´ve been strugulling for quite some time.
I' m writing a python script that will allow me to upload a batch of XML files at once to my SQLite database.
So far, no problems. The point is I want to generate a unique ID for each XML file batch when uploading the infos to SQLite. Here are my codes:
This one reads each XML file pick up a specific value that through a string:
itemNota = 1
notas = []
for item in root.findall("./ns:NFe/ns:infNFe/ns:det", nsNFE):
remessa = (str(uuid.uuid4()))
nfe = self.check_none(root.find("./ns:NFe/ns:infNFe/ns:ide/ns:nNF", nsNFE))
data_emissao = self.check_none(root.find("./ns:NFe/ns:infNFe/ns:ide/ns:dhEmi", nsNFE))
data_emissao = F"{data_emissao[8:10]}/{data_emissao[5:7]}/{data_emissao[:4]}"
data_venc = self.check_none(root.find("./ns:NFe/ns:infNFe/ns:cobr/ns:dup/ns:dVenc", nsNFE))
data_venc = F"{data_venc[8:10]}/{data_venc[5:7]}/{data_venc[:4]}"
# Dados Emitentes
chave = self.check_none(root.find("./ns:protNFe/ns:infProt/ns:chNFe", nsNFE))
cnpj_emitente = self.check_none(root.find("./ns:NFe/ns:infNFe/ns:emit/ns:CNPJ", nsNFE))
nome_emitente = self.check_none(root.find("./ns:NFe/ns:infNFe/ns:emit/ns:xNome", nsNFE))
ie_emitente = self.check_none(root.find("./ns:NFe/ns:infNFe/ns:emit/ns:IE", nsNFE))
cnpj_emitente = self.format_cnpj(cnpj_emitente)
uf_emitente = self.check_none(root.find("./ns:NFe/ns:infNFe/ns:emit/ns:enderEmit/ns:UF", nsNFE))
valorNfe = self.check_none(root.find("./ns:NFe/ns:infNFe/ns:total/ns:ICMSTot/ns:vNF", nsNFE))
data_importacao = date.today()
data_importacao = data_importacao.strftime('%d/%m/%Y')
dados = [chave, cnpj_emitente, nome_emitente, ie_emitente, uf_emitente, cnpj_destinatario, cpf_destinatario,
nome_destinatario, ie_destinatario, uf_destinatario, nfe,
data_emissao, data_venc, valorNfe, data_importacao, remessa]
notas.append(dados)
itemNota += 1
return dados
I want my ID batch to be the"remessa" string using uuid library:
remessa = (str(uuid.uuid4()))
This is the code to insert the data on the SQlite table:
def insert_data(self, full_dataset):
cursor = self.connection.cursor()
campos_tabela = ('chave', 'cnpj_emitente', 'nome_emitente', 'ie_emitente',
'uf_emitente', 'cnpj_destinatario', 'cpf_destinatario', 'nome_destinatario',
'ie_destinatario', 'uf_destinatario', 'nfe', 'data_emissao', 'data_venc',
'valorNfe', 'data_importacao', 'remessa')
columns = (str(campos_tabela)).replace("'", "")
values = ','.join(str('?' * 16))
query_insert_data = self.query_insert_data(columns, values)
cursor.execute(query_insert_data, full_dataset)
self.connection.commit()
After executing, the table stays like this:
Table for XML´s files
But I want the column "remessa" to have the same id for each row and only changing when I insert new XML´s batch files.
Does anyone had acrossed something like this?

empty files in python

I am trying to create a file with all the magnetic information in my lists. I've used this code before as well. for some reason the file it returns is empty. I'm not sure why.
here is my code:
magnetosheath_Bx = JSS_Bx[12339:13795]
magnetosheath_By = JSS_By[12339:13795]
magnetosheath_Bz = JSS_Bz[12339:13795]
magnetosheath_B = JSS_Bmag[12339:13795]
magnetosheath_time = epochtime_magdata[12339:13795]
magnetosheath_r = new_RJSE[12339:13795]
Magnetosheath_data = zip(magnetosheath_Bx, magnetosheath_By, magnetosheath_Bz, magnetosheath_B, magnetosheath_time, magnetosheath_r)
filenew= open('Ulysses_Magnetoseath.txt' , 'w')
filenew.write("hello")
for magnetic_data in Magnetosheath_data:
filenew.write('{} {} {} {} {} {}\n'.format(magnetic_data[0], magnetic_data[1], magnetic_data[2], magnetic_data[3],magnetic_data[4],magnetic_data[5] ))
filenew.write("hello")

Python help, reading and writing to a txt file

I have posted the relevant part of my code below. Before that are just load functions, which I am pretty sure have no errors.
I am recieving error
IndexError: list index out of range( "namestaj["Naziv"] = deon[1]")
Does anyone see something out of order?
#load furniture from a txt file
def ucitajNamestaj():
listaNamestaja = open("namestaj.txt", "r").readlines()
namestaj = []
for red in listaNamestaja:
namestaj.append(stringToNamestaj(red))
return namestaj
#String to Furniture, dictionary
def stringToNamestaj(red):
namestaj = {}
deon = red.strip().split("|")
namestaj["Sifra"] = deon[0]
namestaj["Naziv"] = deon[1]
namestaj["Boja"] = deon[2]
namestaj["Kolicina"] = int(deon[3])
namestaj["Cena"] = float(deon[4])
namestaj["Kategorija"] = deon[5]
namestaj["Dostupan"] = deon[6]
return namestaj

Couple of things first, try always to provide a mcve and make sure you use properly the SO code directives, otherwise your question is unreadable.
Now, probably what's happening is your file has some empty lines and you're not skipping those, try this:
def ucitajNamestaj():
listaNamestaja = open("namestaj.txt", "r").readlines()
namestaj = []
for red in listaNamestaja:
if red.strip() == "":
continue
namestaj.append(stringToNamestaj(red))
return namestaj
def stringToNamestaj(red):
namestaj = {}
deon = red.strip().split("|")
namestaj["Sifra"] = deon[0]
namestaj["Naziv"] = deon[1]
namestaj["Boja"] = deon[2]
namestaj["Kolicina"] = int(deon[3])
namestaj["Cena"] = float(deon[4])
namestaj["Kategorija"] = deon[5]
namestaj["Dostupan"] = deon[6]
return namestaj

Working with Python and pickle to save complex data in object

I am experimenting with python to do a script for a program that works with python, and I need to save an object (with custom classes and arrays inside) to a file so that I can read it afterwards (so that I don't have to remake the object everytime, which takes hours)
I was reading in many forums that the easiest way to do that is to use pickle, but I am making a mistake in some place and I don't understand where...
Now, the code would be:
First I define this class:
class Issue_class:
Title_ID = None
Publisher_ID = None
Imprint_ID = None
Volume = None
Format = None
Color = None
Original = None
Rating = None
Issue_Date_Month = None
Issue_Date_Year = None
Reprint = None
Pages = None
Issue_Title = None
Number = None
Number_str = None
Synopsis = None
Characters_ID = None
Groups_ID = None
Writer_ID = None
Inker_ID = None
Colorist_ID = None
Letterer_ID = None
CoverArtist_ID = None
Penciller_ID = None
Editor_ID = None
Alternatives_ID = None
Reprints_ID = None
Story_ID = None
Multi = None
Multistories = None
then I define a list/array for this class:
Issuesdata = []
then during a loop I fill and append these to the list:
Issuedata = Issue_class()
Issuedata.Color = "unknown"
Issuedata.Tagline = "none"
Issuedata.Synopsis = "none"
Issuedata.Format = "none"
Issuedata.Publisher_ID = "none"
Issuedata.Imprint_ID = -1
Issuedata.Title_ID = -1
Issuedata.Volume = "none"
Issuedata.Number = -1
Issuedata.Number_str = "none"
Issuedata.Issue_Title = "none"
Issuedata.Rating = -1
Issuedata.Pages = -1
Issuedata.Issue_Date_Year = 0
Issuedata.Issue_Date_Month = 0
Issuedata.Original = True
Issuedata.Reprint = False
Issuedata.Multi= True
Issuedata.Letterer_ID = []
Issuedata.Characters_ID = []
Issuedata.Story_ID = []
Issuedata.Groups_ID = []
Issuedata.Writer_ID = []
Issuedata.Penciller_ID = []
Issuedata.Alternatives_ID = []
Issuedata.Reprints_ID = []
Issuedata.Inker_ID = []
Issuedata.Colorist_ID = []
Issuedata.Editor_ID = []
Issuedata.CoverArtist_ID = []
Issuedata.Multistories = []
Then I work with the data inside the object, and when it is complete, I append it to the list:
Issuesdata.append(Issuedata)
After that I print some info inside one of the objects in the list to be sure everything is ok:
print Issuesdata[3].Title_ID
print Issuesdata[3].Publisher_ID
print Issuesdata[3].Imprint_ID
print Issuesdata[3].Volume
print Issuesdata[3].Format
etc...
And everything is ok, the printed data is perfect
Now, I try to save the list to a file with:
filehandler = open("data.dat","wb")
pickle.dump(Issuesdata,filehandler)
filehandler.close()
This create the file with info inside... but when I try to read it with:
file = open("data.dat",'rb')
Issuesdat = pickle.load(file)
file.close()
The Python console tells me "'module' object has no attribute 'Issue_class'"
The first thing I thought was that I was reading the file wrong... But then I open the saved file with notepad and inside it it was full of "wrong data", like name of files or name of classes outside the code... which makes me suspect I am dumping the data wrong in the file...
Am I using pickle wrong?

Ok, I found the problem... It seems you have to define the class of your object in the main module for pickle to see it... I had it defined in the module I was working and calling the pickle command...

Try using pandas library with simple functions like:
DataFrame.to_pickle(file-path) to save pandas Dataframe in pickle.
pandas.read_pickle(file-path) to read pickle file.
Here you can find pandas reference to_pickle read_pickle.

How do I add a .csv extension to a file download in Google App Engine?

I'm using the csv module in python to create a download from one of the datastore tables in Google App Engine. The download works alright but you have to manually add an extension so that you can open it in Excel. I can't figure out how to modify the response so that the file download has a .csv extension. I could leave it like this however this web app is meant for a broad audience so I wanted to make it as easy as possible for them to use.
class fuCheckUp(webapp2.RequestHandler):
def get(self):
schedule_query = emailSchedule.all()
follow_up_num = schedule_query[0].follow_up_num
email_job_query = emailJobs.all()
email_job_query.order('consent_date')
header_tuple = ('last_modified', 'trigger_id', 'recipient_id', 'test_data', 'unsubscribe', 'start_date_local', 'consent_date', 'fu_period', 'last_fu_sent')
data_tuples = ()
variable_list = []
for i in range(1, follow_up_num + 1):
i = str(i)
fu_due = 'fu' + i
fu_sent = 'fu' + i + '_email_sent'
variable_list.append(fu_due)
variable_list.append(fu_sent)
data_tuples = data_tuples + (fu_due, fu_sent)
final_data_tuple = header_tuple + data_tuples
data = [final_data_tuple]
for part in email_job_query:
last_modified = str(part.last_modified)
trigger_id = str(part.trigger_id)
recipient_id = str(part.recipient_id)
test_data = str(part.test_data)
unsubscribed = str(part.unsubscribed)
start_date_local = str(part.start_date_local)
consent_date = str(part.consent_date)
fu_period = str(part.fu_period)
last_fu_sent = str(part.last_fu_sent)
var_list = []
for var in variable_list:
fu_var = getattr(part, var)
var_list.append(str(fu_var))
var_tuple = tuple(var_list)
fixed_tuple = (last_modified, trigger_id, recipient_id, test_data, unsubscribed, start_date_local, consent_date, fu_period, last_fu_sent)
csv_tuple = fixed_tuple + var_tuple
data.append((csv_tuple))
self.response.headers['Content-Type'] = 'application/csv'
writer = csv.writer(self.response.out)
for item in data:
writer.writerow(item)

Add another response header like this:
Content-Disposition: attachment;filename=example.csv