Develop Reference

iteration over 2d list in python and printing headings for each element

I am trying to work out how to iterate over a list and print out each item with a print statement describing what element is. my project is to create a user management system and print out something similar to the image I have attached.
The output I am trying to produce
The output I am getting
My code:
records = 0
userFirst = ["John"]
userLast = ["Doe"]
autoUsername = ["Johndoe91"]
autoPassword = ["123456789"]
hiddenPassword = ["*****789"]
userRole = ["User"]
userDept = ["Administration"]
users = []
confidentialUserDetails = []
users.append(userFirst + userLast + userRole + userDept + autoUsername + autoPassword)
confidentialUserDetails.append(users)
for row in range(len(confidentialUserDetails)):
records += 1
print("-" * 25)
print("Record: ", records)
for col in range(len(confidentialUserDetails[row])):
print(confidentialUserDetails[row][col])
Any help would be greatly appreciated. :)

Your data structures are unusual. I'm assuming that those lists are going to be provided to your code somehow and will, in practice, have multiple user details appended to them so that they are all the same length.
Anyhow, you can achieve the output you're looking for with some readable f-strings like this:
from functools import reduce
userFirst = ["John"]
userLast = ["Doe"]
autoUsername = ["Johndoe91"]
autoPassword = ["123456789"]
hiddenPassword = ["*****789"]
userRole = ["User"]
userDept = ["Administration"]
for row in range(len(userFirst)):
s = (f"""\
Name : {userFirst[row]} {userLast[row]}
Role : {userRole[row]}
Department : {userDept[row]}
Username : {autoUsername[row]}
Password : {hiddenPassword[row]}""")
maxlen = reduce(lambda x,y: max(x, len(y)), s.split("\n"), 0)
print(f"{s}\n{'-'*maxlen}\n")
Output:
Name : John Doe
Role : User
Department : Administration
Username : Johndoe91
Password : *****789
------------------------------

I created a dictionary called user instead of your list and after that I appended it to the second list and finally I printed the key and the value of the dictionary.
Also to get the full name I joined userFirst and userLast as string.
Code:
records = 0
userFirst = ["John"]
userLast = ["Doe"]
autoUsername = ["Johndoe91"]
autoPassword = ["123456789"]
hiddenPassword = ["*****789"]
userRole = ["User"]
userDept = ["Administration"]
confidentialUserDetails = [] # 2d list for asterisked passwords
users={'Name' : [' '.join(userFirst + userLast)] ,'Role' : userRole , 'Departement' : userDept ,'Username' : autoUsername ,'Password' : hiddenPassword }
confidentialUserDetails.append(users)
for user in confidentialUserDetails:
records += 1
print("-" * 25)
print("Record: ", records)
for ele,v in user.items():
print(ele,':',v[0])
Output:
-------------------------
Record: 1
Name : John Doe
Role : User
Departement : Administration
Username : Johndoe91
Password : *****789

Using a dictionary or f strings like the two other answers suggested is probably the best. But if you just want to use your current code to print your desired output, you can simply grab each item by its index number in your print statement.
Change the line:
print(confidentialUserDetails[row][col])
To something like this:
print("Name : ", confidentialUserDetails[row][col][0], confidentialUserDetails[row][col][1])
print("Role: : ", confidentialUserDetails[row][col][2])
Output:
-------------------------
Record: 1
Name : John Doe
Role: : User

Why using bulk_create on Django to insert data with foreign keys returns "property object is not callable"?

I'm using Django 2.17 with a SQLite 3.26 database and trying to insert data from a csv file. I was using the get_or_create method, but it was too slow. So I start to try to insert using bulk_create.
I have the following fields of Models being used:
class SENSOR_TEMPERATURA(models.Model):
ID_Sensor_Temperatura = models.AutoField(primary_key = True)
class VALOR_TEMPERATURA(models.Model):
ID_Valor_Temperatura = models.AutoField(primary_key = True)
ID_Sensor_Temperatura = models.ForeignKey(SENSOR_TEMPERATURA, on_delete = models.PROTECT, null = False, db_column = 'VATE_CD_ID_Sensor_Temperatura')
Data_De_Medição = models.DateTimeField(default = datetime.now(), null = False)
Valor = models.DecimalField(default = 0, null = False, max_digits = 30, decimal_places = 15)
The code that I'm trying to run is:
print (datetime.datetime.now())
reader = csv.reader(f)
insert_CSV = []
count = 1
for row in reader:
insert_CSV.append([
VALOR_TEMPERATURA.pk(ID_Valor_Temperatura = count),
VALOR_TEMPERATURA(Data_De_Medição = datetime.datetime.strptime(row[0] + " UTC-0300",'%d/%m/%Y %H:%M:%S %Z%z')),
VALOR_TEMPERATURA(Valor = float(row[1])),
VALOR_TEMPERATURA(ID_Sensor_Temperatura = SENSOR_TEMPERATURA.objects.get(ID_Sensor_Temperatura = 4))
])
count = count + 1
print (datetime.datetime.now())
VALOR_TEMPERATURA.objects.bulk_create(insert_CSV)
print (datetime.datetime.now())
The part that I think is put me in trouble is "ID_Sensor_Temperatura = SENSOR_TEMPERATURA.objects.get(ID_Sensor_Temperatura = 4))", but it is exactly how I defined the Foreign Key when using get_or_create, so I can't figure out what is the problem.
I'm getting the following error:
6 for row in reader:
7 insert_CSV.append([
8 VALOR_TEMPERATURA.pk(VATE_CD_ID_Valor_Temperatura = count),
9 VALOR_TEMPERATURA(VATE_DF_Data_De_Medição = datetime.datetime.strptime(row[0] + " UTC-0300",'%d/%m/%Y %H:%M:%S %Z%z')),
10 VALOR_TEMPERATURA(VATE_VL_Valor = float(row[1])),
TypeError: 'property' object is not callable
What may be the problem?

This isn't how you write Python. You need to create an instance of the object, passing it the values.
insert_CSV.append(
VALOR_TEMPERATURA(
ID_Valor_Temperatura=count,
Data_De_Medição=datetime.datetime.strptime(row[0] + " UTC-0300",'%d/%m/%Y %H:%M:%S %Z%z')),
Valor=float(row[1]),
ID_Sensor_Temperatura=SENSOR_TEMPERATURA.objects.get(ID_Sensor_Temperatura=4)
)
)
Note also, your models should not be defined in ALL_CAPS, as they are not constants. They shoudl be called ValorTemperatura and SensorTemperatura.

Combine ~Q and F in Django?

Query
Balance.objects.filter(~Q(fax_date=F('paused_date')))
returns empty qs even though I do have objects that fit the condition "fax date field not equal to paused date". Is it possible to use ~Q and F together like that?
ran a test like this:
deals = Deal.objects.all()
balance_pre = Balance.objects.filter(~Q(fax_date=F('paused_date')), fax_date__isnull=False, reserved=False)
agr_nums = list(deals.filter(agr_name__isnull=False).values_list('agr_name', flat=True).distinct())
agrs_with_fax = 0
for agr_num in agr_nums:
try:
balance_agr = Balance.objects.get(number__icontains=agr_num)
if balance_agr.fax_date is not None and balance_agr.fax_date != balance_agr.paused_date and not balance_agr.reserved:
agrs_with_fax += 1
except Balance.DoesNotExist:
pass
agrs_with_fax2 = 0
for agr_num in agr_nums:
try:
balance_pre.get(number__icontains=agr_num)
agrs_with_fax2 += 1
except Balance.DoesNotExist:
pass
r = [agrs_with_fax, agrs_with_fax2, balance_agr.fax_date, balance_agr.paused_date, balance_agr.reserved]
r returned is
[55, 0, datetime.date(2018, 7, 11), None, False]
I don't see my error, both cycles should return same result.

I created a Balance model in a fresh project just to test that print(qs.query) will show you the generated query(not in all cases) in this case. I used also exclude as #daniel-roseman suggested to prove that they were equivalent. I hope this help you.
>>> from django.db.models import F, Q
>>> qs = Balance.objects.filter(~Q(fax_date=F('paused_date')))
>>> print(qs.query)
SELECT "so_balance"."id", "so_balance"."fax_date", "so_balance"."paused_date"
FROM "so_balance" WHERE NOT ("so_balance"."fax_date" =
("so_balance"."paused_date"))
>>> qs = Balance.objects.exclude(fax_date=F('paused_date'))
>>> print(qs.query)
SELECT "so_balance"."id", "so_balance"."fax_date", "so_balance"."paused_date"
FROM "so_balance" WHERE NOT ("so_balance"."fax_date" =
("so_balance"."paused_date"))

Python and string accents

I am making a web scraper.
I access google search, I get the link of the web page and then I get the contents of the <title> tag.
The problem is that, for example, the string "P\xe1gina N\xe3o Encontrada!" should be "Página Não Encontrada!".
I tried do decode to latin-1 and then encode to utf-8 and it did not work.
r2 = requests.get(item_str)
texto_pagina = r2.text
soup_item = BeautifulSoup(texto_pagina,"html.parser")
empresa = soup_item.find_all("title")
print(empresa_str.decode('latin1').encode('utf8'))
Can you help me, please?
Thanks !

You can change the retrieved text variable to something like:
string = u'P\xe1gina N\xe3o Encontrada!'.encode('utf-8')
After printing string it seemed to work just fine for me.
Edit
Instead of adding .encode('utf8'), have you tried just using empresa_str.decode('latin1')?
As in:
string = empresa_str.decode('latin_1')

Not the most elegant solution, but worked for me :
def remove_all(substr, str):
index = 0
length = len(substr)
while string.find(str, substr) != -1:
index = string.find(str, substr)
str = str[0:index] + str[index+length:]
return str
def latin1_to_ascii (unicrap):
xlate={ 'xc3cb3':'o' , 'xc3xa7':'c','xc3xb5':'o', 'xc3xa3':'a', 'xc3xa9':'e',
'xc0':'A', 'xc1':'A', 'xc2':'A', 'xc3':'A', 'xc4':'A', 'xc5':'A',
'xc6':'Ae', 'xc7':'C',
'xc8':'E', 'xc9':'E', 'xca':'E', 'xcb':'E',
'xcc':'I', 'xcd':'I', 'xce':'I', 'xcf':'I',
'xd0':'Th', 'xd1':'N',
'xd2':'O', 'xd3':'O', 'xd4':'O', 'xd5':'O', 'xd6':'O', 'xd8':'O',
'xd9':'U', 'xda':'U', 'xdb':'U', 'xdc':'U',
'xdd':'Y', 'xde':'th', 'xdf':'ss',
'xe0':'a', 'xe1':'a', 'xe2':'a', 'xe3':'a', 'xe4':'a', 'xe5':'a',
'xe6':'ae', 'xe7':'c',
'xe8':'e', 'xe9':'e', 'xea':'e', 'xeb':'e',
'xec':'i', 'xed':'i', 'xee':'i', 'xef':'i',
'xf0':'th', 'xf1':'n',
'xf2':'o', 'xf3':'o', 'xf4':'o', 'xf5':'o', 'xf6':'o', 'xf8':'o',
'xf9':'u', 'xfa':'u', 'xfb':'u', 'xfc':'u',
'xfd':'y', 'xfe':'th', 'xff':'y',
'xa1':'!', 'xa2':'{cent}', 'xa3':'{pound}', 'xa4':'{currency}',
'xa5':'{yen}', 'xa6':'|', 'xa7':'{section}', 'xa8':'{umlaut}',
'xa9':'{C}', 'xaa':'{^a}', 'xab':'<<', 'xac':'{not}',
'xad':'-', 'xae':'{R}', 'xaf':'_', 'xb0':'{degrees}',
'xb1':'{+/-}', 'xb2':'{^2}', 'xb3':'{^3}', 'xb4':'',
'xb5':'{micro}', 'xb6':'{paragraph}', 'xb7':'*', 'xb8':'{cedilla}',
'xb9':'{^1}', 'xba':'{^o}', 'xbb':'>>',
'xbc':'{1/4}', 'xbd':'{1/2}', 'xbe':'{3/4}', 'xbf':'?',
'xd7':'*', 'xf7':'/'
}
unicrap = remove_all ('\\', unicrap)
unicrap = remove_all('&', unicrap)
unicrap = remove_all('u2013', unicrap)
r = unicrap
for item,valor in xlate.items():
#print item, unicrap.find(item)
r = r.replace(item,valor)
return r

pretty print assertEqual() for HTML strings

I want to compare two strings in a python unittest which contain html.
Is there a method which outputs the result in a human friendly (diff like) version?

A simple method is to strip whitespace from the HTML and split it into a list. Python 2.7's unittest (or the backported unittest2) then gives a human-readable diff between the lists.
import re
def split_html(html):
return re.split(r'\s*\n\s*', html.strip())
def test_render_html():
expected = ['<div>', '...', '</div>']
got = split_html(render_html())
self.assertEqual(expected, got)
If I'm writing a test for working code, I usually first set expected = [], insert a self.maxDiff = None before the assert and let the test fail once. The expected list can then be copy-pasted from the test output.
You might need to tweak how whitespace is stripped depending on what your HTML looks like.

I submitted a patch to do this some years back. The patch was rejected but you can still view it on the python bug list.
I doubt you would want to hack your unittest.py to apply the patch (if it even still works after all this time), but here's the function for reducing two strings a manageable size while still keeping at least part of what differs. So long as all you didn't want the complete differences this might be what you want:
def shortdiff(x,y):
'''shortdiff(x,y)
Compare strings x and y and display differences.
If the strings are too long, shorten them to fit
in one line, while still keeping at least some difference.
'''
import difflib
LINELEN = 79
def limit(s):
if len(s) > LINELEN:
return s[:LINELEN-3] + '...'
return s
def firstdiff(s, t):
span = 1000
for pos in range(0, max(len(s), len(t)), span):
if s[pos:pos+span] != t[pos:pos+span]:
for index in range(pos, pos+span):
if s[index:index+1] != t[index:index+1]:
return index
left = LINELEN/4
index = firstdiff(x, y)
if index > left + 7:
x = x[:left] + '...' + x[index-4:index+LINELEN]
y = y[:left] + '...' + y[index-4:index+LINELEN]
else:
x, y = x[:LINELEN+1], y[:LINELEN+1]
left = 0
cruncher = difflib.SequenceMatcher(None)
xtags = ytags = ""
cruncher.set_seqs(x, y)
editchars = { 'replace': ('^', '^'),
'delete': ('-', ''),
'insert': ('', '+'),
'equal': (' ',' ') }
for tag, xi1, xi2, yj1, yj2 in cruncher.get_opcodes():
lx, ly = xi2 - xi1, yj2 - yj1
edits = editchars[tag]
xtags += edits[0] * lx
ytags += edits[1] * ly
# Include ellipsis in edits line.
if left:
xtags = xtags[:left] + '...' + xtags[left+3:]
ytags = ytags[:left] + '...' + ytags[left+3:]
diffs = [ x, xtags, y, ytags ]
if max([len(s) for s in diffs]) < LINELEN:
return '\n'.join(diffs)
diffs = [ limit(s) for s in diffs ]
return '\n'.join(diffs)

Maybe this is a quite 'verbose' solution. You could add a new 'equality function' for your user defined type (e.g: HTMLString) which you have to define first:
class HTMLString(str):
pass
Now you have to define a type equality function:
def assertHTMLStringEqual(first, second):
if first != second:
message = ... # TODO here: format your message, e.g a diff
raise AssertionError(message)
All you have to do is format your message as you like. You can also use a class method in your specific TestCase as a type equality function. This gives you more functionality to format your message, since unittest.TestCase does this a lot.
Now you have to register this equality function in your unittest.TestCase:
...
def __init__(self):
self.addTypeEqualityFunc(HTMLString, assertHTMLStringEqual)
The same for a class method:
...
def __init__(self):
self.addTypeEqualityFunc(HTMLString, 'assertHTMLStringEqual')
And now you can use it in your tests:
def test_something(self):
htmlstring1 = HTMLString(...)
htmlstring2 = HTMLString(...)
self.assertEqual(htmlstring1, htmlstring2)
This should work well with python 2.7.

I (the one asking this question) use BeautfulSoup now:
def assertEqualHTML(string1, string2, file1='', file2=''):
u'''
Compare two unicode strings containing HTML.
A human friendly diff goes to logging.error() if there
are not equal, and an exception gets raised.
'''
from BeautifulSoup import BeautifulSoup as bs
import difflib
def short(mystr):
max=20
if len(mystr)>max:
return mystr[:max]
return mystr
p=[]
for mystr, file in [(string1, file1), (string2, file2)]:
if not isinstance(mystr, unicode):
raise Exception(u'string ist not unicode: %r %s' % (short(mystr), file))
soup=bs(mystr)
pretty=soup.prettify()
p.append(pretty)
if p[0]!=p[1]:
for line in difflib.unified_diff(p[0].splitlines(), p[1].splitlines(), fromfile=file1, tofile=file2):
logging.error(line)
raise Exception('Not equal %s %s' % (file1, file2))