Develop Reference

Checking for Duplicates twice over in a File - Python

config.yml example,
DBtables:
CurrentMinuteLoad:
CSV_File: trend.csv
Table_Name: currentminuteload
GUI image,
This may not be the cleanest route to take.
I'm making a GUI that creates a config.yml file for another python script I'm working with.
Using pysimplegui, My button isn't functioning the way I'd expect it to. It currently and accurately checks for the Reference name (example here would be CurrentMinuteLoad) and will kick it back if it exists, but will skip the check for the table (so the ELIF statement gets skipped). Adding the table still works, I'm just not getting the double-check that I want. Also, I have to hit the Okay button twice in the GUI for it to work?? A weird quirk that doesn't quite make sense to me.
def add_table():
window2.read()
with open ("config.yml","r") as h:
if values['new_ref'] in h.read():
sg.popup('Reference name already exists')
elif values['new_db'] in h.read():
sg.popup('Table name already exists')
else:
with open("config.yml", "a+") as f:
f.write("\n " + values['new_ref'] +":")
f.write("\n CSV_File:" + values['new_csv'])
f.write("\n Table_Name:" + values['new_db'])
f.close()
sg.popup('The reference "' + values['new_ref'] + '" has been included and will add the table "' + values['new_db'] + '" to PG Admin during the next scheduled upload')

When you use h.read(), you should save the value since it will read it like a stream, and subsequent calls for this method will result in an empty string.
Try editing the code like this:
with open ("config.yml","r") as h:
content = h.read()
if values['new_ref'] in content:
sg.popup('Reference name already exists')
elif values['new_db'] in content:
sg.popup('Table name already exists')
else:
# ...

You should update the YAML file using a real YAML parser, that will allow you
to check on duplicate values, without using in, which will give you false
positives when a new value is a substring of an existing value (or key).
In the following I add values twice, and show the resulting YAML. The
first time around the check on new_ref and new_db does not find
a match although it is a substring of existing values. The second time
using the same values there is of course a match on the previously added
values.
import sys
import ruamel.yaml
from pathlib import Path
def add_table(filename, values, verbose=False):
error = False
yaml = ruamel.yaml.YAML()
data = yaml.load(filename)
dbtables = data['DBtables']
if values['new_ref'] in dbtables:
print(f'Reference name "{values["new_ref"]}" already exists') # use sg.popup in your code
error = True
for k, v in dbtables.items():
if values['new_db'] in v.values():
print(f'Table name "{values["new_db"]}" already exists')
error = True
if error:
return
dbtables[values['new_ref']] = d = {}
for x in ['new_cv', 'new_db']:
d[x] = values[x]
yaml.dump(data, filename)
if verbose:
sys.stdout.write(filename.read_text())
values = dict(new_ref='CurrentMinuteL', new_cv='trend_csv', new_db='currentminutel')
add_table(Path('config.yaml'), values, verbose=True)
print('========')
add_table(Path('config.yaml'), values, verbose=True)
which gives:
DBtables:
CurrentMinuteLoad:
CSV_File: trend.csv
Table_Name: currentminuteload
CurrentMinuteL:
new_cv: trend_csv
new_db: currentminutel
========
Reference name "CurrentMinuteL" already exists
Table name "currentminutel" already exists

Python winreg - How to write a linefeed to a REG_SZ value

I need some assistance in writing a line feed to a registry value. The Value is of type REG_SZ.
I'm able to do this manually, by adding a "0A" to each break position when modifying the hex value in the Registry, but I'm not sure how to do this programmatically.
This is my current code to write the String to Registry, which WORKS, but does not allow for the line feeds:
(identifying information redacted, and text shortened. The "#" is the position for a line feed)
import os
import _winreg
from _winreg import *
Key = r"SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System"
Field = [_winreg.REG_SZ, _winreg.REG_BINARY]
Sub_Key = ["legalnoticetext",
"legalnoticecaption"]
value = ["Terms of Use",
"By logging in to this PC, users agree to the Terms of Use"
"\nThe User agrees to the following:#- The User may not alter, in any way, "
"with Settings on this PC without the approval from any Authorised Persons."]
parmLen = len(Sub_Key)
z = 0 # Loop Counter for list iteration
try:
Key = r"SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System"
if not os.path.exists(Key):
key = _winreg.CreateKey(HKEY_LOCAL_MACHINE, Key)
Registrykey = OpenKey(HKEY_LOCAL_MACHINE, Key, 0, KEY_WRITE)
while z < parmLen:
_winreg.SetValueEx(Registrykey, Sub_Key[z], 0, Field[z], value[z])
print ("Setting <" + Sub_Key[z] + "> with value: " + value[z])
z += 1
CloseKey(Registrykey)
print ("SUCCESSFUL! Procedure: Show Logon Terms of Use")
except WindowsError:
print ("Error")
I've tested the following code to see if I can write directly in Hex, as I have the modified hex value, but it results in the value being interpreted as a string, and then formatting it incorrectly (again) to Hex.
import os
import _winreg
from _winreg import *
Key = r"SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System"
Field = [_winreg.REG_SZ, _winreg.REG_BINARY]
Sub_Key = ["legalnoticetext",
"legalnoticecaption"]
value = ["Terms of Use",
"42,00,79,00,20,00,6C,00,6F,00,67,00,67,00,69,00,6E,00,67,00,20,00,69,00,"
"6E,00,20,00,74,00,6F,00,20,00,74,00,68,00,69,00,73,00,20,00,50,00,43,00,"
"2C,00,20,00,75,00,73,00,65,00,72,00,73,00,20,00,61,00,67,00,72,00,65,00,"
"65,00,20,00,74,00,6F,00,20,00,74,00,68,00,65,00,20,00,54,00,65,00,72,00,"
"6D,00,73,00,20,00,6F,00,66,00,20,00,55,00,73,00,65,00,0A,00,54,00,68,00,"
"65,00,20,00,55,00,73,00,65,00,72,00,20,00,61,00,67,00,72,00,65,00,65,00,"
"73,00,20,00,74,00,6F,00,20,00,74,00,68,00,65,00,20,00,66,00,6F,00,6C,00,"
"6C,00,6F,00,77,00,69,00,6E,00,67,00,3A,00,0A,00,2D,00,20,00,54,00,68,00,"
"65,00,20,00,55,00,73,00,65,00,72,00,20,00,6D,00,61,00,79,00,20,00,6E,00,"
"6F,00,74,00,20,00,61,00,6C,00,74,00,65,00,72,00,2C,00,20,00,69,00,6E,00,"
"20,00,61,00,6E,00,79,00,20,00,77,00,61,00,79,00,2C,00,20,00,77,00,69,00,"
"74,00,68,00,20,00,53,00,65,00,74,00,74,00,69,00,6E,00,67,00,73,00,20,00,"
"6F,00,6E,00,20,00,74,00,68,00,69,00,73,00,20,00,50,00,43,00,20,00,77,00,"
"69,00,74,00,68,00,6F,00,75,00,74,00,20,00,74,00,68,00,65,00,20,00,61,00,"
"70,00,70,00,72,00,6F,00,76,00,61,00,6C,00,20,00,66,00,72,00,6F,00,6D,00,"
"20,00,61,00,6E,00,79,00,20,00,41,00,75,00,74,00,68,00,6F,00,72,00,69,00,"
"73,00,65,00,64,00,20,00,50,00,65,00,72,00,73,00,6F,00,6E,00,73,00,2E,00,"
"00,00"]
parmLen = len(Sub_Key)
z = 0 # Loop Counter for list iteration
try:
Key = r"SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System"
if not os.path.exists(Key):
key = _winreg.CreateKey(HKEY_LOCAL_MACHINE, Key)
Registrykey = OpenKey(HKEY_LOCAL_MACHINE, Key, 0, KEY_WRITE)
while z < parmLen:
_winreg.SetValueEx(Registrykey, Sub_Key[z], 0, Field[z], value[z])
print ("Setting <" + Sub_Key[z] + "> with value: " + value[z])
z += 1
CloseKey(Registrykey)
print ("SUCCESSFUL! Procedure: Show Logon Terms of Use")
except WindowsError:
print ("Error")
RTFM'ing doesn't prove to be very helpful with this specific issue, so any guidance would be appreciated!

You have a couple of issues going on. os.path.exists checks for file path existence. Feeding it the key string won't check the registry.
It looks like you are using Python 2.7, please consider upgrading to 3. The issue you are running into is that writing a REG_BINARY expects binary data; in Python 3, this means you just need to encode your string. Here is the code in Python 3.
import winreg
Key_str = r"SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System"
Field = [winreg.REG_SZ, winreg.REG_BINARY]
Sub_Key = ["legalnoticetext", "legalnoticecaption"]
value = ["Terms of Use",
"By logging in to this PC, users agree to the Terms of Use"
"\nThe User agrees to the following:#- The User may not alter, in any way, "
"with Settings on this PC without the approval from any Authorised Persons."]
try:
key = winreg.OpenKey(winreg.HKEY_LOCAL_MACHINE, Key_str)
except FileNotFoundError:
key = winreg.CreateKey(winreg.HKEY_LOCAL_MACHINE, Key_str)
for sk, f, v in zip(Sub_Key, Field, value):
if f == winreg.REG_BINARY:
winreg.SetValueEx(key, sk, 0, f, v.encode('latin-1'))
else:
winreg.SetValueEx(key, sk, 0, f, v)
key.Close()
In Python 2, your standard strings are byte strings, so there is no need to encode the string to a bytes encoding. Here is the Python 2 code:
import _winreg
Key_str = r"SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System"
Field = [_winreg.REG_SZ, _winreg.REG_BINARY]
Sub_Key = ["legalnoticetext", "legalnoticecaption"]
value = ["Terms of Use",
"By logging in to this PC, users agree to the Terms of Use"
"\nThe User agrees to the following:#- The User may not alter, in any way, "
"with Settings on this PC without the approval from any Authorised Persons."]
try:
key = _winreg.OpenKey(_winreg.HKEY_LOCAL_MACHINE, Key_str)
except WindowsError:
key = _winreg.CreateKey(_winreg.HKEY_LOCAL_MACHINE, Key_str)
for sk, f, v in zip(Sub_Key, Field, value):
_winreg.SetValueEx(key, sk, 0, f, v)
key.Close()

Retrieve bulk data from YAML using Python

I have a yaml file of the form below:
Solution:
- number of solutions: 1
number of solutions displayed: 1
- Gap: None
Status: optimal
Message: bonmin\x3a Optimal
Objective:
objective:
Value: 0.010981105395
Variable:
battery_E[b1,1,1]:
Value: 0.25
battery_E[b1,1,2]:
Value: 0.259912707017
battery_E[b1,2,1]:
Value: 0.120758408109
battery_E[b2,1,1]:
Value: 0.0899999972181
battery_E[b2,2,3]:
Value: 0.198967393893
windfarm_L[w1,2,3]:
Value: 1
windfarm_L[w1,3,1]:
Value: 1
windfarm_L[w1,3,2]:
Value: 1
Using Python27, I would like to import all battery_E values from this YAML file. I know I can iterate over the keys of battery_E dictionary to retrieve them one by one (I am already doing it using PyYAML) but I would like to avoid iterating and do it in one go!

It's not possible "in one go" - there will still be some kind of iteration either way, and that's completely OK.
However, if the memory is a concern, you can load only values of the keys of interest during YAML loading:
from __future__ import print_function
import yaml
KEY = 'battery_E'
class Loader(yaml.SafeLoader):
def __init__(self, stream):
super(Loader, self).__init__(stream)
self.values = []
def compose_mapping_node(self, anchor):
start_event = self.get_event()
tag = start_event.tag
if tag is None or tag == '!':
tag = self.resolve(yaml.MappingNode, None, start_event.implicit)
node = yaml.MappingNode(tag, [],
start_event.start_mark, None,
flow_style=start_event.flow_style)
if anchor is not None:
self.anchors[anchor] = node
while not self.check_event(yaml.MappingEndEvent):
item_key = self.compose_node(node, None)
item_value = self.compose_node(node, item_key)
if (isinstance(item_key, yaml.ScalarNode)
and item_key.value.startswith(KEY)
and item_key.value[len(KEY)] == '['):
self.values.append(self.construct_object(item_value, deep=True))
else:
node.value.append((item_key, item_value))
end_event = self.get_event()
node.end_mark = end_event.end_mark
return node
with open('test.yaml') as f:
loader = Loader(f)
try:
loader.get_single_data()
finally:
loader.dispose()
print(loader.values)
Note however, that this code does not assume anything about the position of battery_E keys in the tree inside the YAML file - it will just load all of their values.

There is no need to retrieve each entry using PyYAML, you can load the data once, and then use Pythons to select the key-value pairs with the following two lines:
data = yaml.safe_load(open('input.yaml'))
kv = {k:v['Value'] for k, v in data['Solution'][1]['Variable'].items() if k.startswith('battery_E')}
after that kv contains:
{'battery_E[b2,2,3]': 0.198967393893, 'battery_E[b1,1,1]': 0.25, 'battery_E[b1,1,2]': 0.259912707017, 'battery_E[b2,1,1]': 0.0899999972181, 'battery_E[b1,2,1]': 0.120758408109}

opening shelf file / dbm file returns error dbmError created file using dbm.open and shelve.Shelf.open

python 3.4.2, on Linux
I'm pretty new to this language but I'm coding this project. It started as a simple program that displayed a dictionary . Well I'm trying to expand on it based on tutorials that i am reading. I came to one about shelving and being able to preserve info in a save file in a format much like a dictionary. So far i have a program that takes input and updates the dictionary based on the input. It's very basic but it works on a simple level but naturally I would want to save what I entered. Following is the code so far.
the updateSaveP1() function is what is giving me trouble. Although not coded like this currently, i would ultimately like the function to take 2 arguments, one to name the key in the shelf file, and one to reference the target dictionary/ list etc. Currently its not even saving to the file.
the loadINV() function is a place holder and doent work as coded currently. i need to figure out the dbm problem first as i get the same dbmError with the load function too.
I originally just .opened the file. found documentation here at Stack that i should open it with either so that it creates the right file. I have tried both to no avail.
NOTICE**** this code will create an empty file on your system at the working directory of python named savedata.db
Much Thanks and appreciation for any help.
`import pygame, math, sys, os, pprint, shelve, dbm,
SAVE_LOCATION = os.path.join(os.getcwd() + '/savedata')
SAVE_FILE_LIST = os.listdir(os.getcwd())
SOURCE = os.path.dirname('inventory.py')
YES = ["y","Y"]
NO = ["n","N"]
playerStash = {"rope": 77,
"giant's toe" : 1,
"gold" : 420}
'''def loadINV():
fileTOload = dbm.open('savedata.db', 'w')
print('opened savedata file')
#for eachLine in fileTOload:
playerStash.update(str(fileTOload.read())
)
print('updated dict')
fileTOload.close()'''
def checkSavesFile():
while True:
if os.path.exists(SAVE_LOCATION):
print('Save file found')
break
elif os.path.exists(SAVE_LOCATION + '.db'):
print('.db Save file found')
loadINV()
break
else:
updateSaveP1()
print('New Save Created')
break
def updateSaveP1():
with dbm.open('savedata', 'c') as save:
save['player1'] = str(playerStash)
save.close()
#print(SAVE_LOCATION) #debugging - file name format verification
#pprint.pprint(SAVE_FILE_LIST) debugging will pretty print list of files
checkSavesFile() # runs the save file check
def askAboutInv(player):
while True:
print("What item would you like to add? \n\
(leave blank and press enter to quit)")
name = input() # Reads input and checks for duplicates or non entries
if name == '':
break # Stop loop
elif name in playerStash.keys():
# the check to see if input was in dictionary
dict_quant = int(playerStash.get(name, 0))
# "dict_quant" represents the value in dictionary as an integer
dict_item = str(playerStash.get(name, 0))
# "dict_item represents the value in dictionary as a string
addedItem = dict_quant + 1
#handles adding the value of the input
print("You have " + dict_item + " already, \n\
would you like to add more Y/N?")
# prints " You have "dictionary number" already"
answer = input()
# checks for input if you want to add more to inventory
if answer in YES: #checks to see if y or Y is entered
playerStash[name] = addedItem
# adds +1 to the quantity of "name" per the dict_quant variable
print("you have " + str(addedItem) + " now")
# prints " you have "new dictionary number" now"
if answer in NO: #checks to see if n or N was entered
print("Nothing added") #prints
break #ends loop
else: #if none others statements are true
if name not in playerStash.keys():
#if "name" / input is not in the dictionary
playerStash[name] = playerStash.setdefault(name, 1)
# add the item to the dictionary with a value of 1
print('Inventory updated.')
# prints
updateSaveP1()
def inventoryDisp(player):# displays dictionary pointing towards argument
print("Inventory")
item_total = 0
for eachOne in playerStash.items():
print(eachOne) # looks at and prints each item/ key in dictionary
for i, q in playerStash.items():
item_total = item_total + q #adds all the quantities / values up
print("Total number of items: " + str(item_total))
# prints total number of items in inventory
def updatedInv(player): #same as above just reads "updated inventory"
print("Updated Inventory")
item_total = 0
for eachOne in playerStash.items():
print(eachOne)
for i, q in playerStash.items():
item_total = item_total + q
print("Total number of items: " + str(item_total))
inventoryDisp(playerStash)
askAboutInv(playerStash)
updateSaveP1()
updatedInv(playerStash)`
Update*****
after changing this:
`def updateSaveP1():
with dbm.open('savedata', 'c') as save:
save['player1'] = str(playerStash)
save.close()`
to this:
`def updateSaveP1():
save = openShelf()
#save = shelve.Shelf(dbm.open('savedata', 'c')) #old code
save['player1'] = str(playerStash)
print(save['player1'])
save.close()`
it would seem that the dictionary does get saved. now the loadINV function is giving me trouble. This:
def loadINV():
fileTOload = dbm.open('savedata.db', 'w')
print('opened savedata file')
#for eachLine in fileTOload:
playerStash.update(str(fileTOload.read())
)
print('updated dict')
fileTOload.close()
is now this:
def loadINV():
file = openShelf()
print('opened savedata file')
playerStash.update(file['player1'])
print('updated dict')
fileTOload.close()
but the .update() method returns this error: which i can't seem to find any info.
Traceback (most recent call last):
File "/home/pi/inventoryShelve.py", line 58, in <module>
checkSavesFile() # runs the save file check
File "/home/pi/inventoryShelve.py", line 40, in checkSavesFile
loadINV()
File "/home/pi/inventoryShelve.py", line 25, in loadINV
playerStash.update(file['player1'])
ValueError: dictionary update sequence element #0 has length 1; 2 is required

Turned out I was saving (into the shelf) the data for the inventory dictionary as something other than a dict:
def updateSaveP1():
save = openShelf()
save['player1'] = str(playerStash)
#print(save['player1'])
save.close()
became
def updateSaveP1():
save = openShelf()
save['player1'] = dict(playerStash)
#print(save['player1'])
save.close()

TypeError: 'DictWriter' object is not iterable

I'm working on creating a short simple program for a nonprofit fundraiser to validate ticket numbers as guests check in to make sure no duplicate tickets are redeemed. I'm running Python 3.4.3 on a Windows 10 machine. Once the program is finalized it will be used on a Raspberry Pi with touchscreen at the fundraiser.
I've tried a couple different methods to build the list, save it, and search for duplicates. Ideally the list will be stored in a CSV file, but a plain text or other format is ok too.
Can you help me with the traceback error (TypeError: 'DictWriter' object is not iterable) due to the looping function to check ticket #'s against a list stored in a file to make sure no duplicate tickets are redeemed?
Thank you in advance for your help!
version = "v1.4"
fname="tickets.csv"
import csv
import datetime
import os.path
print("\nWelcome to TicketCheck", version)
extant = os.path.isfile(fname)
with open(fname, 'a', newline='') as csvfile:
fieldnames = ['ticketid', 'timestamp']
ticketwriter = csv.DictWriter(csvfile, fieldnames=fieldnames)
if extant == False:
ticketwriter.writeheader()
while True:
ticket = ""
print("Please enter a ticket # to continue or type exit to exit:")
ticket = str(input())
if ticket == "":
continue
if ticket == "exit":
break
print("You entered ticket # %s." % (ticket))
print("Validating ticket...")
for row in ticketwriter:
if row[0] == ticket:
print("\n\n\n===== ERROR!!! TICKET # %s ALREADY CHECKED IN =====\n\n\n" % (ticket))
continue
time = datetime.datetime.now()
print("Thank you for checking in ticket # %s at %s \n\n\n" % (ticket, time))
print("Ticket is now validated.")
ticketwriter.writerow({'ticketid': ticket, 'timestamp': time})
csvfile.flush()
continue
csvfile.close()
print("All your work has been saved in %s.\n Thank you for using TicketCheck %s \n" % (fname, version))

Hmm, I think you might be over-complicating this a bit! For something like that there's really no need to go to all that trouble. This is a great spot to use a dictionary, and for something with only two inputs, the id and the check-in time, you can easily just make a .txt log. I get the feeling this might be more of what you are looking for.
import time
go = True
while go:
the_guestlist = {}
the_ticket = input().strip()
file = open('somefile.txt', 'r')
for line in file:
my_items = line.split(',')
the_guestlist[my_items[0]] = my_items[1]
file.close()
if the_ticket in the_guestlist.keys():
print("Sorry, that ticket has been entered at {}".format(the_guestlist[the_ticket]))
elif the_ticket == 'exit':
go = False
print('Exiting...')
else:
the_guestlist[the_ticket] = '{}'.format(time.asctime())
file = open('somefile.txt', 'a')
file.write(the_ticket +','+the_guestlist[the_ticket]+'\n')
file.close()

Objects of the csv.DictWriter class are not iterable, i.e. you cannot iterate over them like you would a dictionary, list, or even string, hence your error message. It does not store the data you have previously written to file, only the file you wrote to stores that data.
To achieve your goal, you could do two things: either open your CSV file every time a new ticket needs to be validated, and check if the ticket number is present, or - since you are using relatively small amounts of data - store a dictionary in memory, and only write it out at the end of use, checking from that if the ticket is valid.