The following works when typed directly in the mongodb shell—I receive the correct output:
db.serial_key.find({key: {$ne : "5SNT0V"}})
However, in Python 3, it's not working. Every time, only the if block runs. It does not matter if I use str or not in the query.
for x in keys:
i +=1;
print('key %d = ' %i , keys[i-1]) #out put: key 1 = 3ZOHSH
# place values in dictionary
keyrecord_record = {'key':keys[i-1],'b_p_code':x1}
if(db.serial_key.find({'key':{ '$ne':str(keys[i-1])}})):
db.insert(keyrecord_record)
else:
print('Record in DB')
Please, I expect some expert help. I'm trying to do this within one day. I only want to write values that are not already in the database.
I found this question:
Mongo db not equal to query not working
...but it does not answer my question.
===================full main class code==========================
from pymongo import MongoClient
from algo import algo
#take in put data
x1 = int(input("Enter a number(Brand_name+Pack_size) : "))
y1 = int(input("Enter a number: key Quntity "))
#create connection
client = MongoClient()
#create emty list
alist = []
#make database_table
db = client.product.serial_key
keyrecord_record = {}
#find table existing entry that code
f_cursor = db.find({"b_p_code": x1})
for document in f_cursor:
alist.append(document['key'])
#print(document['key']) //print expected result
if(x1 == "" or y1==""):
print("please enter valid no")
else:
x = algo(x1,y1,alist)
keys =x.id_generator()
keys.sort(reverse=True)
print('\n')
i=0;
for x in keys:
i +=1;
print('key %d = ' %i , keys[i-1])
# place values in dictionary
keyrecord_record = {'key':keys[i-1],'b_p_code':x1}
#not recived expected result. if key in database again print key
if(db.serial_key.find({'key':{ '$ne':str(keys[i-1])}})):
db.insert(keyrecord_record)
else:
print('Record in DB')
print("\n")
print("Batch No: come from db ")
print('Generate Key beetween %d00000 - %d00000' % (x1 ,(x1+1)) )
print("Startin Serial : " , keys[0])
print("Ending Serial : " , keys[len(keys)-1])
print('\n Database Details \n')
#print details
cursor = db.find()
for document in cursor:
print(document['key'])
#print(document)
Image showing console out put.
From mongodb docs:
$ne operator selects the documents where the value of the field is not equal (i.e. !=) to the specified value. This includes documents that do not contain the field.
db.serial_key.find({'key':{ '$ne':str(keys[i-1])}}) will return a mongo cursor. Thus your if condition will always be true resulting in document addition regardless of the presence of the same value.
If you want to verify that the key already exists you can query the value and check the mongo cursor count.
The following code should do the trick.
for x in keys:
i +=1;
print('key %d = ' %i , keys[i-1]) #out put: key 1 = 3ZOHSH
# place values in dictionary
keyrecord_record = {'key':keys[i-1],'b_p_code':x1}
#check if already exists
if(db.find({'key':str(keys[i-1])}).limit(1).count() == 0):
db.insert(keyrecord_record)
else:
print('Record in DB')
Related
I am working on a Python script that reads data from a big excel file and stores the necessary data on corresponding dictionary variables.
Once all of the file's data is read and processed, the script iterates through all the dictionaries by 100 records and calls on a function that will insert them to a table using the executemany function.
The thing is, there are some records that are NOT being inserted to a specific table, which means there are more tables that are not inserting properly, even though I see the data is being passed.
Ex: record ID: IH1-01-01 is taken into consideration to be inserted but it doesn't in the end.
I don't see any fatal errors or anything that points to this mishap happening.
What else could be causing these records to not be inserted?
Here's part of the code for your review
#function that doesn't work properly
def InsertarAnaquelPisoLetraMasivo(queryAnaquel):
try:
query = "INSERT INTO Anaquel(anaq_id,anaq_nombre,anaq_piso, anaq_letra, anaq_cantidad,anaq_movimiento, anaq_almacen) VALUES (NULL,%s, %s, %s, %s, %s, 'Mex')"
cursor.executemany(query, queryAnaquel)
#added so I can see what data was being passed
#I can confirm all records are there but don't get inserted for some reason
for line in queryAnaquel:
print line
except:
print ("Error ", sys.exc_info()[0])
#iterate through excel file
for row in range(2, sheet.max_row + 1):
#...
#read item's location
ubicacion = sheet['H' + str(row)].value if sheet['H' + str(row)].value is not None else ""
if revisarSiLlaveEsConsiderada(ubicaciones, ubicacion):
ubicaciones[ubicacion] += 1
else:
ubicaciones[ubicacion] = 1
#part that processes data and calls for insert
print "Configurando anaqueles"
for key in ubicaciones:
ubicacionSplit = key.split("-")
tipoAnaquel = "Alto"
if len(ubicacionSplit) > 2:
tipoAnaquel = "Isla"
pis = ubicacionSplit[1]
let = ubicacionSplit[2]
elif len(ubicacionSplit)==1:
pis = ''
let = ''
else:
pisoLe = ubicacionSplit[1]
if len(pisoLe) >1:
if len(pisoLe) == 2:
pis = pisoLe[0]
let = pisoLe[1]
else:
pis = pisoLe
let = ''
else:
pis = pisoLe
let = ''
ubs = ubicacionSplit[0]
cantidad = ubicaciones[key]
detalleAnaquel = detalleUbicacion(ubs, pis, let)
if detalleAnaquel == "":
#Here's where to look
value = (ubs,pis, let, cantidad, tipoAnaquel)
queryAnaquel.append(value)
if cuentaQueryAnaquel == 100:
InsertarAnaquelPisoLetraMasivo(queryAnaquel)
cuentaQueryAnaquel = 1
queryAnaquel = []
else:
cuentaQueryAnaquel += 1
else:
idAnaquel = detalleAnaquel
updateAnaquel(idAnaquel, cantidad)
#In case we haven't reached 100 after iterating through all recordds
if len(queryAnaquel) > 0:
InsertarAnaquelPisoLetraMasivo(queryAnaquel)
queryAnaquel = []
cuentaQueryAnaquel=1
Table's config:
CREATE TABLE IF NOT EXISTS `Anaquel` (
`anaq_id` int(11) NOT NULL AUTO_INCREMENT,
`anaq_nombre` varchar(50) NOT NULL,
`anaq_movimiento` enum('Alto','Mediano','Bajo','Caja','Isla','') NOT NULL DEFAULT 'Bajo',
`anaq_piso` varchar(20) NOT NULL,
`anaq_letra` varchar(1) NOT NULL,
`anaq_cantidad` int(11) NOT NULL,
`anaq_capacidad` int(1) NOT NULL,
`anaq_almacen` enum('Mex','Usa') NOT NULL,
PRIMARY KEY (`anaq_id`),
KEY `anaq_nombre` (`anaq_nombre`)
) ENGINE=InnoDB AUTO_INCREMENT=26701 DEFAULT CHARSET=latin1;
Problem data, as was printed by command prompt:
(u'IH1', u'1', u'4', 38, 'Isla')
Edit: I previously removed warnings on the script and I just commented out the call on the warning removal.
I see a lot of warnings that say something like this:
C:\Python27\lib\site-packages\MySQLdb\cursors.py:206: Warning: Incorrect string value: '\xCC\x81s qu...' for column 'booName_title' at row 1
r = r + self.execute(query, a)
I am learning Python programming to build some automations, part of the script I am building iterates over object, I want to be able to search for the same items within a MySQL DB but not quite sure how to do so.
Using the MariaDB tutorial I have successfully retrieved the rows I want, however I'm not sure how I can search for an item within the cursor list without having to manually iterate through each result.
Here is the code I have so far:
cursor = mariadb_connection.cursor(dictionary=True)
# here I get a list of items from an external service
playlists = sp.user_playlists(username)
# here I retrieve my playlists from a DB
cursor.execute("SELECT id, name, monitor FROM playlists")
rows = cursor.fetchall()
while playlists:
for i, playlist in enumerate(playlists['items']):
print("%4d %s %s" % (i + 1 + playlists['offset'], playlist['id'], playlist['name']))
# while iterating through the playlists from the external service, I want to see if its included within my "rows" dictionary.
key = "1"
for row in rows:
print("ID: {}, Name: {}, Monitor: {}".format(row['id'],row['name'], row['monitor']))
# this does not work, there is an ID
if key in rows:
print("found key")
break
else:
print("NOT found key")
if playlists['next']:
playlists = sp.next(playlists)
else:
playlists = None
I have made comments around the area that I need some assistance from.
First, a few comments. In your code you set key to a constant "1", which never varies, and that can't be right. rows, which is returned from the query, will be a list dictionaries. The statement if key in rows: is checking a key value not against the keys for each dictionary but rather against each row, which is an entire dictionary, and that is not right. This should be if key == row['id']:. So a couple of changes should help:
cursor = mariadb_connection.cursor(dictionary=True)
# here I get a list of items from an external service
playlists = sp.user_playlists(username)
# here I retrieve my playlists from a DB
cursor.execute("SELECT id, name, monitor FROM playlists")
rows = cursor.fetchall()
# build a dictionary of keys:
id_dict = {row['id']: row for row in rows}
while playlists:
for i, playlist in enumerate(playlists['items']):
print("%4d %s %s" % (i + 1 + playlists['offset'], playlist['id'], playlist['name']))
# while iterating through the playlists from the external service, I want to see if its included within my "rows" dictionary.
key = playlist['id'] # the key
if key in id_dict:
row = id_dict[key]
print("ID: {}, Name: {}, Monitor: {}".format(row['id'],row['name'], row['monitor']))
print("found key")
else:
print("NOT found key")
if playlists['next']:
playlists = sp.next(playlists)
else:
playlists = None
Another, More Efficient Way
Don't query all the playlists from the database initially:
cursor = mariadb_connection.cursor(dictionary=True)
# here I get a list of items from an external service
playlists = sp.user_playlists(username)
# here I retrieve my playlists from a DB
while playlists:
for i, playlist in enumerate(playlists['items']):
print("%4d %s %s" % (i + 1 + playlists['offset'], playlist['id'], playlist['name']))
# while iterating through the playlists from the external service, I want to see if its included within my "rows" dictionary.
id = playlist['id']
cursor.execute("SELECT id, name, monitor FROM playlists where id = %s", (id,))
row = cursor.fetchone() # this the match, if any
if row:
print("ID: {}, Name: {}, Monitor: {}".format(row['id'],row['name'], row['monitor']))
print("found key")
else:
print("NOT found key")
if playlists['next']:
playlists = sp.next(playlists)
else:
playlists = None
So I want to compare the input value with my database value. IF the input value is the same value as the database, I want to print(inputvalue). But if it's not the same, I want to print("Data Does Not Exist")
So I've tried this code :
cur = connection.cursor()
query = """ SELECT * FROM `foo` """
cur.execute(query)
result = cur.fetchall()
inputvalue = input("Input= ")
for x in result:
if inputvalue not in x:
print("Data Does Not Exist")
else:
print(inputvalue)
and this is the output:
inputvalue= 1728192
Data Does Not Exist
Data Does Not Exist
Data Does Not Exist
Data Does Not Exist
1728192
Data Does Not Exist
Data Does Not Exist
Data Does Not Exist
I expect the output is
Inputvalue= 1728192
Data Does Not Exist
If the Data Does Not Exist,
And this output:
Inputvalue= 1728192
1728192
If the Data Exist
Any answer would be appreciated!
Instead of loading all the rows from foo into python, I would suggest to query if that specific value is in your database. Databases are optimised for such a request. And if there is a lot of data in your database, your approach would need a lot of time to load everything into memory (which can easily result in a memoryError).
So I guess you store your values in a column called 'bar':
inputvalue = input("Input= ")
cur = connection.cursor()
query = """ SELECT * FROM `foo` WHERE bar = inputvalue """
cur.execute(query)
row_count = cur.rowcount
if row_count > 0:
print(inputvalue)
else:
print("Data Does Not Exist")
And for you to understand why your approach is not working as expected: With for x in result: you loop through every row in your table, and for each row, you check if inputvalue is in there.
you can make boolean object and after executing compare your answer :
cur = connection.cursor()
query = """ SELECT * FROM `foo` """
cur.execute(query)
result = cur.fetchall()
inputvalue = input("Input= ")
temp = False
for x in result:
if inputvalue in x:
temp = True
if temp:
print(inputvalue)
else:
print("Data Does Not Exist")
how can I print unique code according to the loop if there is the same value?
FullChar = 'CEFLMPRTVWXYK0123456789'
total = 1000
count = 10
count = int(count)
entries = []
bcd = ""
flg = ""
rll = ""
try:
conn = psycopg2.connect(host="192.168.13.10",database="postgres", port="5432", user="postgres", password="potatona1")
cursor = conn.cursor()
def inputDatabase(data):
postgres_insert_query = """INSERT INTO unique_code(unique_code, barcode, flag, roll) VALUES (%s,%s,%s,%s)"""
cursor.executemany(postgres_insert_query, data)
conn.commit()
for i in range(5):
for x in range(total): # banyaknya code yang di print
unique_code = ''.join(random.sample(FullChar, count - 1))
unique_code = ''.join(random.sample(unique_code, len(unique_code)))
entry = (unique_code, bcd, flg, rll)
entries.append(entry)
inputDatabase(entries)
print(i)
count = cursor.rowcount
print (count, "Record inserted successfully into mobile table")
except (Exception, psycopg2.DatabaseError) as error:
print(error)
conn.rollback()
exmp :
if this code MTY9X4L2E show up again/ duplicate, the loop will stop
and i get this message
duplicate key value violates unique constraint "unique_code_pkey"
To keep track of unique values, use a set.
unique_codes = set()
...
for i in range(5):
for x in range(total): # banyaknya code yang di print
unique_code = ''.join(random.sample(FullChar, count - 1))
unique_code = ''.join(random.sample(unique_code, len(unique_code)))
if unique_code in unique_codes:
# The unique code has already been used.
# Do something?
else:
# Add the code to the set of used codes.
unique_codes.add(unique_code)
It's not very clear what those loops are doing; unique_code gets overwritten in every iteration of the inner loop.
The example code has another problem: the entries list is never cleared, so the second iteration of the outer loop will cause a duplicate key error because entries contains not only the new data but also the data from the previous iteration. entries should be cleared or reinitialised after each call to inputDatabase.
inputDatabase(entries)
entries.clear()
In an effort to learn SQL I've been using it at work. I keep a list of every account I work on and the information I gather while working on that particular account. My database has two tables that I update. One is the customer's information and it always gets updated with each new account and the other is an institution table that gets updated sometimes but not every time. These two tables are relational.
As I said, I am new to SQL and have been using the command line to update these tables and it's getting annoying. What I thought would be an easy solution is to run a series of Python prompts that queries the user for each column and then executes the INSERT command at the end of the prompt. It would then ask if I wanted to create an entry for the second table (since I don't always add to this one).
These are my tables.
> create table stu_info (
> id Integer,
> name text,
> CEEB integer,
> degree text,
> terms integer,
> comm text
> );
> create table coll_info (
> CEEB integer,
> name text,
> Accred text,
> Hours text,
> comm text
> );
In Python I figure it'd be easy to just use raw_input() and add an int() around it when required. Then use a loop so that after adding each new row to the database it starts over until I'm done for the day. My problem is I cannot figure out how to execute sqlite commands through Python and I can't figure out how to access the database.
Whenever I run
import sqlite3
conn = sqlite3.connect(stu.db)
c = conn.cursor()
enter code here
I get a NameError: name 'stu' is not defined
Any help? I imagine this is easy and my Google-fu is bad and inexperience has led even good search results to being useless...
It looks like it's as simple as needing quotes.
conn = sqlite3.connect("stu.db")
The error you're getting is because there is no variable stu in scope. So long as "stu.db" is in the working directory then the snippet above should work.
Once I found out, thanks to Jamie, that the filename needs to be in quotes I was able to connect to the database. I then was able to work my way through the problem. Here is my solution:
import sqlite3
conn = sqlite3.connect('stu.db')
c = conn.cursor()
def add_coll():
cceeb = int(raw_input("CEEB:> "))
cname = raw_input("Name:> ")
ccred = raw_input("Accred:> ")
ccal = raw_input("Calendar:> ")
ccomm = raw_input("Comments:> ")
c.execute("INSERT INTO coll VALUES (%d, '%s', '%s', '%s', '%s')" %
(cceeb, cname, ccred, ccal, ccomm))
var = 1 #This creates an infinite loop
while var == 1:
input_stu = raw_input("Add Student?:> ")
if input_stu == 'no' or input_stu == 'n':
add_coll()
else:
id = int(raw_input("ID:> "))
name = raw_input("Name:> ")
ceeb = int(raw_input("CEEB:> "))
deg = raw_input("Degree:> ")
terms = int(raw_input("Terms:> "))
comm = raw_input("Comments:> ")
c.execute("INSERT INTO stu VALUES (%d, '%s', %d, '%s', %d, '%s')" %
(id, name, ceeb, deg, terms, comm))
input_coll = raw_input("Add College?:> ")
if input_coll == 'yes' or input_coll == 'y':
#cceeb = int(raw_input("CEEB:> "))
#cname = raw_input("Name:> ")
#ccred = raw_input("Accred:> ")
#ccal = raw_input("Calendar:> ")
#
#c.execute("INSERT INTO coll VALUES (%d, '%s', '%s', '%s')" %
# (cceeb, cname, ccred, ccal))
add_coll()
conn.commit()