I'm trying to use the document storage of MySQL 8 in my python project(python 3.8). The version of MySQL-connector python is 8.0.20. According to the API reference and the X DevAPI User Guide, I tried to get the auto increment document ID after adding a document into the DB. Each time after the execution, the data would be inserted into DB successfully, but '-1' would be returned after get_autoincrement_value() was invoked.
My code is just like below:
try:
schema = session.get_schema('my_schema')
collection = schema.get_collection('my_collection')
topic_dict = protobuf_to_dict(topic)
doc_id = collection.add(topic_dict).execute().get_autoincrement_value()
logger.debug('doc_id: {}', doc_id)
return doc_id
except Exception as e:
logger.exception("failed to add topic to db, topic: {}, err: {}", topic, e)
Is there anything wrong with my usage? Thank you all~
Seems like you are interested in the document id that has been auto-generated. If that is the case, you should instead use get_generated_ids:
doc_id = collection.add(topic_dict).execute().get_generated_ids()[0]
In this case, the method returns a list of all the ids that were generated in the scope of the add() operation.
The documentation is probably not clear enough, but get_auto_increment_value() only contains useful data if you are inserting a row with either session.sql() or table.insert() on a table containing an AUTO_INCREMENT column. It has no meaning in the scope of NoSQL collections because in the end a collection is just a table created like (condensed version):
CREATE TABLE collection (
`doc` json DEFAULT NULL,
`_id` varbinary(32),
PRIMARY KEY (`_id`)
)
Which means there isn't anything to "auto increment".
Disclaimer: I'm the lead developer of the MySQL X DevAPI Connector for Node.js
Related
I currently am using SQL Alchemy Core specifically with the SQL Expression Language.
I have a table that is currently using the GENERATED ALWAYS AS IDENTITY parameter.
CREATE TABLE mytable(id INT PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
col1 VARCHAR(100),col2 VARCHAR(100));
Everytime I try insert in the table, i'm getting the error:
DETAIL: Column "id" is an identity column defined as GENERATED ALWAYS.
HINT: Use OVERRIDING SYSTEM VALUE to override.
I know that if I just to use postgres I could:
INSERT INTO mytable (id,col1,col2) OVERRIDING SYSTEM VALUE
VALUES (%s,%s,%s) ON CONFLICT (id) DO NOTHING;
But how would do this using the sql expression language that sqlalchemy provides?
I am currently upserting like this:
insert_stmt = postgresql.insert(target).values(vals)
primary_keys = [key.name for key in inspect(target).primary_key]
stmt = insert_stmt.on_conflict_do_nothing(index_elements=primary_keys)
conn.execute(stmt)
I wanted OVERRIDING SYSTEM VALUE to use fixed IDs in my tests.
As far as I can see, SQLAlchemy doesn't support this at the moment.
I hacked it in this way:
#compiles(Insert)
def set_inserts_overriding_system_value(the_insert, compiler, **kw):
text = compiler.visit_insert(the_insert, **kw)
text = text.replace(") VALUES (", ") OVERRIDING SYSTEM VALUE VALUES (")
return text
You can probably create some weird tables or insert queries on purpose, that will be messed up by this text replace. But it won't ever happen by accident.
This question is a bit related to another question: Get List of Primary Key Columns in Snowflake.
Since INFORMATION_SCHEMA.COLUMNS does not provide the required information regarding the primary keys. And the method proposed by Snowflake itself, where you would describe the table followed by a result_scan, is unreliable when queries are run in parallel.
I was thinking about using SHOW PRIMARY KEYs IN DATABASE. This works great when querying the database from within Snowflake. But as soon as I try to do this in python, I get results for the column name like 'Built-in function id'. Which is not useful when dynamically generating sql statements.
The code I am using is as follows:
SQL_PK = "SHOW PRIMARY KEYS IN DATABASE;"
snowflake_service = SnowflakeService(username=cred["username"], password=cred["password"])
snowflake_service.connect(database=DATABASE,role=ROLE, warehouse=WAREHOUSE)
curs = snowflake_service.cursor
primary_keys = curs.execute(SQL_PK).fetchall()
curs.close()
snowflake_service.connection.close()
Is there something I am doing wrong? Is it even possible to do it like this?
Or is the solution that Snowflake provides reliable enough, when sending these queries as one string? Although with many tables, there will be many round trips required to get all the data needed.
where you would describe the table followed by a result_scan, is unreliable when queries are run in parallel
You could search for specific query run using information_schema.query_history_by_session and then refer to resultset using retrieved QUERY_ID.
SHOW PRIMARY KEYS IN DATABASE;
-- find the newest occurence of `SHOW PRIMARY KEYS`:
SET queryId = (SELECT QUERY_ID
FROM TABLE(information_schema.query_history_by_session())
WHERE QUERY_TEXT LIKE '%SHOW PRIMARY KEYS IN DATABASE%'
ORDER BY ENDTIME DESC LIMIT 1);
SELECT * FROM TABLE(RESULT_SCAN($queryId));
I am using Python and I would like to have a list of IDs stored in disk preserving some of the functionalities of a set (that is, efficiently checking if an ID is contained). To this end, I think using SQLite library is a wise decision (at least that is my impression after googling and stacking a bit). However, I am a beginner in SQL world and could not find any post explaining what I am looking for.
How can I store IDs (strings) in SQLite and later check if a specific ID appears or not in the database?
import sqlite3
id1 = 'abc'
id2 = 'def'
# Initialization of the database
define_database()
# Update the database by inserting a new ID
insert_in_database(id1)
insert_in_database(id2)
# Check if the specified ID is contained in the database (returns a Boolean)
check_if_exists_in_database(id1)
PS: I am aware of the sqlite3 library.
Thanks!
Just use a table with a single column. This column must be indexed (explicitly, or by making it the primary key) for lookups over large data to be efficient:
db = sqlite3.connect('...filename...')
def define_database():
db.execute('CREATE TABLE IF NOT EXISTS MyStuff(id PRIMARY KEY)')
(Use a WITHOUT ROWID table if your Python version is recent enough to have a modern version of the SQLite library.)
Inserting is done with standard SQL:
def insert_in_database(value):
db.execute('INSERT INTO MyStuff(id) VALUES(?)', [value])
To check whether a value exists, just try to read its row:
def check_if_exists_in_database(value):
for row in db.execute('SELECT 1 FROM MyStuff WHERE id = ?', [value])
return True
else:
return False
I'm trying to figure out if it's possible to replace record values in a Microsoft Access (either .accdb or .mdb) database using pyodbc. I've poured over the documentation and noted where it says that "Row Values Can Be Replaced" but I have not been able to make it work.
More specifically, I'm attempting to replace a row value from a python variable. I've tried:
setting the connection autocommit to "True"
made sure that it's not a data type issue
Here is a snippet of the code where I'm executing a SQL query, using fetchone() to grab just one record (I know with this script the query is only returning one record), then I am grabbing the existing value for a field (the field position integer is stored in the z variable), and then am getting the new value I want to write to the field by accessing it from an existing python dictionary created in the script.
pSQL = "SELECT * FROM %s WHERE %s = '%s'" % (reviewTBL, newID, basinID)
cursor.execute(pSQL)
record = cursor.fetchone()
if record:
oldVal = record[z]
val = codeCrosswalk[oldVal]
record[z] = val
I've tried everything I can think bit cannot get it to work. Am I just misunderstanding the help documentation?
The script runs successfully but the newly assigned value never seems to commit. I even tried putting "print str(record[z])this after the record[z] = val line to see if the field in the table has the new value and the new value would print like it worked...but then if I check in the table after the script has finished the old values are still in the table field.
Much appreciate any insight into this...I was hoping this would work like how using VBA in MS Access databases you can use an ADO Recordset to loop through records in a table and assign values to a field from a variable.
thanks,
Tom
The "Row values can be replaced" from the pyodbc documentation refers to the fact that you can modify the values on the returned row objects, for example to perform some cleanup or conversion before you start using them. It does not mean that these changes will automatically be persisted in the database. You will have to use sql UPDATE statements for that.
I'm using PyGreSQL to access my DB. In the use-case I'm currently working on; I am trying to insert a record into a table and return the last rowid... aka the value that the DB created for my ID field:
create table job_runners (
id SERIAL PRIMARY KEY,
hostname varchar(100) not null,
is_available boolean default FALSE
);
sql = "insert into job_runners (hostname) values ('localhost')"
When I used the db.insert(), which made the most sense, I received an "AttributeError". And when I tried db.query(sql) I get nothing but an OID.
Q: Using PyGreSQL what is the best way to insert records and return the value of the ID field without doing any additional reads or queries?
INSERT INTO job_runners
(hostname,is_available) VALUES ('localhost',true)
RETURNING id
That said, I have no idea about pygresql, but by what you've already written, I guess it's db.query() that you want to use here.
The documentation in PyGreSQL says that if you call dbconn.query() with and insert/update statement that it will return the OID. It goes on to say something about lists of OIDs when there are multiple rows involved.
First of all; I found that the OID features did not work. I suppose knowing the version numbers of the libs and tools would have helped, however, I was not trying to return the OID.
Finally; by appending "returning id", as suggested by #hacker, pygresql simply did the right thing and returned a record-set with the ID in the resulting dictionary (see code below).
sql = "insert into job_runners (hostname) values ('localhost') returning id"
rv = dbconn.query(sql)
id = rv.dictresult()[0]['id']
Assuming you have a cursor object cur:
cur.execute("INSERT INTO job_runners (hostname) VALUES (%(hostname)s) RETURNING id",
{'hostname': 'localhost'})
id = cur.fetchone()[0]
This ensures PyGreSQL correctly escapes the input string, preventing SQL injection.