I have an association object defined using SQLAlchemy to represent a many-to-many relationship between 2 tables. The reason I am using the association object pattern is because the association table contains extra columns. I have a unique constraint on the name column in the data_type table. When I try to insert data into source_key, and create the relationships, it results in the error below. My question is, how would I say, "Get the ID if it exists and add to association table; otherwise, create the record in data_type, then add to association table"?
error
the-librarian-backend-1 | sqlalchemy.exc.IntegrityError: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "ix_data_type_name"
the-librarian-backend-1 | DETAIL: Key (name)=(str) already exists.
the-librarian-backend-1 |
the-librarian-backend-1 | [SQL: INSERT INTO data_type (name) VALUES (%(name)s) RETURNING data_type.id]
the-librarian-backend-1 | [parameters: ({'name': 'str'}, {'name': 'str'}, {'name': 'str'}, {'name': 'str'}, {'name': 'str'}, {'name': 'date'}, {'name': 'list'}, {'name': 'int'} ... displaying 10 of 747 total bound parameter sets ... {'name': 'date'}, {'name': 'str'})]
models
# source_key.py
class SourceKey(Base):
__tablename__ = 'source_key'
id = Column(Integer, primary_key=True, index=True)
source_id = Column(Integer, ForeignKey('source.id'), nullable=False)
key_id = Column(Integer, ForeignKey('key.id'), nullable=False)
description = Column(Text)
data_types = relationship("SourceKeyDataType", back_populates="source_keys")
# data_type.py
class DataType(Base):
__tablename__ = 'data_type'
id = Column(Integer, primary_key=True, index=True)
name = Column(Text, index=True, nullable=False, unique=True)
source_keys = relationship("SourceKeyDataType", back_populates="data_types")
# Association Object
class SourceKeyDataType(Base):
__tablename__ = 'source_key_data_type_assoc'
source_key_id = Column(ForeignKey('source_key.id'), primary_key=True)
data_type_id = Column(ForeignKey('data_type.id'), primary_key=True)
count = Column(BigInteger)
source_keys = relationship("SourceKey", back_populates="data_types")
data_types = relationship("DataType", back_populates="source_keys")
code
source_keys = [
{
"key": {
"name": "total"
},
"description": "the total cost of all items",
"data_types": [
{
"name": "str",
"count": 1904165
}
]
},
{
"key": {
"name": "item_value"
},
"description": "the cost of a single item",
"data_types": [
{
"name": "str",
"count": 2079817
}
]
}
]
for source_key in source_keys:
source_key_obj = {k: v for k, v in item.items() if isinstance(v, (str, int, bool, float))}
source_key_db_obj = SourceKey(**source_key_obj)
for dt in source_key.get("data_types") or []:
a = SourceKeyDataType(is_inferred=item.get("is_inferred", False), count=item.get("count", 0))
a.data_types = models.DataType(name=item["name"])
source_key_db_obj.data_types.append(a)
db.add(source_key_db_obj)
db.commit()
db.refresh(source_key_db_obj)
My question is, how would I say, "Get the ID if it exists and add to association table; otherwise, create the record in data_type, then add to association table"?
Your code needs to do exactly that. Let's look at a simplified example that uses an association table instead of an association object. To set up the test:
from sqlalchemy import Column, create_engine, ForeignKey, Integer, select, String, Table
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import declarative_base, relationship, Session
engine = create_engine("sqlite://")
Base = declarative_base()
post_tag = Table(
"post_tag",
Base.metadata,
Column("post_id", Integer, ForeignKey("post.id"), primary_key=True),
Column("tag_id", Integer, ForeignKey("tag.id"), primary_key=True),
)
class Post(Base):
__tablename__ = "post"
id = Column(Integer, primary_key=True)
title = Column(String)
tags = relationship("Tag", secondary=post_tag)
class Tag(Base):
__tablename__ = "tag"
id = Column(Integer, primary_key=True)
name = Column(String, unique=True)
Base.metadata.create_all(engine)
# add test data into empty tables
with Session(engine) as sess:
sess.add(
Post(
title="getting unique constraint violation",
tags=[Tag(name="SQLAlchemy")],
)
)
sess.commit()
First let's try adding a new Post the simplistic way:
# 1st try: adding a new post and blindly creating a new Tag object
with Session(engine) as sess:
sess.add(
Post(
title="some other issue",
tags=[Tag(name="SQLAlchemy")],
)
)
try:
sess.commit()
except IntegrityError:
print("1st try: An IntegrityError has occurred")
# this error gets printed
Now let's check to see if the Tag already exists:
# 2nd try: check for existing tag first
with Session(engine) as sess:
sqla_tag = sess.scalars(
select(Tag).where(Tag.name == "SQLAlchemy")
).first()
if not sqla_tag:
# Tag object does not already exist, so create it
sqla_tag = Tag(name="SQLAlchemy")
sess.add(
Post(
title="some other issue",
tags=[sqla_tag],
)
)
sess.commit()
print("2nd try: Success.")
# no error this time
That's the most straightforward solution. A more advanced technique is to use an association proxy to automate the process, but some users find them difficult to work with.
Below is the model named 'Dataset' containing three fields name, desc and data_file.
class Dataset(models.Model):
name = models.CharField(max_length=256)
desc = models.TextField()
data_file = models.FileField(upload_to='datasets/')
I created a model object with python command.
>>> d = Dataset()
>>> d.save()
>>> d.name, d.desc, d.data_file
('', '', <FieldFile: None>)
Django allowed this object to be saved. Even when blank = False is the default for every field.
How can I may sure that dataset objects cannot be created with these three fields empty ?
Below is the sqlite3 schema:
CREATE TABLE IF NOT EXISTS "datasets_dataset"(
"id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
"name" varchar(256) NOT NULL,
"data_file" varchar(100) NOT NULL,
"desc" text NOT NULL
);
I am building a flask application over an already existing database so there was no need to declare the whole models fully. I have this table:
class Users(db.Model):
__tablename__ = 'my_users'
__table_args__ = {
'autoload': True,
'autoload_with': db.engine
}
the table has about 10 columns but when i query the data i can see that the attribute:
.__dict__
only returns the first 4 columns. i have tried using filter and also filter by but data returned only contains the first 4 columns. Here is my query:
users = Users.query.filter(
section_serial == sectionserial
).all()
I am using the postgres database. Here is a minimal example:
CREATE TABLE public.my_users
(
user_serial integer NOT NULL DEFAULT nextval('my_users_seq'::regclass),
user_name character varying(16) NOT NULL,
user_password character varying(42) NOT NULL,
id_number character varying(155) NOT NULL,
date_added timestamp without time zone NOT NULL DEFAULT now(),
is_enabled boolean NOT NULL DEFAULT true,
expiry_date date NOT NULL DEFAULT (('now'::text)::date + 30),
phone_number character varying(254),
notes text,
section_serial integer,
full_name character varying(155) NOT NULL,
zip_code boolean NOT NULL DEFAULT false,
CONSTRAINT user_serial_pkey PRIMARY KEY (user_serial)
);
After querying the data i only get user_serial, user_name, user_password and id_number. I cannot get the rest of the columns
The problem was it was conflicting with a login model i had created though with a different name. I think models should just be declared once.
class SystemUsers(db.Model):
__tablename__ = 'my_users'
userserial = db.Column(
'user_serial', db.Integer, primary_key=True)
username = db.Column('user_name ', db.String)
password= db.Column('user_password ', db.String)
idnumber = db.Column('id_number', db.String)
isactive = True
isanonymous = False
authenticated = False
I'm trying to insert a JSON string into a snowflake table defined below:
from snowflake.sqlalchemy import VARIANT
class Identify(Base):
__tablename__ = 'identify'
event_id = Column(Integer, Sequence('event_id_seq'), unique=True, primary_key=True)
user_id = Column(String, nullable=False)
traits = Column(VARIANT, nullable=False)
The json_str that I'm trying to insert is:
json_str = '''
{
"address": "xyz",
"age": 32,
"avatar": "xyz",
"birthday": {
"seconds": 20,
"nanos": 10
}
}
'''
I'm using the following code to insert the json_str into the table
session = Session()
obj = Identify(user_id=request.user_id,
traits=json_str)
session.add(obj)
session.commit()
session.close()
Which yields the following error: snowflake.connector.errors.ProgrammingError) 002023 (22000): SQL compilation error: Expression type does not match column data type, expecting VARIANT but got VARCHAR(3038) for column TRAITS
Is there a way to insert the json_str or a dict without writing an SQL insert statement that uses the TO_VARIANT conversion function?
Is that valid JSON? Looks to me like you have unmatched brackets...
-Paul-
The API doc says VARIANT is supported for fetching data by converting to a string.
I've found no other way to store data than an INSERT converting STRING to VARIANT.
I am using sqlalchemy 0.7 and MySQL server version 5.1.63.
I have the following table on my database:
CREATE TABLE `g_domains` (
`id` BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT,
`name` VARCHAR(255) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE INDEX `name` (`name`)
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
The corresponding model is :
class GDomain(Base):
__tablename__ = 'g_domains'
__table_args__ = {
'mysql_engine': 'InnoDB',
'mysql_charset': 'utf8',
'mysql_collate': 'utf8_general_ci'
}
id = Column(mysql.BIGINT(unsigned=True), primary_key=True)
name = Column(mysql.VARCHAR(255, collation='utf8_general_ci'),
nullable=False, unique=True)
The following query in sql alchemy returns no rows :
session.query(GDomain).filter(GDomain.name.in_(domain_set)).
limit(len(domain_set)).all()
where domain_set is a python list containing some domain names like
domain_set = ['www.google.com', 'www.yahoo.com', 'www.AMAZON.com']
Although the table has a row (1, www.amazon.com) the above query returns only
(www.google.com, www.yahoo.com).
When I run the sql query :
SELECT * FROM g_domains
WHERE name IN ('www.google.com', 'www.yahoo.com', 'www.AMAZON.com')
Do you have an idea why this is happening?
Thanks in advance
What is the model_domain variable? Usually it looks like this:
session.query(GDomain).filter(GDomain.name.in_(domain_set)).
limit(len(domain_set)).all()
Note that the GDomain is used in both places. Alternatively you can use aliases:
domains = orm.aliased(GDomain, name='domain')
session.query(domains).filter(domains.name.in_(domain_set))
You can always try debugging, print the query that produced by sqlalchemy (see: SQLAlchemy: print the actual query)