Develop Reference

How to fix 'Invalid input syntax for integer' issue while using sqlalchemy Declarative API

I'm actually building a little utils which aims to take flat csv/excel file and populate a target database on MS Access - as I'm working on a Mac, i'm developping it using Postgres...
So I developped a part which deals with messy input (csv/excel) forms (several heading, etc) but that's not my issue at the moment.
On the other hand, I made my Database model using SQLAlchemy Declarative Base API.
I'm facing issue when importing data in some tables:
- Split flat record to several objects
- Check (SELECT) if the record doesn't exists yet based on uniqueness contraints
- If it doesn't exists I create object else I use the existing one
- Propagate keys information to related object
For some tables I'm using the auto_increment arguments but sometimes the record has its own ID (in input file) so I should it for insert/select in my tables and sometimes no ID so I have to create a new technical Id for my table.
Example: I have a record with for primary key -obsr25644- and sometimes nothing so I use a default value created with uuid.
So below the stacktrace when doing selectoperation on a my table. The same error occurs when working on existing data - obsr25644 - and generated uuid - 'a8098c1a-f86e-11da-bd1a-00112444be1e'
sqlalchemy.exc.DataError: (psycopg2.errors.InvalidTextRepresentation) **invalid input syntax for integer**: "obsr25644"
LINE 3: WHERE "Location"."Id_observer" = 'obsr25644'
As you can see below, "Location"."Id_observer" is declared as String(255). I don't understand why the error is related to 'integer'.
[SQL: SELECT "Location"."Id_location" AS "Location_Id_location", [...], "Location"."Id_observer" AS "Location_Id_observer",
FROM "Location"
WHERE "Location"."Id_observer" = %(Id_observer_1)s
LIMIT %(param_1)s]
[parameters: {'Id_observer_1': 'obsr25644', 'param_1': 1}]
class LocationModel(UniqueMixin, Base):
__tablename__ = 'Location'
# Primary key
Id_location = Column(Integer, primary_key=True, autoincrement=True)
[...]
Id_observer = Column(String(255), ForeignKey('Observer.Id_observer'))
observer = relationship("ObserverModel", load_on_pending=True, back_populates="location")
class ObserverModel(UniqueMixin, Base):
__tablename__ = 'Observer'
# Primary key
Id_observer = Column(String(255), primary_key=True, default=UniqueMixin.unique_hash())
[...]
# Relationship
location = relationship("LocationModel", load_on_pending=True, back_populates="observer")
Note :UniqueMixin.unique_hash() returns uuid.uuid4().hex

Create many to many on one table

Flask-SQLAlchemy gives an example of how to create a many to many relationship. It is done between two different tables.
Is it possible to create a many to many relationship on the same table? For example a sister can have many sisters, who would also have many sisters. I have tried:
girl_sister_map = db.Table('girl_sister_map',
db.Column('girl_id',
db.Integer,
db.ForeignKey('girl.id')),
db.Column('sister_id',
db.Integer,
db.ForeignKey('girl.id')))
class Girl(db.Model):
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String)
sisters = db.relationship('Girl',
secondary=girl_sister_map,
backref=db.backref('othersisters', lazy='dynamic'))
But when I try to add a sister to a girl I get:
sqlalchemy.exc.AmbiguousForeignKeysError: Could not determine join condition between parent/child tables on relationship Girl.sisters - there are multiple foreign key paths linking the tables via secondary table 'girl_sister_map'. Specify the 'foreign_keys' argument, providing a list of those columns which should be counted as containing a foreign key reference from the secondary table to each of the parent and child tables.
Is this possible? How should I be doing it?

You are trying to build what is called an adjacency list. That is you have a table with foreign key to itself.
In your specific case it is a self referencial many to many relationship.
This is supported in SQLAlchemy as you will discover by following the previous link. The doc contains several examples.
Basically, you will need the primaryjoin and secondaryjoin arguments to establish how you would like to join the table. Straight from the doc:
Base = declarative_base()
node_to_node = Table("node_to_node", Base.metadata,
Column("left_node_id", Integer, ForeignKey("node.id"), primary_key=True),
Column("right_node_id", Integer, ForeignKey("node.id"), primary_key=True)
)
class Node(Base):
__tablename__ = 'node'
id = Column(Integer, primary_key=True)
label = Column(String)
right_nodes = relationship("Node",
secondary=node_to_node,
primaryjoin=id==node_to_node.c.left_node_id,
secondaryjoin=id==node_to_node.c.right_node_id,
backref="left_nodes"
)

How to achieve that every record in two tables have different id?

I have two tables inherited from base table ( SQLALCHEMY models)
class Base(object):
def __tablename__(self):
return self.__name__.lower()
id = Column(Integer, primary_key=True, nullable=False)
utc_time = Column(Integer, default=utc_time(), onupdate=utc_time())
datetime = Column(TIMESTAMP, server_default=func.now(), onupdate=func.current_timestamp())
and inherited tables Person and Data
How to achieve that every Person and Data have different id, every id to be unique for two tables ? ( Person when generate of id to be aware of Data ids and vice versa)

if you're using Postgresql, Firebird, or Oracle, use a sequence that's independent of both tables to generate primary key values. Otherwise, you need to roll some manual process like an "id" table or something like that - this can be tricky to do atomically.
Basically if I were given this problem, I'd ask why exactly would two tables need unique primary key values like that - if the primary key is an autoincrementing integer, that indicates it's meaningless. It's only purpose is to provide a unique key into a single table.

SQLAlchemy one-to-many without the child table having a primary key

Is it possible to create a table without a primary key in SQLAlchemy? The relationship I want to define is as follows:
class TPost(Base):
__tablename__ = "forum_post"
id = Column(Integer, primary_key = True)
topic_id = Column(Integer, ForeignKey("forum_topic.id"))
index = Column(Integer)
page = Column(Integer)
user_id = Column(Integer, ForeignKey("forum_user.id"))
posted_at = Column(DateTime)
post_text = Column(String)
has_quotes = Column(Boolean)
quotes = relationship("TQuote")
class TQuote(Base):
__tablename__ = "forum_quotes"
id = Column(Integer, ForeignKey("forum_post.id"))
is_direct = Column(Boolean)
quoted_text = Column(String)
quoted_id = Column(Integer)
As you can see I don't really need a primary key, and I don't intend to extend the Quote relationship in the future.
My problem specifically is represented by this error message :
sqlalchemy.exc.ArgumentError: Mapper Mapper|TQuote|forum_quotes
could not assemble any primary key columns for mapped table 'forum_quotes'
edit :
The (id,quoted_id) pair is unique, and it present for the majority of the data, however when the quote is not direct(and doesn't have a quoted_id in that case), I inline the quoted text directly into the quote relationship. I could use a dual table approach (where the indrect quotes have a table with a primary key), but I'd really rather implement this as a single one-to-many relationship. I don't want to have to do more than a single join.
edit 2:
I'll number the quotes and use the foreign-key + app generated number as a pkey, still annoying tho. Now to figure out the syntax.
edit 3:
Solved the problem as outlined in edit 2. Quite annoyed with sql alchemy since it has all the information it needs to implement the relatioship even when modelling the data at a high level. I understand the reasons why Sql Alchemy wants to have a primary key (makes the orm easier to implement).
I am beginning to question why I am using Sql Alchemy, without it I could implement one way UPSERT or CREATE_IF_NOT_EXIST asynchronous operations using psycopg2. ORM's really need to catch up.

I am assuming #TokenMacGuy is right, and you really are confusing the notions of PrimaryKey, and a surrogate key. In which case the answer to your question is:
NO, SA does not support tables (and therefore relations to tables) without a primary key
and NO, you do not need to create a surrogate key for each table for the purpose of serving as a primary key. You can define a PK using any combination of columns with are unique.
See the code below for an example:
class TPost(Base):
__tablename__ = 'forum_post'
id = Column(Integer, primary_key = True)
post_text = Column(String)
quotes = relationship("TQuote", backref="post")
class TQuote(Base):
__tablename__ = "forum_quotes"
id = Column(Integer, ForeignKey("forum_post.id"))
is_direct = Column(Boolean)
quoted_text = Column(String)
quoted_id = Column(Integer)
__table_args__ = (PrimaryKeyConstraint(id, quoted_id),)

Add an additional column to give the quotes a index, and then add make a composite key of this new column + the foreign key.

Modeling a database many to many relationship between three tables using sqlalchemy in python

Let's consider the following table models for sqlalchemy in python.
class Worker(Base):
id Column(Integer, primary = True)
name Column(String, nullable = False)
class Project(Base):
id Column(Integer, primary = True)
name Column(String, nullable = False)
class Rating(Base):
id Column(Integer, primary = True)
description Column(String, nullable = False)
Let's consider some restrictions on these tables. (Don't question the sense of this example please, it is the best I came up with to describe what I'd like to learn. :-))
A worker may have multiple projects to work on.
A project may have multiple workers assigned to work on it.
For each tuple (worker, project) you have a distinct rating
I understand, that I have three many to many relationship. Those are:
worker and project
worker and rating
rating and project
I have no problems defining a many to many relationship between two tables. I just would have to add the folowing association table and a realtionship to my models.
worker_project = Table('worker_project', METADATA,
Column('worker_id', Integer, ForeignKey('workers.id')),
Column('project_id', Integer, ForeignKey('projects.id')),
class Worker(Base):
# [...]
papers = relationship('Project',
secondary=worker_project,
backref='workers')
What I do fail in, is to link the table "ratings" like described above. Any help is appreciated.

Make the ratings table something like this.
Rating
---------------------
rating_id (primary key)
worker_id (foreign key - worker(worker_id))
project_id (foreign key - project(project_id))
description
If I'm understanding this correctly that is.