Develop Reference

How I get model data without primary key

I'm having trouble updating the data I get from my database. When I want to update directly from my model object, I get this error
arg = {k: v for k, v in kv_generator(self, arg.items())}
AttributeError: 'NoteORM' object has no attribute 'items'.
How do I get the data of my model excluding the primary key?
class NoteORM(Base):
__tablename__ = "Notes"
NoteID = Column(Integer, primary_key=True)
Reciver = Column(Integer, ForeignKey("Users.UserID"))
Note = Column(String, nullable=False)
FilePath = Column(Text)
IsSend = Column(String)
created_at = Column(DateTime, server_default=func.now())
updated_at = Column(DateTime, onupdate=func.now())
u_notes = relationship("UsersORM", backref="Notes")
# getting attirbute of object
def props(cls):
return [i for i in cls.__dict__.keys() if i[:1] != '_']
entity = session.query(NoteORM).get(11)
fields = props(entity) #I also get primary key :) when ı add underscore my primary key
#_NoteID also get error Invalid column name '_NoteID'
how can i update via object, is there a better way?

how can i update via object, is there a better way?
SQLAlchemy's Session.merge() method takes care of the details for you:
from sqlalchemy import Column, Integer, String, create_engine
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session, declarative_base
engine = create_engine("sqlite://")
Base = declarative_base()
class Note(Base):
__tablename__ = "note"
id = Column(Integer, primary_key=True, autoincrement=False)
note_text = Column(String, nullable=False)
Base.metadata.create_all(engine)
# create an existing "note 1"
with Session(engine) as sess:
sess.add(Note(id=1, note_text="original note 1 text"))
sess.commit()
# create a new Note object that we can use to update "note 1"
new_note_1 = Note(id=1, note_text="new note 1 text")
# simply adding new_note_1 to the session won't work
with Session(engine) as sess:
sess.add(new_note_1)
try:
sess.commit()
except IntegrityError as ie:
print(ie)
"""
(sqlite3.IntegrityError) UNIQUE constraint failed: note.id
[SQL: INSERT INTO note (id, note_text) VALUES (?, ?)]
[parameters: (1, 'new note 1 text')]
"""
# merging new_note_1 into the session works
engine.echo = True
with Session(engine) as sess:
sess.merge(new_note_1)
"""
SELECT note.id AS note_id, note.note_text AS note_note_text
FROM note
WHERE note.id = ?
[generated in 0.00072s] (1,)
"""
sess.commit()
"""
UPDATE note SET note_text=? WHERE note.id = ?
[generated in 0.00075s] ('new note 1 text', 1)
"""

Creating a join on multiple foreign keys using Sqlalchemy

I am trying to query my database using Sqlalchemy ORM methods. I have created the tables and the engine as well as testing raw sql against it. I want to be able to use a Location code as a parameter from the locations table and pull the origin / destination from the trip table. Here is the code below:
Base = declarative_base () # generated base class orm
class Trip(Base):
__tablename__="trips"
id = Column("id", Integer, primary_key=True, autoincrement=True)
route =Column("Route", String(25))
origin_id = Column("origin_id", String(10), ForeignKey("locations.locationCode"))
destination_id = Column("destination_id", String(10), ForeignKey("locations.locationCode"))
origin = relationship("Location", foreign_keys=[origin_id])
destination = relationship("Location", foreign_keys=[destination_id])
class Location(Base):
__tablename__ = "locations"
locationCode = Column("locationCode",String(10), primary_key = True)
latitude = Column("latitude", String(25))
longitude = Column("longitude", String(25))
facilityOwnedByCarvana = Column("facilityOwnedByCarvana",Integer)
engine = create_engine("sqlite:///carvana.db")
Session = sessionmaker(bind=engine)
session = Session()
Base.metadata.create_all(engine)
locationsDF = pd.read_csv("data/locations.csv")
tripsDf = pd.read_csv("data/trips.csv")
locationsDF.to_sql(con=engine, name=Location.__tablename__, if_exists="replace", index=False)
tripsDf.to_sql(con=engine, name=Trip.__tablename__,if_exists="replace", index=False)
Here is my attempt at the query
q = (
session.query(Location)
.outerjoin(Trip, Location.locationCode == Trip.destination_id)
.filter(Location.locationCode == "BALT")
.order_by(Location.locationCode)
.limit(10)
)

General:
db_session.query(class_table1)
.join(class_table2,
class_table2.key_table2 == class_table1.key_table1,
isouter=True)
Specific:
db_session.query(Location)
.join(Trip,
Trip.destination_id== Location.locationCode ,
isouter=True)

SQLAlchemy: dynamically apply filter on relationship column ahead of time

I have two SQLAlchemy classes representing a many-to-one relationship, say:
class Person
id = Column(Integer(), primary_key=True)
name = Column(String(30))
known_addresses = relationship('Address', backref='person')
class Address:
id = Column(Integer(), primary_key=True)
person_id = Column(Integer(), ForeignKey(Person.id, ondelete='cascade'))
city = Column(String(30))
zip_code = Column(String(10))
Now, say I have a function that returns a Person queryset (a Select object) filtered by zip codes:
def get_persons_in_zip_code(zip_code):
return session.query(Person).\
join(Address).\
where(Address.zip_code == zip_code)
Once I return the query set, I have no control over it and it is expected that this will encapsulate all the data that the framework I'm using (in my case Django/DRF) to render a list of persons along with their addresses (so the code iterates the query set, calling .addresses for each person and rendering that as well).
Here's the catch: I want to ensure that calling .addresses will return only the addresses matched in the original, zip_code filtered query - not all addresses related to that person.
Is there a way to achieve this in SQLAlchemy without accessing Person objects returned at later stages? That is, I can only modify my get_persons_in_zip_code function, or the original SQLAlchemy classes, but have no access to the Person objects returned from the query, as that happens deep inside the framework rendering code.
EDIT: It's also important that calling count() on the returned query object yields the number of expected Person objects, not the number of Address objects.

It seems like what you are looking for is contains_eager.
EDIT: An updated version that monkeypatches the .count() function to return only the distinct Person count.
from sqlalchemy import Integer, Column, String, ForeignKey
from sqlalchemy import create_engine, func, distinct
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import relationship, sessionmaker, contains_eager
from types import MethodType
engine = create_engine('sqlite:///:memory:', echo=True)
Session = sessionmaker(bind=engine)
session = Session()
Base = declarative_base()
class Person(Base):
__tablename__ = "person"
id = Column(Integer(), primary_key=True)
name = Column(String(30))
known_addresses = relationship('Address', backref='person')
def __repr__(self):
return "<Person {}>".format(self.name)
class Address(Base):
__tablename__ = "address"
id = Column(Integer(), primary_key=True)
person_id = Column(Integer(), ForeignKey(Person.id, ondelete='cascade'))
city = Column(String(30))
zip_code = Column(String(10))
def __repr__(self):
return "<Address {}>".format(self.zip_code)
Base.metadata.create_all(engine)
p1 = Person(name="P1")
session.add(p1)
p2 = Person(name="P2")
session.add(p2)
session.commit()
a1 = Address(person_id=p1.id, zip_code="123")
session.add(a1)
a2 = Address(person_id=p1.id, zip_code="345")
session.add(a2)
a3 = Address(person_id=p2.id, zip_code="123")
session.add(a3)
a4 = Address(person_id=p1.id, zip_code="123")
session.add(a4)
session.commit()
def get_persons_in_zip_code(zip_code):
return session.query(Person).\
join(Person.known_addresses).\
filter(Address.zip_code == zip_code).\
options(contains_eager(Person.known_addresses))
def distinct_person_count(q):
count_q = q.statement.with_only_columns([func.count(distinct(Person.id))])
return q.session.execute(count_q).scalar()
results = get_persons_in_zip_code("123")
results.count = MethodType(distinct_person_count, results)
print(results.count())
for person in results:
print(person)
for address in person.known_addresses:
print(address)
Output:
2
<Person P1>
<Address 123>
<Address 123>
<Person P2>
<Address 123>

Sanity check whether my SQLAlchemy class and query is functionally equivalent to a raw SQL statement

New to SQLALchemy, and this is my attempt at recreating this query:
SELECT
database_server_id || 'dr' database_server_id
, CASE
WHEN database_server_id LIKE 'cluster1' AND name IS NULL THEN 'canvas'
WHEN database_server_id LIKE 'cluster4' AND name IS NULL THEN 'canvas'
WHEN name IS NULL THEN database_server_id
ELSE name
END AS database_name
,SUBSTRING(database_server_id FROM '\d+$')::INT cluster_id
,CASE
WHEN name IS NULL THEN SUBSTRING(database_server_id FROM '\d+$')::INT
ELSE SUBSTRING(name FROM '\d+$')::INT
END AS shard_id
FROM canvas.switchman_shards
WHERE database_server_id LIKE 'cluster%'
ORDER BY cluster_id, shard_id
I've used declarative to create the class, and then used hybrid_properties to take care of the additional logic. Let me know what errors I have made!
class SwitchmanShards(Base):
__tablename__ = 'switchman_shards'
id = Column(Integer, primary_key=True)
database_server_id = Column(String)
name = Column(String, nullable=True)
default = Column(Boolean)
settings = Column(String)
delayed_jobs_shard_id = Column(Integer, nullable=True)
def __init__(self,
id,
db_server_id,
name,
default,
settings,
del_job_shard_id):
self.id = id
self.database_server_id = db_server_id
self.name = name
self.default = default
self.settings = settings
self.delayed_jobs_shard_id = del_job_shard_id
#hybrid_property
def cluster_id(self):
# Clusters that do not use their cluster name as the schema and need to
# be recast.
cluster_list = ('cluster1dr', 'cluster4dr')
if self.database_server_id in cluster_list and self.name is None:
return 'canvas'
elif self.name is None:
return self.database_server_id
else:
return self.name
#hybrid_property
def shard_id(self):
shard_regex = re.compile(r'(\d+$)')
if self.name is None:
res = shard_regex.search(self.database_server_id)
return res.group(1)
else:
res = shard_regex.search(self.name)
return res.group(1)
def return_shards():
Session = sessionmaker(bind=engine)
session = Session()
shards = session.query(SwitchmanShards).filter(
SwitchmanShards.database_server_id.like('cluster%')
).all()
return shards
As a side note, I'm trying to figure out the value of SQLALchemy due to the fact that it took me about 3 times as many lines to write the same code. Arguably it is more readable, that's the big plus I can see.
EDIT: I've noticed that this brings in 10 less records than the raw SQL query, not sure why this is.

Creating a query with few related tables in Pyramid with SQLAlchemy

I have defined few tables in Pyramid like this:
# coding: utf-8
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Integer, Float, DateTime, ForeignKey, ForeignKeyConstraint, String, Column
from sqlalchemy.orm import scoped_session, sessionmaker, relationship, backref,
from zope.sqlalchemy import ZopeTransactionExtension
DBSession = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
Base = declarative_base()
class Codes(Base):
__tablename__ = 'Code'
__table_args__ = {u'schema': 'Locations'}
id = Column(Integer, nullable=False)
code_str = Column(String(9), primary_key=True)
name = Column(String(100))
incoming = relationship(u'Voyages', primaryjoin='Voyage.call == Codes.code_str', backref=backref('Code'))
class Locations(Base):
__tablename__ = 'Location'
__table_args__ = {u'schema': 'Locations'}
unit_id = Column(ForeignKey(u'Structure.Definition.unit_id', ondelete=u'RESTRICT', onupdate=u'CASCADE'), primary_key=True, nullable=False)
timestamp = Column(DateTime, primary_key=True, nullable=False)
longitude = Column(Float)
latitude = Column(Float)
class Voyages(Base):
__tablename__ = 'Voyage'
__table_args__ = (ForeignKeyConstraint(['unit_id', 'Voyage_id'], [u'Locations.Voyages.unit_id', u'Locations.Voyages.voyage_id'], ondelete=u'RESTRICT', onupdate=u'CASCADE'), {u'schema': 'Locations'}
)
uid = Column(Integer, primary_key=True)
unit_id = Column(Integer)
voyage_id = Column(Integer)
departure = Column(ForeignKey(u'Locations.Code.code_str', ondelete=u'RESTRICT', onupdate=u'CASCADE'))
call = Column(ForeignKey(u'Locations.Code.code_str', ondelete=u'RESTRICT', onupdate=u'CASCADE'))
departure_date = Column(DateTime)
voyage_departure = relationship(u'Codes', primaryjoin='Voyage.departure == Codes.code_str')
voyage_call = relationship(u'Codes', primaryjoin='Voyage.call == Codes.code_str')
class Definitions(Base):
__tablename__ = 'Definition'
__table_args__ = {u'schema': 'Structure'}
unit_id = Column(Integer, primary_key=True)
name = Column(String(90))
type = Column(ForeignKey(u'Structure.Type.id', ondelete=u'RESTRICT', onupdate=u'CASCADE'))
locations = relationship(u'Locations', backref=backref('Definition'))
dimensions = relationship(u'Dimensions', backref=backref('Definition'))
types = relationship(u'Types', backref=backref('Definition'))
voyages = relationship(u'Voyages', backref=backref('Definition'))
class Dimensions(Base):
__tablename__ = 'Dimension'
__table_args__ = {u'schema': 'Structure'}
unit_id = Column(ForeignKey(u'Structure.Definition.unit_id', ondelete=u'RESTRICT', onupdate=u'CASCADE'), primary_key=True, nullable=False)
length = Column(Float)
class Types(Base):
__tablename__ = 'Type'
__table_args__ = {u'schema': 'Structure'}
id = Column(SmallInteger, primary_key=True)
type_name = Column(String(255))
type_description = Column(String(255))
What I am trying to do here is to find a specific row from Codes table (filter it by code_str) and get all related tables in return, but under the condition that Location table returns only the last row by timestamp, Voyage table must return only the last row by departure, and it must have all information from Definitions table.
I started to create a query from the scratch and came across something like this:
string_to_search = request.matchdict.get('code')
sub_dest = DBSession.query(func.max(Voyage.departure).label('latest_voyage_timestamp'), Voyage.unit_id, Voyage.call.label('destination_call')).\
filter(Voyage.call== string_to_search).\
group_by(Voyage.unit_id, Voyage.call).\
subquery()
query = DBSession.query(Codes, Voyage).\
join(sub_dest, sub_dest.c.destination_call == Codes.code_str).\
outerjoin(Voyage, sub_dest.c.latest_voyage_timestamp == Voyage.departure_date)
but I have notice that when I iterate through my results (like for code, voyage in query) I am actually iterating every Voyage I get in return. In theory it is not a big problem for me but I am trying to construct some json response with basic information from Codes table which would include all possible Voyages (if there is any at all).
For example:
code_data = {}
all_units = []
for code, voyage in query:
if code_data is not {}:
code_data = {
'code_id': code.id,
'code_str': code.code_str,
'code_name': code.name,
}
single_unit = {
'unit_id': voyage.unit_id,
'unit_departure': str(voyage.departure_date) if voyage.departure_date else None,
}
all_units.append(single_unit)
return {
'code_data': exception.message if exception else code_data,
'voyages': exception.message if exception else all_units,
}
Now, this seems a bit wrong because I don't like rewriting this code_data in each loop, so I put if code_data is not {} line here, but I suppose it would be much better (logical) to iterate in a way similar to this:
for code in query:
code_data = {
'code_id': code.id,
'code_str': code.code_str,
'code_name': code.name,
}
for voyage in code.voyages:
single_unit = {
'unit_id': voyage.unit_id,
'unit_departure': str(voyage.departure) if voyage.departure else None,
}
all_units.append(single_unit)
return {
'code_data': exception.message if exception else code_data,
}
So, to get only single Code in return (since I queried the db for that specific Code) which would then have all Voyages related to it as a nested value, and of course, in each Voyage all other information related to Definition of the particular Unit...
Is my approach good at all in the first place, and how could I construct my query in order to iterate it in this second way?
I'm using Python 2.7.6, SQLAlchemy 0.9.7 and Pyramid 1.5.1 with Postgres database.
Thanks!

Try changing the outer query like so:
query = DBSession.query(Codes).options(contains_eager('incoming')).\
join(sub_dest, sub_dest.c.destination_call == Codes.code_str).\
outerjoin(Voyage, sub_dest.c.latest_voyage_timestamp == Voyage.departure_date)
In case of problems, try calling the options(...) part like so:
(...) .options(contains_eager(Codes.incoming)). (...)
This should result in a single Codes instance being returned with Voyages objects accessible via the relationship you've defined (incoming), so you could proceed with:
results = query.all()
for code in results:
print code
# do something with code.incoming
# actually, you should get only one code so if it proves to work, you should
# use query.one() so that in case something else than a single Code is returned,
# an exception is thrown
of course you need an import, e.g.: from sqlalchemy.orm import contains_eager