I'm trying to set up a database ORM with peewee and am not clear on the use of foreign key relationships.
from peewee import *
db = SqliteDatabase('datab.db')
class datab(Model):
class Meta:
database = db
class Collection(datab):
identifier = CharField()
title = CharField()
class File(datab):
identifier = ForeignKeyField(Collection, related_name='files')
name = CharField()
Later, I do an import of "Collections"
for value in collection:
Collection(**value).save()
Finally, where I am having trouble is adding the Files to the collections
for value in collectionFiles:
File(**value).save()
Within the value dict, there is a keyword pair with key of "identifier" and a value that should associate with the Collection identifier keyword.
However I get an error message:
ValueError: invalid literal for int() with base 10: 'somevalue'
If I change the File(datab): identifier Type to VarChar, it saves the data.
I'm realizing I'm doing it wrong. My assumption was that the unique identifier value in each table would apply the foreign key. After reading the documentation, it looks like the foreign key setup is a bit different. Do I need to do something like
Collections.File.files(**values).save() ? In other words, instead of doing a data import, loading the collection object and then adding the file associated fields through peewee?
Values that make up class File
{'crc32': '63bee49d',
'format': 'Metadata',
'identifier': u'somevalue',
'md5': '34104ffce9e4084fd3641d0decad910a',
'mtime': '1368328224',
'name': 'lupi.jpg_meta.txt',
'sha1': '1448ed1159a5d770da76067dd1c53e94d5a01535',
'size': '1244'}
I think the naming of your fields might be part of the confusion. Rather than calling the foreign key from File -> Collection "identifier", you might call it "collection" instead.
class File(datab):
collection = ForeignKeyField(Collection, related_name='files')
name = CharField()
Peewee prefers that, when setting the value of a Foreign Key, it be a model instance. For example, rather than doing:
File.create(collection='foobar', name='/secret/password')
It is preferable to do something like this:
collection = Collection.get(Collection.identifier == 'foobar')
File.create(collection=collection, name='/secret/password')
As a final note, if the Collection "identifier" is the unique primary key, you can set it up thus:
class Collection(datab):
identifier = CharField(primary_key=True)
title = CharField()
(I'm not familiar with peewee, but if it's like Django then this should work.)
class File has a ForeignKeyField and a CharField, so you can't simply save a pair of strings with File(**value). You need to convert a string to a key first, like this:
for value in collectionFiles:
identifier = value['identifier']
name = value['name']
collection_entity = Collection.objects.filter(identifier=identifier).get()
File(identifier=collection_entity, name=name).save()
Related
I try to automap my existing database to classes. Between two tables I have multiple join paths and I have troubles to manage those properly with SQLAlchemy.
Here's a sample schema from my database:
CREATE SCHEMA shop;
CREATE TABLE shop.address (
id SERIAL PRIMARY KEY,
name text,
address text
)
CREATE TABLE shop.orders (
id SERIAL PRIMARY KEY,
items text,
billingaddr_id integer REFERENCES address,
shippingaddr_id integer REFERENCES address
);
I have declared relationships for those foreign keys as follows:
from sqlalchemy import create_engine
from sqlalchemy.orm import relationship, Session
from sqlalchemy.ext.automap import automap_base
engine = create_engine(
"postgresql://postgres:postgres#localhost:5432/postgres",
future=True
)
Base = automap_base()
class Order(Base):
__tablename__ = 'orders'
__table_args__ = {"schema": "shop"}
billingaddr = relationship('address', foreign_keys="Order.billingaddr_id", backref="orders_billed")
shippingaddr = relationship('address', foreign_keys="Order.shippingaddr_id", backref="orders_shipped")
Base.prepare(engine, schema='shop', reflect=True)
Address = Base.classes.address
Now when creating a new Address object (jack):
jack = Address(name='Jack', address='57815 Cheryl Unions')
I get a warning:
"SAWarning: relationship 'Order.address' will copy column address.id to column orders.shippingaddr_id, which conflicts with relationship(s): 'address.orders_shipped' (copies address.id to orders.shippingaddr_id), 'Order.shippingaddr' (copies address.id to orders.shippingaddr_id). If this is not the intention, consider if these relationships should be linked with back_populates, or if viewonly=True should be applied to one or more if they are read-only. For the less common case that foreign key constraints are partially overlapping, the orm.foreign() annotation can be used to isolate the columns that should be written towards. The 'overlaps' parameter may be used to remove this warning."
How should this be solved?
That address relationship is the one automap creates automatically and is now messing with me. I don't actually need that anymore since I have created relationships by myself. Can I somehow prevent automap from creating it by default or can I delete created unnecessary relationship? I have tried to set address = None in the class declaration but it didn't work. address relationship is still created.
Answering my own question since I got this solved (Thanks Mike from sqlalchemy mailing list).
By default sqlalchemy 1.14 uses same name for both relationships so the latter overrides the previously created. Naming of those relationships can be overridden passing custom implementations of those naming methods to the automap.base.prepare. I implemented those to give unique name for each relationship. This way I don't need to even use the explicit declarative class style.
from sqlalchemy import create_engine
from sqlalchemy.orm import relationship, sessionmaker, backref
from sqlalchemy.ext.automap import automap_base, generate_relationship
engine = create_engine(
"postgresql://postgres:postgres#localhost:5432/postgres",
future=True
)
Base = automap_base()
def name_for_scalars(base, local_cls, referred_cls, constraint):
if local_cls.__name__ == 'orders' and referred_cls.__name__ == 'address':
if constraint.name == 'orders_billingaddr_id_fkey':
return 'billingaddr'
elif constraint.name == 'orders_shippingaddr_id_fkey':
return 'shippingaddr'
return referred_cls.__name__.lower()
def name_for_collections(base, local_cls, referred_cls, constraint):
if local_cls.__name__ == 'address' and referred_cls.__name__ == 'orders':
if constraint.name == 'orders_billingaddr_id_fkey':
return 'orders_billed'
elif constraint.name == 'orders_shippingaddr_id_fkey':
return 'orders_shipped'
return referred_cls.__name__.lower() + "_collection"
Base.prepare(
autoload_with=engine,
schema='shop',
name_for_scalar_relationship=name_for_scalars,
name_for_collection_relationship=name_for_collections
)
Order = Base.classes.orders
Address = Base.classes.address
I'm creating a small python program and i use the ORM peewee to manage my Sqlite database. I want to create a table without primary key using peewee. The problem is that peewee is adding a primary key if i dont specify one. After read the docs, i found the parameter without_rowid supposed to prevent peewee to create this primary key. But it doesnt works.
Here is a small example of code :
#!/usr/bin/python
import peewee
import traceback
db_proxy = peewee.Proxy()
class BaseModel(peewee.Model):
class Meta:
database = db_proxy
class table1(BaseModel):
id = peewee.IntegerField(primary_key=True)
field1 = peewee.CharField()
field2 = peewee.IntegerField()
class table2(BaseModel):
table1_id = peewee.IntegerField()
field1 = peewee.CharField()
class Meta:
without_rowid = True
try:
# create database
db = peewee.SqliteDatabase("test.db")
db_proxy.initialize(db)
db.create_tables([table1, table2])
except Exception as e:
traceback.print_exc(e)
Same with the without_rowid, peewee automatically add an id primary key to my table2 table. Here is the schema of the created database :
CREATE TABLE IF NOT EXISTS "table1"(
"id" INTEGER NOT NULL PRIMARY KEY,
"field1" VARCHAR(255) NOT NULL,
"field2" INTEGER NOT NULL
);
CREATE TABLE IF NOT EXISTS "table2"(
"id" INTEGER NOT NULL PRIMARY KEY,
"table1_id" INTEGER NOT NULL,
"field1" VARCHAR(255) NOT NULL
) WITHOUT ROWID;
Do you know a way to solve this problem and permit me to create a table without rowid ?
ps : if you're asking why i want to create a table without primary key, it's because i need to insert data from a csv file. (sqlite3 database.db ".mode csv" ".import file.csv table1"). As i have an AUTO INCREMENT PRIAMRY KEY on my table, i need to trick a little bit by importing the csv file in a temporary table without primary key as explained here : http://objectiveme.com/populating-a-sqlite-database-using-csv/
Thx ! :)
Peewee documentation is also VERY CLEAR about how to disable primary key:
http://docs.peewee-orm.com/en/latest/peewee/models.html#models-without-a-primary-key
Please READ THE DOCS.
To create a Peewee model without a primary key (which is distinctly different from "WITHOUT ROWID"), you can specify "primary_key = False":
class NoPKModel(Model):
key = TextField()
data = TextField()
class Meta:
primary_key = False
This will not automatically create an "id" field.
Without ROWID is a SQLite optimization with rather specific use-case. Please read the SQLite documentation and understand the SQLite data-model before using it: https://www.sqlite.org/rowidtable.html
Let's say I have a User model with attributes id, name, email and a relationship languages.
Is it possible to create a User instance from existing data that behaves like I would have queried it with dbsession.query(User).get(42)?
What I mean in particular is that I want that an access to user.languages creates a subquery and populates the attribute.
Here a code example:
I have a class User:
class User(Base):
id = Column(Integer, primary_key=True)
name = Column(String(64))
email = Column(String(64))
languages = relationship('Language', secondary='user_languages')
I already have a lot of users stored in my DB.
And I know that I have, for example, this user in my DB:
user_dict = {
'id': 23,
'name': 'foo',
'email': 'foo#bar',
}
So I have all the attributes but the relations.
Now I want to make a sqlalchemy User instance
and kind of register it in sqlalchemy's system
so I can get the languages if needed.
user = User(**user_dict)
# Now I can access the id, name email attributes
assert user.id == 23
# but since sqlalchemy thinks it's a new model it doesn't
# lazy load any relationships
assert len(user.languages) == 0
# I want here that the languages for the user with id 23 appear
# So I want that `user` is the same as when I would have done
user_from_db = DBSession.query(User).get(23)
assert user == user_from_db
The use-case is that I have a big model with lots of complex
relationships but 90% of the time I don't need the data from those.
So I only want to cache the direct attributes plus what else I need
and then load those from the cache like above and be able to
use the sqlalchemy model like I would have queried it from the db.
From the sqlalchemy mailing list:
# to make it look like it was freshly loaded from the db
from sqlalchemy.orm.session import make_transient_to_detached
make_transient_to_detached(user)
# merge instance in session without emitting sql
user = DBSession.merge(user, load=False)
This answer was extracted from the question
I trying to select some data using Peewee ORM,but I'm confused how to use foreign key correctly.
I wanna select post_title,user_name,act_title by Act.id(default primary key in act).
So I use this
Post.select(Post.post_tile,User.user_name,Act.act_title).join(Act).join(User).where(Act.id==actId)
But I got this:
[{"post_title": null,"user": {}, "act": {}}]
Here is my model:
class User(BaseModel):
user_name = CharField(max_length=30,unique=True)
user_email = CharField(max_length=60,unique=True)
class Act(BaseModel):
user = ForeignKeyField(User, related_name='users_act_id') #foreignkey
act_title = CharField(max_length=30)
class Post(BaseModel):
act = ForeignKeyField(Act,related_name='acts_id') #foreignkey
user = ForeignKeyField(User,related_name='users_post_id') #foreignkey
post_title = CharField(max_length=30)
If you want to join both the User and the Act table on the Post table, you need to put a switch in the query, so you would have
Post.select(Post.post_tile,User.user_name,Act.act_title).join(Act).switch(Post).join(User).where(Act.id==actId)
Although it's possible that's not the result you're looking for because you have user as a foreign key in both the Act and Post models
I think the only thing you're missing is not looking up the values on the joined instances:
posts = (Post
.select(Post.post_tile,User.user_name,Act.act_title)
.join(Act)
.switch(Post)
.join(User)
.where(Act.id==actId))
for post in posts:
print post.post_title, post.user.user_name, post.act.act_title
If you want the attributes all assigned to the post object, just tack on a call to .naive(), e.g.:
posts = (Post.select()...where(Act.id==actId).naive())
I am trying to import my KML file into a model using GeoDjango's LayerMapping functionality. I've run tests and had no issues when doing regular imports. However, I recently added a foreign key to my model. My model is called PlaceMark and it now has a FK to a model called Layer. I would like to either
override the import and manually set the value of the foreign key field or
update my KML file to contain a new element that connects the PlaceMark to the layer via either the pk or name field of Layer.
Here is how I am testing from the shell and the relevant error:
>>>from locator import load
>>>load.run()
...
TypeError: ForeignKey mapping must be of dictionary type.
....
Here is my load.py file:
import os
from django.contrib.gis.utils import LayerMapping
from models import PlaceMark
placemark_mapping = {
'name' : 'Name',
'description' : 'Description',
# This line below is the one that is suspect #
'layer': 'Layer',
'geom' : 'POINT25D',
}
placemark_kml = os.path.abspath(os.path.join(os.path.dirname(__file__), 'data/claim.kml'))
def run(verbose=True):
lm = LayerMapping(PlaceMark, placemark_kml, placemark_mapping,
transform=False, encoding='iso-8859-1')
lm.save(strict=True, verbose=verbose)
KML File:
<?xml version="1.0" encoding="Windows-1252"?>
<kml xmlns="http://earth.google.com/kml/2.1">
<Folder>
<description><![CDATA[TankSafe_Claims]]></description>
<Placemark>
<name><![CDATA[G2184729A]]></name>
<description><![CDATA[<br><br><br>
<table border="1" padding="0">
<tr><td>Policy_Number</td><td>53645645</td></tr>
<tr><td>Claim_Number</td><td>2342342234</td></tr>
<tr><td>Policy_Type</td><td>TSP</td></tr>
<tr><td>Name</td><td>Al's Total</td></tr>
<tr><td>Street_Address</td><td>555 109th Avenue</td></tr>
<tr><td>City</td><td>Pullman</td></tr>
<tr><td>State</td><td>NY</td></tr>
<tr><td>Zip_Code</td><td>55555</td></tr>
<tr><td>County</td><td>Allegan</td></tr>
]]></description>
<visibility>1</visibility>
<open>0</open>
<Point>
<extrude>1</extrude>
<altitudeMode>relativeToGround</altitudeMode>
<coordinates>-86.092641,42.483953,0</coordinates>
</Point>
<!--- ***Should I add the line below?*** -->
<Layer><name>claims</name></Layer>
</Placemark>
</Folder>
</kml>
My goal is to just get all the PlaceMarks imported with references to the relevant layer. Any ideas?
Thanks!
Larry
layer_mapping = {
'fk': {'nm_field': 'NAME'}, # foreign key field
'this_field': 'THIS',
'that_field': 'THAT',
'geom': 'POLYGON',
}
the error you're receiving that the Foreign Key field should be a dictionary is basically requesting an additional mapping to the model which the foreign key relates.
in the above snippet:
'fk' is the foreign key field name from the model the data is being loaded into (lets call it 'load model')
'nm_field' is the field name from the model the 'load model' has the foreign key relationship to (lets call it 'primary model')
'NAME' is the field name from the data being loaded into 'load model' which holds the relationship to 'primary model'
more explicitly, imagine if 'primary model' is a dataset of lakes and they have a field called 'nm_field' that is the lake name as a string.
now imagine, 'load model' is a dataset of points representing all the buoys on all the lakes and has a field name 'fk' that is a ForeignKey to 'primary model' for the assignment of the lake each buoy belongs to.
finally, the data you're loading into 'load model' has a string field called 'NAME' and it contains the pre-populated name of the lake each buoy belongs to. that string name is the relationship tie. it allows the 'load model' to use that name to identify which lake in the 'primary model' it should establish a foreign key with.
I tricked the LayerMapper into loading the ForeignKey field as a plain data-type after creating the tables.
Give USCounty an FK "state" to USState and run manage.py syncdb
Replace the "state" with "state_id" and the real datatype,
usually models.IntegerField and execute the load.run() LayerMapper.
Return the "state" FK to the USCounty model.
Use Django normally.
In my case below, the "state" keys are 2-character FIPS codes.
class USCounty(models.Model):
state = models.ForeignKey(USState)
## state_id = models.CharField(max_length=2)
...
geom = models.MultiPolygonField(srid=4326)
objects = models.GeoManager()
I worked around this by manually adding a temporary pre_save callback. You can connect it just for the record creation, then disconnect as soon as LayerMapping has done its work.
See 'My Solution' here - the 'black box' method I refer to is in fact exactly this use case.
The code that works for me:
def pre_save_callback(sender, instance, *args, **kwargs):
fkey = some_method_that_gets_the_foreign_key()
instance.type = fkey
# other mappings defined as usual
mapping = {
'key1': 'KEY1',
...,
}
lm = LayerMapping(models.MyModel, PATH_TO_SHAPEFILE, mapping, transform=True)
# temporarily connect pre_save method
pre_save.connect(pre_save_callback, sender=models.MyModel)
try:
lm.save(strict=True)
except Exception as exc:
optional_error_handling()
raise
finally:
# disconnect pre_save callback
pre_save.disconnect(pre_save_callback, sender=models.MyModel)
It doesn't look like there is an easy way to hook into LayerMapping for foreign key fields. I solved this by using a for loop and the get_geoms() call. Thanks to http://invisibleroads.com/tutorials/geodjango-googlemaps-build.html
Here is an example of what I did:
placemark_kml = os.path.abspath(os.path.join(os.path.dirname(locator.__file__), 'data/claim.kml'))
datasource = DataSource(placemark_kml)
lyr = datasource[0]
waypointNames = lyr.get_fields('Name')
waypointDescriptions = lyr.get_fields('Description')
waypointGeometries = lyr.get_geoms()
for waypointName, waypointGeometry, waypointDescription in itertools.izip(waypointNames, waypointGeometries, waypointDescriptions):
placemark = PlaceMark(name=waypointName, description=waypointDescription, geom=waypointGeometry.wkt)
placemark.layer = Layer.objects.get(pk=8)
placemark.save()
Not an answer but hopefully a hint.
The error thrown comes from this part of the code. line ~220 of layermapping.py
elif isinstance(model_field, models.ForeignKey):
if isinstance(ogr_name, dict):
# Is every given related model mapping field in the Layer?
rel_model = model_field.rel.to
for rel_name, ogr_field in ogr_name.items():
idx = check_ogr_fld(ogr_field)
try:
rel_model._meta.get_field(rel_name)
except models.fields.FieldDoesNotExist:
raise LayerMapError('ForeignKey mapping field "%s" not in %s fields.' %
(rel_name, rel_model.__class__.__name__))
fields_val = rel_model
else:
raise TypeError('ForeignKey mapping must be of dictionary type.')
At the beginning of the for loop, it looks for a dict: ogr_name.items()
ogr_name is actually defined as the value part of the mapping dict.
The dict is supposed to be composed of the org field name and the related field name from the related model.
If anyone understands the origin of that ogr_name dict, it would be of great use.