I'm rebuilding a personal project in Django, (a family tree), and I'm working on migrating the actual data from the old awkward database to my new model/schema, both Postgres databases. I've defined them in the DATABASES list on settings.py as 'default' and 'source'.
I've made my models and I'd like to copy the records from the old database into their corresponding table in the new database, but I'm not quite understanding how to set up the models for it to work, since the Django code uses the models/ORM to access/update/create objects, and I only have models reflecting the new schema, not the old ones.
In a coincidental case where I have a table with the exact same schema in the old and new database, I have a management command that can grab the old records from the source using my new ImagePerson model (ported_data = ImagePerson.objects.using('source').all()), since it the expected fields are the same. Then I save objects for them in the 'default': (obj, created_bool) = ImagePerson.objects.using('default').get_or_create(field=fieldvalue, etc), and it works just like I need it to.
However when I have a table where the old version is missing fields that my new model/table have, I can't use the model to access those records (which makes sense). Am I supposed to also make some kind of legacy version of each model for use in the migration? I saw a tutorial mention running ./manage.py inspectdb --database=source > models.py, but doing so didn't seem to add anything else to my file (and it would seem weird to save temporary/legacy models in there anyway). What's the right way to access the old-formatted records? Is the ORM right?
To give a specific example, I have a Notes table to hold a memory about a specific person or about a specific family. The old table used a 'type' field (1 was for person note, 2 was for family note), and a ref_id that would be the id for the person or family the note applies to. The new table instead has a person_id field and a family_id field.
I'd like my management command to be able to pull all the records from the source table, then if type=1, look up the person with id equal to the ref_id field, and save a new object in the new database with that person. I can grab them using the new Notes model with the old database like this: ported_notes = Note.objects.using('source').all(), but then if I try to access any field (like print(note_row.body)), I get an error that the result object is missing the person_id field
django.db.utils.ProgrammingError: column notes.person_id does not exist
What's the right way to approach this?
Creating models for your old schema definitely doesn't seem like the right approach.
One solution would be to write a data-migration, where you could use raw SQL to fetch your old data, and then use the ORW to write it to your new tables/models:
from django.db import migrations, connections
def transfer_data(apps, schema_editor):
ModelForNewDB = apps.get_model('yourappname', 'ModelForNewDB')
# Fetch your old data
with connections['my_old_db'].cursor() as cursor:
cursor.execute('select * from some_table')
data = cursor.fetchall()
# Write it to your new models
for datum in data:
# do something with the data / add any
# additional values needed.
ModelForNewDB.objects.create(...)
class Migration(migrations.Migration):
dependencies = [
('yourappname', '0001_initial'),
]
operations = [
migrations.RunPython(transfer_data),
]
Then simply run your migrations. One thing to note however:
If you have foreignKeys etc. between tables you will need to be careful how you order the migrations. This can be done by editing your dependencies. You may even have to add a migration to allow null values for some foreign keys, and then add another one afterwards to correct this.
Related
Can we do a loosely coupled data access layer design in python?
Lets say,in a scenario i have an oracle table with column name ACTIVITY_ID with column datatype as Number(10). If this column is a foreign key in lot many tables,to hold the data of this column, can i create something like ACTID class (like a java object) and can be used across the code if i want to manipulate/hold the ACTIVITY_ID column data so that i could maintain consistency of business object columns. Is there any such possibility in python ?
Try Django
As I understand it, Python does not natively have any database functionality. There are many different libraries/frameworks/etc. that can be used to provide database functionality to Python. I recommend taking a look at Django. With Django, you create a class for each database table and Django hides a LOT of the details, including allowing using with multiple database engines such as MySQL and PostgreSQL. Django handles Foreign Key relationships very well. Every table normally has a primary key, by default an auto-incremented id field. If you add a field like activity = models.ForeignKey(Activity) then you now have a foreign key field activity_id in one table referencing the primary key field id in the Activity table. The Admin page will take care of cascading deletion of records if you delete an Activity record, and in general things "just work" the way you might expect them to.
I have looked for an answer to this question, but very was able to find very little. I want to extract the names of the tables, references between them, column names so I can graphically visualize that information. I need to this in a Django project.
Since I am a newbie to python I would like to know if there is some kind of API to do this type of thing.
Edit
I have created a model which consists Node, Attribute and Link. Node has attributes, while Link has fields parent_node and child_node. What I want is to connect to a database, read the metadata by which I mean: Table names, Column names and Foreign key constraints. Then I could properly put this data in the model I have created.
You can use inspectdbcommand, with this django reads your database and create models for each table and if your database has relations you get also in django. You can see more info here.
python manage.py inspectdb
I have little to no experience with databases and i'm wondering how i would go about storing certain parts of an object.
Let's say I have an object like the following and steps can be an arbitrary length. How would I store these steps or list of steps into an sql database?
class Error:
name = "" #name of error
steps = [] #steps to take to attempt to solve error
For your example you would create a table called Errors with metadata about the error such as an error_ID as the primary key, a name, date created, etc... then you'd create another table called Steps with it's own id, lets say Step_ID and any fields related to the step. The important part is you'd create a field on the Steps table that relates back to the Error that the steps are for we'll call that field again error_ID, then you'd make that field a foreign key so the database enforces that constraint.
If you want to store your Python objects in a database (or any other language objects in a database) the place to start is a good ORM (Object-Relational Mapper). For example Django has a built-in ORM. This link has a comparison of some Python Object-Relational mappers.
I'm trying to add a new entry by using the admin panel in Django
The problem is that I've already populated my DB with 200 records and if I try to add a new entry from admin I get a duplicated key error msg that keep increasing whenever I try the process again
error:
duplicate key value violates unique constraint "app_entry_pkey"
admin.py:
admin.site.register(Entry)
model:
class Entry(models.Model):
title = models.CharField(max_length=255)
url = models.TextField(max_length=255)
img = models.CharField(max_length=255)
def __unicode__(self):
return self.title
If you created the database table using Django, then most likely your auto_increment value was not updated when you imported the data outside of Django.
It may also be that when you imported the data you did not give the 200 records each their own unique primary key. I think that (some versions of) SQLite will sometimes allow that in mass imports.
MySQL
For example, I’m looking at a MySQL table in Sequel Pro and see that it has an “auto_increment” value of 144. This means that the next primary key value will be 144.
You can see this value for your table (in MySQL) using:
SHOW TABLE STATUS FROM databaseName where name="entry"
Replacing “databaseName” with the name of your Django database. Other database software will likely have different syntax.
You can set the next auto_increment value (in MySQL) using:
ALTER TABLE databaseName.entry AUTO_INCREMENT ###
Again replacing databaseName with the name of your database; and as before, the syntax may vary depending on the database software you’re using.
If this doesn’t help, you may find it useful to show the table’s status and copy that into your question. This might also be useful in tracking down the issue:
SHOW CREATE TABLE databaseName.entry
Postgres
In Postgres, you can get the current value of the auto increment variable (called sequences in Postgres) using something like:
SELECT last_value FROM app_entry_pkey;
And you will likely set it to a new value with something like:
ALTER SEQUENCE app_entry_pkey RESTART WITH ###
or
SELECT setval('app_entry_pkey', ###)
Note, though, that I do not have a Postgres database handy to test these on. You may also find the following commands useful:
SELECT MAX(id) FROM entry
SELECT nextval('app_entry_pkey')
The latter should generally be larger than the former, and note that “id” is the name of the column in your “entry” model’s table; it may be different in your table. See http://www.postgresql.org/docs/8.1/static/functions-sequence.html for more information.
One of my Django models is a subclass and I want to change its superclass to one that is very similar to the original one. In particular, the new superclass describes the same object and has the same primary key. How can I make South create the new OneToOne field and copy the values from the old one to the new one?
In south, there are two kinds of migrations: schema migrations and data migrations.
After you've created the schemamigration, create a corresponding data migration:
./manage.py datamigration <app> <migration_name>
Do not run the migration (yet). Instead, open up the migration file you just created.
You'll find the method named forwards(). Into this you define the procedure by which values from old tables get copied to new tables.
If you're changing the structure of a given table to a more complex layout, a common method is to have two schema migrations around a data migration: the first schema migration adds fields, the data migration translates the old fields to the new fields, and the second schema migration deletes the old fields. You can do just about anything with the database with the forwards() method, so long as you keep track of which schema (previous or current) you're accessing. Generally, you only read from the orm.-related, and write to the traditional Django accessors.
The South Data Migration Tutorial covers this in some detail. It shows you how to use South's orm reference to access the database using the schema prior to the schema migration and gives access to the database without Django complaining about fields it doesn't understand.
If you're renaming a class, that can be tricky-- it involves creating the new table, migrating from one to the other, and deleting the old table. South can do it, but it might take more than one pass through shifting schemas and data migrations.
South also has the backwards() method, which allows you to return your database tables to a previous step. In some cases, this may be impossible; the new table may record information that will be lost in a downgrade. I recommend using throwing an exception in backwards() if you're not in DEBUG mode.