The documentation we can use auto_import if we "need access to the data but not to he web2py table attributes", but this code seems to use the table attributes just fine.
from gluon import DAL, Field
db = DAL('sqlite://storage.sqlite', auto_import=True)
for row in db(db.person).select():
print row.name
The table was defined in a previous run.
db = DAL('sqlite://storage.sqlite', auto_import=True)
db.define_table('person',
Field('name'))
db.person[0] = {'name' : 'dave'}
db.commit()
Doing both auto_import=True and the define_table gives an error about "invalid table name". Doing neither gives an error if I try to access db.table.
With auto_import=True, web2py will get the field names and types directly from the *.table files in the application's "databases" folder. When the documentation refers to "web2py table attributes" that will not be available, it is referring to attributes that are defined in the model (i.e., using db.define_table()) but not stored in the database or *.table files, such as "requires", "widget", "represent", etc. Those attributes are defined only in web2py code and therefore cannot be determined merely by reading the *.table files. Note, the *.table files are used for database migrations, so they only store metadata directly relevant to the database (i.e., field names and types, and database-level contraints, such as "notnull" and "unique"). Attributes like "requires" and "represent" are only used by web2py and have no effect on the database, so are not recorded in the *.table files.
Related
I'm using SQLAlchemy for MySQL.
The common example of SQLAlchemy is
Defining model classes by the table structure. (class User(Base))
Migrate to the database by db.create_all (or alembic, etc)
Import the model class, and use it. (db.session.query(User))
But what if I want to use raw SQL file instead of defined model classes?
I did read automap do similar like this, but I want to get mapper object from raw SQL file, not created database.
Is there any best practice to do this?
This is an example of DDL
-- ddl.sql
-- This is just an example, so please ignore some issues related to a grammar
CREATE TABLE `card` (
`card_id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'card',
`card_company_id` bigint(20) DEFAULT NULL COMMENT 'card_company_id',
PRIMARY KEY (`card_id`),
KEY `card_ix01` (`card_company_id`),
KEY `card_ix02` (`user_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='card table'
And I want to do like
Base = raw_sql_base('ddl.sql') # Some kinda automap_base but from SQL file
# engine, suppose it has two tables 'user' and 'address' set up
engine = create_engine("mysql://user#localhost/program")
# reflect the tables
Base.prepare(engine)
# mapped classes are now created with names by sql file
Card = Base.classes.card
session = Session(engine)
session.add(Card(card_id=1, card_company_id=1))
session.commit() # Insert
SQLAlchemy is not an SQL parser, but the exact opposite; its reflection works against existing databases only. In other words you must execute your DDL and then use reflection / automap to create the necessary Python models:
from sqlalchemy.ext.automap import automap_base
# engine, suppose it has two tables 'user' and 'address' set up
engine = create_engine("mysql://user#localhost/program")
# execute the DDL in order to populate the DB
with open('ddl.sql') as ddl:
engine.execute(ddl)
Base = automap_base()
# reflect the tables
Base.prepare(engine, reflect=True)
# mapped classes are now created with names by sql file
Card = Base.classes.card
session = Session(engine)
session.add(Card(card_id=1, card_company_id=1))
session.commit() # Insert
This of course may fail, if you have already executed the same DDL against your database, so you would have to handle that case as well. Another possible caveat is that some DB-API drivers may not like executing multiple statements at a time, if your ddl.sql happens to contain more than one CREATE TABLE statement etc.
...but I want to get mapper object from raw SQL file.
Ok, in that case what you need is the aforementioned parser. A cursory search produced two candidates:
sqlparse: Generic, but the issue tracker is a testament to how nontrivial parsing SQL is. Is often confused, for example parses ... COMMENT 'card', `card_company_id` ... as a keyword and an identifier list, not as a keyword, a literal, punctuation, and an identifier (or even better, the column definitions as their own nodes).
mysqlparse: A MySQL specific solution, but with limited support for just about anything, and it seems abandoned.
Parsing would be just the first step, though. You'd then have to convert the resulting trees to models.
I'm trying to use https://pandas.pydata.org/pandas-docs/version/0.21/generated/pandas.DataFrame.to_sql.html
When I change the Name argument, e.g. say I set
pd.to_sql(name="testTable",constring)
the actual table name comes up as [UserName].[testTable] rather than just [testTable]
Is there a way I can get rid of the [userName]? which is linked to the user who runs the script?
The [UserName] portion of the table name is the schema that the table is in. I don't know which database engine you're using, but the schema you're looking for might be "dbo".
According to the documentation, you can provide a schema argument:
pd.to_sql(name="testTable",constring, schema="dbo")
Note that if the schema is left blank, it uses the DB user's default schema (as defined when the user was added to the database), which in your case, appears to be the schema of the user.
I'm trying to query the database with:
fields = "property_1, property_2, ... property_n"
query = "SELECT {0} FROM Table WHERE property_{n+1} = '{1}'".format(fields, property_{n+1})
all_objs = CacheDatastore.fetch(query, refresh=True)
The problem is that the returned list is empty, while if the query is like
"SELECT * FROM Table WHERE property_{n+1} ='{1}'", I receive the full set.
I've created the necessary indexes and have deployed them, so it is not from there.
The log says that a Blob key is not found, but none of the properties is different from string, float or int...
It turned to be a bug in the db library which is no longer in development, so I'm leaving here the link to the ticket and the comments on it.
GAE allows indexing of static members of the db.Model class hierarchy, but returns 0 results for projection queries where the static member is included
among the projected properties.
https://code.google.com/p/google-cloud-platform/issues/detail?id=119
In MongoEngine, when deleting an EmbeddedDocument from a ListField which includes a FileField, the referenced file does not get deleted. Currently, I have solved the issue by looping over the whole list field.
for embdoc in doc.embdocs:
if embdoc.filtered == value:
embdoc.dfile.delete()
doc.update(pull__embdocs={'filtered': value})
I was wondering if there was a better way to do this.
By default, MongoDB doesn’t check the integrity of your data, so deleting documents that other documents still hold references to will lead to consistency issues.
You should use ListField with ReferenceFields. ReferenceFields can used with option reverse_delete_rule=mongoengine.PULL or another:
mongoengine.DO_NOTHING
This is the default and won’t do anything. Deletes are fast, but may cause database inconsistency or dangling references.
mongoengine.DENY
Deletion is denied if there still exist references to the object being deleted.
mongoengine.NULLIFY
Any object’s fields still referring to the object being deleted are removed (using MongoDB’s “unset” operation), effectively nullifying the relationship.
mongoengine.CASCADE
Any object containing fields that are refererring to the object being deleted are deleted first.
mongoengine.PULL
Removes the reference to the object (using MongoDB’s “pull” operation) from any object’s fields of ListField (ReferenceField).
I also was in need to delete a file in a list field inside an embedded document, after a lot of searches I came across this soultion
the Document:
class AllTheFiles(me.EmbeddedDocument):
type1 = me.ListField(me.FileField())
type2 = me.ListField(me.FileField())
class MainDocument(me.Document):
files = me.EmbeddedDocumentField(AllTheFiles)
I am assuming here that you have some documents and they have files, in the real world you will need to check if there are files and for documents existns.
So in order to delete the first file(index 0) in the type1 field:
del_here = MainDocument.objects()[0]
del_here.files.type1[0].delete()
del_here.files.type1.pop(0)
del_here.save()
The file will be deleted in the embedded document type1 list and also in "fs.files" and "fs.chuncks" colloctions.
I'm developing a webapp in Django, and for it's database I need to import a CSV file into a particular MySQL database.
I searched around a bit, and found many pages which listed how to do this, but I'm a bit confused.
Most pages say to do this:
LOAD DATA INFILE '<file>' INTO TABLE <tablenname>
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n';
But I'm confused how Django would interpret this, since we haven't mentioned any column names here.
I'm new to Django and even newer to databasing, so I don't really know how this would work out.
It looks like you are in the database admin (i.e. PostgreSQL/MySQL). Others above has given a good explanation for that.
But if you want to import data into Django itself -- Python has its own csv implementation, like so: import csv.
But if you're new to Django, then I recommend installing something like the Django CSV Importer: http://django-csv-importer.readthedocs.org/en/latest/index.html. (You install the add-ons into your Python library.)
The author, unfortunately, has a typo in the docs, though. You have to do from csvImporter.model import CsvDbModel, not from csv_importer.model import CsvDbModel.
In your models.py file, create something like:
class MyCSVModel(CsvDbModel):
pass
class Meta:
dbModel = Model_You_Want_To_Reference
delimiter = ","
has_header = True
Then, go into your Python shell and do the following command:
my_csv = MyCsvModel.import_data(data = open("my_csv_file_name.csv"))
This isn't Django code, and Django does not care what you call the columns in your CSV file. This is SQL you run directly against your database via the DB shell. You should look at the MySQL documentation for more details, but it will just take the columns in order as they are defined in the table.
If you want more control, you could write some Python code using the csv module to load and parse the file, then add it to the database via the Django ORM. But this will be much much slower than the SQL way.
It will likely just add the data to the columns in order, since they are omitted from your SQL statement.
If you want, you can add the fields to the end of the SQL:
LOAD DATA INFILE '<file>' INTO TABLE <tablenname>
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
(#Field1, #Field2, #Field3) /* Fields in CSV */
SET Col1 = #Field1, Col2 = #Field2, Col3 = #Field3; /* Columns in DB */
More in-depth analysis of the LOAD DATA command at MySQL.com
The command is interpreted by MySQL, not Django. As stated in the manual:
By default, when no column list is provided at the end of the LOAD DATA INFILE statement, input lines are expected to contain a field for each table column. If you want to load only some of a table's columns, specify a column list:
LOAD DATA INFILE 'persondata.txt' INTO TABLE persondata (col1,col2,...);
You must also specify a column list if the order of the fields in the input file differs from the order of the columns in the table. Otherwise, MySQL cannot tell how to match input fields with table columns.