Develop Reference

Most python way to write from GCS to Postgresql in airflow

I am building an airflow dag that takes csv files from GCS and inserts them to a postgresql table in cloud SQL. I have several options:
Use sqlalchemy to insert the reows.
Use pandas
Explore PostgreSQL airflow operators (I don't know how to connect them with gcs).
Which is the most pythonic way to do so?

You sould go with COPY.
See https://www.postgresql.org/docs/current/sql-copy.html
COPY moves data between PostgreSQL tables and standard file-system files. COPY TO copies the contents of a table to a file, while COPY FROM copies data from a file to a table (appending the data to whatever is in the table already).

in-memory sqlite in production with python

I am creating a python system that needs to handle many files. Each of the file has more than 10 thousand lines of text data.
Because DB (like mysql) can not be used in that environment, when file is uploaded by a user, I think I will save all the data of the uploaded file in in-memory-SQLite so that I can use SQL to fetch specific data from there.
Then, when all operations by program are finished, save the processed data in a file. This is the file users will receive from the system.
But some websites say SQLite shouldn't be used in production. But in my case, I just save them temporarily in memory to use SQL for the data. Is there any problem for using SQLite in production even in this scenario?
Edit:
The data in in-memory-DB doesn't need to be shared between processes. It just creates tables, process data, then discard all data and tables after saving the processed data in file. I just think saving everything in list makes search difficult and slow. So using SQLite is still a problem?

SQLite shouldn't be used in production is not a one-for-all rule, it's more of a rule of thumb. Of course there are appliances where one could think of reasonable use of SQLite even in production environments.
However your case doesn't seem to be one of them. While SQLite supports multi-threaded and multi-process environments, it will lock all tables when it opens a write transaction. You need to ask yourself whether this is a problem for your particular case, but if you're uncertain go for "yes, it's a problem for me".
You'd be probably okay with in-memory structures alone, unless there are some details you haven't uncovered.

I'm not familiar with the specific context of your system, but if what you're looking for is a SQL database that is
light
Access is from a single process and a single thread.
If the system crashes in the middle, you have a good way to recover from it (either backing up the last stable version of the database or just create it from scratch).
If you meet all these criteria, using SQLite is production is fine. OSX, for example, uses sqlite for a few purposes (e.g. ./var/db/auth.db).

Sqlite database backup and restore in flask sqlalchemy

I am using flask sqlalchemy to create db which in turns create a app.db files to store tables and data. Now for the backup it should be simple to just take a copy of app.db somewhere in the server. But suppose while the app is writing data to app.db and we make a copy at that time then we might have inconsistent app.db file.
How do we maintain the consistency of the backup. I can implement locks to do so. But I was wondering on standard and good solutions for database backup and how is this implemented in python.

SQLite has the backup API for this, but it is not available in the built-in Python driver.
You could
use the APSW library for the backup; or
execute the .backup command in the sqlite3 command-line shell; or
run BEGIN IMMEDIATE to prevent other connections from writing to the DB, and copy the file(s).

I wanted also create backups of my SQL file, which stores data from my Flask.
Since my Flask web don't have too many users, and the database don't carry more than 4 small-medium size tables I did the following:
#Using Linux cmd
cd pathOfMyDb
nano backupDbRub.sh # backupDBRun is the name of the file that will be called periodically
Nano text editor will open a file and you can write the following
#!/bin/bash
current_time=$(date "+%Y.%m.%d-%H.%M.%S");
sep='"';
file_name = '_site.db"';
functionB = '.backup ';
queryWrite = "$functionB$sep$current_time$file_name";
echo "$queryWrite";
sqlite3 yoursqlitefile.db "$queryWrite"
Basically with this code we create a function ($queryWrite) which will do a copy of the current database with a timestamp. Now in Unix you will have to modify the crontab file.
Write in the Unix terminal
crontab -e
in this new file , as explained in this post you should add the periodicity (in my case , for testing purposed, I wanted to run it every minute to check it is working.
* * * * * /backupDbRub.sh
Think that the copies will appear in the folder you choose in the $querywrite sentence. So you can find them there.
In case you want to test that the script works, you must go to the folder where this script and run
/backupDbRub.sh
This should print the 'echo' sentence called in your file. Sometimes you did not provide permissions in this file, so you should add permissions with 'chmod -x', or whatever other you need to set up.

SQLite database backup

For backup of SQLite database, I went through https://www.sqlite.org/backup.html
I came to know that there is one python wrapper over these SQLite online backup APIs.So, I went through https://github.com/husio/python-sqlite3-backup
Here are some doubts that I have about sqlitebck(python package)
I checked out the code in tests, it shows db is copied from :memory: to file and vice versa using sqlitebck.copy(:memory:,dbfile), I am confused about ":memory:" and its use.
Instead from memory, can I copy one database file to other database file like sqlitebck(dbfile1, dbfile2). so that dbfile2 will be the backup of dbfile1?

':memory:' as a "filename" is how you tell sqlite to keep a small DB in memory rather than on-disk. But yes, sqlitebck is fine to copy from file to file -- though the arguments it takes are sqlite connections so you'd need to sqlite3.connect to each file first (and usually might as well just copy the file directly w/o involving sqlite -- as the sqlite page you link to implies, suggesting Unix cp or Windows copy... Python has its own standard library module for file copies, https://docs.python.org/2/library/shutil.html ).

Get sqlite3 database from disk, load it to memory, and save it to disk? [duplicate]

This question already has answers here:
Moving back and forth between an on-disk database and a fast in-memory database?
(2 answers)
Closed 7 years ago.
What I need to do is load an sqlite3 database from the disk into memory, work with it, and when the script exits save the in-memory database to the disk. How would I go about doing this? Thanks!

All you need to do is connect to the database - you can do anything you want to with it then.
from sqlite3 import connect
conn = connect("/path/to/your/sqlite.db")
# Do what you need to with the database here
# Changes (inserts, updates, etc) will be persisted
# to disk whenever you commit a transaction.
If you need to be able to run a series of commands and roll them all back if one does not succeed you should use a transaction.
If you need to make a copy of the existing database, make alterations to it and then save it somewhere else only if the alterations succeed then you can do one of the following things:
Create an in-memory database and then use SQLite's ATTATCH DATABASE command to attach the existing database to the new in-memory database. Once you have copied all of the data over from the existing database into the in-memory database and transformed it you can attach a new database and copy everything over again.
Copy the existing database file, attempt to make alterations to it, and delete the new file if the operations did not succeed.
Use sqlite3's Connection.iterdump to get the contents of the tables as a SQL script which you can then pass to sqlite3.Cursor.executescript as this answer suggests (using either an in-memory or an on-disk connection string for your new database).