I have created a Django app and tested the application's performance by populating some 10,0000 records. Now i want to delete it using a python script. Can somebody help me in doing this. This is the script i created to populate data into sql db.
def dumpdata():
for i in range(2,10):
userName = "Bryan"
designation = 'Technician '
employeeID = 2312
dateOfJoin = '2009-10-10'
EmployeeDetails(userName= "Bryan",designation= 'Technician',employeeID= 2312,dateOfJoin= '2009-10-10').save()
dumpdata()
QuerySet.delete()
EmployeeDetails.objects.filter(...).delete()
Related
The following code works fine if table t is on the connecting user's search_path. I would like to be able to specify, in code, which schema to search. The only way I can make the code work is by issuing a alter user on the backend.
alter user username set search_path=schema_name;
import ibis
con = ibis.postgres.connect(url = "postgresql+psycopg2:///?service=service_name")
t = con.table("t")
y = t.group_by("b").mutate(z=t.a.sum())
How can i specify seach_path on the Postgres backend to ibis?
Regards
I am very new to MongoDB. I create a database within a loop. Each time (Every 2 hours), I get data from some sources and create a data collection by MongoEngine and name each collection based on the creation time (for example 05_01_2021_17_00_30).
Now, on another python code , I want to get the latest database. how can I call the latest database collection without knowing the name of it?
I saw some guidelines in Stackoverflow but codes are old and not working now. Thanks guys.
I came up with this answer:
In mongo_setup.py: when I want to create a database, it will be named after the time of creation and save the name in a text file.
import mongoengine
import datetime
def global_init():
nownow = datetime.datetime.now()
Update_file_name = str(nownow.strftime("%d_%m_%Y_%H_%M_%S"))
# For Shaking hand between Django and the last updated data base, export the name
of the latest database
# in a text file and from there, Django will understand which database is the
latest
Updated_txt = open('.\\Latest database to read for Django.txt', '+w')
Updated_txt.write(Update_file_name)
Updated_txt.close()
mongoengine.register_connection(alias='core', name=Update_file_name)
In Django views.py: we will call the text file and read the latest database's name:
database_name_text_file = 'directory of the text file...'
db_name_file = open(database_name_text_file, 'r')
db_name = db_name_file.read()
# MongoDb Database
myclient = MongoClient(port=27017)
mydatabase = myclient[db_name]
classagg = mydatabase['aggregation__class']
database_text = classagg.find()
for i in database_text:
....
I am fetching data from mysql using pyspark which for only one table.I want to fetch all tables from mysql db. Don't want call jdbc connection again and again. see code below
Is it possible to simplify my code? Thank you in advance
url = "jdbc:mysql://localhost:3306/dbname"
table_df=sqlContext.read.format("jdbc").option("url",url).option("dbtable","table_name").option("user","root").option("password", "root").load()
sqlContext.registerDataFrameAsTable(table_df, "table1")
table_df_1=sqlContext.read.format("jdbc").option("url",url).option("dbtable","table_name_1").option("user","root").option("password", "root").load()
sqlContext.registerDataFrameAsTable(table_df_1, "table2")
you need somehow to acquire the list of the tables you have in mysql.
Either you find some sql commands to do that, or you manually create a file containing everything.
Then, assuming you can create a list of tablenames in python tablename_list, you can simply loop over it like this :
url = "jdbc:mysql://localhost:3306/dbname"
reader = (
sqlContext.read.format("jdbc")
.option("url", url)
.option("user", "root")
.option("password", "root")
)
for tablename in tablename_list:
reader.option("dbtable", tablename).load().createTempView(tablename)
This will create a temporary view with the same tablename. If you want another name, you can probably change the initial tablename_list with a list of tuples (tablename_in_mysql, tablename_in_spark).
#Steven already gave a perfect answer. As he said, in order to find a Python list of tablenames, you can use:
#list of the tables in the server
table_names_list = spark.read.format('jdbc'). \
options(
url='jdbc:postgresql://localhost:5432/', # database url (local, remote)
dbtable='information_schema.tables',
user='YOUR_USERNAME',
password='YOUR_PASSWORD',
driver='org.postgresql.Driver'). \
load().\
filter("table_schema = 'public'").select("table_name")
#DataFrame[table_name: string]
# table_names_list.collect()
# [Row(table_name='employee'), Row(table_name='bonus')]
table_names_list = [row.table_name for row in table_names_list.collect()]
print(table_names_list)
# ['employee', 'bonus']
Note that this is in PostgreSQL. You can easily change url and driver arguments.
I am writing a python script to query about 60 database tables based on a current timestamp and store those as csv file in S3 bucket. There are some global variables that I need access to like engine, aws credentials, current_time etc. I have this file currently as 60 functions each querying a table which then calls a function to write into s3.
How do I organize this code better so I won't have to call these 60 functions from the main function?
More importantly, how do I also organize this code following OOP. I am very new to this and any help would be greatly appreciated.
This is what my current code looks like:
import (bunch of imports)
engine = create_engine('sqlite:///bookdatabase.db', echo=False)
access_key = 'adasdasdasdasd'
access_id = 'asdasdasd'
def table_name():
table_name = 'book'
sql = "select * from book where modified_date < current_date"
mn = pandas.read_sql(sql, engine)
# write_to_s3
def another_table_name():
# .....
# etc. etc.
Functions that do the same thing, only with a single variance are a clue that really those actions can be combined into a better structure.
In your case, you are doing the same thing (calling a database, and updating a bucket), the difference is you call different databases, and read different tables.
So why not create a function like this:
S3_ACCESS_KEY = '....'
S3_ACCESS_ID = '....'
def export_to_s3(db_configuration):
for db, tables in db_configuration.items():
engine = create_engine('sqlite://{}'.format(db), echo=False)
for table_name in tables:
sql = "SELECT * FROM {} WHERE modified_date <
current_date".format(table_name)
cursor = engine.cursor()
cursor.execute(sql)
for result in cursor:
# push result to s3
db_table_names = {'bookdatabase.db': ['book'],
'another.db': ['fruits', 'planets']}
export_to_s3(db_table_names)
I'm playing around with win32com.client for python to try to write/insert a row to a MS Access table. I've found an example of how to connect and query an Access table here. Basically, their code slightly modified for my own use is:
import win32com.client
connection = win32com.client.Dispatch(r'ADODB.Connection')
DSN = 'PROVIDER=Microsoft.Jet.OLEDB.4.0;DATA SOURCE=c:\\testdb.mdb;'
connection.Open(DSN)
recordset = win32com.client.Dispatch(r'ADODB.Recordset')
recordset.Open('SELECT * FROM Table1', connection, 1, 3)
fields_dict = {}
for x in range(recordset.Fields.Count):
fields_dict[x] = recordset.Fields.Item(x).Name
print fields_dict[x], recordset.Fields.Item(x).Value
So this tells me how to execute a select statement on the Access table. I'd like to be able to write rows and data to the table. When using win32com for MS Office products, I tend to dive into the MSDN pages and try to interpret the VBA code for python code, but this one has me a bit handcuffed. Couple that with no examples found on the internet after lengthy searches has made me second guess whether or not this is possible? hopefully someone out there has played with this before and has a suggestion.
As I mentioned in my comment to the question, using pyodbc (or pypyodbc) and the Access ODBC driver is a more common way of doing CRUD operations, but if you really want to use win32com and OLEDB then you could do an UPDATE like this:
import win32com.client
# ADODB constants
adVarWChar = 202
adInteger = 3
adParamInput = 1
connection = win32com.client.Dispatch(r'ADODB.Connection')
DSN = (
r'PROVIDER=Microsoft.Jet.OLEDB.4.0;'
r'DATA SOURCE=C:\Users\Public\mdbTest.mdb;'
)
connection.Open(DSN)
cmd = win32com.client.Dispatch(r'ADODB.Command')
cmd.ActiveConnection = connection
cmd.CommandText = "UPDATE Donors SET LastName = ? WHERE ID = ?"
cmd.Parameters.Append(cmd.CreateParameter("?", adVarWChar, adParamInput, 255))
cmd.Parameters.Append(cmd.CreateParameter("?", adInteger, adParamInput))
cmd.Parameters(0).Value = "Thompson"
cmd.Parameters(1).Value = 10
cmd.Execute()
connection.Close()