Develop Reference

df to sql server replace

im doing scraping web and want to save my dataframe to my sql server that updates everymonth,what should i add to code for the data got replaced not added.im using pyodbc(i cant use sql achemy)thank you in advance.
col_names = ["month", "price", "change"]
df = pd.read_csv("minyak_brent.csv",sep=',',quotechar='\'',encoding='utf8', names=col_names,skiprows = 1) # Replace Excel_file_name with your excel sheet name
for index,row in df.iterrows():
cursor.execute('INSERT INTO dbo.minyak_brent([month],[price],[change]) values (?,?)',
row['month'],
row['price'],
row['change'])
cnxn.commit()
cursor.close()
cnxn.close()

You should use df.to_sql here, which avoids the need to explicitly iterate over your Pandas data frame:
df.to_sql(con=engine, name='dbo.minyak_brent', if_exists='append', dtype={
'month': STRING(255),
'price': FLOAT,
'change': FLOAT, index=False
)
This assumes that the types of the month, price, and change columns are, String(255), FLOAT, and FLOAT, respectively.
If you must stick with your current approach, then fix your insert statement such that it has the right number of placeholders in the VALUES clause:
cursor.execute('''INSERT INTO dbo.minyak_brent([month], [price], [change])
VALUES (?, ?, ?)''', (row['month'], row['price'], row['change']))
Note that we insert a tuple as the second parameter to cursor#execute.

Updating PostgreSQL Table

I have an Issue with updating SQL columns with lists in Python.
I have list list_arrive and I can't update my values in arrive column in PostgresSQL. Included list as an example
import pandas as pd
import psycopg2
import numpy as np
# Getting values from Location.csv and converting it into DF Note:(I have Multiple Empty Rows and column arrive)
df = pd.read_csv("Locations.csv")
connection = psycopg2.connect(user="user",
password="password",
host="localhost",
port="5432",)
cursor = connection.cursor()
query = "SELECT * FROM location"
cursor.execute(query)
# I tried To Convert Df column type to str due to(TypeError: not all arguments converted during string formatting)Error
df['arrive'] = df['arrive'].astype(str)
# Replaced All Empty Rows With "NaN" Values
df['arrive'] = df['arrive'].replace(r'^\s*$', np.NaN, regex=True)
# Replaced "NaN" Values With None String
df['arrive'].fillna("None",inplace= True)
# Created List
list_arrive = list(df['arrive'])
sql_query = "UPDATE location SET arrive = %s"
# I tried:
cursor.execute(sql_update, (list_arrive,))
cursor.execute(sql_update, list_arrive)
# Tried with executemany
# Tried to Loop trough list_arrive
connection.commit()
list_arrive= ['Cranbury, NJ', 'None', 'New Orleans, LA', 'Ashland,VA', 'Lancaster, NY ', 'None', 'Hazelwood,MO', 'Aurora, CO', 'None']
Note that I'm updating only arrive column values in SQL database which has already other columns and values
I'm having three types of problem:
Getting "TypeError: not all arguments converted during string formatting"
When I tried to just execute() it pasted first value over and over with {} brackets
If I use for loop, it loops with just first values.

How to solve invalid input syntax for type double precision

I try to load csv data into postgres. The creating table part is fine. But when I try to load data from csv, it got error. My code and error are attached below. Is %s wrong?
import psycopg2
import csv
conn = psycopg2.connect(host="127.0.0.1", port="5432", database="postgres", user="postgres", password="*******")
print "Opened database successfully"
cur = conn.cursor()
cur.execute('''create table calls_aapl("Ask" float,"Bid" float,"Change" float,"ContractSymbol" varchar(50),"ImpliedVolatility" float,"LastPrice" float,
"LastTradeDate" date,"OpenInterest" int,"PercentChange" float,"Strike" float,"Volume" int);''')
print "Table created successfully"
reader = csv.reader(open('D:/python/Anaconda/AAPL_Data/Calls.csv', 'r'))
for i, row in enumerate(reader):
print(i, row)
if i == 0: continue
cur.execute('''
INSERT INTO "calls_aapl"(
"Ask", "Bid", "Change", "ContractSymbol", "ImpliedVolatility", "LastPrice", "LastTradeDate", "OpenInterest", "PercentChange", "Strike", "Volume"
) values (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)''', row
)
conn.commit()
cur.close()
Error:
(0, ['Ask', 'Bid', 'Change', 'ContractSymbol', 'LastPrice', 'LastTradeDate', 'OpenInterest', 'PercentChange', 'PercentImpliedVolatility', 'Strike', 'Volume'])
(1, ['41.7', '39.75', '1.15', 'AAPL180803C00150000', '41.05', '7/31/2018', '52', '2.88', '154.59', '150', '6'])
DataError: invalid input syntax for type double precision: "7/31/2018"
LINE 4: ...1.7','39.75','1.15','AAPL180803C00150000','41.05','7/31/2018...
^

Using %s is ok because PostgreSQL can cast strings to numbers in an INSERT.
Your problem is a different one. Your INSERT statement specifies a column "ImpliedVolatility" (too late for a warning against mixed case identifiers) which is not in the data.
This causes the fifth column (labeled LastPrice to be inserted into "ImpliedVolatility" and the next column (labeled LastTradeDate) to be inserted into "LastPrice".
The former of these is wrong but works, because both "LastPrice" and "ImpliedVolatility" are float^H^H^H^H^Hdouble precision, but the latter fails because it tries to insert a date string into a double precision column.
Omit the column "ImpliedVolatility" from the INSERT statement.

its just about typo i think,
you should equalize the table column to your insert query.
That "LastTradeDate" is inserted to "LastPrice" which is which is not the right column
thank you

Usually occurs when your column headers and values aren't matched up properly. Try checking to see if the number of values specified are the same and of similar data types.

Python - creating string from dict items (for writing to postgresql db)

I'm writing some code using psycopg2 to connect to a PostGreSQL database.
I have a lot of different data types that I want to write to different tables in my PostGreSQL database. I am trying to write a function that can write to each of the tables based on a single variable passed in the function and I want to write more than 1 row at a time to optimize my query. Luckily PostGreSQL allows me to do that: PostGreSQL Insert:
INSERT INTO films (code, title, did, date_prod, kind) VALUES
('B6717', 'Tampopo', 110, '1985-02-10', 'Comedy'),
('HG120', 'The Dinner Game', 140, DEFAULT, 'Comedy');
I have run into a problem that I was hoping someone could help me with.
I need to create a string:
string1 = (value11, value21, value31), (value12, value22, value32)
The string1 variable will be created by using a dictionary with values. So far I have been able to create a tuple that is close to the structure I want. I have a list of dictionaries. The list is called rows:
string1 = tuple([tuple([value for value in row.values()]) for row in rows])
To test it I have created the following small rows variable:
rows = [{'id': 1, 'test1': 'something', 'test2': 123},
{'id': 2, 'test1': 'somethingelse', 'test2': 321}]
When rows is passed through the above piece of code string1 becomes as follows:
((1, 'something', 123), (2, 'somethingelse', 321))
As seen with string1 I just need to remove the outmost parenthesis and make it a string for it to be as I need it. So far I don't know how this is done. So my question to you is: "How do I format string1 to have my required format?"

execute_values makes it much easier. Pass the dict sequence in instead of a values sequence:
import psycopg2, psycopg2.extras
rows = [
{'id': 1, 'test1': 'something', 'test2': 123},
{'id': 2, 'test1': 'somethingelse', 'test2': 321}
]
conn = psycopg2.connect(database='cpn')
cursor = conn.cursor()
insert_query = 'insert into t (id, test1, test2) values %s'
psycopg2.extras.execute_values (
cursor, insert_query, rows,
template='(%(id)s, %(test1)s, %(test2)s)',
page_size=100
)
And the values are inserted:
table t;
id | test1 | test2
----+---------------+-------
1 | something | 123
2 | somethingelse | 321
To have the number of affected rows use a CTE:
insert_query = '''
with i as (
insert into t (id, test1, test2) values %s
returning *
)
select count(*) from i
'''
psycopg2.extras.execute_values (
cursor, insert_query, rows,
template='(%(id)s, %(test1)s, %(test2)s)',
page_size=100
)
row_count = cursor.fetchone()[0]

With little modification you can achieve this.
change your piece of cod as follows
','.join([tuple([value for value in row.values()]).__repr__() for row in rows])
current output is
tuple of tuple
(('something', 123, 1), ('somethingelse', 321, 2))
After changes output will be
in string format as you want
"('something', 123, 1),('somethingelse', 321, 2)"

The solution that you described is not so well because potentially it may harm your database – that solution does not care about escaping string, etc. So SQL injection is possible.
Fortunately, psycopg (and psycopg2) has cursor's methods execute and mogrify that will properly do all this work for you:
import contextlib
with contextlib.closing(db_connection.cursor()) as cursor:
values = [cursor.mogrify('(%(id)s, %(test1)s, %(test2)s)', row) for row in rows]
query = 'INSERT INTO films (id, test1, test2) VALUES {0};'.format(', '.join(values))
For python 3:
import contextlib
with contextlib.closing(db_connection.cursor()) as cursor:
values = [cursor.mogrify('(%(id)s, %(test1)s, %(test2)s)', row) for row in rows]
query_bytes = b'INSERT INTO films (id, test1, test2) VALUES ' + b', '.join(values) + b';'

strange mysql insert error

I'm trying to insert dict values into database. But i'm getting error .here is my dictionary. I'm using this code .
`cols = filter_dict.keys()
vals = filter_dict.values()
("INSERT INTO table (%s) VALUES (%s)", [cols, vals])
`
When I only print this query ("INSERT INTO table (%s) VALUES (%s)", [cols, vals]).
I'm getting following output.
('INSERT INTO table (%s) VALUES (%s)', [['cinematography', 'name', 'producer', 'caption', 'studio', 'editing'], ['Venkat Prasad', '100% Love', 'Bunny Vasu', '', '100% Love Poster.jpg', 'Chandra Sekhar T Ramesh,Hari Prasad']])
But When I execute this query I"m getting strange output.
the manual that corresponds to your MySQL server version for the right syntax to use near \'table (("\'cinematography\'", "\'name\
Why execute query adding `\' to each columns?? Can any one help me? thanks

The backslashes are likely due to your debugger. Are you running your code in the interactive prompt?
Anyway, the actual problems are:
table is a reserved word. You should put table in backticks.
If your table contains more than one column, you need to have multiple %s and multiple values.
For example:
"INSERT INTO `table` (%s, %s, &s) VALUES (%s, %s, %s)"
You will also need to change (cols, vals) to list the individual values.
(cols[0], cols[1], cols[2] , vals[0], vals[1], vals[2])
I'd also strongly suggest that you try to find a better name for your table, preferably one that:
1) describes what sort of data the table contains and
2) isn't a reserved word