Develop Reference

How to print value of list in Python (array[25,65], array[25,42], array[25,24]) to (25.65, 25.42, 25,24)

We're working on a python program, where we have trouble sending data to our MySQL database. So far, we are selecting data from our database and we want to do something with the data in our python program and then send it back to our database.
Unfortunately, we're having some challenges, which we hope you can help us with.
We're receiving this error:
[SQL: INSERT INTO `Raw_Validated` (time_start, time_end, first_temp_lpn, first_temp_lpn_validated, second_temp_lpn, second_temp_lpn_validated, third_temp_lpn, third_temp_lpn_validated) VALUES (%s, %s, %s, %s, %s, %s, %s, %s)]
[parameters: ('2019-08-29 16:20:00', '2019-08-29 17:20:00', array([25.69]), 1, array([25.21]), 1, array([25.09]), 1)]
And we can conclude that instead of inserting a value, an array is inserted. We have no idea why this is happening or how we can prevent this, but instead of the parameters above, we want it to become like this:
[parameters: ('2019-08-29 16:20:00', '2019-08-29 17:20:00', 25.69, 1, 25.21, 1, 25.09, 1)]
We're running a for loop which iterate 3x times, which means we are receiving 3x 'a_temp' values, which are saved into our list 'list_lpn_temp (for-loop is not shown in code snippet):
list_lpn_temp = []
new_list_lpn_temp = []
engine = create_engine("mysql://xxx:xxx#localhost/xxx")
conn = engine.connect()
a_temp = pd.read_sql('SELECT temperature FROM Raw_Data WHERE topic = "lpn1" AND timestamp > "%s" AND timestamp < "%s" ORDER BY timestamp DESC LIMIT 1' % (x, x+datetime.timedelta(minutes=20)), conn).astype(float).values
list_lpn_temp.extend(a_temp)
We then have another for loop (keep in mind that list_station has not been initialized, but in our program it has been):
for i in range (len(list_lpn_temp)):
if -1.5 < list_station_temp[i]-list_lpn_temp[i] < 1.5:
validated_lpn = 1
list_validated.append(validated_lpn)
new_list_lpn_temp.extend(list_lpn_temp[i])
print(f'New LPN List = {new_list_lpn_temp}')
else:
validated_lpn = 0
list_validated.append(validated_lpn)
We then prepare the data so we can send it further to the database (there are a lot of new uninitialized variables here, which we have initialized in our program, but not here, as they simply dont matter). Only list_lpn_temp[] matters here:
df2 = pd.DataFrame(columns=['time_start', 'time_end', 'first_temp_lpn', 'first_temp_lpn_validated', 'second_temp_lpn', 'second_temp_lpn_validated', 'third_temp_lpn', 'third_temp_lpn_validated'])
df2 = df2.append({'time_start' : time_start, 'time_end' : time_end, 'first_temp_lpn' : list_lpn_temp[0], 'first_temp_lpn_validated' : list_validated[0], 'second_temp_lpn' : list_lpn_temp[1], 'second_temp_lpn_validated' : list_validated[1$
with engine.connect() as conn, conn.begin():
df2.to_sql('Raw_Validated', conn, if_exists='append', index=False)

Just add one more level of indexing to all list_lpn_temp accesses, so list_lpn_temp[0] will become list_lpn_temp[0][0] and list_lpn_temp[1] will become list_lpn_temp[1][0] etc.
df2 = pd.DataFrame(columns=['time_start', 'time_end', 'first_temp_lpn', 'first_temp_lpn_validated', 'second_temp_lpn', 'second_temp_lpn_validated', 'third_temp_lpn', 'third_temp_lpn_validated'])
df2 = df2.append({'time_start' : time_start, 'time_end' : time_end, 'first_temp_lpn' : list_lpn_temp[0][0], 'first_temp_lpn_validated' : list_validated[0], 'second_temp_lpn' : list_lpn_temp[1][0], 'second_temp_lpn_validated' : list_validated[1$ # Your question cut this line off here also.
with engine.connect() as conn, conn.begin():
df2.to_sql('Raw_Validated', conn, if_exists='append', index=False)

Issue using positional parameters in sql script, using pg8000

I have the following SQL query:
query_string = "SELECT sum(unmatched), " \
"TIMESTAMP WITH TIME ZONE 'epoch' + INTERVAL '1 second' * " \
"round(extract('epoch' from time_window) / {}) * {} as time_window " \
"FROM aggregate_counts WHERE reconciliation_name = %s " \
"GROUP BY round(extract('epoch' from time_window) / {})".format(interval_sec, interval_sec, interval_sec)
cur.execute(query_string, (reconciliation_name))
It works fine unless I want to avoid using string replacement for "interval_sec" and use positional parameters instead, like I have for other parameters. Problem is, if I do that:
query_string = "SELECT sum(unmatched), " \
"TIMESTAMP WITH TIME ZONE 'epoch' + INTERVAL '1 second' * " \
"round(extract('epoch' from time_window) / %s) * %s as time_window " \
"FROM aggregate_counts WHERE reconciliation_name = %s " \
"GROUP BY round(extract('epoch' from time_window) / %s)"
cur.execute(query_string, (interval_sec, interval_sec, reconciliation_name, interval_sec))
I get the following error:
Error handler middleware caught the following exception: {'S':
'ERROR', 'V': 'ERROR', 'C': '42803', 'M': 'column
"aggregate_counts.time_window" must appear in the GROUP BY clause or
be used in an aggregate function', 'P': '177', 'F': 'parse_agg.c',
'L': '1344', 'R': 'check_ungrouped_columns_walker'}
File
File "pg8000/core.py", line 1829, in execute
ps = cache['ps'][key]
KeyError: ("SELECT sum(unmatched), TIMESTAMP WITH TIME ZONE 'epoch' + INTERVAL '1 second' * round(extract('epoch' from time_window) / %s) * %s as time_window FROM aggregate_counts WHERE reconciliation_name = %s GROUP BY round(extract('epoch' from time_window) / %s)", ((701, 1, ), (701, 1, ), (705, 0, .text_out at 0x10c58cea0>)))
Can positional parameters only be used in comparisons (=, >=, < etc...)?

So it's basically not possible to have this. Reason being the %s in select clause will get converted to let's say, $x positional argument, and the %s in group by will get converted to $y (x and y being the respective positions.) Now postgres has no way of knowing that after resolving, these two will be the same. Hence it assumes that "aggregate_counts.time_window" is not present in GROUP BY. I understand it's not the perfect explanation, but this is kind of what is happening.

sqlite , python c.execute AttributeError: 'str' object has no attribute 'execute'

Getting an error:
c.execute('CREATE TABLE IF NOT EXISTS top15' +today +'(symbol TEXT, ending_price REAl, volume REAL, percent REAL)')
AttributeError: 'str' object has no attribute 'execute'
Below is original code where no error occurs.
conn = sqlite3.connect('Top15.db')
c = conn.cursor()
def create_table():
c.execute('CREATE TABLE IF NOT EXISTS top15' +today +'(symbol TEXT, ending_price REAl, volume REAL, percent REAL)')
f15= ['hgd', 'bbd.a', 'mri.u', 'iam', 'hnd', 'tth', 'bbd.b', 'bbd.pr.c', 'esp', 'enl', 'rmp', 'amm', 'vrx', 'xtc', 'cxr']
f45=['4,433,389', '2.96', '-13.7', '1,209,421', '2.25', '-13.1', '3,000', '8.60', '-8.5', '1,000', '1.06', '-7.8', '1,180,466', '21.76', '-7.6', '41,777', '0.97', '-7.6', '32,423,597', '1.89', '-7.4', '43,737', '15.20', '-7.3', '87,604', '1.96', '-7.1', '5,239', '34.00', '-6.2', '2,688,261', '1.83', '-5.7', '63,301', '1.39', '-5.4', '1,664,689', '41.83', '-5.4', '63,453', '13.45', '-5.3', '1,642,197', '36.48', '-5.0']
def dynamic_data_entry():
volume = first_45[i]
ending_price = first_45[i+1]
percent = first_45[i+2]
symbol = first_15[z]
c.execute("INSERT INTO top15" +today +"(symbol,ending_price, volume, percent) VALUES (?, ?, ?, ?)",
(symbol,ending_price, volume, percent))
conn.commit()
create_table()
for i, z in zip(range(0,45,3),range(15)):
dynamic_data_entry()
c.close
conn.close
Below is the new setup. Nothing else has changed other than turning the two lists (f15,f45) into a single list (result) with internal lists. However,now I get the c.execute error. Ive read about c.execute errors and cannot find a solution
conn = sqlite3.connect('Top15.db')
c = conn.cursor()
def create_table():
c.execute('CREATE TABLE IF NOT EXISTS top15' +today +'(symbol TEXT, ending_price REAl, volume REAL, percent REAL)')
f15=[15 list items]
f45=[45 list items]
f45i = iter(f45)
result = [[a, c, b, d] for (a, b, c, d) in zip(f15, f45i, f45i, f45i)]
result = filter(lambda l: l[0].count('.') <= 1, result)
print result
result=[['hgd', '2.96', '4,433,389', '-13.7'], ['bbd.a', '2.25', '1,209,421', '-13.1'], ['mri.u', '8.60', '3,000', '-8.5'], ['iam', '1.06', '1,000', '-7.8'], ['hnd', '21.76', '1,180,466', '-7.6'], ['tth', '0.97', '41,777', '-7.6'], ['bbd.b', '1.89', '32,423,597', '-7.4'], ['esp', '1.96', '87,604', '-7.1'], ['enl', '34.00', '5,239', '-6.2'], ['rmp', '1.83', '2,688,261', '-5.7'], ['amm', '1.39', '63,301', '-5.4'], ['vrx', '41.83', '1,664,689', '-5.4'], ['xtc', '13.45', '63,453', '-5.3'], ['cxr', '36.48', '1,642,197', '-5.0']]
def dynamic_data_entry():
symbol = result[i][0]
ending_price = result[i][1]
volume = result[i][2]
percent = result[i][3]
c.execute("INSERT INTO top15" +today +"(symbol, ending_price, volume, percent) VALUES (?, ?, ?, ?)",
(symbol,ending_price, volume, percent))
conn.commit()
create_table()
i=0
while i <len(result):
dynamic_data_entry()
i+=1
c.close
conn.close
Is there something about the list inside the list creating the problem? Not too sure why this would now result in this error. When it worked perfectly fine below

How to use python to ETL between databases?

Using psycopg2, I'm able to select data from a table in one PostgreSQL database connection and INSERT it into a table in a second PostgreSQL database connection.
However, I'm only able to do it by setting the exact feature I want to extract, and writing out separate variables for each column I'm trying to insert.
Does anyone know of a good practice for either:
moving an entire table between databases, or
iterating through features while not having to declare variables for every column you want to move
or...?
Here's the script I'm currently using where you can see the selection of a specific feature, and the creation of variables (it works, but this is not a practical method):
import psycopg2
connDev = psycopg2.connect("host=host1 dbname=dbname1 user=postgres password=*** ")
connQa = psycopg2.connect("host=host2 dbname=dbname2 user=postgres password=*** ")
curDev = connDev.cursor()
curQa = connQa.cursor()
sql = ('INSERT INTO "tempHoods" (nbhd_name, geom) values (%s, %s);')
curDev.execute('select cast(geom as varchar) from "CCD_Neighborhoods" where nbhd_id = 11;')
tempGeom = curDev.fetchone()
curDev.execute('select nbhd_name from "CCD_Neighborhoods" where nbhd_id = 11;')
tempName = curDev.fetchone()
data = (tempName, tempGeom)
curQa.execute (sql, data)
#commit transactions
connDev.commit()
connQa.commit()
#close connections
curDev.close()
curQa.close()
connDev.close()
connQa.close()
One other note is that python allows the ability to explicitly work with SQL functions / data type casting, which for us is important as we work with the GEOMETRY data type. Above you can see I'm casting it to TEXT then dumping it into an existing geometry column in the source table - this will work with MSSQL Server, which is a huge feature in the geospatial community...

In your solution (your solution and your question have a different order of statements) change the lines which start with 'sql = ' and the loop before '#commit transactions' comment to
sql_insert = 'INSERT INTO "tempHoods" (nbhd_id, nbhd_name, typology, notes, geom) values '
sql_values = ['(%s, %s, %s, %s, %s)']
data_values = []
# you can make this larger if you want
# ...try experimenting to see what works best
batch_size = 100
sql_stmt = sql_insert + ','.join(sql_values*batch_size) + ';'
for i, row in enumerate(rows, 1):
data_values += row[:5]
if i % batch_size == 0:
curQa.execute (sql_stmt , data_values )
data_values = []
if (i % batch_size != 0):
sql_stmt = sql_insert + ','.join(sql_values*(i % batch_size)) + ';'
curQa.execute (sql_stmt , data_values )
BTW, I don't think you need to commit. You don't begin any transactions. So there should not be any need to commit them. Certainly, you don't need to commit a cursor if all you did was a bunch of selects on it.

Here's my updated code based on Dmitry's brilliant solution:
import psycopg2
connDev = psycopg2.connect("host=host1 dbname=dpspgisdev user=postgres password=****")
connQa = psycopg2.connect("host=host2 dbname=dpspgisqa user=postgres password=****")
curDev = connDev.cursor()
curQa = connQa.cursor()
print "Truncating Source"
curQa.execute('delete from "tempHoods"')
connQa.commit()
#Get Data
curDev.execute('select nbhd_id, nbhd_name, typology, notes, cast(geom as varchar) from "CCD_Neighborhoods";') #cast geom to varchar and insert into geometry column!
rows = curDev.fetchall()
sql_insert = 'INSERT INTO "tempHoods" (nbhd_id, nbhd_name, typology, notes, geom) values '
sql_values = ['(%s, %s, %s, %s, %s)'] #number of columns selecting / inserting
data_values = []
batch_size = 1000 #customize for size of tables...
sql_stmt = sql_insert + ','.join(sql_values*batch_size) + ';'
for i, row in enumerate(rows, 1):
data_values += row[:5] #relates to number of columns (%s)
if i % batch_size == 0:
curQa.execute (sql_stmt , data_values )
connQa.commit()
print "Inserting..."
data_values = []
if (i % batch_size != 0):
sql_stmt = sql_insert + ','.join(sql_values*(i % batch_size)) + ';'
curQa.execute (sql_stmt, data_values)
print "Last Values..."
connQa.commit()
# close connections
curDev.close()
curQa.close()
connDev.close()
connQa.close()

Data Not Being Inserted Into Sqlite3 Database Using Python 2.7

I'm having trouble inserting data into my table. I have a list of stocks that I pass to the function getStockData.
I use a for loop to iterate through the list and get the data for each ticker symbol. At the end I put all the information into a dictionary. My final step is to insert the data into a table. I've been unsuccessful at inserting the data in the dictionary into my table.
def getStockData(x):
nowdate = raw_input("What Is Todays Date?: ")
print "Todays list has %d stocks on it\n" % len(x)
for stock in x:
stockPrice = ystockquote.get_price(stock)
stockPriceChange = ystockquote.get_change(stock)
originalPrice = float(stockPrice) + (float(stockPriceChange) * -1)
changePercentage = (float(stockPriceChange) / originalPrice) * 100
stockDict = {'Date': nowdate, 'Ticker Symbol': stock, 'Closing Price': stockPrice,
'Price Change': stockPriceChange, 'Percentage Changed': changePercentage}
conn = db.connect('stocks.db')
cursor = conn.cursor()
cursor.execute('insert into losers values (?, ?, ?, ?, ?)', (stockDict['Date'], stockDict['Ticker Symbol'], stockDict['Price Change'],
stockDict['Percentage Changed'], stockDict['Closing Price']) )
conn.close()

I think you forget to commit your data to your DB before close.
Try
conn.commit()