When printing PostgreSQL query results I only see the result values, I would like to see the column name along with the result values
postgreSQL_select_Query = "SELECT epic,timestamp FROM market_data_historic s1 WHERE timestamp = (SELECT MAX(timestamp) FROM market_data_historic s2 WHERE s1.epic = s2.epic)"
cursor.execute(postgreSQL_select_Query)
# Close the connection
result=(cursor.fetchall())
for row in result:
print (row)
This is the result I get:
('CC.D.LCO.USS.IP', datetime.datetime(2019, 11, 13, 22, 0))
('IX.D.DAX.DAILY.IP', datetime.datetime(2019, 7, 23, 4, 0))
('KB.D.ELECTY.DAILY.IP', datetime.datetime(2020, 1, 24, 16, 0))
('CS.D.ALUMINIUM.TODAY.IP', datetime.datetime(2019, 7, 23, 1, 0))
('CS.D.NZDCAD.TODAY.IP', datetime.datetime(2020, 1, 24, 21, 0))
('CS.D.CADCNH.TODAY.IP', datetime.datetime(2020, 1, 16, 8, 0))
How can I get it to be like this:
(epic:'CC.D.LCO.USS.IP',timestamp: datetime.datetime(2019, 11, 13, 22, 0))
(epic:'IX.D.DAX.DAILY.IP',timestamp: datetime.datetime(2019, 7, 23, 4, 0))
(epic:'KB.D.ELECTY.DAILY.IP',timestamp: datetime.datetime(2020, 1, 24, 16, 0))
(epic:'CS.D.ALUMINIUM.TODAY.IP',timestamp: datetime.datetime(2019, 7, 23, 1, 0))
(epic:'CS.D.NZDCAD.TODAY.IP',timestamp: datetime.datetime(2020, 1, 24, 21, 0))
(epic:'CS.D.CADCNH.TODAY.IP',timestamp: datetime.datetime(2020, 1, 16, 8, 0))
Use the cursor.description attribute to retrieve column names and convert results to dictionaries:
result = cursor.fetchall()
columns = [desc[0] for desc in cursor.description]
for row in result:
print (dict(zip(columns, row)))
Alternatively, you can use Real dictionary cursor or Named tuple cursor.
See also DictCursor vs RealDictCursor.
Try the description attribute of cursor:
Read-only attribute describing the result of a query. It is a sequence of Column instances, each one describing one result column in order. The attribute is None for operations that do not return rows or if the cursor has not had an operation invoked via the execute*() methods yet.
For compatibility with the DB-API, every object can be unpacked as a 7-items sequence: the attributes retuned this way are the following. For further details and other attributes available check the Column documentation.
name: the name of the column returned.
type_code: the PostgreSQL OID of the column.
display_size: the actual length of the column in bytes.
internal_size: the size in bytes of the column associated to this column on the server.
precision: total number of significant digits in columns of type NUMERIC. None for other types.
scale: count of decimal digits in the fractional part in columns of type NUMERIC. None for other types.
null_ok: always None as not easy to retrieve from the libpq.
Related
I have a dictionary weeks that is populated using a for loop. The dictionary contains arguments passed to a function. I'd like to update the code to read second argument and pass that to the function.
from datetime import date, datetime, timedelta
weeks = {}
def generate(date):
# generate 8 rows with 4 cells each
for d in range(1, 32, 4):
start = (date + timedelta(days=d)).strftime('%Y-%m-%d')
week = pd.date_range(start, periods=4).to_pydatetime().tolist()
weeks[d] = week
print(weeks)
def create(arg1):
# do something
return ...
This function populates a dictionary with dates as a list of values.
{1: [datetime.datetime(2021, 10, 13, 0, 0), datetime.datetime(2021, 10, 14, 0, 0), datetime.datetime(2021, 10, 15, 0, 0), datetime.datetime(2021, 10, 16, 0, 0)], 5: [datetime.datetime(2021, 10, 17, 0, 0), datetime.datetime(2021, 10, 18, 0, 0), datetime.datetime(2021, 10, 19, 0, 0), datetime.datetime(2021, 10, 20, 0, 0)],.......
Now I'd like to either modify the weeks dictionary or create a new dictionary that I'll iterate over to read the second argument of the function.
The function call:
[create(i)] for i in weeks[1]
I'd like to modify this call to include a second argument which is read from the dictionary with same structure as well. Pseudocode below:
[create(i,j)] for i,j in weeks[1], value[1]
where value is the new dictionary with populate, if we don't modify the existing dict weeks. I'd appreciate advise on which approach makes sense and how can I modify the function call to pass two arguments.
value dict will look like this:
value = {1: [1000, 1200, 1400, 1600], 5: [2000, 2200, 2400, 2600]}
Use zip to iterate over two lists in parallel, it will yield pairs in tuples
[create(i, j) for i, j in zip(weeks[1], value[1]))
I have a dictionary like this:
dict_connected_hosts = {
{'10.0.0.2': [[12564.0, 6844.0, 632711.0, 56589,0, 4856,0], <ryu.controller.controller.Datapath object at 0x7f2b2008a7d0>, '10.0.0.2', '10.0.0.1', 2, datetime.datetime(2017, 9, 26, 2, 24, 12, 301565)]}
{'10.0.0.3': [[3193.0, 621482.0, 6412.0, 2146.0, 98542.0], <ryu.controller.controller.Datapath object at 0x7f2b2008a7d0>, '10.0.0.3', '10.0.0.1', 3, datetime.datetime(2017, 9, 26, 2, 24, 12, 302224)]
{'10.0.0.7': [[4545.0, 51442.0, 325.0, 452.0, 3555.0], <ryu.controller.controller.Datapath object at 0x7f2b2008a7d0>, '10.0.0.7', '10.0.0.1', 3, datetime.datetime(2017, 9, 26, 2, 24, 12, 302250)]
}
how can I sum the first numbers of each list in the value field? In simple terms numbers
`12564.0 + 3193.0 + 4545.0`
thanks
I have debugged your dictionary structure. The relevant part of it should be :
{
'10.0.0.2': [[12564.0, 6844.0, 632711.0, 56589,0, 4856,0]],
'10.0.0.3': [[3193.0, 621482.0, 6412.0, 2146.0, 98542.0]],
'10.0.0.7': [[4545.0, 51442.0, 325.0, 452.0, 3555.0]]
}
Note : ignoring the other elements in the values as they are not relevant to the question (and they have errors I don't care to debug)
So, to get the sum of the first numbers in the first list of each value, you can do it by list comprehension :
#suppose `a` is the dictionary
print([val[0][0] for val in a.values()])
#[12564.0, 3193.0, 4545.0]
print(sum( [val[0][0] for val in a.values()] ))
#20302.0
I would like to create an array or list from values pulled from a SQL query. From research I believe the data I pull from SQL is a tuple.
How do format the data into a list I can use in python?
In my current code I try to use the numpy command np.asarray. I'm not sure if numpy arrays allow datetime.
import numpy as np
import pyodbc
conn = pyodbc.connect('login')
cursor = conn.cursor()
cursor.execute("SELECT PTIME, PVALUE FROM HISTORY_TABLE WHERE POINT = 'Value' AND PTIME> '2017-04-12' AND PTIME<'2017-04-13' AND HISTTYPE='AVG' AND PERIOD=7200")
sample = cursor.fetchall()
rockin = np.asarray(sample)
print rockin
cursor.close()
conn.close()
My result looks like this:
[[datetime.datetime(2017, 4, 12, 0, 0) 232.83]
[datetime.datetime(2017, 4, 12, 2, 0) 131.49]
[datetime.datetime(2017, 4, 12, 4, 0) 36.67]
...,
[datetime.datetime(2017, 4, 12, 18, 0) 82.08]
[datetime.datetime(2017, 4, 12, 20, 0) 368.83]
[datetime.datetime(2017, 4, 12, 22, 0) 435.79]]
From research I believe the data I pull from SQL is a tuple.
Not exactly. pyodbc's fetchall() method does not return a list of tuples, it returns a list of pyodbc.Row objects:
>>> rows = crsr.execute("SELECT 1 AS foo UNION ALL SELECT 2 AS foo").fetchall()
>>> rows
[(1, ), (2, )]
>>> type(rows[0])
<type 'pyodbc.Row'>
It could be that np.asarray does not know how to handle pyodbc.Row objects. If you want to convert each row to an actual tuple you can use
>>> rows_as_tuples = [tuple(x) for x in rows]
>>> rows_as_tuples
[(1,), (2,)]
>>> type(rows_as_tuples[0])
<type 'tuple'>
I haven't been able to find anything about this value error online and I am at a complete loss as to why my code is eliciting this response.
I have a large dictionary of around 50 keys. The value associated with each key is a 2D array of many elements of the form [datetime object, some other info]. A sample would look like this:
{'some_random_key': array([[datetime(2010, 10, 26, 11, 5, 28, 157404), 14.1],
[datetime(2010, 10, 26, 11, 5, 38, 613066), 17.2]],
dtype=object),
'some_other_key': array([[datetime(2010, 10, 26, 11, 5, 28, 157404), 'true'],
[datetime(2010, 10, 26, 11, 5, 38, 613066), 'false']],
dtype=object)}
What I want my code to do is to allow a user to select a start and stop date and remove all of the array elements (for all of the keys) that are not within that range.
Placing print statements throughout the code I was able to deduce that it can find the dates that are out of range, but for some reason, the error occurs when it attempts to remove the element from the array.
Here is my code:
def selectDateRange(dictionary, start, stop):
#Make a clone dictionary to delete values from
theClone = dict(dictionary)
starting = datetime.strptime(start, '%d-%m-%Y') #put in datetime format
ending = datetime.strptime(stop+' '+ '23:59', '%d-%m-%Y %H:%M') #put in datetime format
#Get a list of all the keys in the dictionary
listOfKeys = theClone.keys()
#Go through each key in the list
for key in listOfKeys:
print key
#The value associate with each key is an array
innerAry = theClone[key]
#Loop through the array and . . .
for j, value in enumerate(reversed(innerAry)):
if (value[0] <= starting) or (value[0] >= ending):
#. . . delete anything that is not in the specified dateRange
del innerAry[j]
return theClone
This is the error message that I get:
ValueError: cannot delete array elements
and it occurs at the line: del innerAry[j]
Please help - perhaps you have the eye to see the problem where I cannot.
Thanks!
If you use numpy arrays, then use them as arrays and not as lists
numpy does comparison elementwise for the entire array, which can then be used to select the relevant subarray. This also removes the need for the inner loop.
>>> a = np.array([[datetime(2010, 10, 26, 11, 5, 28, 157404), 14.1],
[datetime(2010, 10, 26, 11, 5, 30, 613066), 17.2],
[datetime(2010, 10, 26, 11, 5, 31, 613066), 17.2],
[datetime(2010, 10, 26, 11, 5, 32, 613066), 17.2],
[datetime(2010, 10, 26, 11, 5, 33, 613066), 17.2],
[datetime(2010, 10, 26, 11, 5, 38, 613066), 17.2]],
dtype=object)
>>> start = datetime(2010, 10, 26, 11, 5, 28, 157405)
>>> end = datetime(2010, 10, 26, 11, 5, 33, 613066)
>>> (a[:,0] > start)&(a[:,0] < end)
array([False, True, True, True, False, False], dtype=bool)
>>> a[(a[:,0] > start)&(a[:,0] < end)]
array([[2010-10-26 11:05:30.613066, 17.2],
[2010-10-26 11:05:31.613066, 17.2],
[2010-10-26 11:05:32.613066, 17.2]], dtype=object)
just to make sure we still have datetimes in there:
>>> b = a[(a[:,0] > start)&(a[:,0] < end)]
>>> b[0,0]
datetime.datetime(2010, 10, 26, 11, 5, 30, 613066)
NumPy arrays are fixed in size. Use lists instead.
This question has a little history — Is there a way to make a query respect the order of the inputted parameters?
I'm new to building "specialized" queries, so I assumed that if I supply an IN clause as part of a SELECT query, it'll return results in the same order. Unfortunately that's not the case.
SELECT * FROM artists WHERE id IN (8, 1, 2, 15, 14, 3, 13, 31, 16, 5, 4, 7, 32, 9, 37)
>>> [7, 32, 3, 8, 4, 2, 31, 9, 37, 13, 16, 1, 5, 15, 14]
(Didn't include the step where I used Python to loop through the result and append the IDs to a list.)
So the question is, is there a way to make Postgres respect the ordering of the parameters given in an IN clause by returning results the same order?
Query results will be returned in non-deterministic order unless you specify an ORDER BY clause.
If you really want to do the query in the manner you are requesting, then you could construct such a clause. Here's an example using part of your data.
create table artists (
id integer not null primary key,
name char(1) not null);
insert into artists
values
(8, 'a'),
(1, 'b'),
(2, 'c'),
(15, 'd'),
(14, 'e'),
(3, 'f'),
(13, 'g');
select *
from artists
where id in (8, 1, 2, 15, 14, 3, 13)
order by
id = 8 desc,
id = 1 desc,
id = 2 desc,
id = 15 desc,
id = 14 desc,
id = 3 desc,
id = 13 desc;
Based on this and on your other question, I think there is something wrong with your model or the way you are trying to do this. Perhaps you should post a more generic question about how to do what you are trying to do.
If you do have artists and ranking tables, you should be able to do something like this (or the equivalent through your ORM).
select
a.*
from
artists a,
rankings r
where
a.id = r.artist_id
order by
r.score desc;
I suggest you let PostGreSQL return the set in any arbitrary order (especially since it's difficult to do fine-grained SQL-level control from a Django interface), then sort it in the way you wish in Python -- theresultset.sort(key=yourlistofids.index) should do fine (when theresultset is the arbitrary-order list resulting from the database and yourlistofids is the list whose order you want to preserve).
Another way:
SELECT *
FROM artists
WHERE id IN (8, 1, 2, 15, 14, 3, 13, 31, 16, 5, 4, 7, 32, 9, 37)
ORDER BY POSITION(id::text in '(8, 1, 2, 15, 14, 3, 13, 31, 16, 5, 4, 7, 32, 9, 37)');