When I use:
for i in Selection:
Q = "SELECT columnA FROM DB WHERE wbcode='"+i+"' and commodity='1'"
cursor.execute(Q)
ydata[i] = cursor.fetchall()
I get:
ydata = {'GBR': [(u'695022',), (u'774291',), (u'791499',)... ]}
How can I change my code to get:
ydata = {'GBR': [695022, 774291, 791499,...]}
Thank you very much.
obs: this is just a a simplified example. try to refrain from making recommendations about sql injection.
[int(x[0]) for x in cursor.fetchall()]
Based on this and another question of yours, you need to understand SQLite's affinity and how you are populating the database. Other databases require that the values stored in a column are all of the same type - eg all strings or all integers. SQLite allows you to store anything so the type in each row can be different.
To a first approximation, if you put in a string for that row then you'll get a string out, put in an integer and you'll get an integer out. In your case you are getting strings out because you put strings in instead of integers.
However you can declare a column affinity and SQLite will try to convert when you insert data. For example if a column has integer affinity then if what you insert can be safely/correctly converted to an integer then SQLite will do so, so the string "1" will indeed be stored as the integer 1 while "1 1" will be stored as the string "1 1".
Read this page to understand the details. You'll find things a lot easier getting data out if you put it in using the correct types.
http://www.sqlite.org/datatype3.html
If you are importing CSV data then start the APSW shell and use ".help import" to get some suggestions on how to deal with this.
Related
I have a list/array of strings:
l = ['jack','jill','bob']
Now I need to create a table in slite3 for python using which I can insert this array into a column called "Names". I do not want multiple rows with each name in each row. I want a single row which contains the array exactly as shown above and I want to be able to retrieve it in exactly the same format. How can I insert an array as an element in a db? What am I supposed to declare as the data type of the array while creating the db itself? Like:
c.execute("CREATE TABLE names(id text, names ??)")
How do I insert values too? Like:
c.execute("INSERT INTO names VALUES(?,?)",(id,l))
EDIT: I am being so foolish. I just realized that I can have multiple entries for the id and use a query to extract all relevant names. Thanks anyway!
You can store an array in a single string field, if you somehow genereate a string representation of it, e.g. sing the pickle module. Then, when you read the line, you can unpickle it. Pickle converts many different complex objects (but not all) into a string, that the object can be restored of. But: that is most likely not what you want to do (you wont be able to do anything with the data in the tabel, except selecting the lines and then unpickle the array. You wont be able to search.
If you want to have anything of varying length (or fixed length, but many instances of similiar things), you would not want to put that in a column or multiple columns. Thing vertically, not horizontally there, meaning: don't thing about columns, think about rows. For storing a vector with any amount of components, a table is a good tool.
It is a little difficult to explain from the little detail you give, but you should think about creating a second table and putting all the names there for every row of your first table. You'd need some key in your first table, that you can use for your second table, too:
c.execute("CREATE TABLE first_table(int id, varchar(255) text, additional fields)")
c.execute("CREATE TABLE names_table(int id, int num, varchar(255) name)")
With this you can still store whatever information you have except the names in first_table and store the array of names in names_table, just use the same id as in first_table and num to store the index positions inside the array. You can then later get back the array by doing someting like
SELECT name FROM names_table
WHERE id=?
ORDER BY num
to read the array of names for any of your rows in first_table.
That's a pretty normal way to store arrays in a DB.
This is not the way to go. You should consider creating another table for names with foreign key to names.
You could pickle/marshal/json your array and store it as binary/varchar/jsonfield in your database.
Something like:
import json
names = ['jack','jill','bill']
snames = json.dumps(names)
c.execute("INSERT INTO nametable " + snames + ";")
I have a dataframe (originally from a csv)
df = pd.read_excel (r'R:\__Test Server\DailyStatuses\DailyWork10.18.xlsx')
I created a dataframe to deal with some null values in the rows. I've also created the table in SQL Server, and defined the column types (datetime, int and varchar).
I'm building an insert string to insert the data into a new table in SQL Server.
insert_query='INSERT INTO [DailyStatuses].[dbo].[StatusReports] VALUES (
for i in range (df.shape[0]):
for j in range(df.shape[1]):
insert_query += (df[df.columns.values[j]][i]) +','
insert_query= insert_query[:-1] + '),('
insert_query = insert_query[:-3] + ');'
My output is:
INSERT INTO [DailyStatuses].[dbo].[StatusReports] VALUES (3916, 2019-10-17 16:45:54...
I'm constantly running into errors about data types, is it best to define everything as a str so it's easier to insert into SQL Server (and define each column in the table as a str) and define data types upon extraction later down the road?
You would be better off using parameters but based on the question you asked, you are going to have to deal with each data type separately.
For int values, you'll be fine with them as is.
For string values, you'll have to put single quotes around them i.e. 'a value' and you'll also need to replace any embedded single quotes with two single quotes.
For datetime values, you should use a format that isn't affected by regional settings and put quotes around them i.e. '20191231 12:54:54'
The other alternative (as you suggest) is to bring them all in as strings, and do the clean-up, and data-type changes within SQL Server. That's often a more reliable direction. Again though, don't forget to double up any embedded single quotes within the values.
I have been trying this for hours now. But this is not working as expected.
I am pushing data to elasticsearch via python script. Below are some fields I want as integers, but they are not being stored as integers. Sometimes, they are of None Type, else they are strings. So, I did this
body['fuel_fee'] = int(rows[a][23] or 0)
body['late_fee'] = int(rows[a][24] or 0)
body['other_fee'] = int(rows[a][26] or 0)
But I see that they are still being stored as strings in elastic. I want sum
operation on these.
I even deleted index and rewrote the entire data, so I can confirm that there is no issue of previous mappings here.
Why am I not getting these fields as integers ? How can I get this done ?
EDIT - I am fetching data from postgres database. And in postgres, these fields are stored as strings, not integers. Can it have any effect ? I think no, as I am type casting in here in python.
The datatype of a field is determined in either of the following ways
When you create mappings (before indexing any real data) and explicitly tell elasticsearch about the field type. In your example, the field fuel_fee will be mapped to long and any record containing non-integral values will throw an error
Based on the first document indexed, elasticsearch determines the field type. It tries to convert the subsequent document field values to the same type thereafter.
Coming back to your question, how do you know that all your fields are stored as strings and not integer? Can you try GET <your-index>/_mapping and see if your assumption is correct.
If the problem persists, try any of the following:
Create mappings before indexing any data.
Index only 1 document(with kibana or through curl api) and check the mapping output again.
I'm using python and sqlalchemy.
I have a db with a column that is Numeric type.
I want to query the db to check if there is an entry that has a specific value in the column.
Let's assume the value I want to look for is 1.04521, and we know it's in the db.
I've tried
(result,) = session.query( exists().where(MyEntryClass.someNumericValue == 1.0452)
but result is still False even when I know it's in the db.
How do I check to see if there is an entry with a column with a specific Numeric value?
Post Original Question:
After a little more exploration, I think it's due to rounding/representation of the non-integer number.
Actually you are not getting the result because there are query executors like all(), first(), scalar() so use scalar:
result = session.query(exists().where(MyEntryClass.someNumericValue==1.0452)).scalar()
When using the sqlite3 module in python, all elements of cursor.description except the column names are set to None, so this tuple cannot be used to find the column types for a query result (unlike other DB-API compliant modules). Is the only way to get the types of the columns to use pragma table_info(table_name).fetchall() to get a description of the table, store it in memory, and then match the column names from cursor.description to that overall table description?
No, it's not the only way. Alternatively, you can also fetch one row, iterate over it, and inspect the individual column Python objects and types. Unless the value is None (in which case the SQL field is NULL), this should give you a fairly precise indication what the database column type was.
sqlite3 only uses sqlite3_column_decltype and sqlite3_column_type in one place, each, and neither are accessible to the Python application - so their is no "direct" way that you may have been looking for.
I haven't tried this in Python, but you could try something like
SELECT *
FROM sqlite_master
WHERE type = 'table';
which contains the DDL CREATE statement used to create the table. By parsing the DDL you can get the column type info, such as it is. Remember that SQLITE is rather vague and unrestrictive when it comes to column data types.