Develop Reference

Issue with Bigquery table created using Dataframe in Python

I have created a temporary Bigquery table using Python and loaded data from a panda dataframe (code snippet given below).
client=bigquery.Client(project)
client.create_table(tmp_table)
client.load_table_from_dataframe(df,tmp_table)
The table is being created successfully and I can run select queries from web UI.
But when I run a select query using python
query =f"""select * from {tmp_table.project_id}.{tmp_table.dataset_id}.{tmp_table.table_id} """
It throws error select * would expand to zero columns
This is because there python is not able to detect any schema. Below query returns null:
print(tmp_table.schema)
If I hardcode the table name like below, it works fine :
query =f"""select * from project_id.dataset_id.table_id """
Can someone suggest how do I get data from the temporary table using a select query in python? I can't hardcode table name as it's being created at runtime.

Execute sql insert statements via sql alchemy within for loop

I need to extract a bunch of records within a singlestore database and insert the records into another table. For performance, the ideal way to do this is to create a query string with an Insert Into statement and iterate through on a daily basis.
I can't seem to get python to execute the query in the database, but it appears to run successfully?
fh = 'file_containing_insert_select_query.sql'
qry = open(fh).read()
for i in range(2):
qry_new = some_custom_function_to_replace_dates(qry, i)
engine = tls.custom_engine_function()
engine.execute(qry_new)
I've verified that the sql statements created by my custom function can be copy/pasted to a sql editor and executed successfully, but it won't run in python... any thoughts?

After execution of the above query, you need to send a 'commit' query to database using connection.commit()
(Where connection contains the Database connection credentials and ip address) so it can save the no. of rows inserted via python program.

if you want it to run really fast, it’s usually better to use set-oriented SQL, like INSERT INTO targetTbl … SELECT FROM …;
That way, it doesn’t have to round-trip through the client app.

Upload data to Exasol from python Dataframe

I wonder if there's anyways to upload a dataframe and create a new table in Exasol? import_from_pandas assumes the table already exists. Do we need to run a SQL separately to create the table? for other databases, to_sql can just create the table if it doesn't exist.

Yes, As you mentioned import_from_pandas requires a table. So, you need to create a table before writing to it. You can run a SQL create table ... script by connection.execute before using import_from_pandas. Also to_sql needs a table since based on the documentation it will be translated to a SQL insert command.

Pandas to_sql allows to create a new table if it does not exist, but it needs an SQLAlchemy connection, which is not supported for Exasol out of the box. However, there seems to be a SQLAlchemy dialect for Exasol you could use (haven't tried it yet): sqlalchemy-exasol.
Alternatively, I think you have to use a create table statement and then populate the table via pyexasol's import_from_pandas.

python sqlalchemy built temporary table sub-select

I'm working on a very complex raw sql statement that has to be run with sqlalchemy. One part of the statement is a JOIN with a temporary table that is filled with data from a csv file.
The select is looking as follows:
(SELECT * FROM (VALUES ('1','xyz#something.com','+99123456798')) AS t (id, email, phone))
To prevent any sql injections I cannot simply copy paste everything from within the csv into the select.
I know that sqlalchemy has the option of inserting values with :x and then pass the actual value in the execute method, but I have A LOT of values and substituting them will be impossible.
Is there a way to build this temporary table from the csv with the necessary sql injection protection using sqlalchemy?

when pymysql using local_infile, data is not insert database

I created the code to create and execute SQL query statements using pymysql in Python.
If i put the sql query statement generated in code directly in db, it works normally.
However, if i execute the sql query statement created in code with cursor.execute (sql), db will not contain data.
When I connect, I also gave the local_infile = True option. How do I create the code?