Insert pd.dataframe to teradata via bteq - python

When I want to insert some data from a pandas dataframe to a teradata table I do the following steps:
save the pd.DataFrame as a .csv
write a .sql file with the .logon, .import vartext from file, sql insert query and .logoff and save it locally
excecute the .sql file with bteq via the python package subprocess
wait until the process is finished and then continue with the script.
So the issue here is that it is not really save (e.g. there are laying some files locally around with the databasepassword) and a dependent on the right saving of the csv.
So what i would like to do is the following:
create the bteq query internally in the original python process without writing something to a local file (no .csv and no .sql)
Execute this query within in the python process
I know that there is already a teradata python package, hoewever it uses odbc and not bteq (there is a possibility to use bteq, but it is the same with executing a .sql file). And odbc is not installed on the server the script is executed.

Related

Using Python to Import Multiple CSV Data into Microsoft SQL from S3 Browser

How to use python to bulk import the multiple .csv and folder into MySQL from S3 Browser?
Previously, using python to import multiple .csv to Microsoft SQL from own desktop folder. However, it taken longer time for completing the import process.
Had try solution stated in here https://stackoverflow.com/a/55682730/19403948 (MySQL) but failed.
Can anyone assist on this?

Refresh All Data on Excel Workbook in Sharepoint Using Python

To start I managed to successfully run pywin32 locally where it opened the Excel workbooks and refreshed the SQL Query then saved and close them.
I had to download those workbooks locally from Sharepoint and have them sync to apply the changes using one drive.
My Question is would this be possible to do within Sharepoint itself ? Have a python script scheduled on a server and have the process occur there in the backend through a command.
I use this program called Alteryx where I can have batch files execute scripts and maybe I could use an API of some sort to accomplish this on a scheduled basis since thats the only server I have access to.
I have tried looking on this site and other sources but I can't find a post where it would reference this specifically.
I use Jupyter Notebooks to write my scripts and Alteryx to build a workflow with those scripts but I can use other IDEs if I need to.

Import SQL server Management studio backup from an .ipynb file

I have a doubt regarding an ipynb file, it turns out that they send me a database to replicate the structure, they use SQL server Management studio, but I don't know how to import it. I thought it was a simple python script, which could create a SQL database , then Install anaconda, use %%sql statements to recreate it,
Until I realized that they could be imported in SSMS, but there is something that I am not doing well to import it correctly, I understand that it is a problem of correctly parsing the file,
I appreciate any help, thanks!
Install extensions in visual code, anaconda and the necessary libraries for handling SQL in Python, but it all boils down to correctly importing the file created in SSMS.

The ipynb is a notebook that will contain scripts to be executed against a database or create the database and objects as well.
What you are using in SSMS is a tool to import data into tables - these are not the same thing.
As mentioned by #Squirrel, SSMS does not support notebooks, BUT Azure Data Studio does support notebooks. I think that the notebook was created using Azure Data Studio (which will be installed along with SSM on your computer provided you have a recent version of SSMS.
Note that Azure Data Studio is only the name of the tool - it is not restricted to connecting to databases in Azure or running in Azure so you can use it for local or on-premises databases as well.
When you open Azure Data Studio, click on the button for Noptebooks and then the file icon to browse and open your notebook as shown
You will still likely have to set up your connection but that is a similar experience to SSMS.

I would suggest you to follow the below steps:
Open the notebook file in Jupyter notebook and copy all the cell contents as given below:
How to copy multiple input cells in Jupyter Notebook
Copy the content to a single .SQL file.
In the Management Studio, open a new query window and open the file, created in step no. 2 and you can run the sql statements.
Note: Please review the SQL file once to see if everything is falling in place. You might need to add GO statements between batches. Also, it is recommended to put semicolon at the end of statements, to make them running without issues.

Retrieving Data from Elasticsearch-SQL CLI - Insert into Dataframe

I am working with Elasticsearch 6.7 which has an Elasticsearch SQL cli. This allows me to run more standard SQL queries. This is preferred over the API method as the query capabilities are much more robust.
I am attempting to run a query through this CLI and insert those results into a pandas data frame. Is this something I can do via the subprocess method or is there an easier/better way. This will go into production so it needs to run on multiple environments.
This python program will be running on a different host than the Elasticsearch machine.

Python SQLite - How to configure it to use another temporary directory

I have Python code that uses SQLite. However, when running on a large database (7 GB) I get the following error:
sqlite3.OperationalError: database or disk is full
I understand it is related to the temporary directory when it creates the tmp files.
How can I configure the Python SQLite to use another directory (one that I will set it up with enough storage for the tmp files).

Categories

Resources