vertica/sqlalchemy - Permission denied for schema public - python

I am able to generate an engine as follows:
import pandas as pd
import sqlalchemy as sa
url = sa.url.URL(drivername='vertica+pyodbc',
username='username',
password='****',
host='vertica')
engine = sa.create_engine(url)
I can read data, e.g.
pd.read_sql_query("SELECT * FROM my_schema.tablename", engine)
However when I try to write data:
import numpy as np
df = pd.DataFrame(np.random.randn(10, 5))
df.to_sql("my_schema.random_table", engine)
I get the error
Permission denied for schema public\n (4367) (SQLExecDirectW)') u'\nCREATE TABLE "my_schema.random_table"
I believe that the issue is that "my_schema.random_table" is being wrapped by quotes, making Vertica think I want to create this table in the base (public) schema instead of my_schema. Is there a way I can specify the schema I want in the ODBC url?

pandas.DataFrame.to_sql has a schema parameter. Try using that instead of specifying the schema as a prefix to the table name.
df.to_sql('random_table', engine, schema='my_schema')

Related

How can I drop this table using SQLAlchemy?

I am trying to drop a table called 'New'. I currently have the following code:
import pandas as pd
import sqlalchemy
sqlcon = sqlalchemy.create_engine('mssql://ABSECTDCS100TL/AdventureWorks?driver=ODBC+Driver+17+for+SQL+Server')3
df = pd.read_sql_query('SELECT * FROM DimReseller', sqlcon)
df.to_sql('New',sqlcon,if_exists='append', index=False)
sqlalchemy.schema.New.drop(bind=None, checkfirst=False)
I am receiving the error:
AttributeError: module 'sqlalchemy.schema' has no attribute 'New'
Any ideas on what I'm missing here?. Thanks.
You can reflect the table into a Table object and then call its drop method:
from sqlalchemy import Table, MetaData
tbl = Table('New', MetaData(), autoload_with=sqlcon)
tbl.drop(sqlcon, checkfirst=False)
If you want to delete the table using raw SQL, you can do this:
from sqlalchemy import text
with sqlcon.connect() as conn:
# Follow the identifier quoting convention for your RDBMS
# to avoid problems with mixed-case names.
conn.execute(text("""DROP TABLE "New" """))
# Commit if necessary
conn.commit()

Missing column names when importing data from database (python + postgre sql)

I am trying to import some data from the database (Postgre SQL) to work with them in Python. I tried with the code below, which seems quite similar to the ones I've found on the internet.
import psycopg2
import sqlalchemy as db
import pandas as pd
engine = db.create_engine('database specifications')
connection = engine.connect()
metadata = db.MetaData()
data = db.Table(tabela, metadata, schema=shema, autoload=True, autoload_with=engine)
query = db.select([data])
ResultProxy = connection.execute(query)
ResultSet = ResultProxy.fetchall()
df = pd.DataFrame(ResultSet)
However, it returns data without column names. What did I forget?
It turned out the only thing needed is adding
columns = data.columns.keys()
df.columns = columns
There is a great debate about that in this thread.

Importing a .sql file in python

I have just started learning SQL and I'm having some difficulties to import my sql file in python.
The .sql file is in my desktop, as well is my .py file.
That's what I tried so far:
import codecs
from codecs import open
import pandas as pd
sqlfile = "countries.sql"
sql = open(sqlfile, mode='r', encoding='utf-8-sig').read()
pd.read_sql_query("SELECT name FROM countries")
But I got the following message error:
TypeError: read_sql_query() missing 1 required positional argument: 'con'
I think I have to create some kind of connection, but I can't find a way to do that. Converting my data to an ordinary pandas DataFrame would help me a lot.
Thank you
This is the code snippet taken from https://www.dataquest.io/blog/python-pandas-databases/ should help.
import pandas as pd
import sqlite3
conn = sqlite3.connect("flights.db")
df = pd.read_sql_query("select * from airlines limit 5;", conn)
Do not read database as an ordinary file. It has specific binary format and special client should be used.
With it you can create connection which will be able to handle SQL queries. And can be passed to read_sql_query.
Refer to documentation often https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_sql_query.html
You need a database connection. I don't know what SQL flavor are you using, but suppose you want to run your query in SQL server
import pyodbc
con = pyodbc.connect(driver='{SQL Server}', server='yourserverurl', database='yourdb', trusted_connection=yes)
then pass the connection instance to pandas
pd.read_sql_query("SELECT name FROM countries", con)
more about pyodbc here
And if you want to query an SQLite database
import sqlite3
con = sqlite3.connect('pathto/example.db')
More about sqlite here

Data insertion in SQL server with pandas

I'm trying to upload a dataframe in SQL server using pandas (to_sql) function, I get the below error
[SQL Server Native Client 11.0]Invalid character value for cast
specification (0) (SQLExecDirectW)')
I checked for variables' names and types and they are exactly the same in the SQL database and pandas dataframe.
How can I fix this?
Thanks
df.to_sql(raw_table, connDB, if_exists='append', index=False )
plz try this , this code use to juypter note book and SQL workbench
import mysql.connector
from mysql.connector import Error
from sqlalchemy import create_engine
import pandas as pd
mydata = pd.read_csv("E:\\Hourly_Format\\upload.csv")
engine = create_engine("mysql://root:admin#localhost/pythondb", pool_size=10, max_overflow=20)
mydata.to_sql(name='emp',con=engine,if_exists='append', index=False)
jupyter :-
workbench :-

Inserting to schema-specific table with python's odo

I'm using python's odo to move data from a pandas dataframe to a postgresql database. The goal is that each "user" sees their own data in their schema, but with an identical data model and table/view naming schema between "users". With normal SQL I can do this:
CREATE SCHEMA my_schema;
CREATE TABLE my_schema.my_table AS select 1;
My DB URI looks like this
db_uri = 'postgresql://localhost/postgres::my_schema.my_table'
This gives me tables in the default schema named "my_schema.my_table", including the '.' in the table name, instead of tables named "my_table" in the schema "my_schema".
I've tried different combinations based on this github issue, such as:
db_uri = 'postgresql://localhost/postgres.schema::tmp')
which gives me this Traceback
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL: database "postgres/schema" does not exist
And also this one
db_uri = 'postgresql://localhost/postgres::my_schema/my_table'
which gives me tables named "my_schema/my_table".
Here's a sample code:
import pandas as pd
from odo import odo
db_uri = 'postgresql://localhost/postgres::my_schema.my_table'
odo(pd.DataFrame([{'a': 1}, {'a': 1}]), db_uri)
Hidden deep in a mailing list for blaze is a mention of the schema parameter
d = Data(resource('postgresql://localhost/db::t', schema='myschema'))
which can be used with odo with the following format:
from odo import odo, drop
drop(db_uri, schema='my_schema') # to drop table in specific schema
odo(data, db_uri, schema='my_schema')
working code
import pandas as pd
from odo import odo
db_uri = 'postgresql://localhost/postgres::my_table'
odo(pd.DataFrame([{'a': 1}, {'a': 1}]), db_uri, schema='my_schema')

Categories

Resources