I am using Python with spyder and would like to directly connect to the our postgres datawarehouse to directly query data from there for further analysis.
This is what I found under the following link:
[https://www.psycopg.org/docs/]
import psycopg
Connect to your postgres DB
conn = psycopg.connect("dbname=*** user=***")
Open a cursor to perform database operations
cur = conn.cursor()
Execute a query
cur.execute("SELECT * FROM my_data")
Retrieve query results
records = cur.fetchall()
However, I get the following error message:
runfile('C:/Users/.../.spyder-py3/untitled2.py', wdir='C:/Users/.../.spyder-py3')
Traceback (most recent call last):
File "C:\Users....spyder-py3\untitled2.py", line 9, in
import psycopg
ModuleNotFoundError: No module named 'psycopg'
https://www.psycopg.org/docs/ -> this is what you are looking for.
Next step might be using pandas read_sql() to load the sql table into to your variable exploirer if it is not too big.
Does this help?
Posting this in the Spyder Editor actually worked to establish a connection with the PostgreSQL Database:
import psycopg2
connection = psycopg2.connect(user="...",
password="...",
host="...",
port="...",
database="...")
cursor = connection.cursor() cursor.execute("SELECT * FROM data.table25 limit 5")
After running the above command you can check for your rows with:
rows=cur.fetchall()
Related
This question already has an answer here:
Postgres error after updating TimescaleDB on Ubuntu: file not found
(1 answer)
Closed 2 years ago.
I was trying to hook up the sqlalchemy with my underlying postgresql, which uses the timescaledb extension. All queries work fine when I try them from the psql terminal client. But when I try to use python & sqlalchemy to do it, it keeps throwing me an error.
Here's the very basic code snippet that I try to test it with:
engine = create_engine('postgres://usr:pwd#localhost:5432/postgres', echo=True)
engine.execute('select 1;')
And it always shows the following the error message:
File "/home/usr/.local/share/virtualenvs/redbird-lRSbFM0t/lib/python3.6/site-packages/psycopg2/extras.py", line 917, in get_oids
""" % typarray)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not access file "timescaledb-0.9.0": No such file or directory
The connection to the db is fine, otherwise it won't know the db is using timescaledb.
Any one has any insights?
UPDATE: I try to use psycopg2 directly. It basically gives the same error. DB is connected successfully, but timescaledb-0.9.0 cannot be accessed.
Here's the code snippted
conn_string = "host='localhost' dbname='db' user='usr' password='pwd'"
print("Connecting to database\n ->%s " % (conn_string))
conn = psycopg2.connect(conn_string)
cursor = conn.cursor()
print("Connected!\n")
cursor.execute("\dx")
records = cursor.fetchall()
Here's the exact same error message:
Connecting to database
Connected!
Traceback (most recent call last):
File "/home/usr/Workspace/somepath/web/model/model.py", line 21, in <module>
cursor.execute("\dx")
psycopg2.OperationalError: could not access file "timescaledb-0.9.0": No such file or directory
This seems very similar to my issue.
I guess you also updated to a new version of Timescale? The thing is: After each update of the timescale package you don't just have to make sure the library is preloaded (as the warning says on the command line) - you also have to upgrade each database that uses the extension manually via psql.
See my own answer to my issue for the steps.
--
This snipplet works for me:
#! /usr/bin/python
# -*- coding: utf-8 -*-
import psycopg2
# Connect to an existing database.
conn = psycopg2.connect(dbname='my-db-name',
user='postgres',
password='super-secret',
host='localhost',
port='5432')
# Open a cursor to perform database operations.
cur = conn.cursor()
# Query the database and obtain data as Python objects.
cur.execute('SELECT * FROM my-table-name LIMIT 100 ;')
# Print results.
results = cur.fetchall()
for result in results:
print(result)
# Close communication with the database.
cur.close()
conn.close()
Using the cursor to executue psql commands also doesn't work for me. I don't think it is supposed to. But what works reliably is doing SQL:
# Check if the database has the timescaledb extension installed.
# This is about the same as xecuting '\dx' on psql.
cur.execute('SELECT * from pg_extension;')
Can anyone please guide me in how to run .hql queries using Python. Tried Hiveutils lib but its not present in the dev environment. Any other way to execute the queries?
You can use PyHive: PyHive is a collection of Python DB-API and SQLAlchemy interfaces for Presto and Hive.
Example:
from pyhive import hive
cursor = hive.connect('localhost').cursor()
cursor.execute('SELECT * FROM my_awesome_data LIMIT 10')
print(cursor.fetchone())
print(cursor.fetchall())
You can try using pyhs2, try working like the below given example
import pyhs2
with pyhs2.connect(host='localhost',
port=10000,
authMechanism="PLAIN",
user='your_user',
password='your_password',
database='your_default_db') as conn:
with conn.cursor() as cur:
print cur.getDatabases()
cur.execute("select * from table")
#Return info from query
print cur.getSchema()
This is my first question here. So, I am sorry if it is repeated or the formatting is off. I searched through other questions and the error is common but appears on multiple situations.
I have a very simple python code where I want to execute a procedure in MSSQL from pyodbc.
import pyodbc
conn = pyodbc.connect(r'DSN=myDSN')
cursor = conn.cursor()
query = r'{call myproc}'
cursor.execute(query)
I am using call instead of exec after reading that ODBC uses call for executing procedures in MSSQL.
The error I am getting is the following:
Traceback (most recent call last):
File "myscript.py", line 26, in <module>
cursor.execute(query)
pyodbc.ProgrammingError: ('42000', '[42000] [Microsoft][SQL Server Native Client 11.0][SQL Server]The current transaction has aborted, and any pending changes have been rolled back. Cause: A transaction in a rollback-only state was not explicitly rolled back before a DDL, DML or SELECT statement. (111233) (SQLExecDirectW)')
Thanks for the help
In case someone is having the same issue. I was able to find out what the problems was. When you open a connection with DSN, the autocommit is set to False. For some reason, this should be True for the code to work (this depends largely on what I was doing on MSSQL).
import pyodbc
conn = pyodbc.connect(r'DSN=myDSN', autocommit=True)
cursor = conn.cursor()
query = r'{call myproc}'
cursor.execute(query)
This runs well!
Here are two examples on how you can execute a stored proc in MS SQL Server through pyodbc:
Passing a NULL, and the VARCHAR 'tallen' as positional parameter variables:
cursor.execute('EXEC usp_get_user_data ?, ?', None, 'flipperpa')
Passing two VARCHAR values as named parameter variables:
cursor.execute('EXEC usp_get_user_data #user_full_name = ?, #user_username = ?', 'flip', 'flipperpa')
Then to loop through the returned data:
rows = cursor.fetchall()
for row in rows:
# Do stuff
print(row.user_id)
Good luck!
I had issue with executing the Stored procedure using the SQL server 2008. The first step I did is to go to control panel > Administrative tools > Data Sources (ODBC) > add the driver
The only change I made in my python code is use "{call procdev_2017}". I got an error when I tried using exec instead of call
import pyodbc
conn = pyodbc.connect(driver='{SQL Server Native Client 11.0}', server='XXXXXXX', database='XXX',
trusted_connection='yes', autocommit=True)
bepcur = conn.cursor()
ipcur.execute("{call procdev_2017}")
I'm using Python to access the "cookie" chrome sqlite3 db to retrieve the host keys, but getting error below
import sqlite3
conn = sqlite3.connect(r"C:\Users\tikka\AppData\Local\Google\Chrome\User Data\Default\Cookies")
cursor = conn.cursor()
cursor.execute("select host_key from cookies")
results = cursor.fetchall()
print results
conn.close()
Error
Traceback (most recent call last):
File "C:\Python27\cookies.py", line 4, in <module>
cursor.execute("select host_key from cookies")
DatabaseError: malformed database schema (is_transient) - near "where": syntax error
>>>
thanks to link provided by alecxe was able to fix it by upgrading sqlite3 version from 3.6.21 to 3.9.2. I upgraded by downloading new version from this site and placing the dll in C:\Python27\DLLs
I have case :
import pymysql
conn = pymysql.connect(host='127.0.0.1', unix_socket='/opt/lampp/var/mysql/mysql.sock', user='root', passwd=None, db='test')
cur = conn.cursor()
cur.execute("test < /mypath/test.sql")
cur.close()
conn.close()
I always get error :
1064 , "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'test < /mypath/test.sql' at line 1"
I tried to use source and it still failed. Did you know why?
Thank you.
Your error message says that the MySQL server can't understand
test < /mypath/test.sql' at line 1
If you're a long time *nix user, it seems intuitive that you should be able to use commands like this to pass various sorts of data streams to various programs. But that's not the way the Python sql API (or most language-specific) sql APIs works.
You need to pass a valid SQL query to the execute() method in the API, so the API can pass it to the database server. A vaild query will be something like INSERT or CREATE TABLE.
Look, the server might be on a different host machine, so telling the server to read from /mypath/test.sql is very likely a meaningless instruction to that server. Even if it did understand it, it might say File test.sql not found.
The mysql(1) command line client software package can read commands from files. Is that what you want?
>>> import MySQLdb
>>> db = MySQLdb.connect(host = 'demodb', user = 'root', passwd = 'root', db = 'mydb')
>>> cur = db.cursor()
>>> cur.execute('select * from mytable')
>>> rows = cur.fetchall()
Install MySQL-Python package to use MySQLdb.