I am new to the cloud and to data engineering as well.
I have a large csv file stored in a GCS bucket. I would like to write a python script to bulk-insert the data into a postgresql database on my local machine using a COPY statement. I cannot figure out the authentication though.
I would like to do something like this:
import psycopg2
conn = psycopg2.connect(database=database,
user=user,
password=password,
host=host,
port=port)
cursor = conn.cursor()
file = 'https://storage.cloud.google.com/<my_project>/<my_file.csv>'
sql_query = f"COPY <MY_TABLE> FROM {file} WITH CSV"
cursor.execute(sql_query)
conn.commit()
conn.close()
I get this error message:
psycopg2.errors.UndefinedFile: could not open file "https://storage.cloud.google.com/<my_project>/<my_file.csv>" for reading: No such file or directory
HINT: COPY FROM instructs the PostgreSQL server process to read a file. You may want a client-side facility such as psql's \copy.
The same happens when I run the query in psql.
I assume the problem is in authentication. I have set up Application Default Credentials with Google Cloud CLI and when acting like the authenticated user, I can easily download the file using wget. When I switch to postgres user, I get "access denied" error.
The ADC seem to work only with client libraries and command-line tools.
I use Ubuntu 22.04.1 LTS.
Thanks for any help.
This is not going to work for you. The file will need to be in a location permitted to the server process and also not fetched over http (it's a local file path it is expecting).
You can supply a program/script that will fetch the file for you and print it to STDOUT which the server can consume.
Or - do what the error message suggests and handle it locally with psycopg's copy support.
Related
I'm a database newbie. I'm currently trying to create a db with databases and mysql+aiomysql. I need to initialize some tables by reading local csv files.
First of all, I make a connection to the database I previously created with:
database = Database('mysql+aiomysql://{user}:{passwd}#{host}/{db}?local-infile=1'.format(
host='xxx',
user='xxx',
passwd='xxx',
db='xxx'))
await database.connect()
with ?local-infile=1 to enable client's side local data (I also experimented with ?allowLoadLocalInfile=true). Afterwards, I executed SET GLOBAL local_infile = true to enable local data on server's side. Finally, I also made sure to set secure-file-priv = "".
Nonetheless, when I execute LOAD DATA LOCAL INFILE file INTO TABLE table I get the error:
pymysql.err.OperationalError: (3948, 'Loading local data is disabled;
this must be enabled on both the client and server sides')
Tried also to add LOCAL in the LOAD command above without any luck.
What am I missing here?
Find your my.cnf file (usually in /etc/my.cnf, /etc/mysql/my.cnf, or /usr/local/etc/my.cnf) then add these lines:
[mysqld]
local-infile
secure-file-priv = "/directory/you/wish/to/load/files/"
[mysql]
local-infile
Then restart mysql and try again.
I've created a user for my database in mongodb. I've tested with mongo shell to make sure that the user has proper privileges to access the database.
Now I want to use my Python program to access the database, and I use PyMongo. If I run mongod in unauthorized mode (without option --auth), the Python client works fine. However, when I use --auth option, the Python client doesn't work any more. In fact, it reports unauthorized error, which is easy to understand because I didn't change anything in the code. Here is the code to connect my test database:
from pymongo import MongoClient
client = MongoClient()
db = client.test
cursor = db.restaurants.find()
for document in cursor:
print(document)
My question is how can I change the Python code to use username/password created previously? I've looked at the client documentation but there is not information for it.
client = MongoClient("mongodb://username:password#server/dbname")
This is the normal format for telling the client where and how to connect. The way you are using (with no parameters at all) defaults to a local mongo install, on the default port, with no authentication.
Besides Danielle's answer you can also use authenticate method for that. Your code would look like this:
from pymongo
import MongoClient
client = MongoClient()
db = client.test
db.authenticate('user', 'password', mechanism=<either 'SCRAM-SHA-1' or 'MONGODB-CR', being 'MONGODB-CR' the default authentication mechanism Before MongoDB 3.0>)
cursor = db.restaurants.find()
for document in cursor:
print(document)
I wanted to know the process of connecting to a MySQL database that is hosted on a web server.
I have a basic free webserver for testing on 000webhost on which I created a MySQL database.
I have the credentials for the database which I will pretend are
host - mysql.webhost000.com
user - dummy_user
password - dummy_password
database - dummy_database
and I have a python script executing from my local computer with internet access
import MySQLdb
db = MySQLdb.connect(host="mysql.webhost000.com",
port=3306,
user="dummy_user",
passwd="dummy_password",
db="dummy_database")
I was hoping it would connect as long as I have the right credentials but when I execute the script it just hangs and once I quit it I see the error
Can't connect to MySQL server on 'mysql.webhost000.com' (4)
Am I missing some steps?
There are two possible problems and im not able to recreate the first one. One is the
host="mysql.webhost000.com"
is incorrect and throwing an error. The connection could be listed as another way. The other I noticed is this is usually how I set up my connection script.
import MySQLdb
def connect():
db = MySQLdb.connect(host="mysql.webhost000.com",
port=3306,
user="dummy_user",
passwd="dummy_password",
db="dummy_database")
c = conn.cursor()
return c, db
I have a database in my server called ex.
So in my python script (where sqlite3 is imported) I will try to insert values...
try:
# Initial setup runs one time
connect = sqlite3.connect('ex')
cursor = connect.cursor()
except Exception as e:
error_string = e
# Insert query into the cursor
cursor.execute("INSERT INTO student_urls (user_name,assignment,url,created_at) \
values(?,?,?,date('now'))" ,(user_name,assignment, url))
But the cursor.execute line is giving me issues... In particular I get a 500 internal server error. Any ideas?
Specify the full path of your SQL database, as with sqlite3.connect('/var/www/ex.db'). Also, ensure that the application has sufficient read/write permissions. I encountered an identical problem when hosting my Flask app on a server, and had to both use a full path and chmod -R my app directory.
I'm getting a 'need to login' error when trying to interact with my MongoHQ database through python console on heroku:
...
File "/app/.heroku/venv/lib/python2.7/site-packages/pymongo/helpers.py", line 128, in _check_command_response
raise OperationFailure(msg % response["errmsg"])
pymongo.errors.OperationFailure: command SON([('listDatabases', 1)]) failed: need to login
My applicable code
app/init.py:
from mongoengine import connect
import settings
db = connect(settings.DB, host=settings.DB_HOST, port=settings.DB_PORT, username=settings.DB_USER, password=settings.DB_PASS)
app/settings.py:
if 'MONGOHQ_URL' in os.environ:
url = urlparse(os.environ['MONGOHQ_URL'])
DB = url.path[1:]
DB_HOST = url.hostname
DB_PORT = url.port
DB_USER = url.username
DB_PASS = url.password
os.environ['MONGOHQ_URL'] looks like:
'mongodb://[username]:[password]#[host]:[port]/[db-name]'
This code works (connects and can read and write to mongodb) both locally and from the heroku web server.
According to the docs (http://www.mongodb.org/display/DOCS/Connections), it should at make a 'login' attempt on connection to the server as long as the username and password params are passed to Connection or parseable from the URI. I couldn't think of a way to see if the login attempt was being made and failing silently.
I've tried bypassing mongoengine and using pymongo.Connection and got the same result. I tried all of the several patterns of using the Connection method. I created a new database user, different from the one mongoHQ creates for heroku's production access -> same same.
It's a flask app, but I don't think any app code is being touched.
Update
I found a solution, but it will cause some headaches. I can manually connect to the database by
conn = connect(settings.DB, host=settings.DB_HOST, port=settings.DB_PORT, username=settings.DB_USER, password=settings.DB_PASS)
db = conn[settings.DB]
db.authenticate(settings.DB_USER, settings.DB_PASS)
Update #2
Mongolab just worked out of the box.
Please use the URI method for connecting and pass the information to via the host kwarg eg:
connect("testdb_uri", host='mongodb://username:password#localhost/mongoenginetest')
MongoHQ add-on uses password hashes not actual passwords and that's perhaps the error.
You should change the environment variable MONGOHQ_URL to a real password with the following command:
heroku config:set MONGOHQ_URL=mongodb://...
Once set, you may restart your applications (heroku apps) so the change gets picked up. If you're in the directory of the failing application, config:seting the config var will restart the application.