Develop Reference

Can't load file to MySQL

I'm a database newbie. I'm currently trying to create a db with databases and mysql+aiomysql. I need to initialize some tables by reading local csv files.
First of all, I make a connection to the database I previously created with:
database = Database('mysql+aiomysql://{user}:{passwd}#{host}/{db}?local-infile=1'.format(
host='xxx',
user='xxx',
passwd='xxx',
db='xxx'))
await database.connect()
with ?local-infile=1 to enable client's side local data (I also experimented with ?allowLoadLocalInfile=true). Afterwards, I executed SET GLOBAL local_infile = true to enable local data on server's side. Finally, I also made sure to set secure-file-priv = "".
Nonetheless, when I execute LOAD DATA LOCAL INFILE file INTO TABLE table I get the error:
pymysql.err.OperationalError: (3948, 'Loading local data is disabled;
this must be enabled on both the client and server sides')
Tried also to add LOCAL in the LOAD command above without any luck.
What am I missing here?

Find your my.cnf file (usually in /etc/my.cnf, /etc/mysql/my.cnf, or /usr/local/etc/my.cnf) then add these lines:
[mysqld]
local-infile
secure-file-priv = "/directory/you/wish/to/load/files/"
[mysql]
local-infile
Then restart mysql and try again.

How do you find the parameters for conn = psycopg2.connect(dbname=, user=, password=, host = )

I pushed a Python / Django app to Heroku. The app uses a Postgres database. I now want to access this database from a Raspberry Pi running a simple Python program using
conn = psycopg2.connect(dbname=, user=, password=, host = )
The app used SQLite originally until pushed to Heroku.
How do you find the parameters to use for dbname=, user=, password=, host =?

Heroku will set an environment variable DATABASE_URL:
As part of the provisioning process, a DATABASE_URL config var is added to your app’s configuration. This contains the URL your app uses to access the database.
You should use that to connect, since your credentials can change without notice. psycopg2 can use this value directly, e.g.
import os
import psycopg2
database_url = os.getenv(
'DATABASE_URL',
default='postgres://localhost/postgres', # E.g., for local dev
)
connection = psycopg2.connect(database_url)
Edit: Your Raspberry Pi application won't have access to this directly, but you can use the Heroku CLI to query your config vars:
heroku config:get DATABASE_URL --app your-heroku-app-name
Something like this should work from recent versions of Python:
import subprocess
database_url = subprocess.run(
['heroku', 'config:get', 'DATABASE_URL', '--app', 'your-heroku-app-name'],
stdout=subprocess.PIPE,
).stdout

Navigate to https://data.heroku.com/
Select your database.
Select the Settings tab.
Click View Credentials.
You should be able to see host, database name, user, port, and password.
See also: Heroku Postgres Credentials

flask-restless with mod_wsgi can't connect to MySQL server

I am trying to run a flask-restless app in apache using mod_wsgi. This works fine with the development server. I have read everything I can find and none of the answers I have seen seem to work for me. The app handles non-database requests properly but gives the following error when I try to access a url that requires a database access:
OperationalError: (OperationalError) (2003, "Can't connect to MySQL server on 'localhost' ([Errno 13] Permission denied)") None None
I have whittled down to basically the flask-restless quick-start with my config and my flask-sqlalchemy models imported (from flask import models). Here is my python code:
import flask
import flask.ext.sqlalchemy
import flask.ext.restless
import sys
sys.path.insert(0, '/proper/path/to/application')
application = flask.Flask(__name__, static_url_path = "")
application.debug=True
application.config.from_object('config')
db = flask.ext.sqlalchemy.SQLAlchemy(application)
from app import models
# Create the Flask-Restless API manager.
manager = flask.ext.restless.APIManager(application, flask_sqlalchemy_db=db)
# Create API endpoints, which will be available at /api/<tablename> by
# default. Allowed HTTP methods can be specified as well.
manager.create_api(models.Asset, methods=['GET'])
# start the flask loop
if __name__ == '__main__':
application.run()
I assume that mod_wsgi isn't having a problem finding the config file which contains the database access details since I don't get an error when reading the config and I also don't get an error on from app import models.
My research so far has led me to believe that this has something to do with the sql-alchemy db connection existing in the wrong scope or context and possibly complicated by the flask-restless API manager. I can't seem to wrap my head around it.

Your code under Apache/mod_wsgi will run as a special Apache user. That user likely doesn't have the privileges required to connect to the database.
Even though it says 'localhost' and you think that may imply a normal socket connection, some database clients will see 'localhost' and will automatically instead try and use the UNIX socket for the database. It may not have access to that UNIX socket connection.
Alternatively, when going through a UNIX socket connection it is trying to validate whether the Apache user than has access, but if the database hasn't been setup to allow the Apache user access, it may then fail.
Consider using daemon mode of mod_wsgi and configure daemon mode to run as a different user to the Apache user and one you know has access to the database.

Django setting : psycopg2.OperationalError: FATAL: Peer authentication failed for user "indivo"

I am getting problem in Django project setting with POSTGRESQL.
Here is my setting.py database setting
DATABASES = {
'default':{
'ENGINE':'django.db.backends.postgresql_psycopg2', # '.postgresql_psycopg2', '.mysql', or '.oracle'
'NAME':'indivo', # Required to be non-empty string
'USER':'indivo', # Required to be non-empty string
'PASSWORD':'ritvik',
'HOST':'', # Set to empty string for localhost.
'PORT':'', # Set to empty string for default.
},
}
Now in postgres backend what I have done is .
rohit#rohit-desktop:~$ sudo su - postgres
postgres#rohit-desktop:~$ createuser --superuser indivo # create a super user indivo
postgres#rohit-desktop:~$ psql # open psql terminal
psql (9.1.8)
Type "help" for help.
postgres=# \password indivo # set the password ritvik
Enter new password:
Enter it again:
postgres=# \q #logout
postgres#rohit-desktop:~$ createdb -U indivo -O indivo indivo #create db indivo
Unfortunately when i am trying to syncdb I am getting the error .
psycopg2.OperationalError: FATAL: Peer authentication failed for user "indivo"
Please help me out what might I am doing wrong here .

I have similar problem and solved it with this answer by adding localhost to the database HOST settings in settings.py, so your database settings should look like this:
DATABASES = {
'default':{
'ENGINE':'django.db.backends.postgresql_psycopg2', # '.postgresql_psycopg2', '.mysql', or '.oracle'
'NAME':'indivo', # Required to be non-empty string
'USER':'indivo', # Required to be non-empty string
'PASSWORD':'ritvik',
'HOST':'localhost', # <- Changed from empty string to localhost
'PORT':'', # Set to empty string for default.
},
}

By default in many Linux distros, client authentication is set to "peer" for Unix socket connections to the DB. This is set in the pg_hba.conf config file for postgresql. The psycopg2 python library documentation states:
- *host*: database host address (defaults to UNIX socket if not provided)
So if you leave the host option blank, your script will try to connect and use the Unix username:password for authentication. To fix, you can either:
set the "host" option explicitly to 127.0.0.1 into the config code you pasted into your question.
Modify the pg_hba.conf file to use md5 for socket connections so it uses usernames and passwords stored in the postrgres DB user store (not recommended as this may break authentication for system users)

You need to set pg_hba.conf to use md5 authentication for the user, db and source IP of interest. See the client authentication chapter of the documentation.
Search for pg_hba.conf on Stack Overflow for tons more information.

I too was facing similar problem during setting up postgresql database with Django project.
My System is running RHEL CentOS 7 with postgresql version 10. I search a lot but couldnot find actually what is happening until I saw log at /var/lib/pqsql/10/data/log/postgresql*.log
2017-11-10 00:31:40.499 IST [14129] DETAIL: Connection matched pg_hba.conf line 85: "host all all ::1/128 ident"
So I just changed that particular line in file /var/lib/pgsql/10/data/pg_hba.conf :
from
host all all ::1/128 ident
to
host all all ::1/128 md5
with that I am able to create database & table using manage.py migrate
Thanks for all who has helped me to find this solution. especially #juliocesar and official documentation for postgresql client authentication

Connecting to MongoHQ from heroku console (heroku run python)

I'm getting a 'need to login' error when trying to interact with my MongoHQ database through python console on heroku:
...
File "/app/.heroku/venv/lib/python2.7/site-packages/pymongo/helpers.py", line 128, in _check_command_response
raise OperationFailure(msg % response["errmsg"])
pymongo.errors.OperationFailure: command SON([('listDatabases', 1)]) failed: need to login
My applicable code
app/init.py:
from mongoengine import connect
import settings
db = connect(settings.DB, host=settings.DB_HOST, port=settings.DB_PORT, username=settings.DB_USER, password=settings.DB_PASS)
app/settings.py:
if 'MONGOHQ_URL' in os.environ:
url = urlparse(os.environ['MONGOHQ_URL'])
DB = url.path[1:]
DB_HOST = url.hostname
DB_PORT = url.port
DB_USER = url.username
DB_PASS = url.password
os.environ['MONGOHQ_URL'] looks like:
'mongodb://[username]:[password]#[host]:[port]/[db-name]'
This code works (connects and can read and write to mongodb) both locally and from the heroku web server.
According to the docs (http://www.mongodb.org/display/DOCS/Connections), it should at make a 'login' attempt on connection to the server as long as the username and password params are passed to Connection or parseable from the URI. I couldn't think of a way to see if the login attempt was being made and failing silently.
I've tried bypassing mongoengine and using pymongo.Connection and got the same result. I tried all of the several patterns of using the Connection method. I created a new database user, different from the one mongoHQ creates for heroku's production access -> same same.
It's a flask app, but I don't think any app code is being touched.
Update
I found a solution, but it will cause some headaches. I can manually connect to the database by
conn = connect(settings.DB, host=settings.DB_HOST, port=settings.DB_PORT, username=settings.DB_USER, password=settings.DB_PASS)
db = conn[settings.DB]
db.authenticate(settings.DB_USER, settings.DB_PASS)
Update #2
Mongolab just worked out of the box.

Please use the URI method for connecting and pass the information to via the host kwarg eg:
connect("testdb_uri", host='mongodb://username:password#localhost/mongoenginetest')

MongoHQ add-on uses password hashes not actual passwords and that's perhaps the error.
You should change the environment variable MONGOHQ_URL to a real password with the following command:
heroku config:set MONGOHQ_URL=mongodb://...
Once set, you may restart your applications (heroku apps) so the change gets picked up. If you're in the directory of the failing application, config:seting the config var will restart the application.