Let's see if you could help me with this:
I try to upload a gpkg (geo-referenced) file (table) to a MariadB server (mysql).
Not in postgres, Mariadb-(Mysql)
If I import from an OSGeo4W in the console, it works perfectly for me, being a string, for example:
string=ogr2ogr -f MySQL MySQL:ServerX,host=10.00.00.00,user=userX,password=passX E:\QGIS\cup.shp -nln datatable_name -update -overwrite -lco engine=MYISAM
but I don't see how to integrate it in code from python.
What I do is using os.system,, call that previous string and run it: os.system(string),
I have imported in the module:
import mysql.connector
from osgeo import ogr
etc
but it always returns an error:
ERROR 1: Unable to find driver `MySQL'.
Thank You
I'm working on a legacy script upgrade to Python 3 however the script is hanging during a database delete command (DELETE FROM). The script is showing no error and the logger contains only the is_connected result which is true. Here's my test script based on the main.py file but with only the call to delete the contents of a table and reset it's auto increment prior to repopulating the table.
Here's my test.py file.
#!/usr/bin/env python3
import json
import hashlib
from pprint import pprint
import mysql.connector
import configparser
import re
import random
import requests
import sys
import logging
# Load config for database variables
config = configparser.ConfigParser()
config.read("config.ini")
logFile = "logger.log"
# Set up logging
logging.basicConfig(format='%(asctime)s %(message)s', filename=logFile,level=logging.DEBUG)
# Connect to the MySQL database
cnx = mysql.connector.connect(
host=config["MySQL"]["host"],
user=config["MySQL"]["user"],
port=config["MySQL"]["port"],
passwd=config["MySQL"]["password"],
database=config["MySQL"]["database"]
)
cur = cnx.cursor()
logging.debug(cnx.is_connected())
# Clear the database ready to re-import
# Clear lookup tables first
cur.execute("DELETE FROM member_additional_skills")
logging.debug("Delete Done")
cur.execute("ALTER TABLE member_additional_skills AUTO_INCREMENT = 1")
cnx.commit()
logging.debug("Finished!")
print("Done")
I've left this running for 20 minutes and still nothing else is logged after it declares Teue to being connectedand the process is still running. Is there anything I've missed here?
*** EDIT ***
The process is still in htop but is not using cpu so seems crashed right? And as i write this I have the following as the output to my python3 test.py command line:
client_loop: send disconnect: Broken pipe and the process is no longer in htop
I should point out that this table has no more than 30 rows in it so would expect it to complete in milliseconds
thanks
You can use TrUNCAtE member_additional_skills OR DROP member_additional_skills and CREATE member_additional_skills(....)
that is much faster
You might be having a lock issue:
Sometime with the MySQL connector, the table lock's aren't released properly. Making the table unchangeable.
Try resetting the database and trying again!
I try to execute sql file to create postgresql function in my database. But I have error.
File "/home/saputra/Documents/Development/Exercises/Python/flask/ujicoba06/smartapps/__init__.py", line 2, in <module>
from smartapps.app import create_app
File "/home/saputra/Documents/Development/Exercises/Python/flask/ujicoba06/smartapps/app.py", line 30, in <module>
from smartapps.commands import func_db
File "/home/saputra/Documents/Development/Exercises/Python/flask/ujicoba06/smartapps/commands/func_db.py", line 18
while current_app.open_resource("commands/func_query.sql") as f:
^
SyntaxError: invalid syntax
My structure:
.
|-- smartapps
| |-- app.py
| |-- commands
| | |-- func_db.py
| | |-- func_query.sql
| | |-- init_db.py
| | |-- __init__.py
func_query.sql
--function to get staff who don't have user
CREATE OR replace FUNCTION get_staff_not_have_user()
RETURNS TABLE (
real_name varchar,
email varchar
)
AS $$
BEGIN
RETURN query SELECT
st.real_name,
st.email
FROM
public.staff st
LEFT JOIN
public.users us on us.email = st.email
WHERE
us.email IS NULL;
END;
$$ LANGUAGE 'plpgsql';
func_db.py
# -*- coding: utf-8 -*-
import os
import click
from flask import current_app
from flask.cli import with_appcontext
from sqlalchemy.sql import text
from smartapps.extensions import db
def init_func():
"""Clear existing function and create new one"""
while current_app.open_resource("commands/func_query.sql") as f:
db.session.execute(f.read().decode("utf8"))
#click.command("init-func")
#with_appcontext
def func_query_command():
"""Execute sql script"""
init_func()
click.echo('Initialized function for SMART Systems.')
def init_app(app):
app.cli.add_command(func_query_command)
When I run from terminal, I got error.
How to correct code to execute query from file to create function in postgresql database.
Thanks
The syntax error is due to the wrong keyword in the line:
while current_app.open_resource("commands/func_query.sql") as f:
The first word should be with not while.
This the correct code for func_db.py. I add db.session.commit() after db.session.execute(). This I share the solve code.
func_db.py
import os
import click
from flask import current_app
from flask.cli import with_appcontext
from smartapps.extensions import db
def init_func():
"""Clear existing function and create new one"""
with current_app.open_resource("commands/func_query.sql") as f:
db.session.execute(f.read().decode("utf8"))
db.session.commit()
#click.command("init-func")
#with_appcontext
def func_query_command():
"""Execute sql script"""
init_func()
click.echo('Initialized function for SMART Systems.')
def init_app(app):
app.cli.add_command(func_query_command)
I hope this can save someone one day.
Happy coding!
Is there a way for me to configure PyCharm to run shell_plus instead of the default shell?
I've tried putting the text of the manage command in the 'Starting script' but then I get the folloiwing
django_manage_shell.run("/Users/cmason/counsyl/code/website/counsyl/product")
import os
import sys
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "settings")
# The new Django 1.4 default manage.py wants "from django..." before
# importing settings, but we usually tinker with sys.path in
# settings_local.py, which is called from settings.py. Importing
# settings.py works but does mean some double importing. Luckily that
# module does very little work.
import settings
# appease pyflakes; don't ever do this in
# non-super-meta-namespace-trickery code
settings
from django.core.management import execute_from_command_line
execute_from_command_line("shellplus")
and it hasn't really run shell_plus.
It seems like the 'Starting script' happens in addition to rather than instead of the default.
Shell_plus automatically imports all Django model classes, among other things.
I got the model objects auto-loading by hooking into the shell_plus code. I appended this to the default startup script in Preferences > Build, Execution, Deployment > Console > Django Console:
from django_extensions.management import shells
from django.core.management.color import color_style
imported_items = shells.import_objects({}, color_style())
for k, v in imported_items.items():
globals()[k] = v
This was on PyCharm 2018.3.3 Pro
For completeness, this was the full content of starting script:
import sys; print('Python %s on %s' % (sys.version, sys.platform))
import django; print('Django %s' % django.get_version())
sys.path.extend([WORKING_DIR_AND_PYTHON_PATHS])
if 'setup' in dir(django): django.setup()
import django_manage_shell; django_manage_shell.run(PROJECT_ROOT)
from django_extensions.management import shells
from django.core.management.color import color_style
imported_items = shells.import_objects({}, color_style())
for k, v in imported_items.items():
globals()[k] = v
I've been looking for a solution to the same problem, and I ended up here. I tried solutions proposed by others, but none of those appeared to solve this issue. So I decided to find another solution. This is what I came up with:
The code block below is the original Django Console starting script of PyCharm 2019.2:
import sys, django
print('Python %s on %s' % (sys.version, sys.platform))
print('Django %s' % django.get_version())
sys.path.extend([WORKING_DIR_AND_PYTHON_PATHS])
if 'setup' in dir(django):
django.setup()
import django_manage_shell
django_manage_shell.run(PROJECT_ROOT)
Installing IPython and changing the last two lines as below gets it done in the most proper way:
from IPython.core.getipython import get_ipython
ipython = get_ipython()
from django_extensions.management.notebook_extension import load_ipython_extension
load_ipython_extension(ipython)
To make it work: open PyCharm settings (CTRL+S) and head to Django Console section. Then make changes in Starting script window and apply. Finally, start the new Python Console instance.
I looked at the source code of shell_plus, and noticed you could use a method on a Command class named get_imported_objects({})
In PyCharm, go to: Build, Execution, Deployment > Console > Django Console > Starting script
Add this to the existing code in that box:
from django_extensions.management.commands.shell_plus import Command
globals().update(Command().get_imported_objects({}))
Note: you may have to restart PyCharm to see the effect.
One way to solve this is to create a new Python run configuration. Set the target to module, and select the manage.py file for the project. Then put shell_plus in the Parameters field. Set the Working Directory to the project directory. Then lastly, set the Execution to Run with Python Console. Apply the changes, then run the new configuration.
This isn't a complete answer, but I found this script that at least loads up all the app models. Put this in Settings > Console > Django Console > Starting script:
import sys
import logging
logging.basicConfig(format="%(levelname)-8s %(asctime)s %(name)s %(message)s", datefmt='%m/%d/%y %H:%M:%S', stream=sys.stdout )
log = logging.getLogger("root")
from django.db.models import get_models
from django.conf import settings
from django.core.exceptions import ObjectDoesNotExist, MultipleObjectsReturned
logging.config.dictConfig(settings.LOGGING)
log.debug("Logging has been initialized at DEBUG")
log.setLevel( logging.DEBUG)
log.disabled = False
for _class in get_models():
if _class.__name__.startswith("Historical"): continue
log.debug("Registering model {}".format(_class.__name__))
globals()[_class.__name__] = _class
def debug_sql():
from debug_toolbar.management.commands import debugsqlshell
return
I also submitted this a feature request to JetBrains.
In Django 1.7, following script can be used as a workaround with PyCharm 3.4:
File -> Settings -> Console -> Django Console and manage.py options
In Starting script, put:
import sys
import django
django.setup()
from django.db.models import get_models
for _class in get_models():
globals()[_class.__name__] = _class
This configuration works for me
As django.db.models.get_models no longer exists, here's an updated version that will accomplish the same as Christopher Mason's version.
import sys; print('Python %s on %s' % (sys.version, sys.platform))
import django; print('Django %s' % django.get_version())
import logging
logging.basicConfig(format="%(levelname)-8s %(asctime)s %(name)s %(message)s", datefmt='%m/%d/%y %H:%M:%S', stream=sys.stdout )
log = logging.getLogger("root")
from django.apps import apps
from django.conf import settings
from django.core.exceptions import ObjectDoesNotExist, MultipleObjectsReturned
logging.config.dictConfig(settings.LOGGING)
log.debug("Logging has been initialized at DEBUG")
log.setLevel( logging.DEBUG)
log.disabled = False
for _configs in apps.get_app_configs():
for _class in _configs.get_models():
if _class.__name__.startswith("Historical"): continue
log.debug("Registering model {}".format(_class.__name__))
globals()[_class.__name__] = apps.get_model(_configs.label, _class.__name__)
def debug_sql():
from debug_toolbar.management.commands import debugsqlshell
return
here is my code
import sys
import logging
import traceback
import MySQLdb
def log_except_hook(*exc_info):
text = "".join(traceback.format_exception(*exc_info))
logging.error("Unhandled exception: %s", text)
sys.excepthook = log_except_hook
db = MySQLdb.connect("sdfgdfgsdgd.nl","1","1","1" )
cursor = db.cursor()
You need to setup a logger which will write to a file. See the Python Docs for an example on how to do this.