Simplified example of my code, please ignore syntax errors:
import numpy as np
import pandas as pd
import pymysql.cursors
from datetime import date, datetime
connection = pymysql.connect(host=,
user=,
password=,
db=,
cursorclass=pymysql.cursors.DictCursor)
df1 = pd.read_sql()
df2 = pd.read_sql(
df3 = pd.read_sql()
np.where(a=1, b, c)
df1.append([df2, d3])
path = r'C:\Users\\'
df.to_csv(path+'a.csv')
On Jupyternotebook it outputs the csv file like it supposed to. However, it I download the .py and run with python. It will only output a csv the first time I run it after restarting my computer. Other times it will just run and nothing happens. Why this is happening is blowing my mind.
I think you have added the path wrongly
If you change your path to df.to_csv(path+'\a.csv') then it will be correct
It's hard to say without knowing what your actual code is, but one thought is that the connection you have to your DB is never closed, and is somehow locking the DB so you are unable to make another connection.
The first connection would end, of course, when you restart your computer.
To see if this is an issue, you could use the MySQL command SHOW PROCESSLIST that would list out the different connections for you; if, after running the script the first time, one of the processes is still the same connection from your machine you just made, that could be the issue. Here's the docs on the command: https://dev.mysql.com/doc/refman/8.0/en/show-processlist.html
Alternatively, you could wrap the DB connection code in a try/except block with some print statements for good measure, to ascertain whether or not that's an issue, like so:
try:
print "Right before connection"
connection = pymysql.connect(host=,
user=,
password=,
db=,
cursorclass=pymysql.cursors.DictCursor)
print "Right after connection"
except Exception as e:
print "The Exception is:{}".format(str(e))
Also, you should most definitely print the objects that you're trying to write to CSV, to see if they're still valid the second time around (i.e. make sure you've actually populated those variables and they're not just Nones)
Related
Im learning how to use python with psycopg2 for postgres and honestly I don't know why I'm having this issue now, all I did was move a couple of files into a new subfolder and now reversing the changes doesn't remove the error.
I've tried moving the file in and out of different subfolders as well, using different names. This is the first time I've had something like this.
Here's some code that I'm pulling my hair out on:
import psycopg2 as database
import config from config
def connect():
connect = None
try:
params = config()
print('Attempting to connect to database.')
connect = database.connect(**params)
except (Exception,database.DatabaseError) as error:
print(error)
finally:
if connect is not None:
connect.close()
print('Database connection closed.')
if __name__ == '__main__':
connect()
The error you see will be something similar to:
import config from config
^
SyntaxError: invalid syntax
The key is that it is a SyntaxError, which is roughly the Python interpreter saying "You have written something that is not even Python". It is an error in even reading the source code you have given, before the interpreter even tries to make sense of it.
What this means is that changing the structure of directories, moving files etc. will have no effect on this kind of error. You have to look carefully where the error message tells you (i.e. line 2), and figure out why it is not valid Python.
In this case, you must change the second line from import config from config to from config import config.
Because import ... from ... is not valid Python.
I am trying to learn how to send a list of lists in Python to R -script which runs statistical methods and gives two or three data frames back to Python
I stumbled across the pyRserve package. I was able to follow the manual in their documentation and everything works great in command line (>>> ). When I run a script, it does not stop. I have installed Rserve package and started its service in RStudio. Below is the code:
import pyRserve
print "here1" #prints this line...
conn = pyRserve.connect(host='localhost', port=6311)
print "here2"
a= conn.eval('3+5')
print a
Can anyone please help?
The (docs) suggest:
$ python
>>> import pyRserve
>>> conn = pyRserve.connect()
And then go on with:
To connect to a different location host and port can be specified explicitly:
pyRserve.connect(host='localhost', port=6311)
This is not meant to indicate that both lines should be run. The second line should be viewed as a potential modifier for the first. So if you need an alternate address or port, then it should look like:
$ python
>>> import pyRserve
>>> conn = pyRserve.connect(host='localhost', port=6311)
Also note this caveat for windows users:
Note On some windows versions it might be necessary to always provide ‘localhost’ for connecting to a locally running Rserve instance.
I have a malformed database. When I try to get records from any of two tables, it throws an exception:
DatabaseError: database disk image is malformed
I know that through commandline I can do this:
sqlite3 ".dump" base.db | sqlite3 new.db
Can I do something like this from within Python?
As far as i know you cannot do that (alas, i might be mistaken), because the sqlite3 module for python is very limited.
Only workaround i can think of involves calling the os command shell (e.g. terminal, cmd, ...) (more info) via pythons call-command:
Combine it with the info from here to do something like this:
This is done on an windows xp machine:
Unfortunately i can't test it on a unix machine right now - hope it will help you:
from subprocess import check_call
def sqliterepair():
check_call(["sqlite3", "C:/sqlite-tools/base.db", ".mode insert", ".output C:/sqlite-tools/dump_all.sql", ".dump", ".exit"])
check_call(["sqlite3", "C:/sqlite-tools/new.db", ".read C:/sqlite-tools/dump_all.sql", ".exit"])
return
The first argument is calling the sqlite3.exe. Because it is in my system path variable, i don't need to specify the path or the suffix ".exe".
The other arguments are chained into the sqlite3-shell.
Note that the argument ".exit" is required so the sqlite-shell will exit. Otherwise the check_call() will never complete because the outer cmd-shell or terminal will be in suspended.
Of course the dump-file should be removed afterwards...
EDIT: Much shorter solution (credit goes to OP (see comment))
os.system("sqlite3 C:/sqlite-tools/base.db .dump | sqlite3 C:/sqlite-tools/target.db")
Just tested this: it works. Apparently i was wrong in the comments.
If I understood properly, what you want is to duplicate an sqlite3 database in python. Here is how I would do it:
# oldDB = path to the corrupted db,
# newDB = path to the new db
def duplicateDB(oldDB, newDB):
con = sqlite3.connect(oldDB)
script = ''.join(con.iterdump())
con.close()
con = sqlite3.connect(newDB)
con.executescript(script)
con.close()
print "duplicated %s into %s" % (oldDB,newDB)
In your example, call duplicateDB('base.db', 'new.db'). The iterdump function is equivalent to dump.
Note that if you use Python 3, you will need to change the print statement.
I have a problem with a simple UPDATE statement. I wrote a Python tool which creates a lot of UPDATE statements and after creating them I want to execute them on my Access database but it doesn't work This is one statement for example:
UPDATE FCL_B_COVERSHEET_A SET BRANCH = 0 WHERE OBJ_ID = '1220140910132011062005';
The statement syntax is not the problem. I tested it and it works.
This next code snippet shows the initialization for the connect object.
strInputPathMDB = "C:\\Test.mdb"
DRV = '{Microsoft Access Driver (*.mdb)}';
con = pyodbc.connect('Driver={0};Dbq={1};Uid={2};Pwd={3};'.format(DRV,strInputPathMDB,"administrator",""))
After that I wrote a method which execute one SQL statement
def executeSQLStatement(conConnection, strSQL):
arcpy.AddMessage(strSQL)
cursor = conConnection.cursor()
cursor.execute(strSQL)
conConnection.commit()
and if I execute this code everything seems to work - no error message or anything like that - but also the data is not updated and I don't know what I'm doing wrong ...
for strSQL in sqlStateArray:
executeSQLStatement(con, strSQL)
con.close()
I hope you understand what my problem is. Thanks for your help.
Chris
The issue here was that the .mdb file was in the root folder of the C: drive. Root folders often restrict normal users to read-only access so the database file was being opened as read-only. Moving the .mdb file to a public folder solved the problem.
I'm trying to use an SQLite insert operation in a python script, it works when I execute it manually on the command line but when I try to access it on the web it won't insert it in the database. Here is my function:
def insertdb(unique_id,number_of_days):
conn = sqlite3.connect('database.db')
print "Opened database successfully";
conn.execute("INSERT INTO IDENT (ID_NUM,DAYS_LEFT) VALUES (?,?)",(unique_id,number_of_days));
conn.commit()
print "Records created successfully";
conn.close()
When it is executed on the web, it only shows the output "Opened database successfully" but does not seem to insert the value into the database. What am I missing? Is this a server configuration issue? I have checked the database permissions on writing and they are correctly set.
The problem is almost certainly that you're trying to create or open a database named database.db in whatever happens to be the current working directory, and one of the following is true:
The database exists and you don't have permission to write to it. So, everything works until you try to do something that requires write access (like commiting an INSERT).
The database exists, and you have permission to write to it, but you don't have permission to create new files in the directory. So, everything works until sqlite needs to create a temporary file (which it almost always will for execute-ing an INSERT).
Meanwhile, you don't mention what web server/container/etc. you're using, but apparently you have it configured to just swallow all errors silently, which is a really, really bad idea for any debugging. Configure it to report the errors in some way. Otherwise, you will never figure out what's going on with anything that goes wrong.
If you don't have control over the server configuration, you can at least wrap all your code in a try/except and manually log exceptions to some file you have write access to (ideally via the logging module, or just open and write if worst comes to worst).
Or, you can just do that with dumb print statements, as you're already doing:
def insertdb(unique_id,number_of_days):
conn = sqlite3.connect('database.db')
print "Opened database successfully";
try:
conn.execute("INSERT INTO IDENT (ID_NUM,DAYS_LEFT) VALUES (?,?)",(unique_id,number_of_days));
conn.commit()
print "Records created successfully";
except Exception as e:
print e # or, better, traceback.print_exc()
conn.close()