Im learning how to use python with psycopg2 for postgres and honestly I don't know why I'm having this issue now, all I did was move a couple of files into a new subfolder and now reversing the changes doesn't remove the error.
I've tried moving the file in and out of different subfolders as well, using different names. This is the first time I've had something like this.
Here's some code that I'm pulling my hair out on:
import psycopg2 as database
import config from config
def connect():
connect = None
try:
params = config()
print('Attempting to connect to database.')
connect = database.connect(**params)
except (Exception,database.DatabaseError) as error:
print(error)
finally:
if connect is not None:
connect.close()
print('Database connection closed.')
if __name__ == '__main__':
connect()
The error you see will be something similar to:
import config from config
^
SyntaxError: invalid syntax
The key is that it is a SyntaxError, which is roughly the Python interpreter saying "You have written something that is not even Python". It is an error in even reading the source code you have given, before the interpreter even tries to make sense of it.
What this means is that changing the structure of directories, moving files etc. will have no effect on this kind of error. You have to look carefully where the error message tells you (i.e. line 2), and figure out why it is not valid Python.
In this case, you must change the second line from import config from config to from config import config.
Because import ... from ... is not valid Python.
Related
I'm trying to do a very simple test where my function takes in one parameter, a path to a database, tries to connect to the database using pyodbc.connect(database path) and if the path is invalid raises an error. However, as my test is structured currently, I am unable to make the test pass (raise the error) when I pass the function being tested a bad path.
I've tried to pass the test using mocks and without mocks. I believe the correct way to go would be to use mocks, however I also could not get that to work (see code below). I've found questions that were similar to what issue I'm having, but I could not use the suggestions in those questions to get my test to work.
Function Being Tested
import pyodbc
# When function runs by itself (no test) pyodbc.InterfaceError is raised if database path is bad!
def connect_to_database(database):
try:
conn = pyodbc.connect(database)
return conn
except pyodbc.InterfaceError as err:
raise err
Tests
Without mocking
def test_invalid_path_to_database(self):
self.assertRaises(pyodbc.InterfaceError, connect_to_database, '')
With mocking
def setUp(self):
self.mock_conn= mock.patch('my_package.my_module.pyodbc.connect').start()
self.addCleanup(mock.patch.stopall)
def test_invalid_path_to_database(self):
self.mock_conn.side_effect = pyodbc.InterfaceError
self.assertRaises(pyodbc.InterfaceError, connect_to_database, '')
I expect when the passed database path to the function is not valid an error (in this case InterfaceError) should be raised and the test should pass. This is not happening as the test is structured.
Simplified example of my code, please ignore syntax errors:
import numpy as np
import pandas as pd
import pymysql.cursors
from datetime import date, datetime
connection = pymysql.connect(host=,
user=,
password=,
db=,
cursorclass=pymysql.cursors.DictCursor)
df1 = pd.read_sql()
df2 = pd.read_sql(
df3 = pd.read_sql()
np.where(a=1, b, c)
df1.append([df2, d3])
path = r'C:\Users\\'
df.to_csv(path+'a.csv')
On Jupyternotebook it outputs the csv file like it supposed to. However, it I download the .py and run with python. It will only output a csv the first time I run it after restarting my computer. Other times it will just run and nothing happens. Why this is happening is blowing my mind.
I think you have added the path wrongly
If you change your path to df.to_csv(path+'\a.csv') then it will be correct
It's hard to say without knowing what your actual code is, but one thought is that the connection you have to your DB is never closed, and is somehow locking the DB so you are unable to make another connection.
The first connection would end, of course, when you restart your computer.
To see if this is an issue, you could use the MySQL command SHOW PROCESSLIST that would list out the different connections for you; if, after running the script the first time, one of the processes is still the same connection from your machine you just made, that could be the issue. Here's the docs on the command: https://dev.mysql.com/doc/refman/8.0/en/show-processlist.html
Alternatively, you could wrap the DB connection code in a try/except block with some print statements for good measure, to ascertain whether or not that's an issue, like so:
try:
print "Right before connection"
connection = pymysql.connect(host=,
user=,
password=,
db=,
cursorclass=pymysql.cursors.DictCursor)
print "Right after connection"
except Exception as e:
print "The Exception is:{}".format(str(e))
Also, you should most definitely print the objects that you're trying to write to CSV, to see if they're still valid the second time around (i.e. make sure you've actually populated those variables and they're not just Nones)
Have you seen this pandas sqlite error? ArgumentError("Could not parse rfc1738 URL from string '/')? It only happens in a NOSE test.
From the other questions on SO, the err seems to be related to sqlite. But I can't tell how since there are no others using a simple sqlite3 db like I am.
In my unit test, I call a class method that reads from an sqlite3 DB via pandas read_sql. It works perfectly in any python session as well as Jupyter notebook. But for some reason when I run the code through a nosetest, nose tells me there's an Argument Error. And I cannot reproduce it.
I've confirmed the DB is working properly. As I mentioned, the pd.read_sql works perfectly in any other setting.
In the method definition, I am doing the following read,
# get data
div_query = 'SELECT * FROM Dividends WHERE Symbol == "{stock}"'.format(stock = symbol)
div_history = pd.read_sql(div_query, con=dbcnx)
And in the NOSE test,
def test_shiftBeforeDividend():
# here is where the err occurs
result = filters.shiftBeforeDividend('DUK')
# result now equals ArgumentError("Could not parse rfc1738 URL from string '/'")
def setup():
try:
db = 'mydb.db'
test_class.connectToDB(db)
return test_class
except Exception as e:
return e
def teardown():
try:
filters.closeDBConnection(self.dbcnx[0])
except Exception as e:
return e
# startup
filters = setup()
Any ideas on how to eliminate the issue?
Turns out after some hair pulling that the error was a simple misplaced '/' in the db file path. Hence, sqlite could not connect to a DB and all calls to the DB resulted in an error. In all the help topics I read on similar subjects, it seems that this error always results from an improper DB reference (i.e., file path). So if you run into the issue, make sure your file path exists and is correct before trying operations on your DB.
I would like to use git-multimail as post receive hook in one of my git repositories (no gitolite used). Unfortunately, I cannot get it work, and I have hardly any experience using Python.
What I did so far:
I added the following block to the project.git/config file:
[multimailhook]
mailingList = email#example.com
from = email#example.com
envelopeSender = email#example.com
mailer = smtp
smtpServer = smtp.mydomain.com
smtpUser = myUser
smtpPass = myPassword
Please note that I do not know whether "smtp", which is defined in the mailer variable, is installed on my machine.
I copied the current git_multimail.py file into project.git/hooks.
I created a project.git/hook/post-receive file with the following content. The file is executable, I copied this from https://github.com/git-multimail/git-multimail/blob/master/git-multimail/post-receive.example
#! /usr/bin/env python
import sys
import os
import git_multimail
config = git_multimail.Config('multimailhook')
try:
environment = git_multimail.GenericEnvironment(config=config)
#environment = git_multimail.GitoliteEnvironment(config=config)
except git_multimail.ConfigurationException:
sys.stderr.write('*** %s\n' % sys.exc_info()[1])
sys.exit(1)
mailer = git_multimail.choose_mailer(config, environment)
git_multimail.run_as_post_receive_hook(environment, mailer)
What happens:
When I push a change, a file project.git/hooks/git_multimail.pyc is created, but no email is sent.
Doing a configuration test using GIT_MULTIMAIL_CHECK_SETUP=true python git_multimail.py as described on https://github.com/git-multimail/git-multimail/blob/master/doc/troubleshooting.rst tells me that git-multimail seems properly set up
Is there a way to log something like an output of the script? What can I do to find out what is not working? Are there even errors in my files?
Thanks in advance.
Using post-receive.example is by far not the simplest way to set up git_multimail.py: as the header of post-receive.example script says:
The simplest way to use git-multimail is to use the script
git_multimail.py directly as a post-receive hook, and to configure it
using Git's configuration files and command-line parameters.
In other words, just
cp /path/to/git_multimail.py /path/to/project.git/hooks/post-receive
and you're all set (since you already have project.git/config filled-in). See Invocation in git-multimail's README for a bit more detail.
(note: admitedly, the doc is not so clear for beginners, I'll try to improve that when I get time)
OK guys, the error was probably as little as it could be. I did just one very little mistake in the post receive hook file: The sys.exit(1) command is not indented.
So, the WRONG version from my question:
try:
environment = git_multimail.GenericEnvironment(config=config)
except git_multimail.ConfigurationException:
sys.stderr.write('*** %s\n' % sys.exc_info()[1])
sys.exit(1)
CORRECT is (compare last line):
try:
environment = git_multimail.GenericEnvironment(config=config)
except git_multimail.ConfigurationException:
sys.stderr.write('*** %s\n' % sys.exc_info()[1])
sys.exit(1)
Like I said, I hardly know Python, so I did not pay attention to the indents. After correcting this, the email was sent, so feel free to use the above steps as a little tutorial for setting up git-multimail the "easiest" way. (I did not find a tutorial for this exact solution.)
I have a malformed database. When I try to get records from any of two tables, it throws an exception:
DatabaseError: database disk image is malformed
I know that through commandline I can do this:
sqlite3 ".dump" base.db | sqlite3 new.db
Can I do something like this from within Python?
As far as i know you cannot do that (alas, i might be mistaken), because the sqlite3 module for python is very limited.
Only workaround i can think of involves calling the os command shell (e.g. terminal, cmd, ...) (more info) via pythons call-command:
Combine it with the info from here to do something like this:
This is done on an windows xp machine:
Unfortunately i can't test it on a unix machine right now - hope it will help you:
from subprocess import check_call
def sqliterepair():
check_call(["sqlite3", "C:/sqlite-tools/base.db", ".mode insert", ".output C:/sqlite-tools/dump_all.sql", ".dump", ".exit"])
check_call(["sqlite3", "C:/sqlite-tools/new.db", ".read C:/sqlite-tools/dump_all.sql", ".exit"])
return
The first argument is calling the sqlite3.exe. Because it is in my system path variable, i don't need to specify the path or the suffix ".exe".
The other arguments are chained into the sqlite3-shell.
Note that the argument ".exit" is required so the sqlite-shell will exit. Otherwise the check_call() will never complete because the outer cmd-shell or terminal will be in suspended.
Of course the dump-file should be removed afterwards...
EDIT: Much shorter solution (credit goes to OP (see comment))
os.system("sqlite3 C:/sqlite-tools/base.db .dump | sqlite3 C:/sqlite-tools/target.db")
Just tested this: it works. Apparently i was wrong in the comments.
If I understood properly, what you want is to duplicate an sqlite3 database in python. Here is how I would do it:
# oldDB = path to the corrupted db,
# newDB = path to the new db
def duplicateDB(oldDB, newDB):
con = sqlite3.connect(oldDB)
script = ''.join(con.iterdump())
con.close()
con = sqlite3.connect(newDB)
con.executescript(script)
con.close()
print "duplicated %s into %s" % (oldDB,newDB)
In your example, call duplicateDB('base.db', 'new.db'). The iterdump function is equivalent to dump.
Note that if you use Python 3, you will need to change the print statement.