I have a function the a Celery task calls in my Django app that processes each row in a CSV file and tries to insert information from each row into the PostgreSQL database using a model, like so:
reader = csv.DictReader(csvfile, dialect=csv.get_dialect('excel'))
for row_number, row in enumerate(reader):
try:
...
customer, created = Customer.objects.get_or_create(
first_name = row['First Name'],
last_name = row['Last Name'],
email = row['Email'],
account_owner = account_owner
)
...
except Exception, e:
message = "%s: %s\n" % (type(e).__name__, e)
sys.stderr.write("%s\n" % message)
I'm getting the following error at Customer.objects.get_or_create on the first iteration of the loop:
DatabaseError: no such savepoint
And then I'm getting DatabaseError: current transaction is aborted, commands ignored until end of transaction block for each iteration thereafter.
I have successfully inserted (manually with SQL) a record into this table with the same information the script would provide. I have also tried working with savepoints, with no luck:
sid = transaction.savepoint()
customer, created = Customer.objects.get_or_create(
first_name = row['First Name'],
last_name = row['Last Name'],
email = row['Email'],
account_owner = account_owner
)
transaction.savepoint_commit(sid)
This is happening on my dev machine with PostgreSQL 9.1 installed from Homebrew. Obviously the main goal is to fix this issue, but any helpful pointers for debugging the Celery task or locating my Postgres log files would be helpful also.
After doing some more research, it looks like the underlying cause of this is in an IntegrityError which tries to force a savepoint rollback, which doesn't exist anymore. I fix the IntegrityError and I assume the no savepoint issue is fixed. Still would be great to see the IntegrityError immediately as I'm debugging...
Enabled logging using these instructions and I'm seeing that the insert statement generated by the get_or_create violates a foreign key contraint.
Related
I am running into a weird error trying to append dataframe to MySQL table using pandas to_sql function. I have not been able to find answer to this anywhere. Here is a test example:
test_df = pd.DataFrame(['a','b','c','d'], columns = ['char'])
with engine.begin() as connection:
test_df.to_sql(name='test', con=connection, if_exists='append')
The above runs successfuly, I can see the table being created in my database.
new_df = pd.DataFrame(['e','f'], columns = ['char'])
with engine.begin() as connection:
new_df.to_sql(name='test', con=connection, if_exists='append')
However, when I try to append more data, I get following error:
OperationalError: (MySQLdb._exceptions.OperationalError) (1050, "Table 'test' already exists")
[SQL:
CREATE TABLE test (
`index` BIGINT,
`char` TEXT
)
]
(Background on this error at: https://sqlalche.me/e/14/e3q8)
This is very confusing. Since I did not see anyone encounter this error, could it be a bad installation of my packages. Any help will be greatly appreciated.
Thanks to the comment by Rouhollah. I made the "append' to work by replacing
engine = create_engine(f"mysql://{user}:{password}#{host}:{port}")
with
engine = create_engine(f"mysql://{user}:{password}#{host}:{port}/{database}")
previously, I was accessing database using engine.execute(f"USE {database}") which seems to break the append function.
try:
cursor = connection.cursor()
cursor.execute('update m_password set password = :new_password , updater_id = :updater_id_dic , update_date = :update_date_dic where user_cd = :user_cd_dic and del_flg = :del_flag_dic', queryDictionary)
# Commit
connection.commit()
# Close the cursor
cursor.close()
# Close the database connection
connection.close()
return {
'status': "success",
'errorCode': 0,
'errorMessage': json.dumps("")
}
except: cx_Oracle.IntegrityError as e:
errorObj, = e.args
return {
'status': "error",
'errorCode': errorObj.code,
'errorMessage': json.dumps(errorObj.message)
I am trying to update the value of oracle database . Database connection is alright . I can get the value from database . But update the value is not working . It shows success but value is not actually updated . Also there is no error .
also lambda log showing successfully executed . Please check this image
[1]: https://i.stack.imgur.com/PtLdy.png
I am stuck here .It will be really helpful if i get some help .
Thank you.
It's likely that you're getting an error other than an IntegrityError and don't have an explicit error handler for it. This would cause you to continue towards your this is fine return method.
In fact, updating a password shouldn't throw an integrity error at all since you're not operating on something that effects your schema or is effected by a schema constraint. According to the docs:
This exception is raised when the relational integrity of the data is affected. For example, a duplicate key was inserted or a foreign key constraint would fail.
To troubleshoot, I'd start by accepting any error here. Once you've determined what error type you've caught, you can add per exception error handling as needed.
except Exception as e:
errorObj, = e.args
return {
...
I have a Apache Ignite database running which I want to interact with using Python thin client (pyignite). I've already performed create, read and update operations, but I meet problems with the delete ones. For now, even if the submission of delete request does not raise any error, the entries that are supposed to be deleted are not.
I've tried deleting those same entries running the same delete query in terminal via jdbc:ignite:thin://127.0.0.1/ and this does succesfully remove the targeted entries.
Here is how I unsuccesfully tried to delete data :
self.client = Client()
self.client.connect('127.0.0.1', 10800)
patientID = 5
IS_DEFINED_QUERY = "SELECT * FROM Patients WHERE PatientID = ?"
result = self.client.sql(
IS_DEFINED_QUERY,
query_args=[patientID]
)
try:
next(result)
DELETE_QUERY = "DELETE FROM Patients WHERE PatientID = ?"
self.client.sql(
DELETE_QUERY,
query_args=[patientID])
except StopIteration:
raise KeyDoesNotExist()
Any help would be greatly appreciated, thanks !
EDIT: I've got some suggestions saying it might come from database settings that would prevent Thin Client from executing deletions, any thought about it ?
I am creating a simple app in pyramid . While inserting data i am getting db locked error.
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked [SQL: 'INSERT INTO users (name, email, number) VALUES (?, ?, ?)'] [parameters: ('test', 't#t.com', '123654')]
But first time it inserts the data correctly and on 2nd time it gives this error.
Any idea why this is happening second time ?
Here is my code :
name = request.params['name']
email = request.params['email']
no = request.params['number']
DBSession.add(User(name, email, no))
# Get the new ID and redirect
users = DBSession.query(User).all()
SQLite can only handle 1 concurrent transaction.
Have you tried commit()before performing a query() then close() to end the session?
name = request.params['name']
email = request.params['email']
no = request.params['number']
DBSession.add(User(name, email, no))
# Commit the transaction if complete
DBSession.commit()
# Get the new ID and redirect
users = DBSession.query(User).all()
# Close the session
DBSession.close()
Not sure if there's a better way to do this but I have a sign up page on my site and after a user signs up I add their initial data(stuff in the __init__ data model) then I start adding some other info in the same section which is giving me a broken pipe error. Oddly the code seems to work since the entries I'm expecting are in the database. I have tried moving around the .flush() command to see if it helps but it doesn't seem to be.
Traceback (most recent call last):
File "/Users/me/Dropbox/code/eclipseWorkSpace/website/pyramidwiki/lib/python2.7/site-packages/waitress-0.8.1-py2.7.egg/waitress/channel.py", line 134, in handle_write
flush()
File "/Users/me/Dropbox/code/eclipseWorkSpace/website/pyramidwiki/lib/python2.7/site-packages/waitress-0.8.1-py2.7.egg/waitress/channel.py", line 249, in _flush_some
num_sent = self.send(chunk)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/asyncore.py", line 365, in send
result = self.socket.send(data)
error: [Errno 32] Broken pipe
Here's my code:
if 'form.submitted' in request.params:
firstname = request.params['firstname']
lastname = request.params['lastname']
email = request.params['email']
password = request.params['password']
try:
new_user = Users(email, firstname, lastname, password)
DBSession.add(new_user)
#DBSession.flush() #commit so we get error if any
#add some other info
user_data = DBSession.query(Users).filter(Users.email==email).first()
user_data.join_date = datetime.datetime.now()
#create random number for verification url
user_data.vertified = id_generator(50)
DBSession.flush() #doesn't seem to make a difference where the flush is
return HTTPFound(location = request.route_url('new'))
Any ideas?
This may not answer you question directly, but "you're doing it all wrong"(tm) :)
You don't need to re-query User object after you added it to the session - what's more, trying to query it from the database without doing session.flush() first will result in an error because there's no record in the database yet. I'd do something like this:
if 'form.submitted' in request.params:
firstname = request.params['firstname']
lastname = request.params['lastname']
email = request.params['email']
password = request.params['password']
try:
new_user = Users(email, firstname, lastname, password)
new_user.join_date = datetime.datetime.now()
new_user.verified = id_generator(50)
DBSession.add(new_user)
DBSession.flush() # should fail if user email is in the database
return HTTPFound(location = request.route_url('new'))
Also, you need to check that all branches of execution (i.e. the except: clause, the else: clause of "if 'form.submitted' in request.params" return something meaningful - you may be getting the exception because your view function returns None in some conditions. Actually, that's probably what was happening - the "user_data = DBSession.query(Users)" line was raising an exception, and the except: part was not returning anything
I too struggled with the same problem on my Pyramid project and contrary to comments on github, it wasn't with waitress.
In my case, the problem of Error: Broken Pipe only occured whenever I used redirects (HTTPFound, HTTPMovedPermanently etc.). Since your function also uses HTTPFound, I think the problem is the same.
Atleast for me, this error was due to pyramid_debugtoolbar extension. The reason is probably that whenever our view does soemthing like
return HTTPFound(foo)
it sends out a 302 in the header and a Connection:Close. The pyramid_debugtoolbar extension adds a lengthy body to the response. The client on seeing the header, closes the connection and does not accept the lengthy debug body. Hence the Broken Pipe message (Atleast that is what Wireshark shows).
Try disabling the pyramid_debugtoolbar in your app's .ini, it might help.