I was again working with my Pymongo file (really sorry I am asking soo many questions I am really new to pymongo.) and I get this error.. Could you explain what this means becuase I am really new...
ret = await coro(*args, **kwargs)
File "d:\python_projects\Jeli-Bot\cogs\warns.py", line 36, in warn
if collection.count_documents({"memberid":id}) == 0:
File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\pymongo\collection.py", line 1827, in count_documents
return self.__database.client._retryable_read(
File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\pymongo\mongo_client.py", line 1525, in _retryable_read
return func(session, server, sock_info, secondary_ok)
File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\pymongo\collection.py", line 1821, in _cmd
result = self._aggregate_one_result(
File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\pymongo\collection.py", line 1686, in _aggregate_one_result
result = self._command(
File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\pymongo\collection.py", line 238, in _command
return sock_info.command(
File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\pymongo\pool.py", line 726, in command
self._raise_connection_failure(error)
File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\pymongo\pool.py", line 710, in command
return command(self, dbname, spec, secondary_ok,
File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\pymongo\network.py", line 121, in command
request_id, msg, size, max_doc_size = message._op_msg(
File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\pymongo\message.py", line 743, in _op_msg
return _op_msg_uncompressed(
bson.errors.InvalidDocument: cannot encode object: <property object at 0x000001AA06B13AE0>, of type: <class 'property'>
I would suggest to copy-paste the chunk of code you are running, also details on what you were trying to accomplish.
With the information provided I would think you are trying to pass an object to a function that shouldn't be receiving one.
When you try and run a pymongo operation, the data you pass to the operation must match defined python types that pymongo understands. The values it understands are represented in the table here: https://pymongo.readthedocs.io/en/stable/api/bson/index.html
Your issue is that you passing a type that isn't in the Python Type column of the above link. For example, maybe you're passing a set, a datetime.date or an arbitrary class object. If this is the case, you will get the InvalidDocument: cannot encode error you are seeing.
Double check the document structure you are passing into the pymongo command.
Related
I am running a Pyramid + Zope transaction manager + SQLAlchemy + PostgreSQL. On some occasions, I have seen StaleDataError error on a Pyramid web application which should very trivial view of updating one row in a database. As the error happens outside the normal view boundary and is not repeatable, it is quite tricky to debug.
I guess this might have something to do with broken database connections or transaction lifecycle. However I don't know how to start debugging the system, so I am asking what could cause this and furthermore how one can pin down errors like this.
UPDATE statement on table 'xxx' expected to update 1 row(s); 0 were matched.
Stacktrace (most recent call last):
File "pyramid/tweens.py", line 20, in excview_tween
response = handler(request)
File "pyramid_tm/__init__.py", line 94, in tm_tween
reraise(*exc_info)
File "pyramid_tm/compat.py", line 15, in reraise
raise value
File "pyramid_tm/__init__.py", line 82, in tm_tween
manager.commit()
File "transaction/_manager.py", line 111, in commit
return self.get().commit()
File "transaction/_transaction.py", line 280, in commit
reraise(t, v, tb)
File "transaction/_compat.py", line 55, in reraise
raise value
File "transaction/_transaction.py", line 271, in commit
self._commitResources()
File "transaction/_transaction.py", line 417, in _commitResources
reraise(t, v, tb)
File "transaction/_compat.py", line 55, in reraise
raise value
File "transaction/_transaction.py", line 389, in _commitResources
rm.tpc_begin(self)
File "/srv/pyramid/trees/venv/lib/python3.4/site-packages/zope/sqlalchemy/datamanager.py", line 90, in tpc_begin
self.session.flush()
File "sqlalchemy/orm/session.py", line 2004, in flush
self._flush(objects)
File "sqlalchemy/orm/session.py", line 2122, in _flush
transaction.rollback(_capture_exception=True)
File "sqlalchemy/util/langhelpers.py", line 60, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "sqlalchemy/util/compat.py", line 182, in reraise
raise value
File "sqlalchemy/orm/session.py", line 2086, in _flush
flush_context.execute()
File "sqlalchemy/orm/unitofwork.py", line 373, in execute
rec.execute(self)
File "sqlalchemy/orm/unitofwork.py", line 532, in execute
uow
File "sqlalchemy/orm/persistence.py", line 170, in save_obj
mapper, table, update)
File "sqlalchemy/orm/persistence.py", line 692, in _emit_update_statements
(table.description, len(records), rows))
This is most likely scenario:
You have 2 requests that first select an object and try to update/delete it in the datastore and you end up with a "race condition".
Lets say you want to do something like fetch an object and then update it.
If transaction takes some time and you do not select the object with "for update" thus locking the rows - if the object gets deleted in first request and 2nd transaction tries to issue update to row that is not present in the db anymore you can end up with this exception.
You can try doing some row locking to prevent this from happening - subsequent transaction will "wait" for the first operation to finish. Before it gets executed.
http://docs.sqlalchemy.org/en/rel_1_0/orm/query.html?highlight=for_update#sqlalchemy.orm.query.Query.with_for_update
and
http://docs.sqlalchemy.org/en/rel_1_0/orm/query.html?highlight=with_lockmode#sqlalchemy.orm.query.Query.with_lockmode
Describe some of the sqlalchemy machinery you can use to resolve this.
Another option:
TL,DR: if you have "first()" in some cases, you need to remove this in alchemy if you update multiple records
db.session.query(xxx).filter_by(field=value).first()
This command expects the update to affect only one line. And it should if your table has only one record with field=value. This is particularly the case if the field is your ID.
HOWEVER - if your ID is not unique, you might have multiple records with the same ID.
In this case, your can update all by removing "first()"
BTW, use the following to debug your SQL queries (which wouldn't have helped this time...)
import logging
logging.basicConfig()
logging.getLogger('sqlalchemy.engine').setLevel(logging.INFO)
I am attempting to write a string directly into a Django FileField by way of ContentFile.
In doing so, I get a reproducible
TypeError: Unicode-objects must be encoded before hashing
error when attempting to save the contents of this file to the database, which traces through the s3boto3 lib.
The exact source of this error is difficult to suss out.
But let's state the question plainly, in Python 3, on Django 2.2.x, what is the correct way to take a csv file created with the csv lib from Python, and save that into a Django FileField backed by Amazon S3?
This question, and my approach, is inspired by this entry on SO Django - how to create a file and save it to a model's FileField? - however, given the age of the answer, some detail relevant to newer versions of Django appear to have been left out? Difficult to tell.
Example code producing the error in question, truncated for privacy and relevance
def campaign_to_csv_string(campaign_id):
csv_string = io.StringIO()
campaign = Campaign.objects.get(pk=campaign_id)
checklist = campaign.checklist
completed_jobs = JobRecord.objects.filter(appointment__campaign=campaign)
writer = csv.writer(csv_string)
# A bunch of writing to the writer here
# string looks good at this point
return csv_string.getvalue()
calling function
csv_string = campaign_to_csv_string(campaign_report.campaign.pk)
campaign_report.last_run = datetime.datetime.now()
campaign_report.report_file.save(str(campaign_report_pk) + '.report', ContentFile(csv_string))
campaign_report.processing = False
campaign_report.save()
My guess here is that s3boto3 is taking issue with ContentFile but the debugging information sent back to me gives me no clear path forward.
edit
Stack trace by request
TypeError: Unicode-objects must be encoded before hashing
File "celery/app/trace.py", line 385, in trace_task
R = retval = fun(*args, **kwargs)
File "celery/app/trace.py", line 648, in __protected_call__
return self.run(*args, **kwargs)
File "main/tasks.py", line 94, in produce_basic_campaign_report
campaign_report.report_file.save(str(campaign_report_pk) + '.report', csv_file)
File "django/db/models/fields/files.py", line 87, in save
self.name = self.storage.save(name, content, max_length=self.field.max_length)
File "django/core/files/storage.py", line 52, in save
return self._save(name, content)
File "storages/backends/s3boto3.py", line 491, in _save
self._save_content(obj, content, parameters=parameters)
File "storages/backends/s3boto3.py", line 506, in _save_content
obj.upload_fileobj(content, ExtraArgs=put_parameters)
File "boto3/s3/inject.py", line 621, in object_upload_fileobj
ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
File "boto3/s3/inject.py", line 539, in upload_fileobj
return future.result()
File "s3transfer/futures.py", line 106, in result
return self._coordinator.result()
File "s3transfer/futures.py", line 265, in result
raise self._exception
File "s3transfer/tasks.py", line 126, in __call__
return self._execute_main(kwargs)
File "s3transfer/tasks.py", line 150, in _execute_main
return_value = self._main(**kwargs)
File "s3transfer/upload.py", line 692, in _main
client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
File "botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "botocore/client.py", line 642, in _make_api_call
request_signer=self._request_signer, context=request_context)
File "botocore/hooks.py", line 360, in emit_until_response
return self._emitter.emit_until_response(aliased_event_name, **kwargs)
File "botocore/hooks.py", line 243, in emit_until_response
responses = self._emit(event_name, kwargs, stop_on_response=True)
File "botocore/hooks.py", line 211, in _emit
response = handler(**kwargs)
File "botocore/handlers.py", line 212, in conditionally_calculate_md5
calculate_md5(params, **kwargs)
File "botocore/handlers.py", line 190, in calculate_md5
binary_md5 = _calculate_md5_from_file(body)
File "botocore/handlers.py", line 204, in _calculate_md5_from_file
md5.update(chunk)
The csv string needs to be encoded as bytes when instantiating the ContentFile
The error can be reproduced this way:
from django.core.files.base import ContentFile
from botocore.handlers import _calculate_md5_from_file
_calculate_md5_from_file(ContentFile('throws error'))
TypeError: Unicode-objects must be encoded before hashing.
content isn't internal converted to bytes unless it is a gzip mimetype or explicitly compressed. https://github.com/jschneier/django-storages/blob/1.7.2/storages/backends/s3boto.py#L417
_calculate_md5_from_file is expecting a file containing bytes and this is the same for the underlying boto3 s3 client put_object method.
I suggest encoding csv_string as bytes.
campaign_report.report_file.save(
str(campaign_report_pk) + '.report',
ContentFile(
csv_string.encode()
)
)
I am using pandas.read_sql function with hive connection to extract a really large data. I have a script like this:
df = pd.read_sql(query_big, hive_connection)
df2 = pd.read_sql(query_simple, hive_connection)
The big query take a long time, and after it is executed, python returns the following error when trying to execute the second line:
raise NotSupportedError("Hive does not have transactions") # pragma: no cover
It seems there is something wrong with the connection.
Moreover, If I replace the second line with multirpocessing.Manager().Queue(), It returns the following error:
File "/usr/lib64/python3.6/multiprocessing/managers.py", line 662, in temp
token, exp = self._create(typeid, *args, **kwds)
File "/usr/lib64/python3.6/multiprocessing/managers.py", line 554, in _create
conn = self._Client(self._address, authkey=self._authkey)
File "/usr/lib64/python3.6/multiprocessing/connection.py", line 493, in Client
answer_challenge(c, authkey)
File "/usr/lib64/python3.6/multiprocessing/connection.py", line 732, in answer_challenge
message = connection.recv_bytes(256) # reject large message
File "/usr/lib64/python3.6/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/usr/lib64/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
buf = self._recv(4)
File "/usr/lib64/python3.6/multiprocessing/connection.py", line 383, in _recv
raise EOFError
EOFError
It seems this kind of error are related to exit function being messed up, in the connection.py. Moreover, when I changed the query in the first command to extract smaller data that doesn't take too long, Everything works fine. So I assume it may be that because it takes too long to execute the first query, something is improperly terminated. which caused the two error, both of which are so different in nature but both are related to broken connection issues.
I came across this problem when I tried some scraping code. I defined a class MongoCache to cache the html pages:
class MongoCache:
def __init__(self, client=None, expires=timedelta(days=30)):
self.client = MongoClient('localhost', 27017) if client is None else client
self.db = self.client.cache
self.db.webpage.create_index('timestamp1', expireAfterSeconds=expires.total_seconds())
when I build the object:
cache = MongoCache()
the failure information came out.
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "F:\pythoncode\webscraping\mongo_cache.py", line 20, in __init__
File "D:\python27\lib\site-packages\pymongo\collection.py", line 1958, in create_index
self.__create_index(keys, kwargs, session, **cmd_options)
File "D:\python27\lib\site-packages\pymongo\collection.py", line 1860, in __create_index
session=session)
File "D:\python27\lib\site-packages\pymongo\collection.py", line 244, in _command
retryable_write=retryable_write)
File "D:\python27\lib\site-packages\pymongo\pool.py", line 579, in command
unacknowledged=unacknowledged)
File "D:\python27\lib\site-packages\pymongo\network.py", line 150, in command
parse_write_concern_error=parse_write_concern_error)
File "D:\python27\lib\site-packages\pymongo\helpers.py", line 155, in _check_command_response
raise OperationFailure(msg % errmsg, code, response)
OperationFailure: Index with name: timestamp_1 already exists with different options
I tried some solutions from stackoverflow, but those are not for pymongo, and I cannot even use the method drop_index().
I used win10, python2.7 on pycharm, and the MongoDB server version is 4.0.3.
I have spent two days to figure out the problm, and gave up..
Now, I tried the question again, and found that the problem may be in timestamp used for the index.
I define an object with no input parameter, everything's ok.
cache = MongoCache()
but, using the timestamp, it comes again:
cache = MongoCache(expires=timedelta())
The function for saving value for url is:
def __setitem__(self, url, result):
record = {
'result': Binary(zlib.compress(pickle.dumps(result))),
'timestamp': datetime.utcnow()}
self.db.webpage.update({'_id': url}, {'$set': record}, upsert=True)
I'm working on an application in which a server and client are being created; the ServerAPI is using SimpleXMLRPCServer and the ClientAPI is using xmlrpclib. The client is initiated with:
class w_Client:
def __init__(self, ServerIP, ServerPort, ClientIP):
self.conn = xmlrpclib.ServerProxy("http://" + ServerIP + ":" + str(ServerPort))
self.ClientIP = ClientIP
upon a button being pressed in the application, an xml specification file is created and passed thru
def Create(self, XMLstring):
return self.conn.Create(XMLstring, self.ClientIP)
I've already checked to make sure that the XMLstring is valid XML; however, when I get press the button, I get the following error:
Traceback (most recent call last):
File "/home/app/UI/MainWindow.py", line 461, in compile
xmlFile = compiler.compile()
File "/home/app/Core/Compiler.py", line 75, in compile
self.compile_top()
File "/home/app/Core/Compiler.py", line 354, in compile_top
status = mainWidgets["w_client"].Create(xmlString)
File "/home/app/Wireless/ClientAPI.py", line 12, in Create
return self.conn.Create(XMLstring, self.ClientIP)
File "/usr/lib/python2.7/xmlrpclib.py", line 1233, in __call__
return self.__send(self.__name, args)
File "/usr/lib/python2.7/xmlrpclib.py", line 1591, in __request
verbose=self.__verbose
File "/usr/lib/python2.7/xmlrpclib.py", line 1273, in request
return self.single_request(host, handler, request_body, verbose)
File "/usr/lib/python2.7/xmlrpclib.py", line 1306, in single_request
return self.parse_response(response)
File "/usr/lib/python2.7/xmlrpclib.py", line 1482, in parse_response
return u.close()
File "/usr/lib/python2.7/xmlrpclib.py", line 794, in close
raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault 1: "<type 'exceptions.TypeError'>:'NoneType' object has no attribute '__getitem__'">
I've also made sure that the ClientIP is passed correctly. Otherwise, I'm not entirely sure what's going on or how to even go about fixing it.
<type 'exceptions.TypeError'>:'NoneType' object has no attribute '__getitem__'
This exception may have been generated by the xmlrpc method you were calling (i.e. server side).
I suggest that you add verbose=True to your instantiation of the server proxy:
xmlrpclib.ServerProxy("http://" + ServerIP + ":" + str(ServerPort),verbose=True)
This will allow you to see what you're sending and receiving.
It seems the method you're calling is expecting a dict