In the t.cursor() method, an exception from the Twython library is thrown for a small percentage of ids. However, whenever the exception occurs, the actual line in my code where it's thrown is the for-loop that comes after the try/except block, which prevents the continue from being called.
How is it that an exception can be thrown in the try block, not be caught by the except block, and then show up later during some (mostly) unrelated code?
And yes, it's a 401 error, but that's just the Twitter API returning the wrong code. In reality I'm correctly authenticating. I also know I can just move the except block to after the for-loop, but I just want to know how this can happen at all.
from twython import Twython
t = Twython(...)
# ...
for id in ids:
try:
# exception is truly caused by the following line
followers = t.cursor(t.get_followers_ids, id=id)
except:
# this block is never run
print("Exception with user " + str(id))
continue
# this line actually throws the exception, inexplicably
for follower_id in followers:
values.append((follower_id, id, scrape_datetime))
# ...
The traceback:
Traceback (most recent call last):
File "/root/twitter/nightly.py", line 5, in <module>
t.get_followers(t.current_tweeters)
File "/root/twitter/tweets.py", line 81, in get_followers
for follower_id in followers:
File "/usr/local/lib/python3.3/dist-packages/twython-3.0.0-py3.3.egg/twython/api.py", line 398, in cursor
content = function(**params)
File "/usr/local/lib/python3.3/dist-packages/twython-3.0.0-py3.3.egg/twython/endpoints.py", line 212, in get_followers_ids
return self.get('followers/ids', params=params)
File "/usr/local/lib/python3.3/dist-packages/twython-3.0.0-py3.3.egg/twython/api.py", line 231, in get
return self.request(endpoint, params=params, version=version)
File "/usr/local/lib/python3.3/dist-packages/twython-3.0.0-py3.3.egg/twython/api.py", line 225, in request
content = self._request(url, method=method, params=params, api_call=url)
File "/usr/local/lib/python3.3/dist-packages/twython-3.0.0-py3.3.egg/twython/api.py", line 195, in _request
retry_after=response.headers.get('retry-after'))
twython.exceptions.TwythonAuthError: Twitter API returned a 401 (Unauthorized), An error occurred processing your request.
Looks like t.cursor(...) returns a generator that doesn't actually execute until you iterate through it. While it might appear that the connection happens at:
followers = t.cursor(t.get_followers_ids, id=id)
It doesn't until you iterate through the generator with your for loop. Sort of mentioned here
If you have a need to defer processing until later but still want to catch the exception, turn the generator into a list. This will exhaust the generator and save the data for later.
followers = t.cursor(t.get_followers_ids, id=id)
followers = list(followers)
What you probably got with
followers = t.cursor(t.get_followers_ids, id=id)
is a cursor to a piece of code that will fetch your list. But because this cursor is lazy no Twyton code is yet executed. The acual fetching code is only executed when it is first used .... at the line that throws your exception.
So wrap that in the exception handling also.
Related
this is the code but its giving me error. maybe get by entity not working in this code.i have tried changing the entity type and everything else but that also doesnt works. i give parameters as required in the script but still it doesnt do anything but only gives me error"some error in adding".
try:
user_to_add = client.get_entity(int(user['user_id']))
print(user_to_add)
client(InviteToChannelRequest(entity, [user_to_add]))
usr_id = user['user_id']
print(f'{attempt}{g} Adding {usr_id}{rs}')
print(f'{sleep}{g} Sleep 20s{rs}')
time.sleep(20)
except PeerFloodError:
#time.sleep()
os.system(f'del {file}')
sys.exit(f'\n{error}{r} Aborted. Peer Flood Error{rs}')
except UserPrivacyRestrictedError:
print(f'{error}{r} User Privacy Restriction{rs}')
continue
except KeyboardInterrupt:
print(f'{error}{r} Aborted. Keyboard Interrupt{rs}')
update_list(users, added_users)
if not len(users) == 0:
print(f'{info}{g} Remaining users logged to {file}')
logger = Relog(users, file)
logger.start()
except:
print(f'{error}{r} Some Other error in adding{rs}')
continue
#os.system(f'del {file}')
input(f'{info}{g}Adding complete...Press enter to exit...')
sys.exit()
give me this error
> Traceback (most recent call last):
File "C:\Users\Noni\Desktop\python\telegramscraper-main\addbyid1.py", line 90, in <module>
user_to_add = client.get_entity(int(user['user_id']))
File "C:\Users\Noni\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\telethon\sync.py", line 39, in syncified
return loop.run_until_complete(coro)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\asyncio\base_events.py", line 647, in run_until_complete
return future.result()
File "C:\Users\Noni\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\telethon\client\users.py", line 292, in get_entity
inputs.append(await self.get_input_entity(x))
File "C:\Users\Noni\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\telethon\client\users.py", line 466, in get_input_entity
raise ValueError(
ValueError: Could not find the input entity for PeerUser(user_id=807250194) (PeerUser). Please read https://docs.telethon.dev/en/latest/concepts/entities.html to find out more details.
Can you provide the error?
I think you know that you can't add more than 200 subscribers to the channel manually. Please check the current number of subscribers. If it is more than 200 your code will not work.
I am running the following code inside a transaction.atomic block in Django.
#transaction.atomic()
def test():
a.save()
sid = transaction.savepoint()
try:
b.save()
transaction.savepoint_commit(sid)
except IntegrityError as e:
transaction.savepoint_rollback(sid)
c.save()
This code gives me the following Error -
TransactionManagementError
An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.
I followed the following link from the official documentation. https://docs.djangoproject.com/en/1.10/topics/db/transactions/#s-savepoint-rollback
What am I missing here?
EDIT:-
Adding the stacktrace.
File "/Users/vibhor/Documents/juggernaut/user-venv-new/lib/python2.7/site-packages/django/db/models/manager.py", line 122, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/Users/vibhor/Documents/juggernaut/user-venv-new/lib/python2.7/site-packages/django/db/models/query.py", line 401, in create
obj.save(force_insert=True, using=self.db)
File "/Users/vibhor/Documents/juggernaut/user-persistence-service/books/models/books.py", line 243, in save
transaction.savepoint_rollback(sid)
File "/Users/vibhor/Documents/juggernaut/user-venv-new/lib/python2.7/site-packages/django/db/transaction.py", line 66, in savepoint_rollback
get_connection(using).savepoint_rollback(sid)
File "/Users/vibhor/Documents/juggernaut/user-venv-new/lib/python2.7/site-packages/django/db/backends/base/base.py", line 328, in savepoint_rollback
self._savepoint_rollback(sid)
File "/Users/vibhor/Documents/juggernaut/user-venv-new/lib/python2.7/site-packages/django/db/backends/base/base.py", line 288, in _savepoint_rollback
cursor.execute(self.ops.savepoint_rollback_sql(sid))
File "/Users/vibhor/Documents/juggernaut/user-venv-new/lib/python2.7/site-packages/django/db/backends/utils.py", line 79, in execute
return super(CursorDebugWrapper, self).execute(sql, params)
File "/Users/vibhor/Documents/juggernaut/user-venv-new/lib/python2.7/site-packages/django/db/backends/utils.py", line 59, in execute
self.db.validate_no_broken_transaction()
File "/Users/vibhor/Documents/juggernaut/user-venv-new/lib/python2.7/site-packages/django/db/backends/base/base.py", line 429, in validate_no_broken_transaction
"An error occurred in the current transaction. You can't "
TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.
I think you're running into this issue described in the documentation:
Savepoints may be used to recover from a database error by performing a partial rollback. If you’re doing this inside an atomic() block, the entire block will still be rolled back, because it doesn’t know you’ve handled the situation at a lower level! To prevent this, you can control the rollback behavior with the following functions...
As noted there, you probably want to do a transaction.set_rollback(False) to prevent the whole atomic block from being rolled back.
Now, is there a reason you're doing this manually? The code you posted could accomplish the same thing with a nested atomic block, and as the documentation notes:
When the atomic() decorator is nested, it creates a savepoint to allow partial commit or rollback. You’re strongly encouraged to use atomic() rather than the functions described below.
As explained in the documentation :
In order to guarantee atomicity, atomic disables some APIs. Attempting to commit, roll back, or change the autocommit state of the database connection within an atomic block will raise an exception.
The proper way of achieving what you want is to create another atomic block :
#transaction.atomic()
def test():
a.save()
try:
with transaction.atomic():
b.save()
except IntegrityError:
#handle exception here...
...
c.save()
#transaction.atomic()
def test():
a.save()
sid = transaction.savepoint()
try:
with transaction.atomic():
b.save()
transaction.savepoint_commit(sid)
except IntegrityError as e:
transaction.savepoint_rollback(sid)
c.save()
please check this
I am trying to get my bottle server so that when one person in a game logs out, everyone can immediately see it. As I am using long polling, there is a request open with all the users.
The bit I am having trouble with is catching the exception that is thrown when the user leaves the page from the long polling that can no longer connect to the page. The error message is here.
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 438, in handle_one_response
self.run_application()
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 425, in run_application
self.process_result()
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 416, in process_result
self.write(data)
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 373, in write
self.socket.sendall(msg)
File "/usr/lib/python2.7/dist-packages/gevent/socket.py", line 509, in sendall
data_sent += self.send(_get_memory(data, data_sent), flags)
File "/usr/lib/python2.7/dist-packages/gevent/socket.py", line 483, in send
return sock.send(data, flags)
error: [Errno 32] Broken pipe
<WSGIServer fileno=3 address=0.0.0.0:8080>: Failed to handle request:
request = GET /refreshlobby/1 HTTP/1.1 from ('127.0.0.1', 53331)
application = <bottle.Bottle object at 0x7f9c05672750>
127.0.0.1 - - [2013-07-07 10:59:30] "GET /refreshlobby/1 HTTP/1.1" 200 160 6.038377
The function to handle that page is this.
#route('/refreshlobby/<id>')
def refreshlobby(id):
while True:
yield lobby.refresh()
gevent.sleep(1)
I tried catching the exception within the function, and in a decorator which I put to wrap #route, neither of which worked. I tried making an #error(500) decorator, but that didn't trigger, either. It seems that this is to do with the internals of bottle.
Edit: I know now that I need to be catching socket.error, but I don't know whereabouts in my code
The WSGI runner
Look closely at the traceback: this in not happening in your function, but in the WSGI runner.
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 438, in handle_one_response
self.run_application()
The way the WSGI runner works, in your case, is:
Receives a request
Gets a partial response from your code
Sends it to the client (this is where the exception is raised)
Repeats steps 2-3
You can't catch this exception
This error is not raised in your code.
It happens when you try to send a response to a client that closed the connection.
You'll therefore not be able to catch this error from within your code.
Alternate solutions
Unfortunately, it's not possible to tell from within the generator (your code) when it stops being consumed.
It's also not a good idea to rely on your generator being garbage collected.
You have a couple other solutions.
"Last seen"
Another way to know when an user disconnects would probably be to record a "last seen", after your yield statement.
You'll be able to identify clients that disconnected if their last seen is far in the past.
Other runner
Another, non-WSGI runner, will be more appropriate for a realtime application. You could give tornado a try.
Quite often GAE is not able to upload the file and I am getting the following error:
ApplicationError: 2
Traceback (most recent call last):
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/__init__.py", line 636, in __call__
handler.post(*groups)
File "/base/data/home/apps/picasa2vkontakte/1.348093606241250361/picasa2vkontakte.py", line 109, in post
headers=headers
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/urlfetch.py", line 260, in fetch
return rpc.get_result()
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 592, in get_result
return self.__get_result_hook(self)
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/urlfetch.py", line 355, in _get_fetch_result
raise DownloadError(str(err))
DownloadError: ApplicationError: 2
How should I perform retries in case of such error?
try:
result = urlfetch.fetch(url=self.request.get('upload_url'),
payload=''.join(data),
method=urlfetch.POST,
headers=headers
)
except DownloadError:
# how to retry 2 more times?
# and how to verify result here?
If you can, move this work into the task queue. When tasks fail, they retry automatically. If they continue to fail, the system gradually backs off retry frequency to as slow as once-per hour. This is an easy way to handle API requests to rate-limited services without implementing one-off retry logic.
If you really need to handle requests synchronously, something like this should work:
for i in range(3):
try:
result = urlfetch.fetch(...)
# run success conditions here
break
except DownloadError:
#logging.debug("urlfetch failed!")
pass
You can also pass deadline=10 to urlfetch.fetch to double the default timeout deadline.
I have a piece of code that works in a background process which looks like
from django.db import transaction
try:
<some code>
transaction.commit()
except Exception, e:
print e
transaction.rollback()
In a test, I break <some_code> with data that causes a database error. The exception is following
File "/home/commando/Development/Diploma/streaminatr/stream/testcases/feeds.py", line 261, in testInterrupt
form.save(self.user1)
File "/usr/lib/pymodules/python2.5/django/db/transaction.py", line 223, in _autocommit
return func(*args, **kw)
File "/home/commando/Development/Diploma/streaminatr/stream/forms.py", line 99, in save
print(models.FeedChannel.objects.all())
File "/usr/lib/pymodules/python2.5/django/db/models/query.py", line 68, in `__repr__ `
data = list(self[:REPR_OUTPUT_SIZE + 1])
File "/usr/lib/pymodules/python2.5/django/db/models/query.py", line 83, in `__len__ `
self._result_cache.extend(list(self._iter))
File "/usr/lib/pymodules/python2.5/django/db/models/query.py", line 238, in iterator
for row in self.query.results_iter():
File "/usr/lib/pymodules/python2.5/django/db/models/sql/query.py", line 287, in results_iter
for rows in self.execute_sql(MULTI):
File "/usr/lib/pymodules/python2.5/django/db/models/sql/query.py", line 2369, in execute_sql
cursor.execute(sql, params)
InternalError: current transaction is aborted, commands ignored until end of transaction block
This is what I expect. The bad thing is that I still get the same error when I try to access the DB after transaction.rollback is called. What should I do to rollback the transaction succcessfully and make the connection usable once again?
Btw, I also tried inserting print connection.queries to debug the code, and it always returns an empty list. Could it be that Django is using some other DB connection?
The code is run outside of request-response cycle. I tried switching TransactionMiddleware on and off, but it gave no effect.
I am using Django 1.1 and Postgres 8.4.
Default TestCase does not know anything about transactions, you need to use TransactionalTestCase in this case.
I wrote this decorator based on the transaction middleware source. Hope it helps, works perfectly for me.
def djangoDBManaged(func):
def f(*args, **kwargs):
django.db.transaction.enter_transaction_management()
django.db.transaction.managed(True)
try:
rs = func(*args, **kwargs)
except Exception:
if django.db.transaction.is_dirty():
django.db.transaction.rollback()
django.db.transaction.leave_transaction_management()
raise
finally:
if django.db.transaction.is_managed():
if django.db.transaction.is_dirty():
django.db.transaction.commit()
django.db.transaction.leave_transaction_management()
return rs
# So logging gets the right call info whatever the decorator order is
f.__name__ = func.__name__
f.__doc__ = func.__doc__
f.__dict__ = func.__dict__
return f