My scraper runs fine for about an hour. After a while I start seeing these errors:
2014-01-16 21:26:06+0100 [-] Unhandled Error
Traceback (most recent call last):
File "/home/scraper/.fakeroot/lib/python2.7/site-packages/Scrapy-0.20.2-py2.7.egg/scrapy/crawler.py", line 93, in start
self.start_reactor()
File "/home/scraper/.fakeroot/lib/python2.7/site-packages/Scrapy-0.20.2-py2.7.egg/scrapy/crawler.py", line 130, in start_reactor
reactor.run(installSignalHandlers=False) # blocking call
File "/home/scraper/.fakeroot/lib/python2.7/site-packages/twisted/internet/base.py", line 1192, in run
self.mainLoop()
File "/home/scraper/.fakeroot/lib/python2.7/site-packages/twisted/internet/base.py", line 1201, in mainLoop
self.runUntilCurrent()
--- <exception caught here> ---
File "/home/scraper/.fakeroot/lib/python2.7/site-packages/twisted/internet/base.py", line 824, in runUntilCurrent
call.func(*call.args, **call.kw)
File "/home/scraper/.fakeroot/lib/python2.7/site-packages/Scrapy-0.20.2-py2.7.egg/scrapy/utils/reactor.py", line 41, in __call__
return self._func(*self._a, **self._kw)
File "/home/scraper/.fakeroot/lib/python2.7/site-packages/Scrapy-0.20.2-py2.7.egg/scrapy/core/engine.py", line 106, in _next_request
if not self._next_request_from_scheduler(spider):
File "/home/scraper/.fakeroot/lib/python2.7/site-packages/Scrapy-0.20.2-py2.7.egg/scrapy/core/engine.py", line 132, in _next_request_from_scheduler
request = slot.scheduler.next_request()
File "/home/scraper/.fakeroot/lib/python2.7/site-packages/Scrapy-0.20.2-py2.7.egg/scrapy/core/scheduler.py", line 64, in next_request
request = self._dqpop()
File "/home/scraper/.fakeroot/lib/python2.7/site-packages/Scrapy-0.20.2-py2.7.egg/scrapy/core/scheduler.py", line 94, in _dqpop
d = self.dqs.pop()
File "/home/scraper/.fakeroot/lib/python2.7/site-packages/queuelib/pqueue.py", line 43, in pop
m = q.pop()
File "/home/scraper/.fakeroot/lib/python2.7/site-packages/Scrapy-0.20.2-py2.7.egg/scrapy/squeue.py", line 18, in pop
s = super(SerializableQueue, self).pop()
File "/home/scraper/.fakeroot/lib/python2.7/site-packages/queuelib/queue.py", line 157, in pop
self.f.seek(-size-self.SIZE_SIZE, os.SEEK_END)
exceptions.IOError: [Errno 22] Invalid argument
What could possibly be causing this? My version is 0.20.2. Once I get this error, scrapy stops doing anything. Even if I stop and run it again (using a JOBDIR directory), it still gives me these errors. I need to delete the job directory and start over if I need to get rid of these errors.
Try this:
Ensure that you're running latest Scrapy version (current: 0.24)
Search inside the resumed folder, and backup the file requests.seen
After backed up, remove the scrapy job folder
Start the crawl resuming with JOBDIR= option again
Stop the crawl
Replace the newly created requests.seen with previously backed up
Start crawl again
Related
I have this weird error that I have never seen before. I ran an RF automation regression test using Chrome and after a while, I got this error which basically stops the test execution and does not provide any test results.
I also noticed this new text line that I haven't seen before during text execution:
DevTools listening on
ws://127.0.0.1:12306/devtools/browser/df1c8bcc-867c-41bc-a9b4-c364bdfafec2
(The test scripts have been used before & passed without any issues)
W8-BENE date format - ddMMMMMyyyyDot :: This is to test the differ...
DevTools listening on
ws://127.0.0.1:12537/devtools/browser/6d32afb6-90f2-46d4-a18a-001fa61d2444
[ ERROR ] Unexpected error: IOError: [Errno 0] Error Traceback (most
recent call last): File
"c:\python27\lib\site-packages\robot\utils\application.py", line 83,
in _execute
rc = self.main(arguments, **options)
File "c:\python27\lib\site-packages\robot\run.py", line 445, in main
result = suite.run(settings)
File "c:\python27\lib\site-packages\robot\running\model.py", line
248, in run
self.visit(runner)
File "c:\python27\lib\site-packages\robot\model\testsuite.py", line
161, in visit
visitor.visit_suite(self)
File "c:\python27\lib\site-packages\robot\model\visitor.py", line
87, in visit_suite
suite.tests.visit(self)
File "c:\python27\lib\site-packages\robot\model\itemlist.py", line
76, in visit
item.visit(visitor)
File "c:\python27\lib\site-packages\robot\model\testcase.py", line
74, in visit
visitor.visit_test(self)
File "c:\python27\lib\site-packages\robot\running\runner.py", line
151, in visit_test
result)
File "c:\python27\lib\site-packages\robot\running\runner.py", line
182, in _run_teardown
exception = self._run_setup_or_teardown(teardown)
File "c:\python27\lib\site-packages\robot\running\runner.py", line
201, in _run_setup_or_teardown
StepRunner(self._context).run_step(data, name=name)
File "c:\python27\lib\site-packages\robot\running\steprunner.py",
line 57, in run_step
return runner.run(step, context)
File
"c:\python27\lib\site-packages\robot\running\librarykeywordrunner.py",
line 56, in run
return return_value File "c:\python27\lib\site-packages\robot\running\statusreporter.py", line
54, in exit
context.end_keyword(result)
File "c:\python27\lib\site-packages\robot\running\context.py", line
179, in end_keyword
self.output.end_keyword(keyword)
File "c:\python27\lib\site-packages\robot\output\output.py", line
65, in end_keyword
LOGGER.end_keyword(kw)
File "c:\python27\lib\site-packages\robot\output\logger.py", line
195, in end_keyword
logger.end_keyword(keyword)
File
"c:\python27\lib\site-packages\robot\output\console\verbose.py", line
62, in end_keyword
self._writer.keyword_marker(kw.status)
File
"c:\python27\lib\site-packages\robot\output\console\verbose.py", line
136, in keyword_marker
self._clear_status()
File
"c:\python27\lib\site-packages\robot\output\console\verbose.py", line
124, in _clear_status
self._write_info()
File
"c:\python27\lib\site-packages\robot\output\console\verbose.py", line
90, in _write_info
self._stdout.write(self._last_info)
File
"c:\python27\lib\site-packages\robot\output\console\highlighting.py",
line 51, in write
self.stream.write(console_encode(text, stream=self.stream))
W8-BENE date format - ddMMMMMyyyyDot :: This is to test the differ...
C:\Development\robot-scripts\WebApp>
Has anyone had the same issue? I can't seem to find any solution. I uninstalled and reinstalled everything even update all the web drivers.
I had the same error after update to WIN 10. Just did fresh python install and it works fine for me. also look here: https://github.com/robotframework/robotframework/issues/2709
I followed the instructions here: http://shon.github.io/2014/06/19/ui_testing_and_bdd.html about setting up Splinter with Behaving to run automated tests. I'm able to run a test successfully, but at the end of the test, it throws an error saying:
KeyError: 'browser'
and it won't continue testing any additional feature files. I'm pretty new to python and need some help in troubleshooting this.
Exception KeyError: 'browser'
Traceback (most recent call last):
File "/usr/local/bin/behave", line 11, in <module> sys.exit(main())
File "/Library/Python/2.7/site-packages/behave/__main__.py", line 109, in main
failed = runner.run()
File "/Library/Python/2.7/site-packages/behave/runner.py", line 672, in run
return self.run_with_paths()
File "/Library/Python/2.7/site-packages/behave/runner.py", line 693, in run_with_paths
return self.run_model()
File "/Library/Python/2.7/site-packages/behave/runner.py", line 483, in run_model
failed = feature.run(self)
File "/Library/Python/2.7/site-packages/behave/model.py", line 523, in run
failed = scenario.run(runner)
File "/Library/Python/2.7/site-packages/behave/model.py", line 867, in run
runner.run_hook('before_scenario', runner.context, self)
File "/Library/Python/2.7/site-packages/behave/runner.py", line 405, in run_hook
self.hooks[name](context, *args)
File "features/environment.py", line 48, in before_scenario
context.browser = default_browser
File "/Library/Python/2.7/site-packages/behave/runner.py", line 223, in __setattr__
record = self._record[attr]
KeyError: 'browser'
I found the issue. It is related to the Feature file structure. The Feature file was missing:
Background:
Given a browser
This also required changes to the environment.py file based on the info here: https://github.com/ggozad/behaving
I am using Plone 4.3.3 for creating my Plone site but when I shut-down the server it shows the following error.
Traceback (most recent call last):
File "/Plone/zinstance/parts/instance/bin/interpreter", line 298, in <module>
exec(compile(__file__f.read(), __file__, "exec"))
File "/Plone/buildout-cache/eggs/Zope2-2.13.22-py2.7.egg/Zope2/Startup/run.py", line 76, in <module>
run()
File "/Plone/buildout-cache/eggs/Zope2-2.13.22-py2.7.egg/Zope2/Startup/run.py", line 26, in run
starter.run()
File "/Plone/buildout-cache/eggs/Zope2-2.13.22-py2.7.egg/Zope2/Startup/__init__.py", line 108, in run
self.shutdown()
File "/Plone/buildout-cache/eggs/Zope2-2.13.22-py2.7.egg/Zope2/Startup/__init__.py", line 113, in shutdown
db.close()
File "/Plone/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-i686.egg/ZODB/DB.py", line 624, in close
user#user-Vostro-3300:~/Plone/zinstance$ #self._connectionMap
File "/Plone/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-i686.egg/ZODB/DB.py", line 506, in _connectionMap
self.pool.map(f)
File "/Plone/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-i686.egg/ZODB/DB.py", line 206, in map
self.all.map(f)
File "/Plone/buildout-cache/eggs/transaction-1.1.1-py2.7.egg/transaction/weakset.py", line 58, in map
f(elt)
File "/Plone/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-i686.egg/ZODB/DB.py", line 628, in _
c._release_resources()
File "/Plone/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-i686.egg/ZODB/Connection.py", line 1075, in _release_resources
c._storage.release()
AttributeError: 'NoneType' object has no attribute 'release'
There is an issue with Zope2 shutdown that tries to close a database connection (and in turn, a storage). However, this late-running sequence has some cosmetic side-effect for users of RelStorage. This is annoying, but not fundamentally a problem that should cause any data integrity issues.
Users of FileStorage or ZEO should not see this.
References:
https://github.com/zopefoundation/Zope/commit/5032027470091957a6c0028da04c0fc0a1ed646b
https://mail.zope.org/pipermail/zodb-dev/2013-August/015119.html
I'm running on Windows 7 x64. I followed the install documentation on Buildbot and did some research on the issue I'm having and haven't found a solution yet. When I do a force build, everything works fine. I'm using GitPoller. When it tries to poll for changes, an exception is thrown; why? Let me know if I can supply any more information. Here's what I'm getting on the master's twistd.log every 5 minutes:
2014-10-09 00:19:53-0700 [-] while polling for changes
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\buildbot-0.8.9-py2.7.egg\buildbot\util\misc.py", line 54, in start
d = self.method()
File "C:\Python27\lib\site-packages\buildbot-0.8.9-py2.7.egg\buildbot\changes\base.py", line 70, in doPoll
d = defer.maybeDeferred(self.poll)
File "C:\Python27\lib\site-packages\twisted\internet\defer.py", line 139, in maybeDeferred
result = f(*args, **kw)
File "C:\Python27\lib\site-packages\twisted\internet\defer.py", line 1237, in unwindGenerator
return _inlineCallbacks(None, gen, Deferred())
--- <exception caught here> ---
File "C:\Python27\lib\site-packages\twisted\internet\defer.py", line 1099, in _inlineCallbacks
result = g.send(result)
File "C:\Python27\lib\site-packages\buildbot-0.8.9-py2.7.egg\buildbot\changes\gitpoller.py", line 147, in poll
yield self._dovccmd('init', ['--bare', self.workdir])
File "C:\Python27\lib\site-packages\buildbot-0.8.9-py2.7.egg\buildbot\changes\gitpoller.py", line 292, in _dovccmd
[command] + args, path=path, env=os.environ)
File "C:\Python27\lib\site-packages\twisted\internet\utils.py", line 176, in getProcessOutputAndValue
reactor)
File "C:\Python27\lib\site-packages\twisted\internet\utils.py", line 30, in _callProtocolWithDeferred
reactor.spawnProcess(p, executable, (executable,)+tuple(args), env, path)
File "C:\Python27\lib\site-packages\twisted\internet\posixbase.py", line 358, in spawnProcess
return Process(self, processProtocol, executable, args, env, path)
File "C:\Python27\lib\site-packages\twisted\internet\_dumbwin32proc.py", line 195, in __init__
raise OSError(pwte)
exceptions.OSError: (2, 'CreateProcess', 'The system cannot find the file specified.')
Also, here's the relevant portion of my config file:
from buildbot.changes.gitpoller import GitPoller
c['change_source'] = []
c['change_source'].append(GitPoller(
repourl='https://github.com/solstice333/BuildbotTest.git',
branch='master',
pollinterval=300))
Any ideas?
I have similar issue with HgPoller. Try to specify full path to git
c['change_source'].append(GitPoller(
gitbin='full/path/to/git.exe',
repourl='https://github.com/solstice333/BuildbotTest.git',
branch='master',
pollinterval=300))
I think something wrong with twisted - this dont work with same error
PS Twisted use win32process.CreateProcess and MSDN says about it first argument: The string can specify the full path and file name of the module to execute or it can specify a partial name. In the case of a partial name, the function uses the current drive and current directory to complete the specification. The function will not use the search path.
from twisted.internet import utils
utils.getProcessOutputAndValue("hg.exe", ['init', "test_dir"])
I'm having trouble running tasks. I run ./manage celeryd -B -l info, it correctly loads all tasks to registry.
The error happens when any of the tasks run - the task starts, does its thing, and then I get:
[ERROR/MainProcess] Thread 'ResultHandler' crashed: ValueError('Octet out of range 0..2**64-1',)
Traceback (most recent call last):
File "/Users/jzelez/Sites/my_virtual_env/lib/python2.7/site-packages/celery/concurrency/processes/pool.py", line 221, in run
return self.body()
File "/Users/jzelez/Sites/my_virtual_env/lib/python2.7/site-packages/celery/concurrency/processes/pool.py", line 458, in body
on_state_change(task)
File "/Users/jzelez/Sites/my_virtual_env/lib/python2.7/site-packages/celery/concurrency/processes/pool.py", line 436, in on_state_change
state_handlers[state](*args)
File "/Users/jzelez/Sites/my_virtual_env/lib/python2.7/site-packages/celery/concurrency/processes/pool.py", line 413, in on_ack
cache[job]._ack(i, time_accepted, pid)
File "/Users/jzelez/Sites/my_virtual_env/lib/python2.7/site-packages/celery/concurrency/processes/pool.py", line 1016, in _ack
self._accept_callback(pid, time_accepted)
File "/Users/jzelez/Sites/my_virtual_env/lib/python2.7/site-packages/celery/worker/job.py", line 424, in on_accepted
self.acknowledge()
File "/Users/jzelez/Sites/my_virtual_env/lib/python2.7/site-packages/celery/worker/job.py", line 516, in acknowledge
self.on_ack()
File "/Users/jzelez/Sites/my_virtual_env/lib/python2.7/site-packages/celery/worker/consumer.py", line 405, in ack
message.ack()
File "/Users/jzelez/Sites/my_virtual_env/lib/python2.7/site-packages/kombu-2.1.0-py2.7.egg/kombu/transport/base.py", line 98, in ack
self.channel.basic_ack(self.delivery_tag)
File "/Users/jzelez/Sites/my_virtual_env/lib/python2.7/site-packages/amqplib-1.0.2-py2.7.egg/amqplib/client_0_8/channel.py", line 1740, in basic_ack
args.write_longlong(delivery_tag)
File "/Users/jzelez/Sites/my_virtual_env/lib/python2.7/site-packages/amqplib-1.0.2-py2.7.egg/amqplib/client_0_8/serialization.py", line 325, in write_longlong
raise ValueError('Octet out of range 0..2**64-1')
ValueError: Octet out of range 0..2**64-1
I also must note that this worked on my previous Lion install, and even if I create a blank virtualenv with some test code, when a task runs it gives this error.
This happens with Python 2.7.2 and 2.6.4.
Django==1.3.1
amqplib==1.0.2
celery==2.4.6
django-celery==2.4.2
It appears there is some bug with homebrew install python. I've now switched to the native Lion one (2.7.1) and it works.