I have a python script in which I wrote some unit tests and I am using selenium.
I want to extract the whole output of the console (not only my prints but also the unit test related results), so that I can import them later in my test management tool.
Here is my code:
import unittest
from selenium import webdriver
import json
import requests
import sys
class TestUbuntuHomepage(unittest.TestCase):
global strs
strs = []
def setUp(self):
sys.stdout = open("C:\\Users\\Marialena\\Downloads\\out2.log", 'wt')
self.driver = webdriver.Firefox(executable_path="C:\\Users\\Marialena\\Downloads\\selenium-drivers\\geckodriver")
def testTitle(self):
self.driver.get('http://www.ubuntu.com/')
if self.assertIn('Ubuntu', self.driver.title):
strs.append('test'})
def tearDown(self):
self.driver.quit()
if __name__ == '__main__':
unittest.main(verbosity=2)
Using sys.stdout = open("C:\\Users\\Marialena\\Downloads\\out2.log", 'wt') I get in the file everything I have printed and I also get this exception:
Traceback (most recent call last): File "C:\Program
Files\JetBrains\PyCharm\PyCharm Community Edition
2017.3.3\helpers\pycharm_jb_unittest_runner.py", line 35, in
main(argv=args, module=None, testRunner=unittestpy.TeamcityTestRunner, buffer=not
JB_DISABLE_BUFFERING) File
"C:\Users\Marialena\AppData\Local\Programs\Python\Python36-32\Lib\unittest\main.py",
line 95, in init
self.runTests() File "C:\Users\Marialena\AppData\Local\Programs\Python\Python36-32\Lib\unittest\main.py",
line 256, in runTests
self.result = testRunner.run(self.test) File "C:\Program Files\JetBrains\PyCharm\PyCharm Community Edition
2017.3.3\helpers\pycharm\teamcity\unittestpy.py", line 304, in run
return super(TeamcityTestRunner, self).run(test) File "C:\Users\Marialena\AppData\Local\Programs\Python\Python36-32\Lib\unittest\runner.py",
line 176, in run
test(result) File "C:\Users\Marialena\AppData\Local\Programs\Python\Python36-32\Lib\unittest\suite.py",
line 84, in call
return self.run(*args, **kwds) File "C:\Users\Marialena\AppData\Local\Programs\Python\Python36-32\Lib\unittest\suite.py",
line 122, in run
test(result) File "C:\Users\Marialena\AppData\Local\Programs\Python\Python36-32\Lib\unittest\suite.py",
line 84, in call
return self.run(*args, **kwds) File "C:\Users\Marialena\AppData\Local\Programs\Python\Python36-32\Lib\unittest\suite.py",
line 122, in run
test(result) File "C:\Users\Marialena\AppData\Local\Programs\Python\Python36-32\Lib\unittest\suite.py",
line 84, in call
return self.run(*args, **kwds) File "C:\Users\Marialena\AppData\Local\Programs\Python\Python36-32\Lib\unittest\suite.py",
line 122, in run
test(result) File "C:\Users\Marialena\AppData\Local\Programs\Python\Python36-32\Lib\unittest\case.py",
line 653, in call
return self.run(*args, **kwds) File "C:\Users\Marialena\AppData\Local\Programs\Python\Python36-32\Lib\unittest\case.py",
line 624, in run
result.stopTest(self) File "C:\Program Files\JetBrains\PyCharm\PyCharm Community Edition
2017.3.3\helpers\pycharm\teamcity\unittestpy.py", line 260, in stopTest
output = sys.stdout.getvalue() AttributeError: '_io.TextIOWrapper' object has no attribute 'getvalue'
Any help with this, please? Thank you.
It looks like PyCharm is replacing sys.stdout with its own stream, so when you replace it with a file stream, PyCharm fails to use it.
So, limit your interventions to the scope of one function, to avoid interference with PyCharm.
This is the general idea:
def testTitle(self):
original_stdout = sys.stdout
sys.stdout = open("C:\\Users\\Marialena\\Downloads\\out2.log", 'wt')
# your test code goes here
sys.stdout = original_stdout
Now, from the outside, it will look like sys.stdout was never modified.
Of course, you'll want to improve on some things:
handle exceptions in test - restore stdout in a finally block
avoid deleting the file in each test - open the file in append mode
avoid having to copy-and-paste this code in each test - make a context manager
#contextmanager
def redirected_stdout(filename):
original_stdout = sys.stdout
sys.stdout = open(filename, 'at')
try:
yield
finally:
sys.stdout = original_stdout
and then:
def testTitle(self):
with redirected_stdout("C:\\Users\\Marialena\\Downloads\\out2.log"):
# your test code goes here
Alternatively:
Investigate how PyCharm expects sys.stdout to behave and make your own class which does both: writes to file and provides the API which PyCharm expects.
Related
I am dealing with an existing test suite, where we want to implement a timeout functionality, which will cause a hanging test to time out and then move on with its regular teardown/cleanup.
I am toying with the idea of running each test in a process, which I can terminate after e.g. a timeout of 3 seconds. Ideally, I don't want to modify the test cases and instead just add a decorator indicating the test is affected by this timeout behavior.
This is what I have, a minimal example:
import multiprocessing
import sys
from time import sleep
def timeout(func):
def wrapper():
proc = multiprocessing.Process(target=func)
proc.start()
sleep(3)
proc.terminate()
return wrapper
#timeout
def my_test():
while True:
sleep(1)
if __name__ == "__main__":
my_test()
But for some reason, it seems pickle cannot deal with this and the decorator somehow messes up the reference to the function, as this error is hit:
$ python multiproc.py
2019-11-07 07:34:37.098 | DEBUG | __main__:wrapper:13 - In wrapper
Traceback (most recent call last):
File "multiproc.py", line 30, in <module>
my_test()
File "multiproc.py", line 15, in wrapper
proc.start()
File "C:\Python36\lib\multiprocessing\process.py", line 105, in start
self._popen = self._Popen(self)
File "C:\Python36\lib\multiprocessing\context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Python36\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
File "C:\Python36\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
reduction.dump(process_obj, to_child)
File "C:\Python36\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function my_test at 0x000001C0A7E87400>: it's not the same object as __main__.my_test
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Python36\lib\multiprocessing\spawn.py", line 99, in spawn_main
new_handle = reduction.steal_handle(parent_pid, pipe_handle)
File "C:\Python36\lib\multiprocessing\reduction.py", line 82, in steal_handle
_winapi.PROCESS_DUP_HANDLE, False, source_pid)
OSError: [WinError 87] The parameter is incorrect
Does anyone have an idea if this can be solved without modifying the existing test case?
I am calling an R script file from python using pyrserve. I have rserve running. At arbitrary points in the R script, pyrserve gives an error and quits:
Traceback (most recent call last):
File "scriptV2.py", line 272, in <module>
rConn.eval("source(file.PropensityFlow)")
File "/Users/dipayanmaiti/Py3.3venv/lib/python3.3/site-packages/pyRserve/rconn.py", line 47, in decoCheckIfClosed
return func(self, *args, **kw)
File "/Users/dipayanmaiti/Py3.3venv/lib/python3.3/site-packages/pyRserve/rconn.py", line 119, in eval
return rparse(src, atomicArray=atomicArray)
File "/Users/dipayanmaiti/Py3.3venv/lib/python3.3/site-packages/pyRserve/rparser.py", line 539, in rparse
return rparser.parse()
File "/Users/dipayanmaiti/Py3.3venv/lib/python3.3/site-packages/pyRserve/rparser.py", line 349, in parse
self.lexer.readHeader()
File "/Users/dipayanmaiti/Py3.3venv/lib/python3.3/site-packages/pyRserve/rparser.py", line 94, in readHeader
self.responseCode = struct.unpack(b'<i', self.read(3) + b'\x00')[0]
File "/Users/dipayanmaiti/Py3.3venv/lib/python3.3/site-packages/pyRserve/rparser.py", line 149, in read
raise EndOfDataError()
pyRserve.rparser.EndOfDataError
I have set rserv.conf to the following:
maxinbuf 20000000
maxsendbuf 0
Does anybody know why this happens? This looks like some buffer problem, because the R script runs by itself.
It is a late answer, but in such situations it is useful to run Rserve in debug mode so its output can be monitored in a separate shell.
R CMD Rserve.dbg
In some rare cases I've seen Rserve printing warnings to the console, and when this happened the command sent thru pyRserve didn't return any value from Rserve - which led to the 'EndOfDataError' above.
I am trying to run Pyramid with Jinja2 using new Python 2.7 runtime in threadsafe mode and GAE 1.6.0 pre-release SDK. I've made modifications to my app as outlined here, i.e. I've set runtime: python27, threadsafe: true in app.yaml and got rid of main() function. When I generate response by myself it works fine, but when I try to bring jinja2 into the equation, I get the following exception:
ERROR 2011-11-07 00:10:34,356 wsgi.py:170]
Traceback (most recent call last):
File "/gae/google/appengine/runtime/wsgi.py", line 168, in Handle
[...]
File "/myapp/source/myapp-tip/main.py", line 29, in <module>
config.include('pyramid_jinja2')
File "/myapp/source/myapp-tip/lib/dist/pyramid/config/__init__.py", line 616, in include
c(configurator)
File "lib/dist/pyramid_jinja2/__init__.py", line 390, in includeme
_get_or_build_default_environment(config.registry)
File "/lib/dist/pyramid_jinja2/__init__.py", line 217, in _get_or_build_default_environment
_setup_environment(registry)
File "/lib/dist/pyramid_jinja2/__init__.py", line 253, in _setup_environment
package = _caller_package(('pyramid_jinja2', 'jinja2', 'pyramid.config'))
File "/lib/dist/pyramid_jinja2/__init__.py", line 136, in caller_package
for t in self.inspect.stack():
File "/usr/lib/python2.7/inspect.py", line 1056, in stack
return getouterframes(sys._getframe(1), context)
File "/usr/lib/python2.7/inspect.py", line 1034, in getouterframes
framelist.append((frame,) + getframeinfo(frame, context))
File "/usr/lib/python2.7/inspect.py", line 1009, in getframeinfo
lines, lnum = findsource(frame)
File "/usr/lib/python2.7/inspect.py", line 534, in findsource
module = getmodule(object, file)
File "/usr/lib/python2.7/inspect.py", line 506, in getmodule
main = sys.modules['__main__']
KeyError: '__main__'
I tried to mess around a bit with pyramid_jinja2 code to work around this issue, only to be left with another exception:
ERROR 2011-11-04 12:06:38,720 wsgi.py:170]
Traceback (most recent call last):
File "/gae/google/appengine/runtime/wsgi.py", line 168, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
[...]
File "/myapp/source/myapp-tip/main.py", line 29, in <module>
config.add_jinja2_search_path("templates")
File "/myapp/source/myapp-tip/lib/dist/pyramid/config/util.py", line 28, in wrapper
result = wrapped(self, *arg, **kw)
File "/lib/dist/pyramid_jinja2/__init__.py", line 311, in add_jinja2_search_path
env.loader.searchpath.append(abspath_from_resource_spec(d))
File "/myapp/source/myapp-tip/lib/dist/pyramid/asset.py", line 38, in abspath_from_asset_spec
return pkg_resources.resource_filename(pname, filename)
File "/myapp/source/myapp-tip/pkg_resources.py", line 840, in resource_filename
return get_provider(package_or_requirement).get_resource_filename(
File "/myapp/source/myapp-tip/pkg_resources.py", line 160, in get_provider
__import__(moduleOrReq)
File "/gae/google/appengine/tools/dev_appserver_import_hook.py", line 640, in Decorate
return func(self, *args, **kwargs)
File "/gae/google/appengine/tools/dev_appserver_import_hook.py", line 1756, in load_module
return self.FindAndLoadModule(submodule, fullname, search_path)
File "/gae/google/appengine/tools/dev_appserver_import_hook.py", line 640, in Decorate
return func(self, *args, **kwargs)
File "/gae/google/appengine/tools/dev_appserver_import_hook.py", line 1628, in FindAndLoadModule
description)
File "/gae/google/appengine/tools/dev_appserver_import_hook.py", line 640, in Decorate
return func(self, *args, **kwargs)
File "/gae/google/appengine/tools/dev_appserver_import_hook.py", line 1571, in LoadModuleRestricted
description)
ImportError: Cannot re-init internal module __main__
I'd be happy if anybody could shed some light on what pyramid is trying to do under the hood. Judging by the latter stack trace it seems it's trying to resolve an asset, but why is it trying to reload __main__? I'm not even sure my problem is caused by pyramid or GAE.
Thanks for any insight on this issue.
I'm not familiar with pyramid, but the problem really does seem to be with this line:
config.include('pyramid_jinja2')
Whatever that config thing is, it seems to be doing some dynamic import magic.
Don't do that.
The app engine environment doesn't handle imports the way you would in normal python. Step through that line with a debugger and you'll wind up in the replacement version of the import system, which you'll soon see, only implements a small part of what real python does.
If possible, just use a normal import statement... Otherwise, you're going to have to dig into config.include and get it to play nice with the restricted importing features on GAE.
I managed to make it work using Pyramid 1.3's AssetResolver. First attempt is here. Just not sure what the lifetime/scope of the resolver should be in this case, I will figure it out later.
In pyramid_jinja2/__init__.py add the following code before _get_or_build_default_environment()
class VirtualModule(object):
def __init__(self,name):
import sys
sys.modules[name]=self
def __getattr__(self,name):
return globals()[name]
VirtualModule("__main__")
def _get_or_build_default_environment(registry):
(http://www.inductiveautomation.com/forum/viewtopic.php?f=70&p=36917)
I am trying to get started with using nosegae, however I run into the issue that I can't seem to get it to pass even the simplest of cases when using django.
when running without the --without-sandbox flag both the following tests fail
def test_import_django ():
import django
def test_import_django_http ():
import django.http
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\nose-1.1.2-py2.7.egg\nose\case.py", line 1
97, in runTest
self.test(*self.arg)
File "C:\Users\User\Desktop\TDD_GAE\myproj\tests.py", line 2, in test_import_d
jango
import django
File "C:\Python27\lib\site-packages\nosegae-0.1.9-py2.7.egg\nosegae.py", line
207, in find_module
return super(HookMixin, self).find_module(fullname, path)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\de
v_appserver.py", line 1505, in Decorate
return func(self, *args, **kwargs)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\de
v_appserver.py", line 1998, in find_module
search_path)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\de
v_appserver.py", line 1505, in Decorate
return func(self, *args, **kwargs)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\de
v_appserver.py", line 2119, in FindModuleRestricted
result = self.FindPathHook(submodule, submodule_fullname, path_entry)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\de
v_appserver.py", line 2219, in FindPathHook
return self._imp.find_module(submodule, [path_entry])
Howevere if I do use --without-sandbox at least the first test passes
myproj.tests.test_import_django ... ok
myproj.tests.test_import_django_http ... ERROR
======================================================================
ERROR: myproj.tests.test_import_django_http
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\nose-1.1.2-py2.7.egg\nose\case.py", line 1
97, in runTest
self.test(*self.arg)
File "C:\Users\User\Desktop\TDD_GAE\myproj\tests.py", line 5, in test_import_d
jango_http
import django.http
File "C:\Program Files (x86)\Google\google_appengine\lib\django_1_2\django\htt
p\__init__.py", line 9, in <module>
from mod_python.util import parse_qsl
File "C:\Python27\lib\site-packages\nosegae-0.1.9-py2.7.egg\nosegae.py", line
199, in find_module
mod_path = self.find_mod_path(fullname)
File "C:\Python27\lib\site-packages\nosegae-0.1.9-py2.7.egg\nosegae.py", line
251, in find_mod_path
_sf, path, _desc= self._imp.find_module(top, None)
AttributeError: 'str' object has no attribute 'find_module'
Has anyone encountered and know how I can go about past this?
Edit
It seems that the issue is recursive imports
def test_import_pdb ():
import pdb
pdb.set_trace ()
part of the stack trace is
File "C:\Python27\lib\pdb.py", line 72, in __init__
import readline
notice that an import in __init__ of django.http is also part of the stack trace
Read https://docs.djangoproject.com/en/dev/topics/testing/ about Django testing.
As I know it's better to use unittest or doctest shipped with django as it have several improvements for django-specific testing like form field output testing and some database features. Hovewer it's not essential and if you want to continue using nose - think you missed django environment setup:
from django.test.utils import setup_test_environment
setup_test_environment()
This lines needed to run your tests outside of ./manage.py --test
UPD
Yeah my previous thought's were wrong. So I just digged into sources of nose and nose-gae, and what I think - check HardenedModulesHook definition in your nose version, cause in trunk of nose I've found following:
class HardenedModulesHook(object):
...
def __init__(self,
module_dict,
imp_module=imp,
os_module=os,
dummy_thread_module=dummy_thread,
pickle_module=pickle):
...
That gives following - when noseGAE plugin begin() method is executed -> there self._install_hook(dev_appserver.HardenedModulesHook) is called which declares mixed-hook class and creates it's instance like self.hook = Hook(sys.modules, self._path). <- There is HardenedModulesHook.__init__ called with second argument as mystic '_path' however in NOSE this argument should be 'imp' module by default -> That makes an exception you've got:
_sf, path, _desc= self._imp.find_module(top, None)
AttributeError: 'str' object has no attribute 'find_module'
So I think it might be a problem with nose-gae :(
I write a program :
from PyQt4.QtCore import *
from PyQt4.QtGui import *
def main():
app = QApplication([])
button = QPushButton("hello?")
button.show()
app.exec_()
if __name__=="__main__":
main()
the file name is t.py,
when I run:
pylint t.py
in ubuntu9.10, pyqt4,
I got this:
pylint t.py
No config file found, using default configuration
error while building astng for /home/halida/data/workspace/test/t.py
Traceback (most recent call last):
File "/usr/lib/pymodules/python2.6/logilab/astng/manager.py", line 126, in astng_from_file
astng = ASTNGBuilder(self).file_build(filepath, modname)
File "/usr/lib/pymodules/python2.6/logilab/astng/builder.py", line 118, in file_build
node = self.string_build(data, modname, path)
File "/usr/lib/pymodules/python2.6/logilab/astng/builder.py", line 128, in string_build
return self.ast_build(parse(data + '\n'), modname, path)
File "/usr/lib/pymodules/python2.6/logilab/astng/builder.py", line 147, in ast_build
self.rebuilder.walk(node)
File "/usr/lib/pymodules/python2.6/logilab/astng/rebuilder.py", line 89, in walk
self._walk(node)
File "/usr/lib/pymodules/python2.6/logilab/astng/rebuilder.py", line 109, in _walk
self._walk(child, node)
File "/usr/lib/pymodules/python2.6/logilab/astng/rebuilder.py", line 103, in _walk
handle_leave = node.accept(self)
File "/usr/lib/pymodules/python2.6/logilab/astng/nodes.py", line 159, in accept
return func(self)
File "/usr/lib/pymodules/python2.6/logilab/astng/rebuilder.py", line 188, in visit_from
imported = node.root().import_module(node.modname)
File "/usr/lib/pymodules/python2.6/logilab/astng/scoped_nodes.py", line 282, in import_module
return MANAGER.astng_from_module_name(self.relative_name(modname, level))
File "/usr/lib/pymodules/python2.6/logilab/astng/manager.py", line 172, in astng_from_module_name
return self.astng_from_module(module, modname)
File "/usr/lib/pymodules/python2.6/logilab/astng/manager.py", line 207, in astng_from_module
astng = ASTNGBuilder(self).module_build(module, modname)
File "/usr/lib/pymodules/python2.6/logilab/astng/builder.py", line 80, in module_build
node = self.inspect_build(module, modname=modname, path=path)
File "/usr/lib/pymodules/python2.6/logilab/astng/builder.py", line 95, in inspect_build
self.object_build(node, module)
File "/usr/lib/pymodules/python2.6/logilab/astng/builder.py", line 195, in object_build
self.object_build(class_node, member)
File "/usr/lib/pymodules/python2.6/logilab/astng/builder.py", line 198, in object_build
object_build_methoddescriptor(node, member)
File "/usr/lib/pymodules/python2.6/logilab/astng/raw_building.py", line 150, in object_build_methoddescriptor
func = build_function(member.__name__, doc=member.__doc__)
AttributeError: 'PyQt4.QtCore.pyqtSignal' object has no attribute '__name__'
************* Module t
F: 1: <class 'logilab.astng._exceptions.ASTNGBuildingException'>: Unable to load module t ('PyQt4.QtCore.pyqtSignal' object has no attribute '__name__')
Compilation exited abnormally with code 1 at Sat Dec 26 10:43:54
in windows XP, with pythonxy,
I only got a error message, why?
Looks like a bug in astng. They try to read the name of a function which does not publish it (native extension func). I'd report a bug to both astng and pyqt projects. The first one would be that they should handle a no-name situation better. The second one would be that every sane extension should publish at least the function names.
I would check if you're using the very latest astng, pylint, etc.