I have about 10 tests in same file and each one them has following set to execute in order
import pytest
#pytest.mark.order1
.
.
.
#pytest.mark.order10
But the tests never run in the order they are assigned. They always run in the order they are arranged. Anything I am missing ?
even #pytest.mark.tryfirst didn't work. One thing I noticed is, the #pytest.mark.order never shows up in suggestions while atleast #pyetst.mark.tryfirst was there in pycharm.
It looks like you're using pytest-ordering. That package is indeed "alpha quality" -- I wrote it and I haven't spent much time keeping it updated.
Instead of decorating with #pytest.mark.order1, try decorating with #pytest.mark.run(order=1). I believe the Read the Docs documentation is out of date.
I faced the same issue, you can use this command to order it:
#pytest.mark.run(order=1)
But before, install it from this site.
Then it will work fine.
The pytest marks do not do anything special, except for marking the tests. The marks can be used only for the purpose of filtering them with -m CLI option.
This is all pytest alone can do with the marks. Well, and few little things like parameterization & skipif's.
Specifically, there is no such special mark as tryfirst. It is a parameter to the hook declaration, but this is not applicable for the tests/marks.
Some external or internal plugins can add special behavior which is dependent on the marks.
Pytest executes the tests in the order they were found (collected). In some cases, pytest can reorder (regroup) the tests for better fixture usage. At least, this is declared; not sure if actually done.
The tests are assumed to be completely independent by design. If your tests depend on each another, e.g. use the state of the system under test from the previous test-cases, you have a problem with the test design. That state should be somehow converted to the fixture(s).
If you still want to force some dependencies or order of the tests (contrary to the test design principles), you have to install a plugin for the test ordering based on marks, e.g., http://pytest-ordering.readthedocs.io/en/develop/, and mark the tests according to its supported mark names.
It looks like you're using pytest-ordering. Make sure you already have installed pytest-ordering in you env, here de docs: https://github.com/ftobia/pytest-ordering and try to use the following decoration
#pytest.mark.run(order=1)
I was also facing the same issue multiple times and tried everything available online. However, what worked for me is to install ordering using pip3 install pytest-ordering and restart pycharm.
Related
As there are huge number of test cases,it takes lot of time to complete the entire execution.I am not using selenium. Is there any way that can help to achieve parallel execution in robot framework or python .Any example would be of great help.
Pabot is likely what you're looking for. Though you should know that this will not magically make your tests thread-safe. In other words Pabot can only help you with the execution part, but your test cases will need to be designed with parallelization in mind. For example, test cases that make changes to a database or edit a global file may not be parallelization-friendly and will need to be redesigned with parallelization in mind.
PabotLib can help you design thread-safe test cases when needed.
See pabot here : https://github.com/mkorpela/pabot
First install pabot:
pip3 install -U robotframework-pabot
Then you can run rest under case directory in parallel as simple as:
pabot --testlevelsplit case
I am enjoying PyCharm's optimizing of Python imports - as well as removing unused imports, following PEP8 gives them a sensible layout and makes them easier to read. Is there any way to get PyCharm to additionally alphabetize them (which would make scanning them faster, for me at least)?
PyCharm does this automatically now by use of Code -> Optimize Imports. It also sorts them into groups per PEP 8.
See https://www.jetbrains.com/help/pycharm/optimizing-imports.html for details.
PyCharm sorts imports only according to groups specified in PEP-8, not alphabetically.
As already mentioned it seems that Code -> Optimize imports cannot do that.
There is however a plugin named Lines sorter (exists in the PyCharm repositories) with which one can mark all import lines and chose Edit -> Sort lines. Not perfect but easier than doing it all by hand.
This would be a nice addition to PyCharm indeed.
Until that time there's a command line tool that does what you want called isort. It doesn't come with a plug-in for PyCharm, but can be integrated via an External command and the Synchronize files after execution option, as described in their docs. You could even hook it to the original key binding.
Additional answer for the more vim-minded. You can install IdeaVim and use Shift + V, select lines to be sorted, and a call to :sort to do it.
... and a pony! No, seriously. I am looking for a way to organize tests that "just works". Most things do work, but not all pieces fit together. So here is what I want:
Having tests automatically discovered. This includes doctests. Note that the sum of doctests must not appear as a single test. (i.e. not what py.test --doctest-modules does)
Being able to run tests in parallel. (Something like py.test -n from xdist)
Generating a coverage report.
Make python setup.py test just work.
My current approach involves a tests directory and the load_tests protocol. All files contained are named like test_*.py. This makes python -m unittest discover just work, if I create a file test_doctests.py with the following content.
import doctest
import mymodule1, mymodule2
def load_tests(loader, tests, ignore):
tests.addTests(doctest.DocTestSuite(mymodule1))
tests.addTests(doctest.DocTestSuite(mymodule2))
return tests
This approach also has the upside that one can use setuptools and supply setup(test_suite="unittest2.collector").
However this approach has a few problems.
coverage.py expects to run a script. So I cannot use unittest2 discovery here.
py.test does not run load_tests functions, so it does not find the doctests and the --doctest-modules option is crap.
nosetests runs the load_tests functions, but does not supply any parameters. This appears totally broken on the side of nose.
How can I make things work better than this or fix some of the issues above?
This is an old question, but the problem still persists for some of us! I was just working through it and found a solution similar to kaapstorm's, but with much nicer output. I use py.test to run it, but I think it should be compatible across test runners:
import doctest
from mypackage import mymodule
def test_doctest():
results = doctest.testmod(mymodule)
if results.failed:
raise Exception(results)
What I end up with in a failure case is the printed stdout output that you would get from running doctest manually, with an additional exception that looks like this:
Exception: TestResults(failed=1, attempted=21)
As kaapstrom mentioned, it doesn't count tests properly (unless there are failures) but I find that isn't worth a whole lot provided the coverage numbers come back high :)
I use nose, and found your question when I experienced the same problem.
What I've ended up going with is not pretty, but it does run the tests.
import doctest
import mymodule1, mymodule2
def test_mymodule1():
assert doctest.testmod(mymodule1, raise_on_error=True)
def test_mymodule2():
assert doctest.testmod(mymodule2, raise_on_error=True)
Unfortunately it runs all the doctests in a module as a single test. But if things go wrong, at least I know where to start looking. A failure results in a DocTestFailure, with a useful message:
DocTestFailure: <DocTest mymodule1.myfunc from /path/to/mymodule1.py:63 (4 examples)>
I am about to get a bunch of python scripts from an untrusted source.
I'd like to be sure that no part of the code can hurt my system, meaning:
(1) the code is not allowed to import ANY MODULE
(2) the code is not allowed to read or write any data, connect to the network etc
(the purpose of each script is to loop through a list, compute some data from input given to it and return the computed value)
before I execute such code, I'd like to have a script 'examine' it and make sure that there's nothing dangerous there that could hurt my system.
I thought of using the following approach: check that the word 'import' is not used (so we are guaranteed that no modules are imported)
yet, it would still be possible for the user (if desired) to write code to read/write files etc (say, using open).
Then here comes the question:
(1) where can I get a 'global' list of python methods (like open)?
(2) Is there some code that I could add to each script that is sent to me (at the top) that would make some 'global' methods invalid for that script (for example, any use of the keyword open would lead to an exception)?
I know that there are some solutions of python sandboxing. but please try to answer this question as I feel this is the more relevant approach for my needs.
EDIT: suppose that I make sure that no import is in the file, and that no possible hurtful methods (such as open, eval, etc) are in it. can I conclude that the file is SAFE? (can you think of any other 'dangerous' ways that built-in methods can be run?)
This point hasn't been made yet, and should be:
You are not going to be able to secure arbitrary Python code.
A VM is the way to go unless you want security issues up the wazoo.
You can still obfuscate import without using eval:
s = '__imp'
s += 'ort__'
f = globals()['__builtins__'].__dict__[s]
** BOOM **
Built-in functions.
Keywords.
Note that you'll need to do things like look for both "file" and "open", as both can open files.
Also, as others have noted, this isn't 100% certain to stop someone determined to insert malacious code.
An approach that should work better than string matching us to use module ast, parse the python code, do your whitelist filtering on the tree (e.g. allow only basic operations), then compile and run the tree.
See this nice example by Andrew Dalke on manipulating ASTs.
built in functions/keywords:
eval
exec
__import__
open
file
input
execfile
print can be dangerous if you have one of those dumb shells that execute code on seeing certain output
stdin
__builtins__
globals() and locals() must be blocked otherwise they can be used to bypass your rules
There's probably tons of others that I didn't think about.
Unfortunately, crap like this is possible...
object().__reduce__()[0].__globals__["__builtins__"]["eval"]("open('/tmp/l0l0l0l0l0l0l','w').write('pwnd')")
So it turns out keywords, import restrictions, and in-scope by default symbols alone are not enough to cover, you need to verify the entire graph...
Use a Virtual Machine instead of running it on a system that you are concerned about.
Without a sandboxed environment, it is impossible to prevent a Python file from doing harm to your system aside from not running it.
It is easy to create a Cryptominer, delete/encrypt/overwrite files, run shell commands, and do general harm to your system.
If you are on Linux, you should be able to use docker to sandbox your code.
For more information, see this GitHub issue: https://github.com/raxod502/python-in-a-box/issues/2.
I did come across this on GitHub, so something like it could be used, but that has a lot of limits.
Another approach would be to create another Python file which parses the original one, removes the bad code, and runs the file. However, that would still be hit-and-miss.
I've gotten omnicompletion with Pysmell to work before, but I can't seem to do it again.
I tried following some steps online, but most, if not all, of them are to vague and assume too much that you know what you are doing to some extent.
Can someone post a full, step-by-step tutorial on how to get code completion working properly, for complete Vim newbies (for dummies?)?
There's also Ctrl+n in insert mode which will autocomplete based on the words it has seen in any of the open buffers (even in other tabs).
You may try Pydiction (Excerpt below)
Description Pydiction allows you to
Tab-complete Python code in Vim,
including: standard, custom and
third-party modules and packages. Plus
keywords, built-ins, and string
literals.
Pyflakes has a vim plugin that does this pretty awesomely. Unlike Pydiction, you don't need to build a dictionary beforehand (so if you're bouncing between different virtualenvs it's a bit less hassle.) I haven't been using it long but it seems very slick.
Try hitting Ctrl-p while typing mid-word. Ctrl-p inserts the most recent word that starts with the prefix you're typing and Ctrl-n inserts the next match. If you have several possibilities, you can hit ctrl-p more than once to substitute each candidate in order.