I have a SCons project that builds set of Python modules (mostly as shared objects compiled from C++).
This part works flawlessly, all dependencies seem to be fine.
However, now I reached point where I started trying to add tests for those modules. Those tests are supposed to be run as part of build. Tests are written in Python and when run they need to be run under environment that already has all modules built (so that import statements can find them).
I got that part working up to the point of dependencies for those tests. Tests are run, however I can't seem to find a way to generate dependencies for them from import statements.
I have found modulefinder which does exactly what I want. What's more I am able to run it in built environment and get expected results after my project is built.
I wanted to use modulefinder in emitter to scan for files the test/Python script depends on.
Problem is that dependency scanning/building + running of emitters happens before SCons has built all modules and before it can set up environment properly for tests, hence also modulefinder can't work.
I can't seem to work out how to make SCons look for dependencies for specific targets after some other targets are already built.
edit:
I found ParseDepends in SCons documentation which seems to talk about the same type of issue (well, almost exactly same with exception to langauge).
This limitation of ParseDepends leads to unnecessary recompilations. Therefore, ParseDepends should only be used if scanners are not available for the employed language or not powerful enough for the specific task.
I am still hopeful though that there is some clean solution to my problem.
You can't change dependencies in SCons during the compilation phase. SCons creates its dependency tree and after it runs it. You can't change it.
I suggest you to write a Scanner for your builder. In C++, SCons uses a Scanner to find include dependencies.
[http://www.scons.org/doc/1.1.0/HTML/scons-user/c3742.html][1]
After a lot of playing I found, not-so-clean-but-not-too-horrible-way-that-seems-to-work which I've wrapped in a helper Scanner-derived class:
from SCons.Node import Node
from SCons import Scanner
import logging
_LOGGER = logging.getLogger(__name__)
class DeferredScanner(Scanner.Current):
"""
This is a helper class for implementing source scanners that need
to wait for specific things to be built before source scanning can happen.
One practical example of usage is when you are you generating Python
modules (i.e. shared libraries) which you want to test.
You have to wait for all your modules are ready before dependencies
of your tests can be scanned.
To do this approach with this scanner is to collect all generated modules
and `wait_for` them before scanning dependncies of whatever this scanner
is used for.
Sample usage:
py_scanner = DeferredScanner(
wait_for = all_generated_modules,
function = _python_source_scanner,
recursive = True,
skeys = ['.py'],
path_function = FindENVPathDirs('PYTHONPATH'),
)
"""
def __init__(self, wait_for, **kw):
Scanner.Current.__init__(
self,
node_factory = Node,
**kw
)
self.wait_for = wait_for
self.sources_to_rescan = []
self.ready = False
env = wait_for[0].get_build_env()
env.AlwaysBuild(wait_for)
self._hook_implicit_reset(
wait_for[0],
'built',
self.sources_to_rescan,
lambda: setattr(self, 'ready', True),
)
def _hook_implicit_reset(self, node, name, on, extra = None):
# We can only modify dependencies in main thread
# of Taskmaster processing.
# However there seems to be no hook to "sign up" for
# callback when post processing of built node is hapenning
# (at least I couldn't find anything like this in SCons
# 2.2.0).
# AddPostAction executes actions in Executor thread, not
# main one, se we can't used that.
#
# `built` is guaranteed to be executed in that thread,
# so we hijack this one.
#
node.built = lambda old=node.built: (
old(),
self._reset_stored_dependencies(name, on),
extra and extra(),
)[0]
def _reset_stored_dependencies(self, name, on):
_LOGGER.debug('Resetting dependencies for {0}-{1} files: {2}'.format(
# oh, but it does have those
self.skeys, # pylint: disable=no-member
name,
len(on),
))
for s in on:
# I can't find any official way to invalidate
# FS.File (or Node in general) list of implicit dependencies
# that were found.
# Yet we want to force that Node to scan its implicit
# dependencies again.
# This seems to do the trick.
s._memo.pop('get_found_includes', None) # pylint: disable=protected-access
def __call__(self, node, env, path = ()):
ready = self.ready
_LOGGER.debug('Attempt to scan {0} {1}'.format(str(node), ready))
deps = self.wait_for + [node]
if ready:
deps.extend(Scanner.Current.__call__(self, node, env, path))
self.sources_to_rescan.append(node)
# In case `wait_for` is not dependent on this node
# we have to make sure we will rescan dependencies when
# this node is built itself.
# It boggles my mind that SCons scanns nodes before
# they exist, and caches result even if there was no
# list returned.
self._hook_implicit_reset(
node,
'self',
[node],
)
return deps
This seems to work exactly as I'd have hoped and do the job.
It is probably as efficient as you can get.
Probably should also note that this works with SCons 2.2.0, but I suspect it shouldn't be much different for newer ones.
Related
Probably related to globals and locals in python exec(), Python 2 How to debug code injected by the exec block and How to get local variables updated, when using the `exec` call?
I am trying to develop a test framework for our desktop applications which uses click bot like functions.
My goal was to enable non-programmers to write small scripts which could work as a test script. So my idea is to structure the test scripts by files like:
tests-folder
| -> 01-first-test.py
| -> 02-second-test.py
| ... etc
| -> fixture.py
And then just execute these scripts in alphabetical order. However, I also wanted to have fixtures which would define functions, classes, variables and make them available to the different scripts without having the scripts to import that fixture explicitly. If that works, I could also have that approach for 2 or more directory levels.
I could get it working-ish with some hacking around, but I am not entirely convinced. I have a test_sequence.py which looks like this:
from pathlib import Path
from copy import deepcopy
from my_module.test import Failure
def run_test_sequence(test_dir: str):
error_occured = False
fixture = {
'some_pre_defined_variable': 'this is available in all scripts and fixtures',
'directory_name': test_dir,
}
# Check if fixture.py exists and load that first
fixture_file = Path(dir) / 'fixture.py'
if fixture_file.exists():
with open(fixture_file.absolute(), 'r') as code:
exec(code.read(), fixture, fixture)
# Go over all files in test sequence directory and execute them
for test_file in sorted(Path(test_dir).iterdir()):
if test_file.name == 'fixture.py':
continue
# Make a deepcopy, so scripts cannot influence one another
fixture_copy = deepcopy(fixture)
fixture_copy.update({
'some_other_variable': 'this is available in all scripts but not in fixture'
})
try:
with open(test_file.absolute(), 'r') as code:
exec(code.read(), fixture_locals, fixture_locals)
except Failure:
error_occured = True
return error_occured
This iterates over all files in the directory tests-folder and executes them in order (with fixture.py first). It also makes the local variables, functions and classes from fixture.py available to each test-script.
A test script could then just be arbitrary code that will be executed and if it raises my custom Failure exception, this will be noted as a failed test.
The whole sequence is started with a script that does
from my_module.test_sequence import run_test_sequence
if __name__ == '__main__':
exit(run_test_sequence('tests-folder')
This mostly works.
What it cannot do, and what leaves me unsatisfied with this approach:
I cannot debug the scripts itself. Since the code is loaded as string and then interpreted, breakpoints inside the test scripts are not recognized.
Calling fixture functions behaves weird. When I define a function in fixture.py like:
from my_hello_module import print_hello
def printer():
print_hello()
I will receive a message during execution that print_hello is undefined because the variables/modules/etc. in the scope surrounding printer are lost.
Stacktraces are useless. On failure it shows the stacktrace but of course only shows my line which says `exec(...)' and the insides of that function, but none of the code that has been loaded.
I am sure there are other drawbacks, that I have not found yet, but these are the most annoying ones.
I also tried to find a solution through __import__ but I couldn't get it to inject my custom locals or globals into the imported script.
Is there a solution, that I am too inexperienced to find or another builtin Python function that actually does, what I am trying to do? Or is there no way to achieve this and I should rather have each test-script import the fixture and file/directory names from the test-scripts itself. I want those scripts to have as few dependencies and pythony code as possible. Ideally they are just:
from my_module.test import *
click(x, y, LEFT)
write('admin')
press('tab')
write('password')
press('enter')
if text_on_screen('Login successful'):
succeed('Test successful')
else:
fail('Could not login')
Additional note: I think I had the debugger working when I still used execfile, but it is not available in python3 environments.
This question already has answers here:
How to deal with Pylint's "too-many-instance-attributes" message?
(4 answers)
Closed 1 year ago.
I am writing an application in Python using Flask, now during creation of a Resource class for an endpoint I get a Pylint 'too-many-instance-attributes' warning. Now I do not know anymore if what I am doing is even a 'correct' way of writing Resources.
I inject dependencies into the resource like this:
api.add_resource(TicketsQuery, '/tickets/query',
'/ticket/query/<int:ticketID>',
resource_class_kwargs={'cleaner': Cleaner(StrategyResponseTickets()),
'machine_cleaner': Cleaner(StrategyResponseMachines()),
'db': soap_caller,
'cache': cache,
'field_map': app.config['FIELD_FILTER_MAP']['TICKETS'],
'endpoint_permission': TicketsQueryPermission
})
Which then shows up in the Resource as a kwargs argument. I also decorate the functions inside the init since I need a variable from the class (to do the decoration itself).
class TicketsQuery(Resource):
def __init__(self, **kwargs):
# Dependencies
self.cleaner = kwargs['cleaner']
self.machine_cleaner = kwargs['machine_cleaner']
self.db = kwargs['db']
self.cache = kwargs['cache']
self.field_map = kwargs['field_map']
self.endpoint_permission = kwargs['endpoint_permission']
# Permissions of endpoint method calls, implemented using wrapper
self.get = authorization_required(self.endpoint_permission, UserType.GENERIC_EMPLOYEE)(self.get)
self.post = authorization_required(self.endpoint_permission, UserType.GENERIC_EMPLOYEE)(self.post)
def get(self, permission_set: TicketsPermissionSet, ticketID=-1):
Is this a correct way of writing Resources in Flask? Or is there a better structure to adhere to? Any insight or tips are appreciated!
Don't be discouraged by a linter warning ~ it's there to remind you to stick to a specific style. If you don't yet have a .pylintrc in the project's root folder then the linter will be using the default rules; this is simply a style warning against "too many variables in instance" ~ i.e. too many self.*'s - it is 100% a pylint thing, and nothing to do with Flask.
You can suppress the warning on this but keep it active for any others by adding a comment # pylint: disable=too-many-instance-attributes above the class's def __init__(...):.
If you'd like a more pylintesque way of solving it, run pylint --generate-rcfile (assuming you've pip install pylint'd and these messages aren't just coming from the IDE) in your project's root folder to generate a .pylintrc.
From there you can add the error's code R0902 to the disable= list ~ i.e. disable=R0902, or disable=A*,B*,R0902 ~ if you're disabling multiple warnings (which accept globbing, so you could just disable ALL R* messages, but it's best to only do that if you're sure you know what warnings you're turning off). OR, you can find the line max-attributes= under [DESIGN] and set/up that number to a higher value you'd consider more reasonable.
For reference, the resources to find this information if you'd like to explore further ~ here's a list of the warnings you can get from pylint which you can search either by the name of the error too-many-instance-attributes or by its code R0902, and here's a sample pylintrc, which you can mostly browse by the code i.e. R0902 to find the property in it that affects that warning.
Lastly, here's another SO article on the same pylint error message, if you want more examples ~ How to deal with Pylint's "too-many-instance-attributes" message?.
I have a program that interacts with and changes block devices (/dev/sda and such) on linux. I'm using various external commands (mostly commands from the fdisk and GNU fdisk packages) to control the devices. I have made a class that serves as the interface for most of the basic actions with block devices (for information like: What size is it? Where is it mounted? etc.)
Here is one such method querying the size of a partition:
def get_drive_size(device):
"""Returns the maximum size of the drive, in sectors.
:device the device identifier (/dev/sda and such)"""
query_proc = subprocess.Popen(["blockdev", "--getsz", device], stdout=subprocess.PIPE)
#blockdev returns the number of 512B blocks in a drive
output, error = query_proc.communicate()
exit_code = query_proc.returncode
if exit_code != 0:
raise Exception("Non-zero exit code", str(error, "utf-8")) #I have custom exceptions, this is slight pseudo-code
return int(output) #should always be valid
So this method accepts a block device path, and returns an integer. The tests will run as root, since this entire program will end up having to run as root anyway.
Should I try and test code such as these methods? If so, how? I could try and create and mount image files for each test, but this seems like a lot of overhead, and is probably error-prone itself. It expects block devices, so I cannot operate directly on image files in the file system.
I could try mocking, as some answers suggest, but this feels inadequate. It seems like I start to test the implementation of the method, if I mock the Popen object, rather than the output. Is this a correct assessment of proper unit-testing methodology in this case?
I am using python3 for this project, and I have not yet chosen a unit-testing framework. In the absence of other reasons, I will probably just use the default unittest framework included in Python.
You should look into the mock module (I think it's part of the unittest module now in Python 3).
It enables you to run tests without the need to depened in any external resources while giving you control over how the mocks interact with your code.
I would start from the docs in Voidspace
Here's an example:
import unittest2 as unittest
import mock
class GetDriveSizeTestSuite(unittest.TestCase):
#mock.patch('path/to/original/file.subprocess.Popen')
def test_a_scenario_with_mock_subprocess(self, mock_popen):
mock_popen.return_value.communicate.return_value = ('Expected_value', '')
mock_popen.return_value.returncode = '0'
self.assertEqual('expected_value', get_drive_size('some device'))
I need to develop a plugin for GIMP and would like to stay with PyCharm for Python editing, etc.
FYI, I'm on Windows.
After directing PyCharm to use the Python interpreter included with GIMP:
I also added a path to gimpfu.py to get rid of the error on from gimpfu import *:
This fixes the error on the import, even when set to Excluded.
I experimented with setting this directory to Sources, Resources and Excluded and still get errors for constants such as RGBA-IMAGE, TRANSPARENT_FILL, NORMAL_MODE, etc.
Any idea on how to contort PyCharm into playing nice for GIMP plugin development?
Not really running any code from PyCharm, it's really just being used as a nice code editor, facilitate revisions control, etc.
As you find this variables are part of .pyd files (dll files for Python). PyCharm can't get signatures for content of this files.
For Python builtins (like abs, all, any, etc.) PyCharm has it's own .py files that uses only for signatures and docs. You can see it if you'll click on some of this funcs and go to it's declaration:
PyCharm will open builtins.py file in it's folder with following content:
def abs(*args, **kwargs): # real signature unknown
""" Return the absolute value of the argument. """
pass
def all(*args, **kwargs): # real signature unknown
"""
Return True if bool(x) is True for all values x in the iterable.
If the iterable is empty, return True.
"""
pass
def any(*args, **kwargs): # real signature unknown
"""
Return True if bool(x) is True for any x in the iterable.
If the iterable is empty, return False.
"""
pass
As you see functions are defined and documented, but have no implementation, because their implementation created with C and placed somewhere in binary file.
Pycharm can't provide such wrapper for every library. Usually people who created .pyd files provide their .py wrappers (for example, PyQt module: no native python implementation, just signatures).
Looks like Gimp doesn't have such wrapper for some of variables. Only way I see is to create some sort of own wrapper manually. For example, create gimpfu_signatures.py with following content:
RGBA_IMAGE = 1
TRANSPARENT_FILL = 2
NORMAL_MODE = 3
And import it while you're creating plugin:
from gimpfu import *
from gimpfu_signatures import * # comment on release
Not elegant, but better then nothing.
...
One more note about gimpfu.py's path. If I understand correctly, you just added this path to project. It may work, but correct way is to add it to project's PYTHONPATH (in project preferences). See this link for detailed manual.
I write a module that wraps functionality of an external binary.
For example, I wrap ls program into a python module my_wrapper.py
import my_wrapper
print my_wrapper.ls('some_directory/')
# list files in some_directory
and in my_wrapper.py I do:
# my_wrapper.py
PATH_TO_LS = '/bin/ls'
def ls(path):
proc = subprocess.Popen([PATH_TO_LS, path], ...)
...
return paths
(of course, I do not wrap ls but some other binary)
The binary might be installed with an arbitrary location, like /usr/bin/, /opt/ or even at the same place as the python script (./binaries/)
Question:
What would be the cleanest (from the user perspective) way to set the path to the binary?
Should the user specify my_wrapper.PATH_TO_LS = ... or invoke some my_wrapper.set_binary_path(path) at the beginning of his script?
Maybe it would be better to specify it in env, and the wrapper would find it with os.environ?
If the wrapper is distributed as egg, can I require during the installation, that the executable is already present in the system (see below)?
egg example:
# setup.py
setup(
name='my_wrapper',
requires_binaries=['the_binary'] # <--- require that the binary is already
# installed and on visible
# on execution path
)
or
easy_install my_wrapper BINARY_PATH=/usr/local/bin/the_binary
Create a "configuration object" with sane defaults. Allow the consumer to modify the values as appropriate. Accept a configuration object instance to your functions, taking the one you created by default.