Develop Reference

How can I use fixtures in helper functions?

I asked the same question in GitHub.
I learned about pytest-helpers-namespace from s0undt3ch in his very helpful answer. However I found a usecase I cant seem to find an obvious workaround. Here is the paste of my original question on GitHub.
How can I use the fixtures already declared in my conftest within my helper functions?
I am have a large, memory heavy configuration object (for simplicity, a dictionary) in all test, but I dont want to tear it down and rebuild this object, thus scoped as session and reused. Often times, I want to grab values from the configuration object within my test.
I know reusing fixtures within fixtures, you have to pass a reference
# fixtures
#pytest.fixture(scope="session")
def return_dictionary():
return {
"test_key": "test_value"
}
#pytest.fixture(scope="session")
def add_random(return_dictionary):
_temp = return_dictionary
_temp["test_key_random"] = "test_random_value"
return _temp
Is it because pytest collects similar decorators, and analyzes them together? I would like someone's input into this. Thanks!
Here is a few files I created to demonstrate what I was looking for, and what the error I am seeing.
# conftest.py
import pytest
from pprint import pprint
pytest_plugins = ["helpers_namespace"]
# fixtures
#pytest.fixture(scope="session")
def return_dictionary():
return {
"test_key": "test_value"
}
# helpers
#pytest.helpers.register
def super_print(_dict):
pprint(_dict)
#pytest.helpers.register
def super_print_always(key, _dict=return_dictionary):
pprint(_dict[key])
# test_check.py
import pytest
def test_option_1(return_dictionary):
print(return_dictionary)
def test_option_2(return_dictionary):
return_dictionary["test_key_2"] = "test_value_2"
pytest.helpers.super_print(return_dictionary)
def test_option_3():
pytest.helpers.super_print_always('test_key')
key = 'test_key', _dict = <function return_dictionary at 0x039B6C48>
#pytest.helpers.register
def super_print_always(key, _dict=return_dictionary):
> pprint(_dict[key])
E TypeError: 'function' object is not subscriptable
conftest.py:30: TypeError

is this possible to create sub-methods in python?

I want to write a method to parse a site with requests library, the method should take a part of url having base_url in it and perform the get request on this, the main problem is that I do not know how to make it better;
What I have in mind now is:
import requests
class Response:
# ...
def site_parser(self, atom):
base_url="https://example.com/"
def category1(self):
return requests.get(base_url + category1/ + atom).text
def category2(self):
return requests.get(base_url + category2/ + atom).text
if __name == "__main__":
def main():
result = Response()
result.site_parser.category1("atom")
result.site_parser.category2("atom")
so needed data has the same base url but different dirs to get into, and I need to gen each dir if only the method was called afterwards. is there a way of doing this properly? I wouuld like to avoid making base url global variable

It seems to me that what you need is another class.
class Response:
# ... Some dark magic here ...
def site_parser(self, atom):
return ResponseParser(self, atom)
class ResponseParser:
def __init__(self, res, atom):
self.atom = atom
self.res = res
self.base_url = "https://example.com/"
def category1(self):
# ... Do stuff ...
def category2(self):
# ... Do stuff ...
Then you call it with
result = Response()
result.site_parser("atom").category1()
If you really insist on getting rid of the parentheses on the site_parser call, you could move the "atom" bit to the categoryN methods and turn site_parser into a property, but IMO that would probably just confuse people more than anything.
As a functional programmer, I love nested functions and closures as much as the next guy, but it seems to me that, based on the limited example you've given, having a second helper class is probably going to be the more readable way to go about this in this case.

Understanding flow of execution of Python code

I'm trying to do home assignment connected with python from Data Manipulation at Scale: Systems and Algorithms at Curesra. Generally I have problems with understanding base code which was presented as an example of MapReduce alogorythm. I would be grateful for helping me understand it in 2 places, details below.
I tired to go step by step through code flow of below two files after running command:
python wordcount.py 'data/books.json'
File wordcount.py is opened
mr = MapReduce.MapReduce() - me object is created
def __init__(self): part from MapReduce.py is
executed
We come back to wordcount.py
Functions def mapper(record): and def reducer(key,list_of_values): are created but for the time being without execution
Python go to if __name__ == '__main__':
` inputdata = open(sys.argv[1]) - json file is assigned to a
variable
mr.execute(inputdata, mapper, reducer) - A call to the function from MapReduce.py.
And here is my first question we haven't deffined mapper or reducer variable/object so far. Is it just null/no value passed to this function or we somehow defined this variable before but I missed this?
Later me move to def execute(self, data, mapper, reducer): in
MapReduce.py
And there we have mapper(record).
So this is reference to a function in wordcount.py, am I right? But if we have reference to a function in different file shouldn't we use import at the beginning of the file and define from which file this function came?
(...) further code execution
wordcount.py file:
import MapReduce
import sys
"""
Word Count Example in the Simple Python MapReduce Framework
"""
mr = MapReduce.MapReduce()
# =============================
# Do not modify above this line
def mapper(record):
# key: document identifier
# value: document contents
key = record[0]
value = record[1]
words = value.split()
for w in words:
mr.emit_intermediate(w, 1)
def reducer(key, list_of_values):
# key: word
# value: list of occurrence counts
total = 0
for v in list_of_values:
total += v
mr.emit((key, total))
# Do not modify below this line
# =============================
if __name__ == '__main__':
inputdata = open(sys.argv[1])
mr.execute(inputdata, mapper, reducer)
MapReduce.py file:
import json
class MapReduce:
def __init__(self):
self.intermediate = {}
self.result = []
def emit_intermediate(self, key, value):
self.intermediate.setdefault(key, [])
self.intermediate[key].append(value)
def emit(self, value):
self.result.append(value)
def execute(self, data, mapper, reducer):
for line in data:
record = json.loads(line)
mapper(record)
for key in self.intermediate:
reducer(key, self.intermediate[key])
#jenc = json.JSONEncoder(encoding='latin-1')
jenc = json.JSONEncoder()
for item in self.result:
print jenc.encode(item)
Thank you in advance for help with that.

In python everything is a object, that include functions, so you can pass a functionA as argument to another functionB (or class or whenever), and if functionB expect that you to do it, it will assume that you give it a functions with the right firm and a proceed as normal.
In yours case
mr.execute(inputdata, mapper, reducer)
here mapper, reducer are the functions previously defined that are passed as argument to the method execute of the instance mr of the class MapReduce and as you can see, said method use it as the functions that it expect.
Thank to this you can, as the that code show, make generic code that do some calculus that can be used in similar way by many applications by given the user the options of supplies his/her own functions.
A much more generic example of this is the function map, this function receive a function that do something, map don't care what it does or where it comefrom, only that receive as many argument as map himself receive (others that say functions) and return a value to build a new list with the results.

How to deal with globals in modules?

I try to make a non blocking api calls for OpenWeatherMap, but my problem is:
When i was doing tests on the file, and run it, the global api was taking effect, but when importing the function, global dont work anymore, and api dident change: api = ""?
Just after declaring the function i put global api, and then when I use print 'The API link is: ' + api I get the exact api, but global dident took effect!
Here is the code: https://github.com/abdelouahabb/tornadowm/blob/master/tornadowm.py#L62
What am I doing wrong?
When I import the file:
from tornadowm import *
forecast('daily', q='london', lang='fr')
The API link is: http://api.openweathermap.org/data/2.5/forecast/daily?lang=fr&q=london
api
Out[5]: ''
When executing the file instead of importing it:
runfile('C:/Python27/Lib/site-packages/tornadowm.py', wdir='C:/Python27/Lib/site-packages')
forecast('daily', q='london', lang='fr')
The API link is: http://api.openweathermap.org/data/2.5/forecast/daily?lang=fr&q=london
api
Out[8]: 'http://api.openweathermap.org/data/2.5/forecast/daily?lang=fr&q=london'
Edit: here is the code, if the Git got updated:
from tornado.httpclient import AsyncHTTPClient
import json
import xml.etree.ElementTree as ET
http_client = AsyncHTTPClient()
url = ''
response = ''
args = []
link = 'http://api.openweathermap.org/data/2.5/'
api = ''
result = {}
way = ''
def forecast(way, **kwargs):
global api
if way in ('weather', 'forecast', 'daily', 'find'):
if way == 'daily':
way = 'forecast/daily?'
else:
way += '?'
for i, j in kwargs.iteritems():
args.append('&{0}={1}'.format(i, j))
a = ''.join(set(args))
api = (link + way + a.replace(' ', '+')).replace('?&', '?')
print 'The API link is: ' + api
def handle_request(resp):
global response
if resp.error:
print "Error:", resp.error
else:
response = resp.body
http_client.fetch(api, handle_request)
else:
print "please put a way: 'weather', 'forecast', 'daily', 'find' "
def get_result():
global result
if response.startswith('{'):
print 'the result is JSON, stored in the variable result'
result = json.loads(response)
elif response.startswith('<'):
print 'the result is XML, parse the result variable to work on the nodes,'
print 'or, use response to see the raw result'
result = ET.fromstring(response)
else:
print '''Sorry, no valid response, or you used a parameter that is not compatible with the way!\n please check http://www.openweathermap.com/api for more informations''

It's the side effect of using global.
When you do from tornadowm import * your forecast() function is, we could say metaphorically, "on its own" and is not "hard-linked" to your global space anymore.
Why? Because any effect you make on your global api will "end" with your function, and the definition of api = "" in your global space will take precedence.
Also, as a side note, it's not considered a good practice to use from something import *. You should do from tornadowm import forecast or even better, import tornadown and then use tornadowm.forecast().
OR
Even better, I just noticed your forecast() function doesn't return anything. Which technically makes it not a function anymore, but a procedure (a procedure is like a function but it returns nothing, it just "does" stuff).
Instead of using a global, you should define api in this function and then return api from it. Like this:
def forecast(blablabla):
api = "something"
blablabla
return api
And then
import tornadowm
api = tornadown.forecast(something)
And you're done.

Globals are global only to the module they're defined in. So, normally, you would expect tornadowm.api to be changed when you call forecast, but not api in some other namespace.
The import * is contributing to your understanding of the problem. This imports api (among other names) into the importing namespace. This means that api and tornadowm.api initially point to the same object. But these two names are not linked in any way, and so calling forecast() changes only tornadowm.api and now the two names point to different objects.
To avoid this, don't use import *. It is bad practice anyway and this is just one of the reasons. Instead, import tornadowm and access the variable in the importing module as tornadowm.api.

I'm afraid this is because global is coupled within module, by the time you from tornadowm import * you have imported the api name, but the global api won't take any effects within another module.

pytest fixture to introspect calling function

I have a test class and a setup function that looks like this:
#pytest.fixture(autouse=True, scope='function')
def setup(self, request):
self.client = MyClass()
first_patcher = patch('myclass.myclass.function_to_patch')
first_mock = first_patcher.start()
first_mock.return_value = 'foo'
value_to_return = getattr(request, 'value_name', None)
second_patcher = patch('myclass.myclass.function_two')
second_mock = second_patcher.start()
second_mock.return_value = value_to_return
#could clean up my mocks here, but don't care right now
I see in the documentation for pytest, that introspection can be done for a module level value:
val = getattr(request.module, 'val_name', None)
But, I want to be able to specify different values to return based on the test I am in. So I am looking for a way to introspect the test_function not the test_module.
http://pytest.org/latest/fixture.html#fixtures-can-introspect-the-requesting-test-context

You can use request.function to get to the test function. Just follow the link on the b wepage you referenced to see what is available on the test request object :)

Maybe the documentation has changed since the time of the accepted answer.
At least for me it was not clear how to
Just follow the link
So I thought I'd update this thread with the link itself:
https://pytest.org/en/6.2.x/reference.html#request
Edit December 2021
Even when the link is correct now I think this statement from the pytest documentation is just not correct:
Fixture functions can accept the request object to introspect the “requesting” test function ...
While I found some examples for getting attributes of the module I did not find a single working example of introspecting the test function that requests the fixture. May be related to collection and runtime order.
What really helped me to get the desired behavior was to use the factory idiom a little further down in the pytest documentation:
https://pytest.org/en/6.2.x/fixture.html#factories-as-fixtures
Set up the fixture factory
#pytest.fixture(scope='function')
def getQueryResult() -> object:
def _impl(_mrId: int = 7622):
return QueryResult(_mrId)
return _impl
Usage
# Concrete value
def test_foo(getQueryResult):
queryResult = getQueryResult(4711)
...
# Default value
def test_bar(getQueryResult):
queryResult = getQueryResult()
...