I am reporting data from some tests, and each test can have a 'summary' and a 'details' section. The 'summary' field for any particular test should be static, with any additional dynamic information going into the details field, as follows:
run_test()
if condition:
report_test_data(summary="condition met", details=f"{component} ended with {state}")
else:
report_test_data(summary="condition not met", details=f{component} ended with {state}")
However, this only applies to these calls to report_test_data, and there is nothing to stop another test from swapping these around, or putting all the data into the 'summary' field:
run_test()
if condition:
report_test_data(summary=f"{component} ended with {state} - condition met", details="")
else:
report_test_data(summary=f"{component} ended with {state} - condition not met", details="")
I am analyzing the test data based off the summary, so any particular ending state (e.g. condition = True) should have a static return string. I thought about making a class that manually defines every possible 'summary' string, but that quickly becomes untenable with more tests when a given test can have tens of possible ending states. The best option I can think of is if I could force the value passed into 'summary' to be a normal string. Is there any way to disallow passing f-strings into a particular function?
Note: I use pylint, so if there's an easy way to make it call these out, that would work as well.
Related
I want to extend the Endian Swapper example of Cocotb, so that, it also checks the contents of the packages outputted by the device under test (DUT). In the provided example code, the model function which generates the expected output appends the unmodified input transaction to the list of expected outputs. This list is given as a parameter to the scoreboard.
To understand how the scoreboarding works and why the model function did not append the byte-swapped transactions, I introduced a design error in the DUT. Within the following code block of endian_swapper.vhdl
if (byteswapping = '0') then
stream_out_data <= stream_in_data;
else
stream_out_data <= byteswap(stream_in_data);
end if;
I just inverted the if condition in the first line to: (byteswapping /= '0').
After re-running the testbench, I would have expected that the test fails, but it still passes:
# 62345.03ns INFO cocotb.regression regression.py:209 in handle_result Test Passed: wavedrom_test
# 62345.03ns INFO cocotb.regression regression.py:170 in tear_down Passed 33 tests (0 skipped)
# 62345.03ns INFO cocotb.regression regression.py:176 in tear_down Shutting down...
It seems, that the compare function is missing in the creation of the scoreboard:
self.scoreboard = Scoreboard(dut)
self.scoreboard.add_interface(self.stream_out, self.expected_output)
It should have a third parameter in the call of add_interface, but this parameter is undocumented.
So, how do I specify this compare function, so that, also the package content is checked?
I am using QuestaSim for simulation and executed the testbench with make SIM=questa. I also clean-upped the build directory between the runs.
If I apply the following diff when using Icarus the tests fail as expected:
diff --git a/examples/endian_swapper/hdl/endian_swapper.sv b/examples/endian_swapper/hdl/endian_swapper.sv
index 810d3b7..a85db0d 100644
--- a/examples/endian_swapper/hdl/endian_swapper.sv
+++ b/examples/endian_swapper/hdl/endian_swapper.sv
## -119,7 +119,7 ## always #(posedge clk or negedge reset_n) begin
stream_out_startofpacket <= stream_in_startofpacket;
stream_out_endofpacket <= stream_in_endofpacket;
- if (!byteswapping)
+ if (byteswapping)
stream_out_data <= stream_in_data;
else
stream_out_data <= byteswap(stream_in_data);
I don't have access to Questa but I'll see what happens on a VHDL simulator. My instinct would be to double check that you ran make clean after making the change and check that Questa isn't caching the built RTL libraries somehow.
You are correct that there are some undocumented keyword arguments to the add_interface method of the scoreboard:
compare_fn can be any callable function
reorder_depth is an integer to permit re-ordering of transactions
If you supply a compare_fn it will be called with the transaction when it's received by a monitor, however this is quite a primitive mechanism. It's not scalable and only there for historical reasons (hence un-documented).
A better way is to subclass the Scoreboard class and define a custom compare method according to the following prototype:
def compare(self, got, exp, log, strict_type=True):
"""
Common function for comparing two transactions.
Can be re-implemented by a subclass.
"""
where got and exp are the received and expected transactions and log is a reference to the logger instance of the monitor (to provide more meaningful messages).
The top-level of the Endian Swapper example is provided as SystemVerilog code as well as VHDL code. The verilog code is used by default if not specified by compile options otherwise.
If I run:
make SIM=questa TOPLEVEL_LANG=vhdl
as given in the Quick Start Guide, everything works as expected. There is no need to specify a compare function in this case.
I am new to unit test.I want to write unit test for web scraper that I wrote.My scraper collects the data from website,which is on local disk where inputting different date gives different results
I have the following function in script.
get_date [returns date mentioned on web page]
get_product_and_cost [returns product mentioned and their cost]
I am not sure what to test in these functions.So far I have written this
class SimplisticTest(unittest.TestCase):
def setUp(self):
data = read_file("path to file")
self.soup = BeautifulSoup(data,'html5lib')
def test_date(self):
self.assertIsInstance(get_date(self.soup), str)
def test_date_length(self):
self.assertEqual(len(get_date(self.soup)),10)
if __name__ == '__main__':
unittest.main()
Usually, it is good to test a known output from a known input. In your case you test for the return type, but it would be even better to test if the returned object corresponds to what you would expect from the input, and that's where static test data (a test web page in your case) becomes useful. You can also test for exceptions with self.assertRaises(ExceptionType, method, args). Refer to https://docs.python.org/3.4/library/unittest.html if you haven't already.
Basically you want to test at least one explicit case (like the test page), the exceptions that can be raised like bad argument type (TypeError or ValueError) or a possible None return type depending on your function. Make sure not to test only for the type of the return or the amount of the return but explicitly for the data, such that if a change is made that breaks the feature, it is found (whereas a change could still return 10 elements, yet the elements might contain invalid data). I'd also suggest to have one test method for each method: get_date would have its test method test_get_date.
Keep in mind that what you want to find is if the method does its job, so test for extreme cases (big input data, as much as it can support or at least the method defines it can) and try to create them such that if the method outputs differently from what is expected based on its definition (documentation), the test fails and breaking changes are caught early on.
I know this is a data object but I'm new to python and I don't understand what user_output is doing. Trying to understand all this.
def user_output(self,check): #data object
output = ""
#loop of all the info for __user_arr
for user in self.__user_arr:
if user.user_bot == str(check):
output += '''
<div class='about'>
<h1>{user.user_name}</h1>
<h2>My favorite band is: {user.user_band} </h2>
<div class='business'>
<p>Why I want the tickets:<br /> {user.user_business}</p>
</div>
</div>
'''
return output.format(**locals())
else: #this will spit an error if the user do not answer the last question correctly.
output = '''<div class='error'><h2>Turns out you are a robot. No tickets for you </h2></div>'''
return output
It will generate for each user a div block with some favorite bands etc. , provided that, the user flag user_bot is equal to check (as far as I can understand a kind of bot check). If the user is a bot (does not pass the check) the program will generate the Turns out you are a robot.. block.
This looks like a function, not an object. It checks whether your user is a robot. It does so by checking if your user (for every user) has their user_bot as the string you get as input, probably a string identifying robots. Then you write something I know nothing about to the output string, probably selling the ticket.
This function seems to do this only for the first user, though because it returns in the first if else clause.
user_output() seems to be a method in some class. It is using a "poor man's" tempting facility to output an HTML fragment.
It appears to use a loop, for user in self.__user_arr:, but this is a bit of a sham. It uses this construct to pick the first item out of self.__user_arr. Because it soon returns from the function under all conditions, it will never return to the head of the loop to check subsequent users (if any).
Also note that the double-understore prefix to self.__user_arr marks it as a private part of the implementation of the object. A deeply private part. A single underscore usually indicates "private." Double underscores don't have a super-precise definition, but "very private" or "very implementation dependent" would be my guess.
If the user_bot property of that first user is a string equal to the check value (as stringified), then the template is instantiated. It does this first by setting a string variable (output) which has embedded in tt the variable markers ({a_variable}) compatible with str's format method. Calling output.format(**locals()) does the string formatting operation using locals() (which is a lookup dictionary of all current variable assignments, a.k.a. a symbol table). The **locals() construction means "use this dict (or dict-like) object as though it were a set of keyword arguments." The effect is that substrings in output like {user.user_band} are directly interpolated into the output string. Then the filled-in string is returned.
If, instead, the user.user_bot == str(check) test fails, a simple error string is returned.
In either case, a string is returned, and the method exits.
The only other case worth considering is what if there are no users in the self.__user_arr collection? Then the loop would never run, and the method would return None implicitly.
Let's say I have following fixtures:
#pytest.fixture(params=['google.com','other-provider.com')
def smtp_server(request):
.... some initialisation ....
return SmtpServer(request.param)
#pytest.fixture(params=['plain_text','html')
def message(request):
.... some initialisation according to email type....
return msg_obj
So if I use them in one test function, I have combination: google+plain, provider+plain, google+html, provider+html.
But what if I want to reuse fixtures, but only in specific combination. Eg I noticed that when I send html email to google, it fails under some circumstances. How can I reuse fixtures and test this situation, without testing sending to other-provider.com, which is pointless?
In other words - how to skip some combination of fixtures in specific test function?
Firstly I'd like to point out this is actually a fairly odd case. Are you really sure the tests are not meaningful at all for the combinations you want to skip?
Having said that the way I have solved this myself is to simply use
if snmtp_server == 'other-provider.com' and message == 'html':
pytest.skip('impossible combination')
inside the test function. It's rudimentary but works good enough for this unusual and rare case.
If I want to check for the existence and if possible retrieve an object, which of the following methods is faster? More idiomatic? And why? If not either of the two examples I list, how else would one go about doing this?
if Object.objects.get(**kwargs).exists():
my_object = Object.objects.get(**kwargs)
my_object = Object.objects.filter(**kwargs)
if my_object:
my_object = my_object[0]
If relevant, I care about mysql and postgres for this.
Why not do this in a try/except block to avoid the multiple queries / query then an if?
try:
obj = Object.objects.get(**kwargs)
except Object.DoesNotExist:
pass
Just add your else logic under the except.
django provides a pretty good overview of exists
Using your first example it will do the query two times, according to the documentation:
if some_queryset has not yet been evaluated, but you
know that it will be at some point, then using some_queryset.exists()
will do more overall work (one query for the existence check plus an
extra one to later retrieve the results) than simply using
bool(some_queryset), which retrieves the results and then checks if
any were returned.
So if you're going to be using the object, after checking for existance, the docs suggest just using it and forcing evaluation 1 time using
if my_object:
pass