Let's start with a simple test definition in
speclj, the BDD testing framework for Clojure.
(describe "my module"
(it "has the correct value"
(for [item {:item1 "value1" :item2 "value2"}]
(should= (val item) (get-value item)))))
This test simple asserts that
my module has the correct value for
item1 and
item2. Pretty simple. Ok, let's say that we add 15 more fields to
my module. Our test will now look like this...
(describe "my module"
(it "has the correct value"
(for [item {:item1 "value1" :item2 "value2" :item3 "value3" (etc.)}]
(should= (val item) (get-value item)))))
Not too bad. For every test that we add all we have to do is add a key and value to the for loop. Seems pretty extensible to me. So later, one of the assertions fails. However, we only get one error. Thats not good. That makes it seem like our entire module is broken. We first have to identify will field the test broken on, then we have to go fix the bug. This could be especially painful if there were more like 1000 fields! Ok, so lets find a better solution. Consider this example.
(describe "my module"
(for [item {:item1 "value1" :item2 "value2"}]
(it (str "has the correct value for " (key (name item)))
(should= (val item) (get-value item)))))
So what is different in this example? Instead of generating assertions inside of the
(it...) block, we loop over it and generate a bunch of
(it...) blocks. So what is useful about this? If we consider the example above were "my module" has 1000 fields. Using this new method, we will be able to tell exactly which field this test failed on. This is extremely helpful because it cuts out the entire time it takes to find which assertion was the culprit of the failure. Also, when this module fails, not every test will fail. We will get better feedback on what is actually being tested and breaking. So, in summary, if you need to test a large list of items, it's better to generate tests for it rather than generating assertions.
1. What if there's a lot of set and each test take a long time to run?
ReplyDelete2. Doesn't speclj print a nice failure message to let you know which assertion failed and why?
3. Could you not provide descriptive test data which points you toward the failure?
4. Do you think that the assertions should take an optional string describing the failure?
1. If each tests is running long, it would be better to group the assertions to avoid to overhead of an it block.
ReplyDelete2, 3. Yes, however sometimes the test data does not lend itself to meaningful asserts on their own, especially when dealing with Booleans.
4. That would be really helpful in some cases, such as the case of the large set of data and long running tests where the assertions must be grouped together for performance reasons.