All talks: https://emacsconf.org/2023/talks/
What I learned by writing test cases for GNU Hyperbole
https://emacsconf.org/2023/talks/test - Mats Lidell - Track: Development
Watch/participate: https://emacsconf.org/2023/watch/dev/
Q&A room: https://media.emacsconf.org/2023/current/bbb-test.html
IRC: Speaker nick: matsl - https://chat.emacsconf.org/#/connect?join=emacsconf,emacsconf-dev or #emacsconf-dev on libera.chat network
Guidelines for conduct: https://emacsconf.org/conduct
See end of file for license (CC Attribution-ShareAlike 4.0 + GPLv3 or later)
----------------------------------------------------------------
Notes, discussions, links, feedback:
----------------------------------------------------------------
Questions and answers go here:
- Q:How many tests do you have for Hyperbole and how wouild you rate the test coverage compared to other packages?
- A:
- With all tests including the interactive we have 354 tests. Havng said that I must point out that the size of the tests can be very different. I tend to split tests so they are logically (in some sense) different. So that if a test fails it will more likely point you to what the error is. This makes it become more tests. Codewise you could collect similar tests to one ert-deftest making the name of the test point out some group or collection of functions, but I don't do that!
- I have not studied other packages so I don't know how our test coverage compares to other packages. In fact I don't know what code coverage we have. That is another thing to look into.
- Q: One small suggestion, to me 'should' means optional, whereas 'shall' or 'must' means required. Not sure if it is too late to make a major grammar change like that :) Very nice presentation. (I see :))
- A: The assertions come from the ert package so any changes would have to be suggested to that. I guess you could make your own version of the assestions using aliases for should et al.
- Q: FYI, you may find this helpful for running Emacs tests/lints, both from a command line and from within Emacs with a Transient menu: https://github.com/alphapapa/makem.sh It also works on remote CI.
- A: Thanks for the suggestion. I did have a look at makem.sh but a long time ago so I don't remember why we did not try to apply it. I might give it another look now when I have used plain ert more.
- Q: Is it easy to run ad hoc tests inside of an Emacs session, given the command line scripts you need to run to get a batch test session running? In other words, can you tweak tests in an Emacs session and run them right away?
- A:
- Yes, in principle you just load your tests and run them all using `ert` and give it the test selector `t`. That runs all loaded tests.
- If you want to modify a test you can do that. You change it, evaluate it, and run it again. Just as you change any function.
- Q: Did you have to change Hyperbole code and design to be more readily testable as you were increasing your test coverage?
- A:
- Yes, we have done that to a small extent but we should do more of that. Some Hyperbole functions are large and by that complicated to test. Splitting them into smaller logical parts can make testing easier.
- Also moving code into pure functions and avoid side effects is a good thing. Pure functions are easier to test. Maybe haveing the side effects separated out into fewer places. This has not been applied but is something I have been thinking about. With side effects I here mean things like adding or modifying text in buffers.
- Q: What's the craziest bug you found when writing these tests?
- A: This is not a bug but I always assumed giving a prefix argument to a cursor movement would give the same result as hitting the key the same amount of times. So like C-u 2 C-f would be the same as hitting the C-f key twise. It is not! When moving over a hidden area, the three dots '...' at the end of folded line in org-mode or outline-mode, you get different behavior. Trying to write a test case for the kotl-mode and its folded behavior teached me that.
- Q: Why do you prefer el-mock to mocking using cl-letf. (Question asked in BBB)
- With cl-letf you need to keep track if the mocked functionality is being called or not. The el-mock package does that for you which is what you normally want. Doing this with cl-letf means the definition becomes longer and more complicated. Sort of blurs the picture. el-mock is more to the point.
- BUT since cl-letf does allow you do define a "new" function it is more powerful and it can be the only option in cases where el-mock is too limited. So it is good to know of this possibility with cl-letf when el-mock does not provide what you need.
----------------------------------------------------------------
Next talks:
Questions/comments related to EmacsConf 2023 as a whole? https://pad.emacsconf.org/2023
----------------------------------------------------------------
This pad will be archived at https://emacsconf.org/2023/talks/test after the conference.
Except where otherwise noted, the material on the EmacsConf pad are dual-licensed under the terms of the Creative Commons Attribution-ShareAlike 4.0 International Public License; and the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) an later version. Copies of these two licenses are included in the EmacsConf wiki repository, in the COPYING.GPL and COPYING.CC-BY-SA files (https://emacsconf.org/COPYING/)
By contributing to this pad, you agree to make your contributions available under the above licenses. You are also promising that you are the author of your changes, or that you copied them from a work in the public domain or a work released under a free license that is compatible with the above two licenses. DO NOT SUBMIT COPYRIGHTED WORK WITHOUT PERMISSION.