Introducing panflute-tests

Written for Panflute by Paul Kuliniewicz on 2010-09-26

The bad news is that I won’t be releasing the next version of Panflute at the end of the month as I had planned; that’s been pushed back to the end of October. That’s because it took much longer than I was expecting to get a new backend feature working: panflute-tests.

As of ths writing, Panflute supports 15 different music players, with even more planned: Amarok, Audacious, Banshee, Exaile, Guayadeque, Listen, MOC, MPD, Muine, Quod Libet, Rhythmbox, Songbird, VLC, XMMS, and XMMS2. Manually testing Panflute with the latest version of each is impractical; expanding that testing to include the last n versions of each of those is completely unreasonable. As a result, I had largely been slacking off doing proper testing, meaning that bugs, especially when the new version of a player changed its interface somehow, were common.

panflute-tests is a tool designed to automate the process of compatibility testing as much as possible. It allows a developer to define a series of test configurations, which are basically a particular installation of a particular version of a player, such as Banshee 1.7.4 installed under $HOME/players/banshee-1.7.4. For each defined configuration, panflute-tests will initialize the player with a known “music” collection and run a series of tests against it. The tests themselves are simple; for example, one checks that when the Play, Pause, and Stop commands are issued, the player responds by reporting the appropriate state change.

Even though each test is simple, the ability to automate running them against lots of configurations makes overall compatibility testing much more thorough and repeatable. Already, it’s discovered bugs that hadn’t been reported to the Panflute bug tracker, such as not retrieving metadata from recent versions of XMMS2 correctly.

[see my blog post for more]

Read more