Last week at the monthly meetup organized by Tabara de Testare four of our colleagues organized a workshop on Testing Tours – we call them content-owners @ Tabara de Testare. Even though many people have written on touring over the past 10 – 15 years, this technique seems very little known and used by practitioners. This is why when my colleagues chose this theme I thought it was a great idea. Also, there are many tours that I’m not familiar with, so it  was a good learning opportunity for me.

About 25 people gathered for this meetup and, after the introduction delivered by the content-owners, we split in teams of 2, 3 or 4 testers and we went out in the garden to get started. This month’s venue was ClujHub, and the great thing about this co-working space, besides the downtown location, is this garden!I teamed up with Gabi Kis, a fellow facilitator from Tabara de Testare. We took the tours cards prepared by the content-owners and went through all of them to see which tour we’d like to perform – all the other teams already chose their tour, so it seemed to me that we were a little behind; now that I’m thinking back, I’m surprised that I was not stressed by this :). Having the cards in a physical form, not on a PowerPoint slide, made it easy for us to take them in the garden – I also noticed that all teams took the card for the chosen tour with them One other aspect I liked at having the cards was that most of them contained enough information to give starting ideas on how to approach the tour.

Out of all the tours, we stopped at the following 3:

Scenario Tour – List realistic scenarios for how different types of users would use the product.

Testability Tour – Find features and tools to help in testing the application.

Complexity Tour – Identify the most complex things present in the application.

For the Scenario Tour we weren’t sure if we just needed to list the scenarios or also perform them. For the Complexity Tour we couldn’t find a working definition for “complexity in the application”. We decided to choose the Testability Tour.

Testability Tour on Slack

The application we had to work with was Slack, and luckily for me I was familiar with it as we use it at Altom and also on several projects. Gabi didn’t know the application very well, so for him it was a good opportunity to discover it.

As we didn’t have access to the internal documentation of Slack, we started listing the items we thought would make the application easier to test. We structured our notes in 3 sections:

1. App Features
   1. API – can be used for generating test data
   2. Websockets – same as API
   3. Logs
   4. Error messages (UI level, API level and logs) – this would help the tester to better understand when something goes bad
   5. File sharing – can we have access to the cloud storage, but not through the application to control the test data?
   6. Bots – can we use them in our chat tests to get responses?
2. Tools – as this is a client-server application, we started to think about what kind of tools we could use to test it:
   1. Proxy (Charles / Fiddler) or Wireshark – to intercept the calls
   2. data sets – Gabi said that he would like to be able to build specific data sets to put the application in different states.
   3. server side
      1. Top / Perfmon / Nagios – to monitor server side resource utilization
      2. Jmeter – to send requests for load and performance testing
   4. client side
      1. Perfmon / Top / Activity monitor – to check the desktop application resource utilization (the desktop client is a web app packaged as a standalone application)
      2. adb / instruments – to check the mobile application resource utilization
      3. Yslow and Developer tools – for the browser client
   5. spider tools – to discover and list different features from the application; one aspect we thought of was that if the app uses custom controls the tool won’t be able to find too many things…
3. App Structure
   1. ease of identifying elements for UI level test automation
   2. how easy it would be to build our own test environment
   3. client app installation – do we have a silent install? We asked ourselves if this is an app feature, as it can be used by sys admin to install Slack on large networks, or a testability feature, as it would allow us to setup the test environment easier.
   4. server side installation – Can it be installed on premise? If yes, how easy can it be set up?

The above list was not created in the order it is now displayed, but more as a brainstorming session: when we identified one item, we would seek to explain why it could be relevant and try to categorise it in one of the main sections. What I find interesting is that we initially started with 2 sections, Features and Tools, and while coming up with new ideas we thought of adding a new section, App Structure (one could argue that this last section could easily be part of the App Features section).

About 45 minutes into the exercise there was still no touring of the application. We thought of taking one item from our list and start investigating it, so we chose the logs feature: we wanted to know if there are any client side logs on the desktop client.

I was using a Mac, so I thought Slack would save the data under the user’s Library folder, but nothing was there. We looked in Application Support and in Cache and we couldn’t see anything relevant. I looked in the System / Library folder, and still nothing.

I googled a bit for Slack logs and local storage, and for some reason Google decided to return https://status.slack.com, which is a nice status overview of the application. This makes me think that there should be more detailed monitoring on the server :). Unfortunately we didn’t find anything else relevant in the google search.

I looked in the application package from /Application. Nothing relevant there either.

The next step was to open Activity Monitor and double-click on the Slack process, and then I went to the Open Files and Ports and noticed that the user files are saved in ~/Library/Containers/com.tinyspeck.slackmacgap/Data

So this is where all the goodies are! Listing the folder content we noticed that most of the folders are aliases for system folders

We looked into Library, and found the same behaviour: lots of Aliases.

We also noticed the Logs folder. Unfortunately it was empty…

Next we went to Application Support and found Crashes and Slack. We digged into Slack and found what we were looking for: the log file.

OS X comes with a pretty neat tool, Console, which helped us to inspect the log. Below is an example of what my Slack log shows today. You can find information about users joining teams and channels, who left a channel, if websockets are opened, etc.

Now that we found our log, we decided to also look in what is saved locally, so we googled for an SQLite browser and found http://sqlitebrowser.org. We downloaded it and first opened localStorage.sqlite, but this had data from 2015. We then opened localCache.sqlite and found the cached data. We also tried to also open localCache.sqlite-wal but it was password-protected.

Going back to inspect the other folders from Application Support, we noticed that Slack has an alias for AddressBook, which made us wonder about the integration from these two applications and if we can use Address Book to inject data in Slack for testing purposes.

One of our last thoughts before the time was up was that we might have to define what we wanted to test. We approached this tour as if we wanted to test “everything” (we thought of client and server features, tools for performance and UI level automation). Had we have started with a more focused mission, our notes would have been very different.

What I liked about this session with Gabi was that we started with a brainstorming based on our previous experiences with similar applications and then chose one aspect – client side logging – and drilled in it. During all this time we tried to take notes so that we would use them for future, maybe more focused, touring. Here is our log, with notes in English and Romanian, to have a better view of what our outcome was after the tour.

After one hour of touring, it was time to go back inside and do a debriefing. Each team presented their work and the discussions were facilitated by the content-owners (what they learned, how they organized their work and notes…).

One thing I learned during the debriefing is that the order in which the tours are performed matters. For example the Scenario Tour more suitable to be done after the User Tour, as one needs a list of users to identify the scenarios, or the Feature and Configuration Tour to get familiar with what the application can do. This makes total sense now.

One interesting discussion was between Team #4 (Variability Tour) and Team #5 (Configuration Tour) as they toured around similar areas, and they were debating if their work was relevant for the tour taken. One of the content owners clarified that the Configuration tour can be seen as a sub-tour of the Variability Tour, the main difference being that the Configuration Tour is focused on persistent changes, while the Variability Tour is focused on temporary changes.

All in all it was a great workshop. People were highly engaged, showing thirst for knowledge and discussion. I challenge you to try testing tours with your team as an exercise and see the benefits for yourself.

P.S. of course such a meetup couldn’t have finished without a beer 😀