Deliberate discovery, exploratory testing and code playground

Have you ever pair-programmed with a colleague of yours who has a separate project, usually named sandbox or playground where all sorts of cool and crazy code snippets and experiments are located?

I have had such moments a few times, certainly. In fact, until recently, I had my own sandbox local project where I stored all sorts of code snippets that I scratched quickly, for example to explore API of a new unknown library. Usually it’s a local project, and unintentionally never shared with anyone to learn from all the exploration done, since those experiments are not part of some production software.

I would like to touch on the idea of deliberate discovery first, before circling back to code snippets with experiments and exploratory testing, since for me these topics sit behind the same underlying theme of discovering unknowns and learning faster (hopefully in a more fun way).

Deliberate discovery

There is an article by Dan North that I really like - Introducing Deliberate Discovery.

Dan North references this thought experiment, that he himself heard from Liz Keogh:

Liz Keogh told me about a thought experiment she came across recently. Think of a recent significant project or piece of work your team completed (ideally over a period of months). How long did it take, end to end, inception to delivery? Now imagine you were to do the same project over again, with the same team, the same organisational constraints, the same everything, except your team would already know everything they learned during the project. How long would it take you the second time, all the way through? Stop now and try it.

It turned out answers in the order of 1/2 to 1/4 the time to repeat the project were not uncommon. This led to the conclusion that “Learning is the constraint”.

The conclusion by Dan North is that “it’s not really learning that’s the constraint — it’s ignorance. More accurately, it’s ignorance about specific aspects of the problem at hand. In other words:

Ignorance is the single greatest impediment to throughput.

In my own reflection and experience, indeed, certain unknowns, had they been discovered earlier, would have made a huge difference for the project. Though I also asked myself if I possibly fell into the hindsight bias¹. Maybe, in the end, it wasn’t so easy or even possible to figure out unknowns in the beginning of the project phase? To really evaluate the quality of my decisions in the past and see if I could have done a better job discovering the unknowns, I ask myself what information was available to me during the decision-making, and did I do a good job taking into account all the information to make a next step?

Hindsight bias occurs when we analyse decisions of past events based on the new information that we obtain after the event passed. For example, you might think that you made a very good decision to invest into some stock 3 months ago, because the stock rose in past 3 months by 50%. But you analysed the quality of your past decision based on the information from the future. This evaluation is done in hindsight, and it doesn’t help you make better decisions in the future. The better question to ask oneself might be like this: given all available information I had when making a decision in the past, did I do a good decision? Such evaluation can be considered more useful, since it helps us to make future decisions given only the available information we have when making a decision.

When I analyse the quality of my previous decisions, I try not to base them on the outcomes, but rather on all the available information I had when making a decision.

Just as Dan North makes an argument about deliberately figuring out unknowns in a project, over time I started to deliberately focus on figuring out unknowns better, given all the available information I have at the moment. I noticed that it’s a skill that can be learned and improved.

To add, ignorance and unknowns are very multidimensional. One day it’s a domain knowledge that I lack to solve a problem efficiently, another day it’s purely code-related etc. While it’s impossible to plan for and tackle unknown unknowns, there are plenty of known unknowns that I can focus on right in the beginning and throughout the project to increase the probability of a project’s success.

Exploratory API testing

Domain-related unknowns in most cases are harder to discover than tech-related unknowns, so establishing a quick feedback loop with a domain expert (or source of expertise) is paramount. In this post though, I would like to focus on some techniques I found are useful for myself to discover tech-related unknowns.

I really like the concept of exploratory api testing. I will explain it based on my recent experience. I had to explore the API of google’s tink library in Go, which provides cryptographic APIs that are secure, e.g. to encrypt/decrypt data, do envelope encryption with the help of KMS etc.

One approach for me could be to immediately start using tink in a project’s repository and start integrating it with other existing code immediately to fulfill some required task, let’s say to encrypt some data. This approach has few minuses imho. I would battle at least two problems at the same time:

1. My brain now needs to think how to integrate tink and encryption feature into the existing code structure.
2. I haven’t yet familiarized myself with tink library, its API and all its capabilities that I might need.

So, instead, I ignore my first problem completely for a while, and decide that I better tackle problem #2 first - getting to know tink. I do exploratory API testing. I create tink_test.go file and start playing around with the library immediately.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

func TestAEAD(t *testing.T) {
	kh, err := keyset.NewHandle(aead.AES256GCMKeyTemplate())
	if err != nil {
		t.Fatal(err)
	}

	a, err := aead.New(kh)
	if err != nil {
		t.Fatal(err)
	}

	ct, err := a.Encrypt([]byte("this data needs to be encrypted"), []byte("associated data"))
	if err != nil {
		t.Fatal(err)
	}

	pt, err := a.Decrypt(ct, []byte("associated data"))
	if err != nil {
		t.Fatal(err)
	}

	assert.Equal(t, "this data needs to be encrypted", string(pt))
}

I copied this excerpt from tink’s docs, and adapted it a bit to wrap it into the go test and add assertion in the end instead of the printf.

Even after reading tink’s docs, I was still confused on its API a bit. There was quite a bit of terminology, concepts, and interfaces to wrap my head around - keyset Handle, KeyTemplate, loading keys and decrypting them, serializing them back securely using a key handle’s AEAD² interface, integrating with KMS providers etc.

For at least an hour or two, my development process looked something like this:

• Wrote a bit of a code snippet in a test. Something broke. Set a breakpoint to debug why I used API incorrectly. Fixed. As a result, had a correct code snippet that worked as expected.
• When I didn’t understand some of tink’s API, I went into its definition in the library code itself and saw how it interacts with other pieces. Tink, particularly, had very good comments inside its library code regarding its API.
• Wrote short tests for all possible scenarios I needed tink for. In my case I needed to write proof of concept for such main scenarios: encrypting the secret key with KMS and persisting it, decrypting it with KMS and loading into memory, using pseudo-random function module in tink to do HMAC_SHA256, encrypting/decrypting data with envelope encryption.

After an hour, I got very familiar and comfortable with the tink’s API and its capabilities. Even more important, I had few exploratory tests, similar to the one above, which served as a proof-of-concept working code that I could later integrate in my existing project’s code structure to implement a feature end-to-end.

During this exploratory phase, I am also very liberal to add TODOs or comments into these tests, making notes for myself and future readers.

By the end I answered a very important tech-related unknown I had previously - would tink provide all necessary capabilities I need for my use-cases? The answer was yes, and I was happy to receive this answer in about two hours, get very familiar with tink’s API, and as a bonus to have a proof-of-concept code that I would use later to implement a necessary feature.

Now, what do we do with this tink_test.go file that now has a couple of exploratory api tests?

Shared playground for experiments and explorations

We have few options for the fate of our tink_test.go:

1. Leave it somewhere in your laptop, locally, never shared with anyone. This has a cons of, well, not being shared with anyone to learn from.
2. Place it inside the relevant project itself, for example under playground folder.
3. Place it inside a dedicated repository, e.g. playground, shared across the company, where different experiments and code explorations are located.

In BetterMedicine, I opted for the third option. We have a dedicated playground repository in our github. We put our code experiments there. There are no rules for this repository, no code quality checks, linters, CI/CD pipeline, it’s not related to any production-related code etc. This is, of course, to avoid any friction of sharing code snippets and explorations with each other.

Unknowns and feedback loops

I find that discovering unknowns is not an easy task, given alone evaluating for myself how good (or bad) I am at discovering known unknowns, while avoiding a hindsight bias. For tech-related unknowns or learning, I figured out for myself that a quick exploratory testing helps me out.

I also observed that no matter if it’s about discovering domain-related or tech-related unknowns, the key is to establish a quick feedback loop. For discovering the domain-related knowledge, the feedback loop might be in the form of shadowing a domain expert often, for example, or having interviews. For discovering tech-related unknowns, exploratory api testing is in my toolbox, as it provides a quick feedback loop too.

And the other observation I had for myself. Well-established feedback loops make things less stressful and more fun in software engineering. When we have a robust test suite that we trust, whenever we refactor and tests pass, we feel calm to deploy. When something is not functioning well inside a team, a regular retrospective, given a correct approach, enables everyone to share concerns and solve them as a team, thus de-stressing the situation and stabilizing things.