Different types of testing

Software development is complex. It involves people talking about an idea or concept and then translating it to code. The code then is compiled, packages or run and we expect a certain result. In all of these (generic) steps, there's room for complexity, interpretation and errors to sneak into the software.

Luckily, testing software has matured and expanded to a level where we can confidently release code that doesn't break on build, only changes where change is needed and the features can even be asserted before writing a single line of code!

When writing code for the web, we can distinguish several types and levels of testing. Apart form human testing (which quickly gets outpaced by scaling and expanding software), most of the test procedures are automated and integral part of an application.

Let's do a short series on testing, because of it's importance in delivering robust coding solutions.

Types of testing

Linting

We can look at testing at several levels: you could consider linting tools a very basic form of testing in the sense that it assures that code doesn't contain syntax errors. Linting is usually integrated with your integrated development environment (IDE). The common setup is able to detect certain coding languages and interprets the lines you've added while your drafting your code. It is comparable to doing a spell check on words you're writing.

Formatting

Formatting is a means of having a standardised markup for your code. There are multiple ways of writing an expression and a formatter (usually part of your IDE as well) can be configured to adopt a set of formatting rules. This is not necessarily part of testing, but I want to address it because it helps catching logic errors in code and helps fellow engineers with assessing code during a visual inspection.

Code analysis

For further (more holistic) analysis, there are tools on the market that perform code analysis, with the goal of improving the general quality of your code by looking and identifying patterns that could indicate code smells, bugs or security vulnerabilities.

With tooling ingesting the entire codebase, it can offer a much better insight than validating on, say, a pull request level. These tools

Unit testing

Testing a unit is the smalles possible actual test you can run against your code. It basically means that you take the smalles possible function and execute all kinds of scenarios against the function. Unit test can be very useful for assessing the robustness of code, since you can zoom in on a specific part and make all sorts of assessments on the function. You can focus on the happy flow, but just as important is the unhappy flow. On this level, on a specific part of your application, you're looking at the least complex moving part. The further you zoom out, the more complex the application (and therefore test cases) would have to be. Having robustness on this level, gives confidence on higher levels.

Mocks

Typically you want to rule out anything that's not part of the "unit" that you're testing, which means you will try to limit or in any case mock any external influences while conducting your test. For this reason, unit testing libraries offer the ability to inject or mock any properties or services that your unit would depend in. Think of APIs, Time/Date functions, Timing events, global variables etc.

Snapshots

When the output of a unit test contains a structure with a certain size, it can be useful to "snapshot" the outcome. By snapshotting, you run the testable function and store the output somewhere. The next time you run this snapshot, the test suite will assert whether the snapshots match. If not, it will highlight differences so you can assess whether it was a desirable change or not. Usually, you have the option to accept the new snapshot if it was indeed a desirable change.

Snapshots are advantageous when comparing relatively large sets of content and structure. At the same time, it's their drawback: with such a tight coupling to (usually) larger parts of a unit, it is more fragile and can sometimes obfuscate the origin of the change in the snapshot.

Visual snapshots

This applies to writing user interfaces. A visual snapshot is the representation of a certain state of a user interface component. This is close to what a user would see when interacting with your application. Visual snapshots are useful to detect changes the rendering of a component. Usually you are able to specify a rendering viewport (and engine) to emulate responsive behaviour.

Visual snapshots work very similar to regular snapshot testing: a rendering is saved as an image on first run. Successive runs will do a pixel to pixel comparison with the baseline and indicate any changes. There's usually a threshold that can be set, since rendering engines sometimes act differently at stages.

Visual snapshots have similar pros and cons to regular snapshots: they can easily identify higher level changes, but you'd have to manually verify them as being a desired change or a bug. Again, the origin can sometimes be tricky to pinpoint.

Component test

Going more into high fidelity tests: a component tests is used to emulate how a user would interact with your component. In this case you'd need to be able to scaffold out your component in a browser engine and define the interactions that the component should support.

Being able to focus on a single component allows you to validate not just the component, but also how generic (or reusable) the component is, but defining and executing different scenario's.

A component test will be able to identify user interface related bugs and the flow of different states. Test suites offer all sorts of browser mimicking APIs, such as resizing, scrolling, clicking, typing etc.

In this (and the next case) you want to be able to target specific elements (such as a submit button, password field), which you can achieve by adding data attributes to your component, which describe an HTML element within the context of the component.

End to End tests

Chaining different components together with actual customer journeys is what happens with end to end (e2e) tests. These are automated instructions that test the core features of your application by interacting with at as being a user. This is very closely related to the Component test, but on more practical implementation. You really want to test your code features here.

At this stage you are also (very probably) manipulating data, so this is something to take into account on the environment you run these tests on. You could either make sure to spin up a "fresh" environment every time you run these tests, or make sure you rollback any change you make to data, so that you keep a reliable dataset.

Synthetic tests

These are very similar to End to End tests in term of mimicking the customer journey. The main difference is that End to End tests run in a controlled environment: you predetermine the variables and the test itself runs typically in isolation. Synthetic test differ in that they are more closer to what a user would experience: they run in an actual browser (not a browser engine) and run in an exposed environment (production is no exception), which blurs the lines between testing and monitoring.

Performance test

This is kind of an outlier, since it mostly holds value on a production environment. Having performance tests (and a performance budget) can help you indicate issues that are not strictly related to blocks of code, more the connections between them. Since performance is reliant on several factors, it is harder to isolate and correlate changes in performance to changes in code.

It is however, from a users' point of view, one of the more important metrics that determine the success of your application.

Because of the lower coupling between deployments and performance metrics, this is one of the few tests that I'd recommend to run at scheduled intervals. Adding a performance step to a post deployment is not a bad idea (if you consider possible caching delays), but I would say this fall more into a monitoring category than specific testing.

Next up: Where are my tests?

Let's see at what point(s) it makes sense to add different types of tests. We'll go a bit over IDEs, CI/CD pipelines and monitoring in general.