The importance of Static Testing in 2021
Hi there. Last time I have ended my article injection on this blog with an epic “I’ll be back!” – so here I am, keeping my promise with more testing goodies for you to unpack. In case you forgot: I am Victor, one of your favourite QA Engineers at EBS Integrator. Today I am going to lead the way through the importance of Static Testing in 2021. As it is with QA, static testing is quite overlooked or underestimated. It does not happen by chance: everyone is afraid of the unknown. To deliver a compelling case for ST, I will begin with some context delivery – to clear-out any muddy waters.
An introduction to initial testing requirements
Quality of a Service (further as QoS) is two main roles: Developers and Product Owners. By rule of thumb, developers ensure software meets technical specifications. At the other end, Product Owners ensure business objectives meet operational needs.
To reach the Valhalla pf QoS, both: developers and product owners alike; must perform tests. Developers ensure high-quality within software development processes and Product Owners guarantee scope achievement.
Alike in football: you have a Coach in charge of strategy; and a team captain (or the quarterback) that does the work. Their common goal is to reach that coveted result and for that, they must work together. As it is with any game, strategies must adhere to a specific set of rules:
Rule #1: Define and understand user needs
To “score”, you must solve a particular issue in the interest of your end-users. In our football analogy, these users would be your supporters.
Rule #2: Foresee critical business scenarios
Your strategy must align with the opponents’ profile, tenacity, and response capacity. Hence, that strategy must adhere to current circumstances and outline specific resolution scenarios.
Rule #3: Develop test scenarios for each business and risk outlook
A good coach will foresee needs and risks. A great one will plan resolution strategies by subjecting those to a trial run. Aka: will run tests in the first half time or based on a record of accomplishment for a particular opponent.
Rule #4:Assign designed assessment scenarios to different test cases
A “coach”, together with the “captain”, will probe their strategies in the first half-time. The “half-time” here representing any planning stage of a particular project. This makes sure everyone keeps course and there’s a chance to score yet again.
Got those main or drafting rules of a strategy design in place? Well, hold your horses. We still face yet another decision: choosing a test design technique.
Deliberating on evaluation techniques
You cannot play a game on the whiteboard, so it is high-noon to roll-up your sleeves and “get some action”. “The rolling” implies deciding what tool is the sharpest in your testing toolbox. //aka choosing a test design technique.
If you thought that is easy, let me put you on a diet. Designing a test case is a high-priority, high-tenacity task. Your end-result depends on it – so focus on what matters! A football coach will not check the colour of the opponent’s socks. This assessment would lead to inconclusive results for the game.
Go ahead – micro-manage and draw inconclusive tests or design them blindfolded. This will result in missing functionality. Not scary enough? Imagine placing a buggy deliverable in production. Now that’s the kind of stress to burn 2000 calories a minute… At large scales, this might be “the straw that breaks” your’ application is back. No camels here, we’re all in to avoid animal cruelty.
Let’s run a captain obvious: your goal is the release of a viable deliverable. That release should be bug-free, at least at the core of your business logic.
This is what you should focus on. In the wild, no one cares your button is 2px shorter as compared to your competitors. They want their main pain managed, not a wider button… Unless of course, button width is their pain.
Now, let’s end this “Development Horror Story” and see what you can do about it. At the core, to ensure QoS, you have the following choices in testing approaches:
Traditional static testing, also known as ”white-box”
This one is a design that relies on testing the inner workings of an app. The main focus is “the personality” of an app, rather than testing the apps’ functionality.
Dynamic testing, also known as “black-box”
Such design strategies focus on testing the functionality of an app. Like most pragmatic strategies, it could not care less about the applications’ internals.
The “Gray-Box” testing – the mid-mile
Not much to say here. Put those two together and you get the perfect combo. Lots of love for both: your application’s internals and its’ functionality.
Black-Box testing characteristics
Now, back to our football analogy. White-box strategies focus on left and right tackles to lead the entire game. Now that’s a gamble!
Here, you risk playing an offensive match. That means blocking any quarterback initiative and building-up disadvantage on your defensive positions.
Your functionality may be fine, but your system internals… Let’s say that application internals would resemble a messy “wonderland”. This means no or little documentation and no capacity assessment. That AWS bill might get as heavy as Jupiter.
Switching to the QA vocabulary. The dynamic, or “black-box” technique validates the functionality of an application. It does that by executing the code (aka playing defence).
Now, dynamic analysis is not focusing on syntax alone but considers state information. Look at it as “tackles” in football, the main role is to assess risk and intervene to protect their team members. In essence, you run “symbolic executions”. These allow checking assumptions about possible values of all variables and statements.
Roughly put: dynamic testing focuses on using the application. Its’ role is to check if a functionality works as designed. You feed data to your app, without accounting for comprehension – then repeat for n scenarios. This is usually the default way of conducting testing activities in most organizations.
White-Box testing characteristics
In football terms: if you design a white-box strategy, the main focus is the quarterback. After all, he yells the signals at the line of scrimmage. Keep in mind though, those actions (the internals at the development end) might not be enough. After all, the quarterback alone cannot play a game.
Now again, in regular terms. A white-box strategy is static. This means there are no changing conditions – hence your trust in the quarterback. It involves evaluating the code, the requirements, the design. Everything relies on documentation and/or artefact. There is no need to execute any deliverable.
The static analysis employs various formal methods. These resemble a holy trinity: abstract interpretation, model checking, and symbolic execution. In general, abstract interpretation or model checking is suitable for software verification. Symbolic execution is more appropriate for bug finding.
You could argue that compilers are getting better at analysing mistake patterns. Wouldn’t that be enough at the development end? Well, the answer is no. Without ST the “depth” that they work to is a lot less than proper tools for this. \
Let us imagine you need QoS across several units. When using ST tools, you can understand if a function may return (for instance) a NULL pointer. If it does, your code will crash because you do not check for NULL values before accessing that pointer. Static analysis tools can also do checking for lock usage. This is something a compiler fails at.
Gray Box testing characteristics
A game engaging the entire team within attack/defence strategies is a golden mix. The tricky thing here is playing it right. The Quarterbacks’ talent, the coaches’ experience, the unity of your team – all must converge.
This means you are finally bringing all the aspects of your game. You play offensive and defensive positions, gaining a higher score chance. All this – to win that match. There is more to the “Gray-box” story. But… Since this piece explores static testing, I will leave this to an eventual article. I may be “supervising” my certified junior QA. She is up to that task.
Back to “white” – or why Static Testing is crucial
Ok. We covered most of the testing ground, let us jump into it. The best approach to build a case for ST is by checking its’ efficiency. As with anything related to software development, efficiency relies on:
- how skilled your devs and QAs are;
- which tools & setting you are employing in the QC process; and
- the general quality of your business goals, objectives, or strategies.
We should also consider how much of the codebase your engineers can cover (in matters of testing ). It is unrealistic to test each code branch. Well, it is, but only if you got lucky with your investors. Despite their level, humans are, alas, humans. Some aspects of your code might slip away and leave your defences naked. This might leave your app open to both: availability as well as security issues.
Proper deployment of specific ST tools is the right way to go. This will ensure proactive issue detection. That means those invisible issues to QA engineers (or testing coverage) will emerge.
5 Gains delivered by Static Testing and basic mechanics
Yes – QA, in general, is expensive, but Crucial in most of the cases. This is why Business owners are oblivious to gains delivered by this practice. For them, this is an additional expense – all until the sewage hits the fan…
Foremost, the Crown Jewel of static testing is spotting defects and anomalies. Before hiding your wallets, debit cards or your offshore accounts, let’s explore cost. For as little as 30% of your development budget (a rough figure), you can detect:
Such “undesirables” can cost. I mean, you could spend about 50% (if you are lucky) of your initial development budget as technical debt. Yes; this leads to technical debt.
Particularly because Business Owners skip on QA to leverage development speed. Here, BOs should understand that any eventual bug fixing and refactoring is billable. And that’s a default. Usually, you would hear excuses like:
“We don’t have time for testing” or “I’d rather spend that budget on development.”
Well then, when the time comes, get ready to pay the piper. The alternative is a nifty investment Static Testing, especially if your dev relationships risk going south (Ahem… When freelancing, it happens all the time).
Deviations from coding standards
If you want to ensure longevity to your code-base, you must always adhere to a coding standard. That will make your base readable to various developers. Otherwise, you will be stuck with one person, and one person only to maintain your app. (Ahem… That lone freelancer that disappeared for 5 weeks). In Essence, you are jumping right back to the above issue. Static testing will not only point at copy-pasting. Besides spotting bad practices, you get the right idea regards dev seniority levels.
Nobody is perfect! Well, at EBS we are pretty flipping close, but still… Use static testing to spot any Architecture defects early. This is again a way to reduce your technical debt. If you are running a project that needs pivoting – ST is a must.
This is useful when running a scrum development model and you need a hold on change management. If something is not there you will be aware of it. This is how you avoid trashing half of your sprint due to broken dependencies.
Inconsistent interface specifications
As in the case of spotting missing requirements, inconsistent UI can limit scaling. Again, this issue contributes to a hefty technical debt and boy, you will not get out of that expense. Remember: it will never be fast, cheap, and good – you must pick two.
As a bonus to those big five, static testing will spot the most obvious candidate issues. Security vulnerabilities, undeclared variables, boundary violations, syntax violations, inconsistent interfaces, cannot stand a chance to ST. Then again, by focusing on those “top 5” gains, you will cover these as well.
Clift notes for Static Testing + one heck of a dramatic exit
I am no lawyer, but I believe everything laid down before your eyes today, builds a compelling case for employing Static Testing. Just to give you a final idea of how crucial this is, think about these consequences of ignoring QA and ST:
- 1985: because of concurrent programming errors (also known as race conditions race condition errors) in the inadequately tested Therac-25 software, 7 cancer patients died in Canada, due to overdoses of radiation.
- 2004: a new Canadian welfare management system, costing several hundred million dollars, was unable to handle a simple benefits rate increase because its system did not pass any tests for such functionality.
- 2004, USA: a software bug was determined to be a major contributor to the 2003 Northeast blackout, the worst power system failure in North American history. The failure involved loss of electrical power to 50 million customers, forced shutdown of 100 power plants, and economic losses estimated at $6 billion.
- September 28, 2020: 911 services went down in multiple US states. States like Arizona, California, Colorado, Delaware, Florida, Illinois, Indiana, Minnesota, Nevada, North Carolina, North Dakota, Ohio, Pennsylvania, and Washington, as well as many other police departments in counties across the US reported congestion and multiple communications outages. Those Systems became useless – and left police departments turning to social media to tell residents there were other ways to contact emergency services.
The list can go on. If you are deploying a high demand service, static testing, as well as other versed QA approaches are “a must” – especially when technical debt is less of a number and more of a life and death risk.
I will not say goodbye just yet. My next heist attempt at this blog will feature a deep dive into the process of dynamic testing.