The Pitfalls of Traditional Strategies

Many engineers resort to various tactics that appear to resolve issues temporarily, only to see the problem resurface later in a more severe form. When this happens, the original engineer may try another workaround, or a different engineer might tackle it successfully.

Common misguided strategies include:

• NIH (Not Invented Here): A complete rewrite of a module simply because it wasn’t originally authored by the engineer.
• OaaS (Obfuscation as a Service): Making minimal changes while complicating the code unnecessarily to showcase proficiency.
• BTC (Blame the Customer): Dismissing the problem as the user's fault when initial checks yield no results.
• BTH (Blame the Hardware/OS): Pointing fingers at the underlying system when issues arise.
• CFCS (Change for Change's Sake): Refactoring code to improve aesthetics without addressing the core problem.

Transitioning to Professional Solutions

Unlike the previous strategies, a structured approach involves several key steps:

1. Analyze the Issue
2. Formulate a Hypothesis
3. Test the Hypothesis
4. Implement Changes
5. Validate the Solution

Step 1: Analyze the Issue — What Triggers the Problem?

The first step involves identifying the specific conditions and events leading to the issue. This can be challenging, as otherwise, it would have been detected during initial testing.

Gathering comprehensive evidence is crucial, including:

• Application logs
• Performance metrics
• Security logs
• Firewall activity
• Monitoring service feedback
• User observations during the incident

Assume the role of an investigative lead and meticulously collect all relevant information, especially from users who encountered the issue. A standardized set of questions can help the support team gather crucial data effectively.

Compile the information into a timeline to visualize the sequence of events, ruling out potential causes as you go. Attempt to replicate the live environment on a testing system, ensuring it mirrors the original conditions as closely as possible.

Exploring Causal Analysis Techniques

Analyze the collected data using methods such as:

• 5 Whys: Digging deeper by asking "Why?" five times.
• Ishikawa (Fishbone Diagram): Visualizing cause-and-effect relationships.
• FMEA (Failure Mode and Effects Analysis): Systematically identifying potential failure modes.
• Pareto Analysis: Applying the 80/20 rule to prioritize issues.

Step 1b: When the Issue Remains Elusive...

Sometimes, problems are sporadic and difficult to replicate. In such cases:

• Organize a brainstorming session with a diverse group of individuals to generate ideas.
• Document all suggestions, even the outlandish ones, as they may lead to breakthroughs later.

After the meeting, evaluate the feasibility and costs of the proposed experiments. Keep management informed about progress and potential expenses, especially for significant outages.

Step 2: Formulate a Hypothesis — Identifying the Problem

After thorough analysis, narrow down to one or a few likely causes and formulate a hypothesis.

Example Hypothesis: The website experiences slowdowns when two or more users simultaneously search on the advanced search page.

Step 3: Testing the Hypothesis — Intensifying the Problem

To validate your hypothesis, simulate conditions to see if the issue occurs under exaggerated scenarios. For example, test with 1,000 users performing complex searches simultaneously to increase the likelihood of observing the slowdown.

Step 4: Implementing the Solution

Once the hypothesis is confirmed, proceed to implement the software changes in the testing environment.

Step 5: Validating the Solution

Conduct tests in the exaggerated environment to ensure the changes effectively resolve the issue. If the system performs well under heavy load, you can have a high level of confidence in the solution.

Conclusion: A Structured Approach to Problem Solving

Understanding and replicating an issue is essential for effective resolution. By following a structured methodology, you can achieve reliable results and prevent recurring problems. Engineering solutions demands rigor, not guesswork, and every system, whether critical or recreational, deserves robust and carefully constructed software.

Further Reading