Let's Talk Statistics - Introduction to Statistics for Software Quality (No.11 Confidence Intervals and Errors: How Much Can We Trust These Results?)

“Let’s Talk Statistics” installment 11 explains the fundamental concepts of inferential statistics—confidence intervals and standard errors—from a practical perspective on how to apply them to reports and quality assessments.

In software quality contexts, quantifying uncertainty with confidence intervals and Z-scores is becoming increasingly important for answering the question, “How much can we trust the results?”

As mentioned in the 8th installment, let’s review briefly.
Understanding the difference between “probability theory” and “statistical inference” is crucial for grasping “confidence intervals” and “estimation.”

Probability theory is the foundation of statistical inference.
Statistical inference is the tool that uses probability theory to draw conclusions from “real data.”

By understanding this difference, your perspective on “how to infer a population from a sample” improves dramatically.

So far, we’ve extracted representative values such as “mean” and “proportion” from quality data to infer the overall situation.
This is nothing more than the act of estimating the population from a limited sample.

This kind of “reading the population from a sample” is called statistical inference.

But you might wonder:

“How accurate is that average?”
“If we measure it again, there’s no guarantee we’ll get the same value, right?”

Statistical inference always involves error.
The expression “mean ± value” quantitatively shows the magnitude of that error.

For example—
if it’s written “the defect rate is 3.2 ± 0.4%,”
what does the “± 0.4” mean?

In short, it indicates the error.
But this “error” is not vague; it is quantitatively defined by the standard error (SE) and the width of the confidence interval (CI).

In other words, it numerically shows the degree of uncertainty, i.e., how much the estimate (3.2%) might fluctuate.

Although the sample mean tends to be close to the population mean, the standard error quantitatively measures how much it fluctuates.

$$SE = \frac{\sigma}{\sqrt{n}}$$

• σ: population standard deviation (spread of the distribution)
• n: sample size

The standard error is the “margin of error of the mean,” and as the sample size increases, SE decreases.
In other words, the more data you have, the more accurate the estimate.

As you can see from SE = σ/√n, as the sample size n increases, the denominator grows, and the standard error (the margin of variation of the mean) becomes smaller.

To understand this visually, let’s compare the “spread of the sampling distribution of the mean” as n changes. The figure below shows the spread for n = 5, 10, 30, and 100.

• When n = 5: the distribution is wide with large variability (large standard error)
• When n = 10: variability is still fairly large, but it starts to concentrate toward the center compared to n = 5
• When n = 30: the distribution becomes narrower and approximates a normal distribution (effect of the Central Limit Theorem)
• When n = 100: the distribution is narrow and peaked with small variability (small standard error)

Thus, the property “the more observations, the more stable the mean” is very important for understanding why confidence intervals narrow.

For example, if someone says, “The 95% confidence interval for this defect rate is 3.2% to 4.8%,” it’s tempting to interpret that as “there’s a 95% probability the true value lies within this range.”
But more precisely, it means:
“If you repeat the sampling and estimation using this method many times, 95% of the resulting intervals will contain the true value.”

In other words, a confidence interval shows only the range of uncertainty in the estimation, not that “a given interval has a 95% probability of containing the true value.”

So, how do we calculate that “range of uncertainty (= confidence interval)?” This is where the Z-score comes in.

A Z-score measures how many standard errors the sample mean is away from the population mean and is widely used as a reference for probabilities in the standard normal distribution.

Using the Z-score, the confidence interval is calculated as:

$$\text{Confidence Interval} = \bar{x} \pm Z \cdot SE$$

• x̄: sample mean
• Z: Z-score corresponding to the confidence level (95% → 1.96, 99% → 2.58, etc.)
• SE: standard error (margin of variability of the mean)

This formula has the structure “estimate ± margin of variability.”

For example:

• Sample mean defect rate = 4.0%
• Standard error = 0.5%
• Confidence level 95% (Z-score = 1.96)

The confidence interval is:

$$4.0\% \pm 1.96 \times 0.5\% = 4.0\% \pm 0.98\%$$

We obtain the confidence interval as “mean ± Z × standard error,” but how should we interpret this result (e.g., 3.02%–4.98%)?

It means:
“If you perform this method 100 times, in 95 of those cases the true defect rate will fall within this interval (3.02%–4.98% in the example above).”
It does not mean that “the true value itself has a 95% probability of lying within this interval.”

This difference is subtle because people tend to confuse “probability” with “reliability.”

Here, we assume “the true defect rate (population parameter) is fixed at a single value.”
The intervals obtained by estimation will vary by sample, but 95% of those intervals will “by chance” contain the true value.

That is—

The “interval” changes (varies)
The true value does not change (it is fixed)

This is the premise.

Statistically meaningful “estimation” differs from merely reporting averages; it’s important to convey the variability and uncertainty.

• Add “±” to the average time required for reviews to create a “prediction interval.”
• Show a confidence interval for test success rates to make decisions that account for variability.

For example, saying “The test success rate for this version is 91.3% ± 2.4% (95% confidence interval)” allows you to interpret it as “at worst 88.9%, at best 93.7%.”

Typically, a 95% confidence interval is used, but you can choose according to the situation:

• Specify not only “± how much” but also what confidence level it corresponds to.
• For particularly critical decisions, consider a 99% confidence interval, adjusting as needed.
• In business settings, rephrasing as “at worst this much, at best this much” tends to be easier to understand.

A confidence interval is the range expected to contain the true value in 95% of repeated studies. In classical (frequentist) statistics, the true value (population parameter) is a fixed single value, assumed to exist somewhere. What behaves probabilistically is the “estimator” (the sample mean or the interval). A confidence interval means “if you repeat this method many times, XX% of the intervals obtained will contain the true value.”

In other words, a “95% confidence interval” does not mean there is a 95% probability that the true value is within it; rather, it asserts that 95% of such intervals would contain the true value. Since the true value is always fixed, there are only two possibilities: it either lies inside the interval or it does not.

Even a small difference can become statistically significant, but it may be negligible in terms of quality.

Even if you can say there is a “statistically significant difference,” it does not necessarily mean there is a “difference that matters in practice.”

For example:

Suppose you compare the launch times of an app under Plan A and Plan B. After collecting sufficient data and performing a statistical test, the results were:

• Average launch time for A: 2.10 seconds
• Average launch time for B: 2.04 seconds

The difference is 0.06 seconds. Because the sample size was large, this difference was statistically significant (p < 0.01).

However, in practice, a user cannot perceive a difference of 0.06 seconds. If adopting Plan B complicates the implementation or design, you might decide that “we don’t need to adopt Plan B and can ignore this difference.”

Lessons:

• “Statistically significant” ≠ “meaningful difference”
• Statistical significance only indicates that the result is unlikely to be due to chance

→ Whether the difference is important should be considered together with context, costs, and on-site judgment.

Moreover:

• If the sample size is large, even a very small difference can become statistically significant.
• Conversely, if the sample size is small, a large difference may not be statistically significant.

→ Statistics is a tool. To draw meaningful conclusions, it’s important to use it in conjunction with human judgment.

In this article, we explained how to judge “how much you can trust the results” from a practical perspective, based on the fundamental concepts of inferential statistics such as confidence intervals, standard errors, and Z-scores.

• What is a confidence interval?

  • An interval that indicates “how much uncertainty there is” for a single estimation
  • It does not mean “there is a 95% probability that the true value is in it,” but “if this method is repeated, 95% of the intervals will contain the true value”

• What is the standard error?

  • A measure of how much the mean fluctuates, which decreases as the number of data points increases
  • Note the formula SE = σ/√n

• Why can we make estimates?

  • Central Limit Theorem → the distribution of the sample mean approaches normal
  • Law of Large Numbers → the sample mean converges to the population mean

• How should we interpret confidence intervals and test results?

  • Even if there is statistical significance, it may not have practical relevance
  • It is important not to trust them excessively but to treat them as decisions that include uncertainty

Next time, we’ll talk about hypothesis testing.
We’ll introduce the logic for discerning what it means to have a “difference”—whether it’s due to chance or truly real.

We’ve compiled related statistical information here.

We hope you find it useful for your data analysis.