Does significant actually mean small? Bear with me here.
X caused a significant increase in Y
When a politician says something will lead to a “significant increase” or is a “significant problem” they are trying to emphasise the scale or importance of the issue. Of course, they are doing so in a way that is subjective and can’t be proven to be wrong. It is meaningless in this context but is also almost certainly essential as you wouldn’t spend money on something that isn’t “significant”. Everything is significant to someone! The intention of the speaker definitely isn’t for significant to mean small.
However, when a scientist says “significant increase” they are commenting on the confidence that can be placed in the observation. It’s a singe word to establish that there is low likelihood that the observation is a false positive (type I error). It is a binary word. A result is either statistically significant or not significant. It isn’t really subjective per se as the significance threshold is defined before the study and explicitly stated elsewhere in the document. Of course, a scientist might also use significant as a general adjective to mean important of meaningful. We’re already ambiguous but could it be even worse; has the writer chosen to include significant in the sentence because the difference is small?
When does significant mean small?
This argument centres on two fundamental principles:
- Your message should be unambiguous
- Sentence structure should provide value to your reader
The first point is clear. You should edit your writing to remove any potential ambiguity, Here writing “statistically significant” or “biologically significant” would be clearer. In results sections this may not be necessary as a reader will assume its stats, but elsewhere you use the label to be sure. Easy. However, now we move on to point 2…
When you state “statistically significant” (or imply the statistical part), then you are explicitly and deliberately emphasising that your observations have reached your pre-defined confidence level.
At this point you might argue that your aren’t emphasising confidence, that you are just stating the results. However, in science writing, these sorts of statement are always accompanied with the descriptive and inferential statistics either in a table or in parenthesis (mean+SD group A, mean +SD group B, p<0.05). Because the inferential statistics are reported there is no need to make any additional comment on the confidence level within the sentence itself. If there is no requirement to repeat the statistical assertion, then it must be a choice. If you are making this choice, then it must be for emphasis to provide value to the reader.
This naturally leads to the follow-on question: when does emphasising confidence add value? There are two primary reasons i) the spread of the data is large but, despite that spread, you are be confident that the effect you have measured is real OR ii) the difference is small but as the spread is small you are confident that the observed effect is real. In option ii, significant means “small but real”
A parallel scenario would be when you start a story by saying “True story…”. The phrase that followed would not be something obvious, “True story, I am wearing a jumper”, but rather you would only use it when you thought it possible that the listener wouldn’t believe you; “True story, I made this jumper from wild sheep wool while lost in the Highlands”. Significant is the same idea, you only need to include it when it is needed!
Just because something is statistically significant, it doesn’t mean it is biologically meaningful!
What about not significant?
If you aren’t sold yet on significant meaning “small but confident” or “spread but confident”, then consider the opposite scenario.
A non-significant increase in Y was observed (compare this to “no difference”)
Here the non-significant emphasis is clear. You are explicitly telling the reader that you cannot be confident about the observation. If your observations have not reached the pre-defined threshold for significance but you still want to report them then you should definitely say non-significant, you cannot pick and choose like with significant.
You might have been told that you can only report statistically significant results. However, in discovery projects, it is quite common that you will not have powered your experiment based on a particular observation. The non significant phrase might expand to: “although we can not be confident in these pilot observations, if they stand up to subsequent hypothesis-testing analyses then they would be biologically important”. The decision on how to report your observations should be based on value. The phrasing and discussion of those findings should reflect the confidence.
An absence of confidence is not an absence of difference. To make confidence statements about a lack of difference you need to test for false negatives (type 2 errors).
What words should we use?
Good news, there is no need to have any ambiguity. There are lots of more precise adjectives available. Here’s a few:
To mean “large” [include the actual magnitude in the phase]
- large (simple and unambiguous)
- pronounced
- major
- substantial (if you can count the thing)
- substantive (if the thing is uncountable)
- considerable
To mean “important” [much more subjective, you should define why it is important]
- important (sometimes simple is best)
- valuable
- meaningful
- material
- concrete
- tangible
- consequential
- impressive
To mean “confident“
- real
- definite
- definitive
Note that you can combine for added emphasis “a large, important and definitive effect”. That might be overkill,
So…. stop saying significant unless you actually want to emphasise small but likely to be real!
This was part 2 of our new “editing for impact” series. Part 1 dealt with emphasis and sentence structure. Link below. Also look our for part 3 coming soon.
You might also like these webpages:
LaNts and Laminins 
6 Comments