On the dangers of subjectivity in your analysis

In recent analysis sessions at a local university, the worrying influence of human judgement in the reproducibility of results was made very clear. Students completing an exercise had shown great consistency in their analysis results using Progenesis SameSpots, right up to the point where they had to make a judgement call. And what was the decision that caused such divergence?

“Will this spot yield a reliable protein identification, if picked?”

The fact that SameSpots supported such reproducible, objective analysis up to this point was great to see. In fact, a large percentage of those taking part had managed to analyse their images 100% in accordance with our own expert’s analysis. This article, however, aims to support you better in making the final step of selecting your picking spots from the list of interesting spots found in your analysis.

We’ll start by looking at the advice sheet given to the students who took part in our exercise. The examples it gives are exaggerated to illustrate problems clearly, but the types of problem are not unusual:

The following table gives a few examples of why a given spot might not produce a reliable protein identification:

Reason for rejection Example
The outline appears to surround more than one spot in the image.
In this situation, the measurements we used to tag our spots could be unreliable.
The spot has been poorly aligned in one or more samples.
This could be fixed by manually adjusting the image’s alignment, but as it stands, the measurements will not be reliable.
The spot is small and poorly defined in all conditions.
This is likely to correspond to a misdetection in the software caused by foreign material on the gel or poor image capture. There’ll be no protein to identify.

As mentioned above, the problems shown are deliberately exaggerated; the case for rejecting the spot is clear. Often, however, it’s not so clear and even 2 experts may disagree over the reliability of a given spot. Where does one draw the line? What is an acceptable tolerance for alignment? What about for intensity? And how can you be sure that’s really 2 spots and not just an uneven running of the second dimension?

So what can you do to reduce the subjectivity here? First and foremost is to make the most of the tools available to you in the software. Secondly, take a quantitative rather than qualitative approach. In Progenesis SameSpots, this means:

Use the controls in the Filtering step to remove very small spots
This is a measure that can be made completely objective within your lab. Choose an area value and use it in your standard analysis protocol. The value itself may be based on the corresponding size of your robot’s picking head, the smallest spot size you can reliably cut by hand, or either of these plus a margin area (to account for the protein’s density profile across the spot area).
Use the contrast control
Sometimes, your spots may not stand out clearly against background, when judged by eye. This often illustrates the strength of robust statistical measures (being able to identify meaningful differences that can’t be done by eye alone), but there’s still a limit to how little protein material can be used to get a reliable identification. For a quantitative measure, define the contrast level at which spots must be readily apparent, and also the number of gels in which it must be evident.
Use the 3D view to confirm merged proteins
Where a spot outline suggests it may actually consist of two partially-separated proteins, the 3D view can be an invaluable tool to confirm this. In the View Results screen, select the 3D Montage tab. The 3D spot image can then be moved around using the mouse. If you can clearly see a valley inside the spot boundary, this may well be a pair of merged proteins. The Peak Scale option can help clarify things further.
Before investigating in the 3D view:
After rotating and increasing the peak scaling view option:
As with the contrast level, you may wish to define the number of gels in which the valley is present before rejection as part of your standard analysis protocol.

Of course, these are just some of the things you can do to address the issue of subjectivity-based variation. Here at Nonlinear, we’ll continue to look for further ways that we can eliminate it in the software itself. That may, ultimately, be an unreachable goal — there’s always another new situation to cater for — but we remain committed to proving that proteomics can work, reliably and repeatably.

In summary

The questions raised when considering a spot’s picking potential are not always simple to answer, but it’s very clear that you need to be aware of the issues. It could mean the difference between identifying a key protein and missing it entirely. For our part, here at Nonlinear, with our analysis sessions’ confirmation of the serious implications of using human judgement in your analysis, we’ll be refocusing our development efforts on increasing objectivity and eliminating subjectivity.

If the results from your current software or analysis service still rely heavily on subjective decisions — and the provider’s not addressing the situation — you may be left wondering why.

One Trackback

  1. […] the reproducibility of your 2D gel running In my last blog post, I talked about the need to minimise subjectivity in your analysis. It’s an area of great importance, as recognised by the Fixing Proteomics […]