We do estimation to serve a purpose, and that purpose is that it helps manage expectations. If we are managing expectations successfully then we don’t really need to worry about any errors in our estimation.
To give an example: Consider a team which regularly plans and estimates stories ahead of its iteration, and it always chooses just what it thinks it will deliver. If the team’s small stories consistently turn out to be difficult, and its big stories consistently turn out to be trivial, and they end up cancelling each other out so that all the planned stories end up being delivered every time, then it’s fine. The estimation of each story is clearly poor, but in the end the team is consistently delivering what it promised, and that’s what matters.
Of course, this example is artificial. In the real world we do need techniques that will help us estimate better. One important technique is to parcel up the work into stories that are as small as possible, and only estimating them then. This decreases error. Another is to not worry about estimating stories as long as they are below a certain size or complexity. Assessing that they are small enough is still an act of estimation, but it’s less granular than the previous approach.
As long as we can meet our stakeholders’ expectations then the mechanics of doing that are secondary.