This answer is going to be based on the assumption that you are versioning your dlls.
If you set SpecificVersion to true (which is the default when adding a reference), then the project will reference to that dll with a particular version (say for instance 1.0.0.0). If, at a later time, you're given a new dll (say 1.0.1.0), then you will have to remove the old dll reference and add the new reference. This is because the project is specifically looking for 1.0.0.0 when you have a new version 1.0.1.0.
The alternative to this is to set the SpecificVersion to false, which tells the project to find the latest available dll and use that one. The problem with this is that the project is now required to "hunt" in various places for the dll you've referenced, which can increase your build time. It will do this even though it knows the path of the dll you've referenced. I'm not sure if this is a bug or if this is done by design. It might be checking to see if there are any newer dlls besides the one you've referenced (perhaps in the GAC or elsewhere).
Here's an article that describes this issue in more detail.
与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…