This significantly improves performance, since we already create the
elements when measuring them, and that effort is now no longer wasted.
We must take care to detach, rather than remove, when clearing the text
layer, so we can add the elements back later if necessary.