What separates a “shaggy dog” story from one that entices a person to listen is whether or not it goes in a straight line.

Too many “side references” stall the story out and put it in neutral as far as the plot goes.  In real life conversations, people may listen.  But on the air, a minute is a long time.  (Want to argue the point?  Hold your breath for one minute, starting right… now.)

There’s also the danger of sounding like you’re interrupting your own thought.  This is a weird habit because at that point you’re just trying to sort out in your own mind how to tell the story.

I don’t have time to listen to that.

You need to be prepared and make the story march at a decent pace, but also be mindful of when a pause is needed.  We do want to sound as natural as possible, but we owe the listener being expedient, too.