In order to ensure pages load as quickly as possible, I've ditched external Digg Javascript and now use the Digg API to keep digg counts up to date.
The buttons look a little different (better, I think), work in much the same way, but don't hinder page loads.
More info in the (new) HEXUS devblog.