User blog comment:Crucially/Explanation of the Sunday outage/@comment-192.68.211.161-20091113104940

I agree with anon. Seperate managment networks, serial console access to all vital hardware, etc etc.

That and, as you found out yourselves, Spanning Tree considered harmful. You need to seperate Layer 2 physically (where VLANs count as physical ;). Also, disable automagic VLAN configuration as that is a larger and more complex valley of pain than spanning tree.