I had a reminder last night of why back-out plans are important. We were 5 hours into a late night outage scheduled for 6 hours. Things were going reasonably well on about half of the cut-over. The other half...not so well. These types of scenarios aren't ideal but they do happen.

The important thing that saved us in this scenario was a clear understanding of the backout plan. In retrospect there are definitely improvements that could have been made to our plan but generally it went fairly well.

The last thing that you want to do at 2am is try to come up with a complex rollback plan. Have one ready!

The other important part of this scenario is that we actually made the decision to roll back. It is completely plausible that we could have figured out our problems last night but it is even more likely that, since we were all getting tired, we would have only caused more issues. I'm not pleased that we had to roll back many of our changes but I am happy that I made the decision.

Have a plan. Use your plan!