Google Apps Incident Report Google Calendar - November 15, 2010 Prepared for Google Apps Customers

The following is the incident report for the Google Calendar access issues experienced by some Google Apps customers on November 15, 2010. We understand that this service issue has affected our valued customers and their users, and we apologize for the impact. Issue Summary On November 15, a performance issue with the Google Calendar servers caused slow response, timeouts, and browser errors for the affected Google Calendar users. Actions and Root Cause Analysis At 7:50 AM PT | 16:00 UTC November 15, Google Calendar Engineering brought a system of servers online too quickly during peak traffic, which overloaded some web servers and led to Calendar processes that would start, receive an error, stop, and restart again. The resulting spike in resource consumption caused temporary browser errors and timeouts for some users trying to access Google Calendar. Within minutes, our monitoring systems detected the event and alerted Engineering. To resolve the issue, Google Calendar Engineering took the affected servers offline, and redirected incoming requests to other servers. By 9:43 AM PT | 17:43 UTC November 15, access to Google Calendar returned to normal. No existing Calendar data was affected during this event. Corrective and Preventative Measures The Engineering team conducted an internal review and analysis, and performed the following actions to help address the underlying causes of the problem and help prevent recurrence: ● ●

Google Engineering has implemented guidelines to enable new server capacity incrementally during peak traffic periods Google Engineering has updated internal training and documentation for bringing servers online.

We appreciate your patience and again apologize for the impact to your organization. We thank you for your business and continued support during this time. Sincerely, The Google Apps Team