Google Apps Incident Report Google Apps Gmail - May 11 through May 13, 2010 Prepared for Google Apps Customers The following is the incident report for the Gmail performance and availability issues experienced by some Google Apps customers between May 11 and 13, 2010. We understand that this service issue has affected our valued customers and their users, and we apologize for the impact. Issue Summary Starting at 7:02 AM PT | 15:02 GMT on Tuesday, May 11, approximately 5% of Gmail users intermittently experienced degraded performance and increased latency due to load balancing issues on the Gmail servers. The symptoms were most noticeable during peak traffic hours in the United States. They continued until 8:50 AM PT | 16:50 GMT on Thursday, May 13. No data was lost. Actions and Root Cause Analysis On Tuesday morning, May 11, Google began preparing for scheduled maintenance on a Gmail server cluster by transferring users from that cluster to other Gmail server clusters. The additional load on those other Gmail server clusters exposed shortcomings in the algorithms used for balancing the load between clusters. Some clusters became overloaded, resulting in the performance and availability issues. To resolve the immediate issue, the Engineering team made manual adjustments to the load balance until the scheduled maintenance was complete. The issues were completely resolved at 8:50 AM PT | 16:50 GMT when the updated Gmail server cluster began serving users. Corrective and Preventative Measures The Engineering team conducted an internal review and analysis, and is performing the following actions to help address the underlying causes of the problem and help prevent recurrence: • Improve the Gmail load balancing algorithm so that it better accounts for expected peak loads based on user location. • Evaluate the resource utilization rates for Gmail server clusters to ensure that they are able to handle increased loads efficiently. • Schedule large maintenance jobs for Gmail server clusters during non-peak hours, such as weekends. We appreciate your patience and again apologize for the impact to your organization. We thank you for your business and continued support during this time. Sincerely, The Google Apps Team
May 11, 2010 - Google Apps Gmail - May 11 through May 13, 2010. Prepared for Google ... for your business and continued support during this time. Sincerely,.
May 6, 2010 - Android is one of the largest users of the Calendar subscription service. ... Calendar was slow to respond to subscription requests, Android ...
Mar 16, 2010 - resources for Gmail routing and greatly increased the number of active Gmail routers. Following an internal investigation and analysis, the ...
Engineering was made of aware of the problem and promptly began to work to manage excessive traffic ... your business and continued support during this time.
A routing configuration file was released to production which incorrectly directed large volumes of global web traffic through ... traffic, resulting in timeouts and access delays to some Google services. The service ... continued support. Sincerely,
Feb 27, 2011 - Google Engineering reverted the storage software update, and halted ... better identify this class of bug during the software development cycle.
We understand this service issue has impacted our valued customers and users, and we apologize to everyone ... At 6:12 AM PDT, a bug in a thirdparty software update caused a partial failure of a Google network router in ... escalated the software iss
Mar 18, 2013 - service disruption was an issue in the network control software. Actions and Root Cause Analysis. At 6:09 AM PT, a portion of Google's network ...
This misconfiguration prevented changes to existing customer data for upgraded users. ... Eliminate the need for server restarts to recover from this type of error.
At 7:50 AM PT | 16:00 UTC November 15, Google Calendar Engineering brought a system of servers ... your business and continued support during this time.
Mar 17, 2014 - Issue Summary. From 8:35 AM to 12:10 PM PT, Google Talk, Google Hangouts (including Chat and Video), Google. Voice, and the Google App ...
Apr 17, 2013 - The following is the incident report for the Google services access ... Talk, Google Sync, the Admin panel, and the Cloud Console, and to a ...
Mar 19, 2013 - Applications using the Google Drive and Docs APIs also returned errors. ... We thank you for your business and continued support. Sincerely,.
Feb 27, 2011 - Google Engineering reverted the storage software update, and halted further deployment. Restoration Process. While analyzing the issue and its root cause, Google Engineering also worked on the process to restore users' accounts. At 6:0
Google Drive list. Applications using ... The Google Engineering team conducted an internal review and analysis of the March 21 event. They ... Modify the Drive software to more reliably serve user requests during short periods where overall.
Google Apps Incident Report. Google Docs Outage - September 7, 2011. Prepared for Google Apps for Business customers. The following is the incident report ...
Google Apps Incident Report. Gmail Outage - September 23, 2011. Prepared for Google Apps for Business customers. The following is the incident report for the ...
Apr 17, 2013 - The following is the incident report for the Google services access disruption that occurred on. April 17 ... Talk, Google Sync, the Admin panel, and the Cloud Console, and to a lesser extent Groups,. Sites, and ... misconfiguration oc
Google Docs Outage - September 7, 2011. Prepared for Google Apps for Business customers. The following is the incident report for the Google Docs access ...
The following is the incident report for the calendar performance and availability ... server issued numerous updates to the Google account management system.
