Google Apps Incident Report Google Docs March 19, 2013 Prepared for Google Apps customers
The following is the incident report for the Google Drive access disruption that occurred on March 19, 2013. We understand this service issue has impacted our valued customers and users, and we apologize to everyone who was affected. Issue Summary
For 9:00 AM to 9:35 AM PT, some users experienced “Server Error 503” messages, long load times, or timeouts when trying to access Google Drive. Applications using the Google Drive and Docs APIs also returned errors. The issue affected up to 25% of all user requests to Google Drive during this period. Users could continue to access individual Drive files by direct link or URL. The root cause of this service disruption was an issue in the software that manages user connections and sessions with Google Drive. Actions and Root Cause Analysis
Note: The cause of this incident is the same as the Drive incident of March 18, 2013. Corrective actions were underway with the Google Drive team when the incident occurred. On March 19, a routine maintenance event temporarily reduced the available server capacity for displaying the Google Drive interface. This resulted in a small increase in processing latency, which does not normally affect the user experience. However, the latency did trigger a bug in the software that manages user connections and sessions with Google Drive. This resulted in errors and timeouts for some users who were attempting to access Google Drive. Corrective and Preventative Measures
The Google Engineering team conducted an internal review and analysis of the March 19 event. They are taking the following preliminary actions to address the underlying causes of the issue and to help prevent recurrence. Some of these actions are also described in the March 18 event. ● ● ● ●
Fix the bug within Drive and change internal structures and resources to make Drive far more resilient to latency and errors. Improve the Drive alert systems and expand monitoring of Drive systems for faster detection of issues. Accelerate the work in progress that ensures user traffic for Drive is properly prioritized during network events. Increase the capacity of the systems that serve Drive requests well beyond peak demand estimates.
Google is committed to continually and quickly improving our technology and operational processes to prevent service disruptions. We appreciate your patience and again apologize for the impact to your organization. We thank you for your business and continued support. Sincerely, The Google Apps Team
Mar 19, 2013 - Applications using the Google Drive and Docs APIs also returned errors. ... We thank you for your business and continued support. Sincerely,.
This misconfiguration prevented changes to existing customer data for upgraded users. ... Eliminate the need for server restarts to recover from this type of error.
At 7:50 AM PT | 16:00 UTC November 15, Google Calendar Engineering brought a system of servers ... your business and continued support during this time.
Feb 27, 2011 - Google Engineering reverted the storage software update, and halted ... better identify this class of bug during the software development cycle.
We understand this service issue has impacted our valued customers and users, and we apologize to everyone ... At 6:12 AM PDT, a bug in a thirdparty software update caused a partial failure of a Google network router in ... escalated the software iss
Mar 18, 2013 - service disruption was an issue in the network control software. Actions and Root Cause Analysis. At 6:09 AM PT, a portion of Google's network ...
Mar 17, 2014 - Issue Summary. From 8:35 AM to 12:10 PM PT, Google Talk, Google Hangouts (including Chat and Video), Google. Voice, and the Google App ...
Apr 17, 2013 - The following is the incident report for the Google services access ... Talk, Google Sync, the Admin panel, and the Cloud Console, and to a ...
Feb 27, 2011 - Google Engineering reverted the storage software update, and halted further deployment. Restoration Process. While analyzing the issue and its root cause, Google Engineering also worked on the process to restore users' accounts. At 6:0
Google Drive list. Applications using ... The Google Engineering team conducted an internal review and analysis of the March 21 event. They ... Modify the Drive software to more reliably serve user requests during short periods where overall.
Google Apps Incident Report. Google Docs Outage - September 7, 2011. Prepared for Google Apps for Business customers. The following is the incident report ...
Google Apps Incident Report. Gmail Outage - September 23, 2011. Prepared for Google Apps for Business customers. The following is the incident report for the ...
Apr 17, 2013 - The following is the incident report for the Google services access disruption that occurred on. April 17 ... Talk, Google Sync, the Admin panel, and the Cloud Console, and to a lesser extent Groups,. Sites, and ... misconfiguration oc
Google Docs Outage - September 7, 2011. Prepared for Google Apps for Business customers. The following is the incident report for the Google Docs access ...
Sep 1, 2009 - On Tuesday, September 1, a small portion of Gmail's web capacity was taken ... request routing automatically directs users' requests to available servers. ... Over the next few weeks, we are dedicated to implementing these ...
Sep 25, 2009 - Between 7:00 AM - 9:50 PDT | 14:00 - 16:50 GMT, Thursday September 24, Google Apps users were unable to access the Contacts feature through the Gmail interface. However, they could view their contacts at an alternate URL. During this p
Sep 25, 2009 - Prepared for Google Apps Premier Edition Customers. Incident ... add users to their Google Apps accounts. ... business and continued support.
Sep 1, 2009 - server. Gmail processing and access through the IMAP/POP interfaces ... Over the next few weeks, we are dedicated to implementing these ...
Mar 16, 2010 - resources for Gmail routing and greatly increased the number of active Gmail routers. Following an internal investigation and analysis, the ...
Engineering was made of aware of the problem and promptly began to work to manage excessive traffic ... your business and continued support during this time.
from the Google Engineering team traced this problem to new code introduced at 3:00 PM PDT |. 22:00 UTC August 19. The Google Engineering team repaired ...
Feb 24, 2009 - The root cause of the problem was a software bug that caused an ... we monitor our systems 24 x 7, we have engineers available to analyze.