Google Apps Incident Report Google Services ­ July 10, 2013 Prepared for Google Apps customers

The following is the incident report for the Google services disruption that occurred on July 10, 2013.  We understand this service issue has impacted our valued customers and users, and we apologize to everyone who was affected. Issue Summary

From 6:12 AM to 7:37 AM PDT, a fraction of users in West Virginia, North Carolina, Nebraska, and Georgia experienced errors when trying to access many of Google services, including Search, Gmail, Google Drive, Google Calendar, Google Groups, Google Sites, and the Google Admin console. During this time, between 0.2% and 0.5% of Google’s traffic was affected. The root cause of this service disruption was a novel failure of a network router. Actions and Root Cause Analysis

Background: Routers take incoming traffic, analyze the information, and then direct users requests to other Google systems for processing and serving content, such as generating search results and accessing Gmail. At 6:12 AM PDT, a bug in a third­party software update caused a partial failure of a Google network router in the Atlanta region. Normally when a network router fails, it alerts Google Engineering of the problem, and redundant network devices quickly reroute traffic. This novel bug caused the router to incorrectly identify a significant portion of traffic as invalid and discard it, but not treat this behavior as a failure. As the result, the router silently dropped traffic for some users in specific areas of the United States, and the affected users found their access to Google services limited or unavailable. Google’s monitoring systems detected the outage at 7:03 AM PDT, and Google Engineering immediately began to investigate. The Engineering team determined the source of the problem, escalated the software issue with the third­party vendor, and removed the router from production at approximately 7:35 AM PDT. Google networking devices then routed traffic around the failure, and the affected users’ access to services was restored within minutes. The third­party vendor continued to research the software bug, and provided a root cause and workaround later in the day. Corrective and Preventative Measures

The Google Engineering team conducted an internal review and analysis of the June 10 event. They are taking the following preliminary actions to address the underlying causes of the issue and to help prevent recurrence: ● ●

Implement tests to detect this mode of router failure during the testing process for software updates and configuration changes. Investigate methods for adding alerts for more quickly detecting a decrease in traffic, and for differentiating unintended traffic drops from intended discards, for example of denial­of­service traffic.



Improve network monitoring for early detection of this type of issue, including monitoring router performance during upgrades and updates, and increasing the scope of monitoring across interconnected network devices.

Corrective and Preventative Measures

Google is committed to continually and quickly improving our technology and operational processes to prevent service disruptions. We appreciate your patience and again apologize for the impact to your organization. We thank you for your business and continued support. Sincerely, The Google Apps Team

Google Apps Incident Report

We understand this service issue has impacted our valued customers and users, and we apologize to everyone ... At 6:12 AM PDT, a bug in a thirdparty software update caused a partial failure of a Google network router in ... escalated the software issue with the thirdparty vendor, and removed the router from production at.

106KB Sizes 1 Downloads 289 Views

Recommend Documents

Google Apps Incident Report
This misconfiguration prevented changes to existing customer data for upgraded users. ... Eliminate the need for server restarts to recover from this type of error.

Google Apps Incident Report
At 7:50 AM PT | 16:00 UTC November 15, Google Calendar Engineering brought a system of servers ... your business and continued support during this time.

Google Apps Incident Report
Nov 15, 2010 - Prepared for Google Apps Customers ... Apps customers on November 15, 2010. ... your business and continued support during this time.

Google Apps Incident Report
Feb 27, 2011 - Google Engineering reverted the storage software update, and halted ... better identify this class of bug during the software development cycle.

Google Apps Incident Report
Mar 18, 2013 - service disruption was an issue in the network control software. Actions and Root Cause Analysis. At 6:09 AM PT, a portion of Google's network ...

Google Apps Incident Report
Mar 17, 2014 - Issue Summary. From 8:35 AM to 12:10 PM PT, Google Talk, Google Hangouts (including Chat and Video), Google. Voice, and the Google App ...

Google Apps Incident Report
Apr 17, 2013 - The following is the incident report for the Google services access ... Talk, Google Sync, the Admin panel, and the Cloud Console, and to a ...

Google Apps Incident Report
Mar 19, 2013 - Applications using the Google Drive and Docs APIs also returned errors. ... We thank you for your business and continued support. Sincerely,.

Google Apps Incident Report
Feb 27, 2011 - Google Engineering reverted the storage software update, and halted further deployment. Restoration Process. While analyzing the issue and its root cause, Google Engineering also worked on the process to restore users' accounts. At 6:0

Google Apps Incident Report
Apr 17, 2012 - Prepared for Google Apps for Business customers. The following is the ... Enhance internal documentation for configuration management.

Google Apps Incident Report
Google Drive list. Applications using ... The Google Engineering team conducted an internal review and analysis of the March 21 event. They ... Modify the Drive software to more reliably serve user requests during short periods where overall.

Google Apps Incident Report
Dec 10, 2012 - Actions and Root Cause Analysis. Background: The load balancing software routes the millions of users' requests to Google data centers.

Google Apps Incident Report
Google Apps Incident Report. Google Docs Outage - September 7, 2011. Prepared for Google Apps for Business customers. The following is the incident report ...

Google Apps Incident Report
Google Apps Incident Report. Gmail Outage - September 23, 2011. Prepared for Google Apps for Business customers. The following is the incident report for the ...

Google Apps Incident Report - PDFKUL.COM
Apr 17, 2013 - The following is the incident report for the Google services access disruption that occurred on. April 17 ... Talk, Google Sync, the Admin panel, and the Cloud Console, and to a lesser extent Groups,. Sites, and ... misconfiguration oc

Google Apps Incident Report - PDFKUL.COM
Google Docs Outage - September 7, 2011. Prepared for Google Apps for Business customers. The following is the incident report for the Google Docs access ...

Incident Report: Google Apps Mail - September 01 2009
Sep 1, 2009 - On Tuesday, September 1, a small portion of Gmail's web capacity was taken ... request routing automatically directs users' requests to available servers. ... Over the next few weeks, we are dedicated to implementing these ...

Google Apps Incident Report 1 - Sept 24 - Service Disruption
Sep 25, 2009 - Between 7:00 AM - 9:50 PDT | 14:00 - 16:50 GMT, Thursday September 24, Google Apps users were unable to access the Contacts feature through the Gmail interface. However, they could view their contacts at an alternate URL. During this p

Google Apps Incident Report 1 - Sept 24 - Service Disruption
Sep 25, 2009 - Prepared for Google Apps Premier Edition Customers. Incident ... add users to their Google Apps accounts. ... business and continued support.

Incident Report: Google Apps Mail - September 01 2009
Sep 1, 2009 - server. Gmail processing and access through the IMAP/POP interfaces ... Over the next few weeks, we are dedicated to implementing these ...

Google Apps Incident Report: Gmail Delay, March 16, 2010
Mar 16, 2010 - resources for Gmail routing and greatly increased the number of active Gmail routers. Following an internal investigation and analysis, the ...

Google Apps Incident Report Gmail Delivery Delays - June 22, 2010
Engineering was made of aware of the problem and promptly began to work to manage excessive traffic ... your business and continued support during this time.

Google Apps Incident Report: August 25, 2010 - Duplicate Mail
from the Google Engineering team traced this problem to new code introduced at 3:00 PM PDT |. 22:00 UTC August 19. The Google Engineering team repaired ...

Incident Report
Feb 24, 2009 - The root cause of the problem was a software bug that caused an ... we monitor our systems 24 x 7, we have engineers available to analyze.