[German]That's not cool: The issues that Microsoft have had with Microsoft 365 on January 29, 2019 also affected Azure SQL databases. Some customers have had SQL databases deleted on Azure. After restoring a snapshot, customers discovered that there were transaction data of up to 5 minutes are missing within their databases.
Advertising
Microsoft 365 outage on January 29, 2019
The last days Microsoft was shaken by failures of its Office365.com and Microsoft 365 services – see article links at the end of this article. Now The Register reported here that Azure's January 29, 2019 failure even claimed some unexpected victims. Customers suddenly discovered that their Azure SQL databases had been deleted or that data was missing. Here is a tweet from a victim:
Spent my whole day on why our Azure SQL database was dropped and a new Azure SQL Database was created with random name appended to dropped database name,
— SANTHANAM SENGUTUVAN (@santhu7289) 30. Januar 2019
Also the following user was probably caught cold by the deleted Azure SQL database.
@AzureSupport @AzureSQLDB Last night around 21.23 some of our azure SQL databases where deleted and not by us. The database is back restored but is empty, we did not do the restore?
I heard from other people that they have the same issue but there databases are not restored— Koen Vanden Bossche (@VdBosscheKoen) 30. Januar 2019
The issue only affected customers using Transparent Data Encryption (TDE). This is to protect the databases against malicious activities of attackers.
Advertising
Azure SQL database deleted
According to the site The Register, during the Azure instabilities that occurred on 01/29/2019, internal automatic task simply deleted the affected SQL databases. Microsoft was able to restore the databases from snapshots. But the snapshots was up to 5 minutes in the past, so customers were missing 5 minutes of data from various transactions.
The issue affected a number of Azure SQL databases that use custom KeyVault keys for transparent data encryption (TDE). The Register cited from a messages Microsoft has send to affected users:
"An automated process, designed to trigger when custom keys are removed from KeyVault, inadvertently caused these TDE databases to be dropped.
We are in the process of restoring a copy of these SQL DBs from a recovery point in time of less than 5 minutes before the database was dropped. These restored databases … are located on the same server as the original database"
Within the notification, Microsoft asks customers to check the databases for lost data and, if necessary, open a support ticket.
We ask that customers, for each database, identify if lost transactions, during this 5 minute timeframe, could impact business processes or applications outside the database. We would ask you to raise a support ticket in this instance. If the restored database is suitable, the database can be renamed to the original name to continue usage of it.
I guess, some customers are pretty pissed. But Microsoft has good news for those affected – the company wrote:
We sincerely apologize for the impact on your service. Azure usage fees are waived for all restored databases for 2 months and for all original databases for 3 months. We are continuously taking steps to improve the Microsoft Azure platform and our processes to ensure that such incidents do not occur again in the future.
That sounds good – or what's your opinion? The Register asked Microsoft what the next step would be. A spokesman said: 'We're working to restore access to resources that were unavailable to a limited subset of customers. Full access has been re-established for most of those customers already.'
