Maintenance Window for Data Processing

Incident Report for Attune

Resolved

We successfully replaced our data processing engine - which concludes today's maintenance window. All services are currently operational.
Posted Feb 23, 2024 - 04:39 UTC

Update

All status and data have recovered and are up to date. We are currently planning to replace our data processing engine on Thursday during a deployment window that will be announced on the platform.
Posted Feb 21, 2024 - 00:09 UTC

Update

Data processing is delayed for some sites and modules may show offline.
Posted Feb 20, 2024 - 21:52 UTC

Update

We successfully replaced the faulty hardware and all status and data have recovered and are up to date
Posted Feb 07, 2024 - 13:37 UTC

Update

Data processing is delayed for some sites and modules may show offline due to some AWS degrades instances. We are investigating
Posted Feb 07, 2024 - 11:57 UTC

Update

All status and data have recovered and are up to date
Posted Jan 29, 2024 - 21:58 UTC

Update

Data processing is delayed for some sites and modules may show offline.
Posted Jan 29, 2024 - 21:00 UTC

Update

All status and data have recovered and are up to date.
Posted Jan 24, 2024 - 06:59 UTC

Update

Data processing is delayed for some sites and modules may show offline.
Posted Jan 24, 2024 - 03:04 UTC

Update

All status and data have recovered and are up to date.
Posted Jan 17, 2024 - 21:58 UTC

Update

Some installs are showing delayed data processing. We are working on restoring those.
Posted Jan 17, 2024 - 20:39 UTC

Update

The Data processing issues are currently resolved. Notifications are enabled and all data is caught up. We are still working on the overhaul of our data processing engine and expect to release it in the coming weeks to prevent further data processing issues as we've been experiencing. in the past months.
Posted Jan 09, 2024 - 23:08 UTC

Update

Notifications are currently disabled
Posted Jan 09, 2024 - 17:22 UTC

Update

We are currently experiencing some delay in data processing and we are investigating the issue and how to minimize the impact.
Posted Jan 09, 2024 - 17:06 UTC

Update

Data processing is currently running normally and data is up to date. Notifications will be reenabled shortly. We are not processing any weather data at the moment. We are still actively working on a more robust replacement for the failing system. We are continuing to monitor the situation closely.
Posted Nov 20, 2023 - 15:40 UTC

Update

As of last night, we’ve been able to temporarily restore data processing. We are currently catching up on backed up data. We are still actively working on replacing that part of the code in order to prevent further outages.
Posted Nov 19, 2023 - 12:41 UTC

Update

Update: today we will attempt to process some data based on some patches we’ve been working on over the past days. In parallel, we are completely rewriting the failing piece of software that prevents us from processing data (very short term) and then will rewrite it completely differently for the longer term to avoid a single point of failure moving forward. We apologize again for the inconveniences this is causing and really appreciate your support and understanding. We will keep posting updates as we have them.
Posted Nov 18, 2023 - 17:32 UTC

Update

We are still actively working on restoring data processing. We do not have an ETR for when things will be restored yet. Apologies for the inconvenience.
Posted Nov 16, 2023 - 22:50 UTC

Update

We are currently processing some status data (nodes are showing online) but no sensor data. We are focusing on status data to minimize sensor data loss.
Posted Nov 15, 2023 - 18:12 UTC

Update

We keep experiencing issues with our data processing engine. We’re actively working on it. Apologies for the inconvenience
Posted Nov 15, 2023 - 14:21 UTC

Update

We’ve successfully re-established communications between the gateways and our infrastructure, allowing the gateways to upload all their stored data. We are now working on the data processing part of our pipeline.
Posted Nov 15, 2023 - 09:05 UTC

Update

We are currently experiencing a major outage, we apologize for any inconvenience while we work on resolving the issue.
Posted Nov 14, 2023 - 11:52 UTC

Update

We are continuing to work on optimizing performance and performance is degraded while we do that.
Posted Nov 13, 2023 - 20:06 UTC

Update

We’ve completed the maintenance window for today. We continue monitoring performance.
Posted Nov 10, 2023 - 23:28 UTC

Update

We are keeping the maintenance window open as we are making some more tweaks. Some modules may show offline, and data in dashboards and alarms may be delayed.
Posted Nov 10, 2023 - 15:36 UTC

Update

We are continuing to monitor the upgrades we made. Notifications are enabled.
Posted Nov 08, 2023 - 17:14 UTC

Monitoring

We successfully increased our data processing capacity and are now monitoring the results and catching up on data. Notifications are still disabled.
Posted Nov 07, 2023 - 13:06 UTC

Update

We started working on the underlying server infrastructure and we are behind on data processing. Notifications have been disabled.
Posted Nov 06, 2023 - 20:13 UTC

Identified

We are opening a Maintenance window to increase the underlying capacity of our data processing. Some modules may show offline, and data in dashboards and alarms may be delayed.
Posted Nov 06, 2023 - 19:32 UTC
This incident affected: Data Processing.