tag:status.aiir.com,2005:/historyAiir Status - Incident History2024-03-28T15:47:56ZAiirtag:status.aiir.com,2005:Incident/202765662024-03-16T11:22:28Z2024-03-16T11:22:28ZSlow loading of CMS<p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>11:22</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>10:15</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor the fix.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>10:04</var> UTC</small><br><strong>Monitoring</strong> - The issue has been identified and a fix has been implemented. Now Playing data may be temporarily delayed. Thank you for your continued cooperation and patience.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>09:52</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue. We appreciate your cooperation and patience.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>09:34</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating a degraded performance issue with logging into managing Aiir websites and mobile apps. We're sorry for any inconvenience this may cause and we'll update you here once we have more information.</p>tag:status.aiir.com,2005:Incident/201951202024-03-12T00:10:56Z2024-03-12T00:10:56ZEssential maintenance for our web services<p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>00:10</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>00:05</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>16:40</var> UTC</small><br><strong>Scheduled</strong> - Our web services will be undergoing essential maintenance, involving one of our database servers. <br />We anticipate this to last no more than 5 minutes, during which time our CMS may be unresponsive, but customer websites should largely continue to function thanks to caching and our CDN.<br />We're sorry for any inconvenience this may cause and we'll update you here once we have the all-clear.</p>tag:status.aiir.com,2005:Incident/201606752024-03-06T10:26:48Z2024-03-06T10:26:48ZStreaming<p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>10:26</var> UTC</small><br><strong>Resolved</strong> - We've completed the switch from our alternative server back to our main server, so everything is now back to normal. Sorry for the interruptions, thank you for bearing with us.</p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>10:20</var> UTC</small><br><strong>Update</strong> - We're about to switch from our alternative server back to our main server, so there may be a brief disruption while we do this. Customers will automatically reconnect to the new server. Listeners may need to refresh their players to continue listening. We're sorry this is necessary and the inconvenience this causes. Thanks for your patience while we complete this work.</p><p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>22:01</var> UTC</small><br><strong>Monitoring</strong> - We've brought an alternative server online, so our streaming service is now returning to normal. Customers should automatically reconnect shortly if they haven't already. We're so sorry for the interruption today folks. Thanks for bearing with us.</p><p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>21:52</var> UTC</small><br><strong>Update</strong> - We're really sorry for the ongoing disruption to our streaming service. Our team are continuing to investigate - we appreciate your cooperation while we get to the bottom of this as quickly as possible.</p><p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>21:44</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating an issue with our streaming service. Customers may not be able to listen or connect to their streams.</p>tag:status.aiir.com,2005:Incident/199824342024-02-15T00:01:43Z2024-02-15T00:01:43ZEssential maintenance for our web services<p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>00:01</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>00:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>10:04</var> UTC</small><br><strong>Scheduled</strong> - Our web services will be undergoing essential maintenance, involving the reboot of one of our database servers. Although we wrote in our last post that it would be a one-time reboot, new information has come to light which means another reboot is required. The last one lasted under 2 minutes and we expect this time to be no different.<br />During this time, our CMS may be unresponsive, but customer websites should largely continue to function thanks to caching and our CDN.<br />We're sorry for any inconvenience this may cause and we'll update you here once we have the all-clear.</p>tag:status.aiir.com,2005:Incident/199735322024-02-14T00:02:29Z2024-02-14T00:02:29ZEssential maintenance for our web services<p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>00:02</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>00:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>09:36</var> UTC</small><br><strong>Scheduled</strong> - Our web services will be undergoing essential maintenance, involving the reboot of one of our database servers. We don't like having to do this, but AWS (our web hosts) require us to make some changes over the coming weeks, and the best way to avoid further disruption is to do a one-time reboot allowing us to make further changes more discreetly. We expect this to be speedy, as no other changes are taking place at this time. During this time, our CMS may be unresponsive, but customer websites should largely continue to function thanks to caching and our CDN.<br />We're sorry for any inconvenience this may cause and we'll update you here once we have the all-clear.</p>tag:status.aiir.com,2005:Incident/197149712024-01-15T21:06:45Z2024-01-15T21:06:45ZPlayoutONE Streaming Maintenance<p><small>Jan <var data-var='date'>15</var>, <var data-var='time'>21:06</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>15</var>, <var data-var='time'>20:01</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>15</var>, <var data-var='time'>18:40</var> UTC</small><br><strong>Scheduled</strong> - From 8PM UK time on 15th January 2024 - we will be performing essential maintenance on our main streaming server. This may cause a brief outage whilst your stream is redirected to our backup server.<br /><br />We apologise for any inconvenience this may cause and we will endeavour to keep outages to a minimum during this maintenance.</p>tag:status.aiir.com,2005:Incident/191382992023-11-16T17:26:33Z2023-11-16T17:26:33ZIncoming messages to Studio Inbox<p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>17:26</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>15:11</var> UTC</small><br><strong>Monitoring</strong> - Our team have performed a fix on the system. The receiving of messages in Studio Inbox should now be restored and we are continuing to monitor.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>15:04</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>15:04</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating an issue that could cause incoming messages to Studio Inbox via email and mobile apps to either not be received or be delayed.</p>tag:status.aiir.com,2005:Incident/191261752023-11-15T11:16:51Z2023-11-15T11:16:51ZIncoming messages to Studio Inbox<p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>11:16</var> UTC</small><br><strong>Resolved</strong> - Our team have performed a fix on the system. The receiving of messages in Studio Inbox should now be restored.</p><p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>10:35</var> UTC</small><br><strong>Investigating</strong> - We are investigating a reoccurrence of the issue causing messages to be received delayed.</p>tag:status.aiir.com,2005:Incident/191256582023-11-15T09:36:14Z2023-11-15T09:36:14ZIncoming messages to Studio Inbox<p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>09:36</var> UTC</small><br><strong>Resolved</strong> - Our team identified an issue that could have caused messages sent via email and mobile apps to arrive delayed in Studio Inbox. SMS was unaffected by this issue.<br /><br />Any messages sent during this time unfortunately won't be received, but messages sent from now on will be received as normal.</p><p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>09:11</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating an issue that could cause incoming messages to Studio Inbox via email, SMS and mobile apps to either not be received or be delayed.</p>tag:status.aiir.com,2005:Incident/190473662023-11-07T17:35:28Z2023-11-07T17:35:28ZIssues accessing CMS<p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>17:35</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>16:48</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>16:04</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>15:58</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.aiir.com,2005:Incident/190471382023-11-07T15:49:30Z2023-11-07T15:49:30ZIssues accessing CMS<p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>15:49</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>15:42</var> UTC</small><br><strong>Monitoring</strong> - The CMS should be available again. We're monitoring to ensure it remains stable.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>15:31</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.aiir.com,2005:Incident/187643432023-10-12T11:53:31Z2023-10-12T11:53:31ZPlatform Unavailable<p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>11:53</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>11:42</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>11:39</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.aiir.com,2005:Incident/185449262023-09-20T08:55:45Z2023-09-20T08:55:45ZFacebook are experiencing a problem with link thumbnails<p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>08:55</var> UTC</small><br><strong>Resolved</strong> - Facebook have not confirmed the bug is fixed, but we can see from all recent posts on various pages that the issue does appear to be resolved.</p><p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>22:20</var> UTC</small><br><strong>Monitoring</strong> - Facebook are yet to confirm that this issue is resolved on their bug tracker, but we are seeing images now correctly appearing alongside shared links. To test one of your links, use their Sharing Debugger: https://developers.facebook.com/tools/debug/</p><p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>19:10</var> UTC</small><br><strong>Update</strong> - Facebook are tracking this bug here: https://developers.facebook.com/support/bugs/280868194805721/<br />They recently commented to say they're passing the issue to their product team to investigate.<br />We can see that the issue is ongoing.</p><p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>11:52</var> UTC</small><br><strong>Identified</strong> - We have received multiple reports today of Facebook not showing a thumbnail image, or the wrong image, for links shared in posts on Facebook pages.<br />We can confirm this issue is not specific to Aiir or Aiir's customers. For example, the same thing can be seen for Sky News (not an Aiir customer) on their page: https://www.facebook.com/skynews<br />We appreciate this is frustrating, but as the issue is not caused by Aiir's products or any changes we have made, we can only await a resolution from Facebook themselves.</p>tag:status.aiir.com,2005:Incident/181624392023-08-22T19:35:51Z2023-08-22T19:35:54ZStreaming Server<p><small>Aug <var data-var='date'>22</var>, <var data-var='time'>19:35</var> UTC</small><br><strong>Resolved</strong> - After all of our tests were completed successfully, the primary server has now been made active.</p><p><small>Aug <var data-var='date'>22</var>, <var data-var='time'>10:58</var> UTC</small><br><strong>Update</strong> - Our team performed additional investigation which has caused the migration back to the primary streaming infrastructure to be moved to 8pm UK time on 22/08/2023.</p><p><small>Aug <var data-var='date'>21</var>, <var data-var='time'>11:32</var> UTC</small><br><strong>Update</strong> - Following the conclusion of our investigation into the issues on the primary streaming infrastructure, we plan to migrate streaming back to the primary throughout Monday 21st August.</p><p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>13:14</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>13:14</var> UTC</small><br><strong>Update</strong> - We have diverted streaming traffic to our backup server so normal service should now be restored whilst we investigate the root cause of the issue with our primary server.</p><p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>12:52</var> UTC</small><br><strong>Identified</strong> - We have identified an issue with our streaming infrastructure. This may cause your stream to drop or be offline at this present time.</p>tag:status.aiir.com,2005:Incident/179638482023-07-26T15:16:33Z2023-07-26T15:16:33ZStreaming Degraded<p><small>Jul <var data-var='date'>26</var>, <var data-var='time'>15:16</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>26</var>, <var data-var='time'>10:43</var> UTC</small><br><strong>Monitoring</strong> - Our streaming servers seem to be stable once again and our team is continuing to monitor the situation.</p><p><small>Jul <var data-var='date'>26</var>, <var data-var='time'>09:09</var> UTC</small><br><strong>Investigating</strong> - We are aware of an issue that is causing our streaming server to be degraded. This may cause your encoder to not connect and listeners to be unable to tune in.</p>tag:status.aiir.com,2005:Incident/173628912023-05-25T11:01:06Z2023-05-25T11:01:06ZShowhost Maintenance<p><small>May <var data-var='date'>25</var>, <var data-var='time'>11:01</var> UTC</small><br><strong>Resolved</strong> - The maintenance is complete.</p><p><small>May <var data-var='date'>25</var>, <var data-var='time'>09:41</var> UTC</small><br><strong>Monitoring</strong> - We're performing some important maintenance on Showhost today. Hopefully (if we've done our sums right) you won't notice any impact at all. But on the off chance something doesn't go to plan, we thought you should know we're working on it and hope to have everything back in business ASAP.<br />You may see pages load but infomration on them not appear. You may also not receive a link to log in via your email.</p>tag:status.aiir.com,2005:Incident/172467612023-05-14T17:47:32Z2023-05-14T17:47:32ZIntermittent availability of customer websites<p><small>May <var data-var='date'>14</var>, <var data-var='time'>17:47</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>14</var>, <var data-var='time'>17:16</var> UTC</small><br><strong>Monitoring</strong> - One of our customer websites has experienced a "distributed denial-of-service" (DDoS) attack. We have taken the specific site offline, the customer has been informed, and this has allowed all our services (except this one site) to return to normal. We will continue to investigate and monitor services. There is no need for you to be concerned about the impact this attack has had on your website, beyond the intermittent availability in the last few hours. No data has been lost and security has not been breached. We're sorry for any inconvenience this caused today.</p><p><small>May <var data-var='date'>14</var>, <var data-var='time'>16:48</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating an issue affecting our customer websites. You may see a "bad gateway" error, or a cached (outdated) version of a page.</p>tag:status.aiir.com,2005:Incident/169199832023-04-30T07:01:05Z2023-04-30T07:01:05ZCloud Automation Reboots<p><small>Apr <var data-var='date'>30</var>, <var data-var='time'>07:01</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'>24</var>, <var data-var='time'>19:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>11:09</var> UTC</small><br><strong>Scheduled</strong> - Due to a driver update, cloud hosted PlayoutONE machines may be rebooted between April 24, 2023 and April 30, 2023 between 6:00PM - 6:00AM BST. PlayoutONE should start back up automatically and begin streaming when the machine reboots.</p>tag:status.aiir.com,2005:Incident/169562292023-04-27T10:37:07Z2023-04-27T10:37:07ZStreaming<p><small>Apr <var data-var='date'>27</var>, <var data-var='time'>10:37</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>10:10</var> UTC</small><br><strong>Update</strong> - We are still monitoring the issue and have found a further problem which we hope to resolve within the next 24 hours.</p><p><small>Apr <var data-var='date'>25</var>, <var data-var='time'>23:32</var> UTC</small><br><strong>Monitoring</strong> - We have implemented a fix. Should this hold over the next few hours we will migrate clients back to the primary server at approximately 11:00 BST</p><p><small>Apr <var data-var='date'>25</var>, <var data-var='time'>09:54</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for our primary server. We hope to have another update at around 22:00 BST.</p><p><small>Apr <var data-var='date'>24</var>, <var data-var='time'>21:17</var> UTC</small><br><strong>Identified</strong> - Our primary server is currently experiencing an outage. We are directing all traffic to our backup server whilst we identify the problem.</p><p><small>Apr <var data-var='date'>24</var>, <var data-var='time'>21:09</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.aiir.com,2005:Incident/169128562023-04-24T23:01:45Z2023-04-24T23:01:45ZTwitter functionality in Studio Inbox<p><small>Apr <var data-var='date'>24</var>, <var data-var='time'>23:01</var> UTC</small><br><strong>Resolved</strong> - We have shared an important update on Twitter and Aiir on our blog:<br />https://blog.aiir.com/an-update-on-twitter-and-aiir-6024e83111ac</p><p><small>Apr <var data-var='date'>19</var>, <var data-var='time'>16:41</var> UTC</small><br><strong>Investigating</strong> - Twitter have introduced new rate limits which are affecting some functionality in Studio Inbox, such as Twitter search columns. We're looking into this further to see what can be done.</p>tag:status.aiir.com,2005:Incident/169289822023-04-21T09:52:29Z2023-04-21T09:52:29ZVoucher PDFs unavailable<p><small>Apr <var data-var='date'>21</var>, <var data-var='time'>09:52</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>21</var>, <var data-var='time'>09:27</var> UTC</small><br><strong>Investigating</strong> - We are investigating an issue affecting the availability of PDFs for purchased vouchers.</p>tag:status.aiir.com,2005:Incident/168720702023-04-15T09:45:13Z2023-04-15T09:45:13ZWYSIWYG editor<p><small>Apr <var data-var='date'>15</var>, <var data-var='time'>09:45</var> UTC</small><br><strong>Resolved</strong> - The CDN which hosts our WYSIWYG editor suffered an outage which meant it wasn't appearing in our CMS. We're sorry for any inconvenience this caused you. Things look to be all better now.</p><p><small>Apr <var data-var='date'>15</var>, <var data-var='time'>03:54</var> UTC</small><br><strong>Investigating</strong> - We're investigating an issue where the WYSIWYG editor is not appearing on pages and other areas of our CMS.</p>tag:status.aiir.com,2005:Incident/167725782023-04-06T20:31:10Z2023-04-06T20:31:10ZDegraded Twitter functionality<p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>20:31</var> UTC</small><br><strong>Resolved</strong> - We've fixed a few more things affecting tweets in Studio Inbox. If you come across any issues or spot anything unexpected involving tweets, please let us know.<br />Thanks for bearing with us while we got this sorted.</p><p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>20:15</var> UTC</small><br><strong>Monitoring</strong> - The issues with Studio Inbox weren't as critical as we first thought, so you may not have been impacted at all. We're continuing to check things over before considering this fully resolved.</p><p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>19:49</var> UTC</small><br><strong>Update</strong> - Issues with Social Publisher are fixed, thanks for bearing with us. We're continuing to investigate issues in Studio Inbox.</p><p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>18:31</var> UTC</small><br><strong>Investigating</strong> - We're currently investigating an issue which is affecting Twitter functionality in our products, specifically Studio Inbox and Social Publisher. We're sorry for any inconvenience this causes you today. We hope to have things back up to speed soon.</p>tag:status.aiir.com,2005:Incident/166845592023-03-29T08:52:03Z2023-03-29T08:52:03ZForms are disabled<p><small>Mar <var data-var='date'>29</var>, <var data-var='time'>08:52</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>29</var>, <var data-var='time'>08:03</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>29</var>, <var data-var='time'>07:41</var> UTC</small><br><strong>Investigating</strong> - We have had to disable all website forms while we investigate an issue. We're very sorry for the inconvenience this causes.</p>tag:status.aiir.com,2005:Incident/166775082023-03-28T22:52:06Z2023-03-28T22:52:06ZMaintenance on Forms<p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>22:52</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>22:01</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>15:29</var> UTC</small><br><strong>Scheduled</strong> - We're going to be performing essential maintenance on Forms. While this is underway, the Forms section of our platform (on.aiir.com) and forms on customer websites will be unavailable. We're sorry for any inconvenenice this causes you. We've done everything we can in advance to keep this disruption to a minimum, and we'll get it wrapped up as quickly as possible.</p>