<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><span id="docs-internal-guid-71bf25e1-7fff-a98a-364d-e1b515cd5480" class=""><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;" class="">Dear colleagues,</span></div><br class=""><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;" class="">Yesterday, we performed an upgrade on the Security world software on our RPKI core servers. The upgrade was finished at approximately 08:45 UTC. We tested the upgrade and verified that everything worked before enabling the RPKI dashboard again.</span></div><br class=""><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;" class="">At approximately 10:50 UTC, we received an alert from our monitoring that showed an error for both our online Hardware Security Modules (HSMs). While we immediately started the investigation of this alert, we also decided to temporarily stop RPKI Core to keep a consistent state. This also meant that we had to temporarily close down the RPKI Dashboard.</span></div><br class=""><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;" class="">At 11:22 UTC we contacted our vendor as we had never seen this behaviour before. A consultant from our vendor advised a reboot of the HSMs, which we performed at 11:55 UTC. After the reboot, the HSMs got back online and we enabled the RPKI Core and RPKI dashboard. It is still unknown whether the upgrade was the direct cause of the errors, as the error was very generic. </span></div><br class=""><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;" class="">While we are working on finding the root cause, we still need to reboot systems and HSMs occasionally, which causes unavailability of the RPKI Dashboard for a few minutes and it will take a bit longer than usual for objects to get published in our repository. As soon as we have more information, we will share it here.</span></div><br class=""><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;" class="">As a result of this outage, we will speed up the process to replace the online HSMs, which we described in our recent </span><a href="https://labs.ripe.net/author/ties/securing-the-ripe-ncc-trust-anchor/" style="text-decoration:none;" class=""><span style="font-size: 11pt; font-family: Arial; color: rgb(17, 85, 204); font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;" class="">RIPE Labs article</span></a><span style="font-size: 11pt; font-family: Arial; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;" class=""> [0].</span></div><br class=""><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;" class="">Kind regards,</span></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;" class="">Stella Vouteva</span></div><br class=""><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;" class="">[0]: <a href="https://labs.ripe.net/author/ties/securing-the-ripe-ncc-trust-anchor/" class="">https://labs.ripe.net/author/ties/securing-the-ripe-ncc-trust-anchor/</a></span></div><br class=""></span></div></body></html>