[routing-wg] Late Revocation of CA Certificates due to Bug in RIPE NCC CA Software

Dear colleagues,

As a result of a software bug introduced in our RPKI CA system on 16 May at
around 08:49 UTC, our CA system failed to revoke certificates for members/End
Users that lost their final resources.

This issue affected two certificates, one containing a /22 and another
containing a single AS Number. In violation of our CPS [0, Section 4.9.5], we
did not revoke the affected certificates within eight hours of changing the
resources. These certificates did not issue any leftover CA products (ROAs).

A fix for this issue was deployed to production today, 17 May at 08:20 UTC, and
the two certificates were correctly revoked at 08:29 UTC on 17 May.

Since the /22 certificate involved the consolidation of resources and no ROAs
were present, we believe there was no impact on the validity of prefixes.
Similarly, there was no impact for the AS Number returned to the free pool.

We have checked the prefixes affected by all transfers that happened during the
time period the bug was present. No other certificates were affected: Either the
CA still had resources, or there was no CA certificate for the member/End User
to lose resources.

To detect bugs like this and to prevent them from being introduced in the
future, we will (1) improve the monitoring that verifies that the resources of
the published certificates match the registry and (2) introduce tests that cover
this scenario.

Kind regards,

Ties de Kock
Specialist Software Engineer
RIPE NCC

[0]: https://www.ripe.net/publications/docs/ripe-751