[routing-wg] HTTP 503 responses for rrdp.ripe.net for locations in the United States

Dear colleagues,

Today, between approximately 6:41 UTC and 10:31 UTC, 503 responses were served
for rrdp.ripe.net from multiple CDN edge locations in the United States. When we
received a report of this issue at 12:09 UTC, we mitigated potential issues by
switching CDN providers before starting our investigation.

The CDN in use at the time uses a cache between the origin and edge locations.
We expect the issue to be limited to CDN edge locations in Atlanta and Dallas
(usatl1-edge1, usatl1-edge2, usdal-edge1, and usdal-edge2). Due to this location,
this issue was not visible to us from our monitoring locations in Europe. We
monitor that the objects in the RRDP snapshot that each CDN provided caches are
in sync with our CA system but do not monitor individual CDN edge locations.

However, during an extended outage, relying party instances will fall back to
rsync. We did not observe a significant increase in the rate of rsync
connections during the outage period. No rsync connections were dropped because
of rsync capacity, with a peak of 200 connections and the 15-minute average
peaking at 68 connections. Based on this, we expect the impact to have been
limited.

We are still investigating the exact impact. Please share any information you
have on this issue with us via rpki at ripe.net.

Kind regards,
Ties de Kock