<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Dear Elvis,<br>
<br>
Thank you for your feedback. You are correct in pointing out that
it isn’t straightforward to compare the work of the Registry
Services in 2013 with the work we do in 2021. That’s why I’m
preparing a deeper analysis that looks at recent changes and
developments, as well as our approach to improving our services. A
lot has changed since you left eight years ago, and you might be
surprised to learn about the added complexity for even seemingly
easy requests.<br>
<br>
To share some more numbers on this situation, in the first eight
months of 2021, the amount of resource-related requests has
increased by almost 25% compared to the same period last year.
Looking at just August of this year, 62% more requests were
created than in August 2020. At the same time, our due diligence
activities have increased due to sanction checks. This is the main
reason it has been challenging to provide the fast response time
that our members are used to.<br>
<br>
We are confident that with the steps taken, as outlined in my
previous email, we will soon be handling this increased workload
more quickly.<br>
<br>
As soon as my overview is ready, I will share it on this list.<br>
<br>
Kind regards,<br>
<br>
Marco Schmidt<br>
Registry Services Assistant Manager<br>
RIPE NCC</p>
<div class="moz-cite-prefix">On 03/09/2021 12:39, Elvis Daniel Velea
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAJZHz-rU9e06xy3F_7ChnPxeoor6_Sfp+ZFZhfhFBCsCyFZpnA@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div>
<div>
<div>
<div dir="auto">Hi Marco,</div>
<div dir="auto"><br>
</div>
<div dir="auto">thank you for your reply. Please see further
comments below. Maybe some will help you with your next
reply.</div>
<div><br>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div dir="ltr" class="gmail_attr">On Fri, Sep 3, 2021 at
01:57 Marco Schmidt <<a href="mailto:mschmidt@ripe.net"
target="_blank" moz-do-not-send="true">mschmidt@ripe.net</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)">Dear
Elvis,<br>
<br>
Your email touches on multiple issues. I‘d like to address
some of them <br>
now, with the understanding that we will share more
detailed information <br>
soon. For now, I can provide a general overview on what
the situation <br>
with wait times is, and how we plan to tackle this issue
in the coming <br>
weeks.</blockquote>
<div dir="auto"><br>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div dir="auto">ok</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div dir="auto"><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"
dir="auto"><br>
<br>
First, a quick historical comparison for context. Back in
2013, the RIPE <br>
NCC had around 9,000 LIRs, and we processed a mere 150
policy transfers <br>
in the whole year, i.e. approximately 12-13 transfer
requests a month. </blockquote>
<div dir="auto"><br>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div dir="auto">Please do not compare apples to oranges. In
2013 the RIPE NCC was processing hundreds of requests for
initial and additional allocations, requests that were way
more complex than some of the current resource requests
(which no longer require any need based justification or
evaluation) or transfer requests that only require the
RIPE NCC to verify (i) if an organization is still active
in their country and (ii) the document you received is
signed by the people with PoA.</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div dir="auto"><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"
dir="auto"><br>
We currently have about 23,700 LIRs, which is 2.5x the
LIRs in 2013. <br>
Last month, we processed over 400 policy transfers. This
is 30x the <br>
number of transfer requests we processed when you were
part of the <br>
Registry Services team. And remember that there are all of
the other <br>
requests on top of this.</blockquote>
<div dir="auto"><br>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div dir="auto">That is why I was asking you to provide
total number of tickets handled, back then, now and every
year in between. A ticket is a ticket, some are simple,
some are more complicated. Same was then as it is now…</div>
<div dir="auto"><br>
</div>
<div dir="auto">You are right, you did not process hundreds
of transfer requests in 2013, but there were many other
types of requests that were way more complex than some of
current transfer requests. So saying that you process 30x
transfer requests is a bit misleading as you are not
showing the complete picture.</div>
</div>
</div>
<div>
<div>
<div dir="auto"><br>
</div>
<div dir="auto">The NCC RS dept. even succeeded to respect
the SLA when it was evaluating requests in pairs, towards
the end of the runout. At that time you were receiving
maaany requests, more than usually. If the department was
able to cope with that situation, why can’t it now? I
still have not seen the answer to this question.</div>
</div>
</div>
</div>
<div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"
dir="auto"><br>
<br>
In 2020, we averaged around 1,600 resources-related requests
per month. <br>
However, from November to December there was a threefold
increase <br>
compared to the rest of the year. It is hard to avoid an
increased wait <br>
time when this happens, and that’s why we prefer to be upfront
when we <br>
expect delays, so members can plan around them.</blockquote>
<div dir="auto"><br>
</div>
</div>
<div>
<div>
<div>
<div dir="auto">I await the numbers as requested. The
numbers that show the whole picture, not just the parts
you want to show, please.</div>
<div dir="auto"><br>
</div>
<div dir="auto">Remember when we had a ‘quota’ of 20-30
requests/replies per day. Multiply that with 20-30
employees and at minimum you should be able to process
2000 requests per week, not per month. There were
employees leaving the NCC and new ones (that needed
training) joining all the time. That did not stop the RS
management to require the department to respect the SLA
daily. </div>
<div dir="auto">Again, and you remember this perfectly as we
used to work in the same department, a day missing the SLA
was a tragedy, why is it now the norm?</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div dir="auto"><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"
dir="auto"><br>
<br>
While these statistics provide a glimpse into how things
have changed <br>
over the past few years, I would like to reiterate that it
is not just <br>
the number of tickets but also the complexity of requests
that has <br>
increased. We report on this regularly at RIPE Meetings.</blockquote>
<div dir="auto"><br>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div dir="auto">Complexity of tickets before the RIPE NCC
reached the last /8 (actually, until the requirements for
additional allocations were simplified in policy) was much
higher than the complexity of current transfer tickets.
Indeed, back then the amount of attempts of fraud were way
lower.. but that doesn’t mean complexity has increased, au
contraire. </div>
<div dir="auto">Complexity has just changed from complex
evaluations of networks, deployment plans, equipment lists
and network diagrams to anti-fraud. The RIPE NCC’s RS
dept. philosophy has also changed from ‘we trust our
members’ to ‘we need to make sure this is not fraud’.
Maybe this change has caused the increased complexity you
are talking about? </div>
</div>
</div>
</div>
<div>
<div>
<div>
<div dir="auto"><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"
dir="auto"><br>
<br>
Despite these challenges, Registry Services strives to
handle incoming <br>
requests efficiently. In order to improve our ticket
handling times, we <br>
have:<br>
- Hired new staff, who are still being trained, but will
help address <br>
the workload once they are up to speed.<br>
- Taken on temporary staff to enter registration details
of resource <br>
holders into our system which will allow us to automate
certain due <br>
diligence checks.<br>
We will publish a RIPE Labs article about these changes in
the coming weeks.</blockquote>
<div dir="auto"><br>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div dir="auto">Not sure exactly what you mean by ‘enter
registration details of resource holders into our system’
but I look forward to reading more details in the
article. </div>
</div>
</div>
</div>
<div>
<div>
<div dir="auto"><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"
dir="auto"><br>
<br>
Some additional clarifications regarding internal targets:<br>
- Every ticket that doesn't get a response by 18:00 (UTC +2)
the <br>
following working day is counted as a ‘miss’ against our
internal targets.<br>
- Our notification emails do not count against these targets
(we don’t <br>
see much point in gaming our internal KPIs).<br>
- A gentle reminder that Saturdays and Sundays are not
working days, so <br>
requests received on a Friday are likely to be addressed the
next <br>
business day, i.e. the Monday after the weekend.</blockquote>
<div dir="auto"><br>
</div>
</div>
</div>
<div>
<div>
<div dir="auto">See ticket #398137, for example. Ticket opened
Aug 26 at 4AM, received an automated message the same day
saying that due to workload… That was on Thursday. Do you
agree with me that this ticket should have been handled on
Friday? If yes, why was it resolved only next Monday? </div>
<div dir="auto">Marco, this is one of many tickets that have
been delayed for no reason. For some I have called the NCC
to see what’s the problem, we even talked a few times about
some of the tickets that were being delayed for no reason..
You know about them. </div>
<div dir="auto"><br>
</div>
<div dir="auto">See the other e-mail sent by Elina. Their
situation is even worse…</div>
<div dir="auto"><br>
</div>
<div>
<div dir="auto">So, Marco, please tell us how many working
days in the past few months/years has the RS dept. missed
the SLA? </div>
<div dir="auto">- actually, what may be easier to calculate
is how many working days you haven’t missed the SLA </div>
<div dir="auto"><br>
</div>
<div dir="auto">Around 2015-2017 (I don’t remember when
exactly) the RS dept. decided to stop showing how many
requests it received and handled daily. Before that it
used to have a page showing the workload and how long it
was taking for a ticket to receive a reply. Although at
that time I asked for that functionality to be restored,
the NCC preferred to hide it’s head in the sand and
decided that too much transparency was not so good. </div>
<div dir="auto">- any plans to become transparent again
about your workload and SLA?</div>
<div dir="auto"><br>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div dir="auto"><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"
dir="auto"><br>
<br>
We can appreciate that it is frustrating to wait on
tickets. However, <br>
while we’re still training additional staff and automating
systems, we <br>
need to rely on your patience and ask that you allow us
time to ensure <br>
requests are not just handled quickly, but also
accurately.</blockquote>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"
dir="auto"><br>
<br>
Since we know from experience that the number of requests
increases <br>
dramatically in the last quarter of the year, I would like
to repeat our <br>
request for members to kindly not wait until the last
minute to send in <br>
a request, and to expect longer handling times in the last
quarter of 2021.</blockquote>
<div dir="auto"><br>
</div>
</div>
</div>
</div>
<div>
<div dir="auto">I think this warning was and is a bad idea.
Instead of warning the members that you will miss the SLA even
more in Q4, you should work on fixing the bottlenecks. This
situation has been ongoing for months/years now… you can’t
come now and say ‘we know we are slow, we will be even slower
in the future’, this should have been solved already so you
don’t have a bigger problem next Q.</div>
</div>
<div>
<div>
<div>
<div dir="auto"><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"
dir="auto"><br>
<br>
Once I have a more detailed overview of our ticket
history, I’ll share <br>
it on this list, along with how we plan to further improve
our services.<br>
<br>
I am aware that I have not addressed all the concerns
raised in your <br>
email, Elvis, but I plan to continue this conversation in
the coming days.</blockquote>
<div dir="auto"><br>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div dir="auto">I look forward to your next e-mails. So far
this one has had almost no new information, you just
repeated the same info you sent last month, just worded it
differently. You also tried to say you are handling more
requests now than years ago, you haven’t convinced me,
instead you tried to trick us with some numbers that were
misleading.</div>
</div>
</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">Looking forward to have an actual conversation
about this.</div>
<div dir="auto"><br>
</div>
<div dir="auto">Elvis</div>
<div>
<div>
<div>
<div>
<div class="gmail_quote">
<div dir="auto"><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px
0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"
dir="auto"><br>
<br>
Kind regards,<br>
Marco Schmidt<br>
Registry Services Assistant Manager<br>
RIPE NCC<br>
<br>
On 31/08/2021 06:06, Elvis Daniel Velea wrote:<br>
> Dear Marco,<br>
><br>
> thank you for the update below. I'd like to ask
you to provide a few <br>
> more details and to detail your plans to fix the
SLA violation by the <br>
> Registration Services department, please see my
questions inline.<br>
><br>
> On 8/4/21 3:43 AM, Marco Schmidt wrote:<br>
>> Dear colleagues,<br>
>><br>
>> We would like to give you an update on the
ticket wait times in <br>
>> Registry Services, after discussions at RIPE
82 [1] and on the <br>
>> mailing list [2].<br>
>><br>
>> We have been addressing the issues causing
delays. As the number of <br>
>> incoming requests continue to exceed previous
years, we have taken on <br>
>> additional staff to help with the workload.
We are also automating <br>
>> some of our due diligence processes which are
currently time consuming.<br>
><br>
> You say that you receive more requests than in
previous years, can you <br>
> please provide some statistics for the past 10
years showing the <br>
> number of employees vs number of requests/replies
received/sent daily?<br>
> - please try, if possible, to show us how many of
these tickets were <br>
> related to transfers, how many to billing or
closures, etc...<br>
><br>
> Can you provide some stats showing how many
replies are necessary for <br>
> a ticket to be resolved? Also, let me know what
is being done so that <br>
> requests get evaluated and resolved in one reply.<br>
><br>
> Also, please let us know how exactly have you
addressed the constant <br>
> SLA violations, what part of which process has
been automated and <br>
> whether you believe that with the additional
staff you will resolve <br>
> the issues swiftly.<br>
><br>
> As you probably remember, I used to work in the
RS department between <br>
> 2007 and 2013. Back then, a day when the SLA was
not respected was <br>
> almost a tragedy... it seems that now it's the
norm and not the <br>
> exception any longer. What happened that your
department ended in this <br>
> situation?<br>
><br>
>><br>
>> You should start to see improvements in our
response times towards <br>
>> the end of this year once our new colleagues
are trained and our <br>
>> automation is ready. In the meantime, you may
experience slower <br>
>> response times. In other words, we will be
slower before we can get <br>
>> up to speed properly, but we think this will
be worth it over the <br>
>> long term.<br>
><br>
> Things are slower and slower. For example, last
week I had a ticket <br>
> receive 1 reply after ~24 hours saying that
because of time <br>
> constraints, the ticket will be evaluated later.
Then Friday, Saturday <br>
> and Sunday passed before I got a reply on Monday.<br>
><br>
> None of the tickets I or my customers opened last
month (August) have <br>
> been replied to within the SLA. NONE! We're
talking about more than 10 <br>
> tickets in a month, none receiving a reply within
the SLA.<br>
><br>
> You are saying that waiting will be worth it over
the long term while <br>
> right now all tickets are being delayed from a
day to another and <br>
> sometimes a ticket that should take 10 minutes to
evaluate is delayed <br>
> for 4 days or more... This is not acceptable and
that is why I'd like <br>
> to ask you for more details to prove to us that
the wait will be worth <br>
> it. As things are getting worse and not better, I
hope you understand <br>
> why I can't take your word for it :)<br>
><br>
><br>
> When you generate those stats, please let us know
if the replies <br>
> saying 'Due to a very high workload, we were
unable to process your <br>
> request at this time and will process it as soon
as possible.' are <br>
> still counted as a violation of the SLA or not.<br>
><br>
> Also, when you generate the stats I ask for,
please also tell us how <br>
> many working days in a month (for the past year
or two) the <br>
> Registration Services dept. has violated the SLA.<br>
><br>
><br>
> I've been trying to help the RS department
evaluate all the requests <br>
> received from V4Escrow and it's customers
quicker. The help I offered <br>
> was refused while the SLA violations got worse
and worse.. maybe it is <br>
> time you accept the help when it is offered to
you.<br>
><br>
> I've had several conversation with RS management
and higher up, all <br>
> the way up to MD. I talked to several teams to
explain the way I <br>
> wanted to help RS and our customers get all the
tickets resolved <br>
> swiftly (at least the ones that are for transfers
of resources). I <br>
> proposed the introduction of an API for
tickets/requests so we can try <br>
> to automate a large part of the transfer ticket,
this automation could <br>
> help RS evaluate requests faster, would help
customers provide all the <br>
> required information in one go; everyone would be
happy. I was told my <br>
> request will be looked at and maybe added to plan
no earlier than <br>
> Q1-2022. Please tell me if that is still the plan
or whether, in the <br>
> light of these constant SLA violation, you could
enable the API faster <br>
> so we, brokers, can help you receive complete and
easy to evaluate <br>
> transfer requests.<br>
><br>
>> We also know from experience that we receive
far more requests <br>
>> towards the end of the year which results in
a bottleneck. We will <br>
>> shift resources to counter this, but we would
also appreciate it if <br>
>> you could get some of these requests in
earlier.<br>
><br>
> Is this a warning that requests will take weeks
to process towards the <br>
> end of the year? They already take days...<br>
><br>
> Please clarify the bottlenecks you have and let's
see if there's any <br>
> way we can help you with them. I am sure that
automating a lot of <br>
> steps in your overly complicated processes will
help but I am looking <br>
> forward to receiving more details from you about
this.<br>
><br>
><br>
> Elvis<br>
><br>
> PS: Note that I wanted to also send an e-mail
about the RIPE NCC's <br>
> ticketing system (zendesk) to the members-discuss
mailing list and to <br>
> the board members. I will wait for an answer to
the e-mail I sent to <br>
> HPH one week ago (which he promised he'll forward
to the relevant team <br>
> for a reply).<br>
><br>
>><br>
>> Kind regards,<br>
>> Marco Schmidt<br>
>> Registry Services Assistant Manager<br>
>> RIPE NCC<br>
>><br>
>> [1] RIPE NCC Operational Update (slides
4-10):<br>
>> <a
href="https://ripe82.ripe.net/archives/video/584/"
rel="noreferrer" target="_blank"
moz-do-not-send="true">https://ripe82.ripe.net/archives/video/584/</a><br>
>><br>
>> [2] Discussion on Members Discuss mailing
list:<br>
>> <a
href="https://www.ripe.net/ripe/mail/archives/members-discuss/2021-May/004353.html"
rel="noreferrer" target="_blank"
moz-do-not-send="true">https://www.ripe.net/ripe/mail/archives/members-discuss/2021-May/004353.html</a>
<br>
>><br>
>><br>
>><br>
>>
_______________________________________________<br>
>> members-discuss mailing list<br>
>> <a href="mailto:members-discuss@ripe.net"
target="_blank" moz-do-not-send="true">members-discuss@ripe.net</a><br>
>> <a
href="https://lists.ripe.net/mailman/listinfo/members-discuss"
rel="noreferrer" target="_blank"
moz-do-not-send="true">https://lists.ripe.net/mailman/listinfo/members-discuss</a><br>
>> Unsubscribe: <br>
>> <a
href="https://lists.ripe.net/mailman/options/members-discuss/elvis%40v4escrow.net"
rel="noreferrer" target="_blank"
moz-do-not-send="true">https://lists.ripe.net/mailman/options/members-discuss/elvis%40v4escrow.net</a><br>
><br>
</blockquote>
</div>
</div>
</div>
</div>
</div>
-- <br>
<div dir="ltr" class="gmail_signature"
data-smartmail="gmail_signature">This message was sent from a
mobile device. Some typos may be possible. </div>
</blockquote>
</body>
</html>