Hi,


I want to test the performance of the ntpd server. Are there any free/open 
source ntp clients that measure the server scalability? Thanks in advance for 
your help.

Thanks,
RC.

On Sep 24, 2010, at 11:23 AM, R C wrote:
> I want to test the performance of the ntpd server. Are there any free/open 
> source ntp clients that measure the server scalability? Thanks in advance for 
> your help.

The ntpd sources come with ntpdate; use "ntpdate -q" in a shell script loop to issue many queries without changing the clock.  Anyway, even on old Pentium-grade hardware and 100Mbs ethernet, you should be able to do thousands to tens of thousands of NTP packets per second.  The bottleneck almost invariably will your upstream network connection, either in terms of bandwidth or in terms of a firewall/NAT box trying to keep state.

Regards,
-- 
-Chuck

On 2010-09-24, R C <rc_work@yahoo.com> wrote:
> Hi,
>
>
> I want to test the performance of the ntpd server. Are there any free/open 
> source ntp clients that measure the server scalability? Thanks in advance for 
> your help.

I think you need to tell us more about exactly what you want to do and
why. 
Your request is so vague that it makes it almost impossible to answer.

On Fri, Sep 24, 2010 at 6:23 PM, R C <rc_work@yahoo.com> wrote:
> I want to test the performance of the ntpd server. Are there any free/open
> source ntp clients that measure the server scalability?

I don't know of any such scalability-testing client software, but I
can tell you ntpd scalability is typically a non-issue.  The traffic
involves small packets, which take very little time to process, so
typically you'll run into router scalability issues before server
ones.

I am involved with a publicly-accessible pool.ntp.org server running
on a 2.5 GHz Celeron.  It uses ntpd built without disabling ntpd's
built-in debugging support, though it is using compiler optimization,
it's also carrying around extra sanity-checking code and activity
tracing that is typically not enabled at runtime, but still causes
extra overhead compared to a ntpd configured with --disable-debugging.

Here's some hard data on its scalability:

hart@pool1> ps ax | fgrep ntpd
 3332  ??  Rs   193:57.22 /usr/local/bin/ntpd -c /etc/ntp.conf -g
 3333  ??  I      0:16.44 /usr/local/bin/ntpd -c /etc/ntp.conf -g
21884  p0  R+     0:00.00 fgrep ntpd
hart@pool1> ntpq -c sysstats
uptime:                 789921
sysstats reset:         789921
packets received:       181690517
current version:        115060504
older version:          50474534
bad length or format:   16152355
authentication failed:  369435
declined:               2605
restricted:             11647
rate limited:           15114153
KoD responses:          1291563
processed for time:     54715
hart@pool1>

There are two ntpd processes, the second is a DNS worker child of the
first.  Apparently there's a DNS name in its ntp.conf which isn't
resolving which it is continuing to retry occasionally.  In 789921s,
it has consumed 11654s of CPU time, or about 1.5%.  Undoubtedly
there's some system overhead not accounted for in that CPU time, but
you can see the CPU requirement to keep up with an average of 230
requests/s is marginal.  Its ntpd performance is great:


hart@pool1> ntpq -crv -p
associd=0 status=0618 leap_none, sync_ntp, 1 event, no_sys_peer,
version="ntpd 4.2.7p50@1.2188-o Thu Sep 16 19:45:06 UTC 2010 (3)",
processor="i386", system="FreeBSD/6.4-STABLE", leap=00, stratum=2,
precision=-20, rootdelay=1.637, rootdisp=10.881, refid=127.67.113.92,
reftime=d04902de.fb29a07c  Sat, Sep 25 2010 23:23:10.981,
clock=d0490453.ef2cc9cb  Sat, Sep 25 2010 23:29:23.934, peer=37589, tc=7,
mintc=3, offset=-0.035, frequency=-67.148, sys_jitter=0.831,
clk_jitter=0.092, clk_wander=0.001
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 192.168.4.255   .BCST.          16 B    -   64    0    0.000    0.000   0.001
-clock.sjc.he.ne .CDMA.           1 u    1  128  377    0.550   -1.135   1.203
+clock.fmt.he.ne .PPS.            1 u  122  128  377    1.092    0.015   0.056
*clock.fmt.he.ne .PPS.            1 u  108  128  377    1.637   -0.203   0.883
+clepsydra.dec.c .GPS.            1 u  116  128  377    0.679    0.017   0.015
+usno.pa-x.dec.c .USNO.           1 u  102  128  377    0.616    0.048   0.235
-psp-os1.ntp.org 192.168.4.22     3 s   63   64  376    0.111   -0.019   0.039
-clock.isc.org   .GPS.            1 u    7  128  377    0.228   -3.965   0.390
 149.20.54.20 (p 131.107.13.100   2 u  109  128  377    0.328    8.792   3.214
hart@pool1>

Would you mind elaborating on what question(s) you hope to answer
through testing?

Cheers,
Dave Hart

Hi all,

Thanks for your prompt response.

My main intension of this scalability testing is to find out 

1. how long it takes for the ntpd server to serve 100,000 clients. 
2. What is the rate at which the server can process client requets.

So, to test the above, I want to be able to simulate a 100,000 clients so i can 
hit my ntpd server with these 100,000 client requests. Please let me know if you 
have more questions.

Thanks,
RC.




________________________________
From: Chuck Swiger <cswiger@mac.com>
To: R C <rc_work@yahoo.com>
Cc: questions@lists.ntp.org
Sent: Sat, September 25, 2010 8:55:36 AM
Subject: Re: Test ntpd performance

On Sep 24, 2010, at 11:23 AM, R C wrote:
> I want to test the performance of the ntpd server. Are there any free/open 
> source ntp clients that measure the server scalability? Thanks in advance for 
> your help.

The ntpd sources come with ntpdate; use "ntpdate -q" in a shell script loop to 
issue many queries without changing the clock.  Anyway, even on old 
Pentium-grade hardware and 100Mbs ethernet, you should be able to do thousands 
to tens of thousands of NTP packets per second.  The bottleneck almost 
invariably will your upstream network connection, either in terms of bandwidth 
or in terms of a firewall/NAT box trying to keep state.

Regards,
-- 
-Chuck

R C wrote:
> Hi all,
> 
> Thanks for your prompt response.
> 
> My main intension of this scalability testing is to find out 
> 
> 1. how long it takes for the ntpd server to serve 100,000 clients. 

Why would they all try to hit it at the same time?

However, if they did, the timing is likely to be dominated by ethernet 
collision back-offs. which will not be well simulated by a single test 
source.  Also the roundtrip time will be poor if they all hit at once.

> 2. What is the rate at which the server can process client requets

Almost certainly, the rate at which the network can deliver them.

On 09/26/2010 02:05 AM, R C wrote:
> Hi all,
> 
> Thanks for your prompt response.
> 
> My main intension of this scalability testing is to find out 
> 
> 1. how long it takes for the ntpd server to serve 100,000 clients. 
> 2. What is the rate at which the server can process client requets.

A crude testing tool I once wrote:

http://n1.taur.dk/permanent/ntpload.c

Do NOT run this against a box you do not have permission
to test. This ntp client makes d-link&co look positively
angelic by comparison.

(2): An Atom N270 box handles 19k requests per second 
(2.4GHz Core2: 39k). That's ~50us per request. To test 
your box, you will need 3+ client machines, each running 
multiple instances of ntpload. Increase the number of 
instances until the box under test hits 100% cpu.

(1) is ambiguous. I assume the question is not about how
long the system takes to recover if you turn on 100.000
clients within a few hundred ms (the answer would be a
few seconds or less depending on buffer sizes), but 
rather CPU time. That's 50usec * 100k clients / 16 sec
= 31% of a 1.6GHz Atom core, if we assume that ALL your 
clients are awful. If they are merely rude, or all 
powered up within a minute, then expect 50usec * 100k 
clients / 64 seconds = 8% of a 1.6GHz Atom core.

You need to worry about the unasked question #3, which
is 'Do I accidentally have stateful firewalling in place,
and if so, does the router/firewall have memory for 
100k additional states?'. 


/Kasper Pedersen

On 2010-09-26, R C <rc_work@yahoo.com> wrote:
> Hi all,
>
> Thanks for your prompt response.
>
> My main intension of this scalability testing is to find out 
>
> 1. how long it takes for the ntpd server to serve 100,000 clients. 

HOwever long it takes those clients to send the requests. 

> 2. What is the rate at which the server can process client requets.
>
> So, to test the above, I want to be able to simulate a 100,000 clients so i can 

Why? Is this some theoretic query or do you really anticiplate servicing
100000 clients. And where are they coming from?

> hit my ntpd server with these 100,000 client requests. Please let me know if you 
> have more questions.

Over the past 10 years I have probably serviced 100000 client requests.
No problem. But that may not be what you meant. Note that the question
is one of networking, switches, etc far more than it is a question of
ntpd. It is like asking how many clients a fast food restaraunt can
service without mentioning it is at the end of a 10 mile muddy mountain
road. 

On Fri, Sep 24, 2010 at 11:23:11AM -0700, R C wrote:
> I want to test the performance of the ntpd server. Are there any free/open 
> source ntp clients that measure the server scalability? Thanks in advance for 
> your help.

Not as good as would be a special ntp client, but you can generate ntp
traffic with tcpdump and tcpreplay. First record an NTP query (e.g.
from ntpdate -p 1) with tcpdump and then you can replay it at specified
packet rate with tcpreplay.

Note that ntpd tends to drop packets before the CPU is fully loaded so
you need to compare the outgoing and incoming packet rate to be sure
it's replying to all queries.

-- 
Miroslav Lichvar

Hi everyone,

Thank you for your responses. Sorry about the ambiguity.

1. Atleast if not a 100,000 all at a time, say 10000 client requests at a time 
hitting the server. 

2. I want to see how long the server takes to serve a 100,000 requests totally, 
if 5000 or 10000 requests are received at the same time.
3. Also, while serving these 100,000 requests, if there are any failures.
4. Are there any established performance results for ntpd already?

Thanks for your help.

RC.



________________________________
From: David Woolley <david@ex.djwhome.demon.invalid>
To: questions@lists.ntp.org
Sent: Sun, September 26, 2010 9:06:42 AM
Subject: Re: Test ntpd performance

R C wrote:
> Hi all,
> 
> Thanks for your prompt response.
> 
> My main intension of this scalability testing is to find out 
> 1. how long it takes for the ntpd server to serve 100,000 clients. 

Why would they all try to hit it at the same time?

However, if they did, the timing is likely to be dominated by ethernet collision 
back-offs. which will not be well simulated by a single test source.  Also the 
roundtrip time will be poor if they all hit at once.

> 2. What is the rate at which the server can process client requets

Almost certainly, the rate at which the network can deliver them.

_______________________________________________
questions mailing list
questions@lists.ntp.org
http://lists.ntp.org/listinfo/questions

On 2010-09-26, R C <rc_work@yahoo.com> wrote:
> Hi everyone,
>
> Thank you for your responses. Sorry about the ambiguity.
>
> 1. Atleast if not a 100,000 all at a time, say 10000 client requests at a time 
> hitting the server. 

They cannot. All network cards are serial devices. They cannot deliver
1000 or even 2 requests at the same time. 
Sso you still have not been very clear as to what you want.

>
> 2. I want to see how long the server takes to serve a 100,000 requests totally, 
> if 5000 or 10000 requests are received at the same time.
> 3. Also, while serving these 100,000 requests, if there are any failures.

YOu will get massive network congestion and collisions if you try. So
yes there will be lots of failures all attributed to the network. YOu
still have not told us what you really want. Why doyou want the answer
to these questions?

The dominant rate limiter is the network-- even if you have a 1Gb
network-- switches, interrupts, etc.
 

You keep being given the answer and you keep asking.

David Woolley <david@ex.djwhome.demon.invalid> wrote:
> However, if they did, the timing is likely to be dominated by
> ethernet collision back-offs.

Collision back-offs? Perhaps - on ancient, hubbed, 100BT or 10BT
networks, but virtually nowhere else :)

rick jones
-- 
denial, anger, bargaining, depression, acceptance, rebirth...
                                     where do you want to be today?
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to rick.jones2 in hp.com  but NOT BOTH...

unruh <unruh@wormhole.physics.ubc.ca> wrote:
> The dominant rate limiter is the network-- even if you have a 1Gb
> network-- switches, interrupts, etc.

Including a 14 byte Ethernet header, an NTP query/response appears to
be 80 bytes or 640 bits.  The 1GbE interframe gap is allowed to be as
low as 64 bit times.  So, call that 640+64 or 704 bits.  That is a
maximum then of a little over 1.42 million queries per second (since
1GbE is full-duplex).  

Now, I have seen aggregate netperf TCP_RR tests achieve in excess of
1.8M transactions (queries) per second over a 10GbE NIC, but never to
a single-threaded process. The 10GbE NIC I was using at the time did
have a low-ish PPS limit.

The most I've seen a single netperf *single-byte* burst-mode test do
is on the order of 350K transactions per second, and that is when
netperf is bound to a core other than the one taking interrupts from
the NIC to get some additional parallelism.  In a netperf TCP_RR test,
netperf does virutally nothing but a send()/recv() pair and a couple
conditionals and adds.

Are there "multi-threaded" NTP server implementations?  If not, I
suspect it is possible for an NTP server to saturate before the 1GbE
network to which it is connected.

rick jones
-- 
oxymoron n, Hummer H2 with California Save Our Coasts and Oceans plates
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

(Shrug)

<BlockQuote> <http://www.pool.ntp.org/en/join.html>
The required bandwidth is relatively low.
 Each client will only send a couple of UDP packets every
  few to every 20 minutes.

Currently most servers get about 5-15 NTP packets per second
  with spikes a couple of times a day of 60-120 packets per second.
 This is roughly equivalent to 10-15Kbit/sec with spikes of
  50-120Kbit/sec.
 In plain terms, you probably need at least 384-512Kbit
  bandwidth (in and out-going).
 Here are a few servers with traffic/load graphs:
  <http://www.pool.ntp.org/scores?ip=147.231.100.5>
</BlockQuote>



<BlockQuote> <http://www.ece.udel.edu/~mills/ntp/html/rate.html>
Some national time metrology laboratories, including NIST
  and USNO, use the NTP reference implementation in their
  very busy public time servers.
 They operate multiple servers behind load-balancing devices
  to support aggregate rates up to ten thousand packets per second.
</BlockQuote>
time-C.timefreq.bldrdoc.gov <http://www.pool.ntp.org/scores/132.163.4.103>



<http://www.google.com/search?q=ntp_stats.txt>

<BlockQuote> <http://www.schlitt.net/ntpstats/ntp_stats.txt>
185074 ntp requests, in total, have been seen since 04/26/09 11:52:37
 56636 (30.6%) are from clients that are still active.
Long term request rate:  8.052 seconds between requests  (0.12 req/sec)
Long term bandwidth in:  0.009 KBytes/s    0.074 Kbits/s
Long term bandwidth in:  0.023 GB/month    0.182 Gb/month
Current request rate:  11.033 seconds between requests  (0.09 req/sec)
Current bandwidth in:  0.007 KBytes/s    0.054 Kbits/s
Current bandwidth in:  0.017 GB/month    0.133 Gb/month
</BlockQuote>

<BlockQuote> <http://tb0.asg-platform.org/ntpstats/ntp_stats.txt>
400691042 ntp requests, in total, have been seen since 03/27/10 18:14:04
 30993181 ( 7.7%) are from clients that are still active.
Long term request rate:  0.039 seconds between requests  (25.92 req/sec)
Long term bandwidth in:  1.924 KBytes/s   15.390 Kbits/s
Long term bandwidth in:  4.755 GB/month   38.042 Gb/month
Current request rate:  0.061 seconds between requests  (16.29 req/sec)
Current bandwidth in:  1.209 KBytes/s    9.670 Kbits/s
Current bandwidth in:  2.988 GB/month   23.904 Gb/month
</BlockQuote>

<BlockQuote> <http://arcdraco.net/ntpstats/ntp_stats.txt>
1244388142 ntp requests, in total, have been seen since 11/13/08 00:48:31
 147157500 (11.8%) are from clients that are still active.
Long term request rate:  0.030 seconds between requests  (33.75 req/sec)
Long term bandwidth in:  2.505 KBytes/s   20.038 Kbits/s
Long term bandwidth in:  6.191 GB/month   49.532 Gb/month
Current request rate:  0.032 seconds between requests  (31.52 req/sec)
Current bandwidth in:  2.339 KBytes/s   18.712 Kbits/s
Current bandwidth in:  5.782 GB/month   46.256 Gb/month
</BlockQuote>

<BlockQuote> <http://riceru.net/ntpstats/ntp_stats.txt>
 677276 ntp requests, in total, have been seen since 07/03/05 03:49:06
 588450 (86.9%) are from clients that are still active.
Long term request rate:  0.074 seconds between requests  (13.46 req/sec)
Long term bandwidth in:  0.999 KBytes/s    7.990 Kbits/s
Long term bandwidth in:  2.469 GB/month   19.750 Gb/month
Current request rate:  0.074 seconds between requests  (13.51 req/sec)
Current bandwidth in:  1.003 KBytes/s    8.022 Kbits/s
Current bandwidth in:  2.479 GB/month   19.831 Gb/month
(NTP packets are usually 76 bytes, UDP overhead included, in each direction.)
</BlockQuote>

<BlockQuote> <http://meine.uhr.geht.net/meine.uhr.geht.net/ntp_stats.txt>
 38491757 ntp requests, in total, have been seen since 04/26/10 00:40:07
 17270364 (44.9%) are from clients that are still active.
Long term request rate:  0.406 seconds between requests  (2.46 req/sec)
Long term bandwidth in:  0.183 KBytes/s    1.463 Kbits/s
Long term bandwidth in:  0.452 GB/month    3.617 Gb/month
Current request rate:  0.409 seconds between requests  (2.44 req/sec)
Current bandwidth in:  0.181 KBytes/s    1.451 Kbits/s
Current bandwidth in:  0.448 GB/month    3.586 Gb/month
(NTP packets are usually 76 bytes, UDP overhead included, in each direction.)
</BlockQuote>

-- 
E-Mail Sent to this address <BlackList@Anitech-Systems.com>
  will be added to the BlackLists.

>The most I've seen a single netperf *single-byte* burst-mode test do
>is on the order of 350K transactions per second, and that is when
>netperf is bound to a core other than the one taking interrupts from
>the NIC to get some additional parallelism.  In a netperf TCP_RR test,
>netperf does virutally nothing but a send()/recv() pair and a couple
>conditionals and adds.

Is that sending 1 byte of data per packet, or batching up
several/many bytes per packet?  (and mostly measuring
the CPU time for a one byte send/recv pair)


Even if you have a test program that blasts lots of packets,
that won't mimmic real traffic.  It's bypassing all the setup
of ARP and router slots.

The only really useful data I've seen is the numbers from the
people running heavily loaded servers.


-- 
These are my opinions, not necessarily my employer's.  I hate spam.

Hal Murray <hal-usenet@ip-64-139-1-69.sjc.megapath.net> wrote:

> >The most I've seen a single netperf *single-byte* burst-mode test do
> >is on the order of 350K transactions per second, and that is when
> >netperf is bound to a core other than the one taking interrupts from
> >the NIC to get some additional parallelism.  In a netperf TCP_RR test,
> >netperf does virutally nothing but a send()/recv() pair and a couple
> >conditionals and adds.

> Is that sending 1 byte of data per packet, or batching up
> several/many bytes per packet?  (and mostly measuring the CPU time
> for a one byte send/recv pair)

I seek to avoid batching by setting TCP_NODELAY on the socket and then
do spoit-checks on the link-level packet counts.

> Even if you have a test program that blasts lots of packets, that
> won't mimmic real traffic.  It's bypassing all the setup of ARP and
> router slots.

I'm not quite sure if I'm parsing that accurately - yes, such a
single-connection test between a pair of systems isn't doing much
to/with the ARP cache or routing tables, which means that if anything
for the 100000 client case the 350K transactions per second is
optimistic (*).  I should have been more clear - my main intent was to
show/suggest that the server could very likely indeed bottleneck
before a 1GbE link could.  Sort of an inverted (term?) "New York, New
York" (the song argument) where "If you can't make it there, you won't
make it anywhere" where netperf is "there" and the ntpd would be
"anywhere."

rick jones

* then again, I'd hope there aren't 100000 clients in the same
broadcast domain, which suggests (ok, yes, I'm extrapolating) that
almost all the clients will be reached by the ntp server via its
default route, and thus also have only one active ARP cache entry -
much like the single-connection netperf test between systems on the
same LAN.

-- 
firebug n, the idiot who tosses a lit cigarette out his car window
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Hi--

On Sep 28, 2010, at 10:07 AM, Rick Jones wrote:
>> Even if you have a test program that blasts lots of packets, that
>> won't mimmic real traffic.  It's bypassing all the setup of ARP and
>> router slots.
> 
> I'm not quite sure if I'm parsing that accurately - yes, such a
> single-connection test between a pair of systems isn't doing much
> to/with the ARP cache or routing tables, which means that if anything
> for the 100000 client case the 350K transactions per second is
> optimistic (*).  I should have been more clear - my main intent was to
> show/suggest that the server could very likely indeed bottleneck
> before a 1GbE link could. [ ... ]
> 
> rick jones
> 
> * then again, I'd hope there aren't 100000 clients in the same
> broadcast domain, which suggests (ok, yes, I'm extrapolating) that
> almost all the clients will be reached by the ntp server via its
> default route, and thus also have only one active ARP cache entry -
> much like the single-connection netperf test between systems on the
> same LAN.

In point of fact, most ethernet switches are limited to a few thousand MAC addresses-- for example, the ProCurve 26xx switches have a MAC table size of 8000.  Even a core switch is unlikely to support larger than 65K MAC table size.  But, if you wanted to assume that you have 100K clients locally, at default maxpoll they'd be generating one query every 1024 seconds, or an average of 2000pps, which ntpd will have no problems handling.

For a more normal case, the NTP server is handling requests from the Internet which are not coming from machines in the local broadcast domain-- in which case, agreed, there would only be a single ARP entry between the NTP server and it's upstream router.  In such a case, your bottleneck is still much more likely to be the capability of the router/firewall to handle large numbers of tiny packets, especially if NAT is being used or if connection state is being kept by a firewall.

Anyone adding their NTP server to the pool will discover these issues, especially if they are using consumer-grade DSL/FiOS/cable router boxes rather than telco-grade T1/T3/100Mbs/OC3/OC12/GBE routers for their Internet connectivity.  They'll also discover spikes of thousands to tens of thousands of NTP requests per second when your IP gets rotated into the pool's active DNS and Turk Telekom hits you.  That probably does correspond to on the order of 100+K clients....

Regards,
-- 
-Chuck