Hi,

This probably isn't the place to post this, if so, I will apologise in
advance..

I have  a problem that really baffles me, having run ntp servers for
years.

I have two ntp servers set up in the local network, from which another
40 odd servers synchronise.

Some of the servers do function as expected, and keep time quite
accurately, however, the
majority of   servers rapport an error that I cannot lock down, and it's
currently baffling me...  
(It's probably  something obvious I'm overlooking or so I hope).

The logs on the systems that have problems usually show the following
pattern


Sep  7 17:43:48 sn ntpd[11721]: synchronized to 10.7.100.28, stratum 2
Sep  7 17:57:51 sn ntpd[11721]: time reset +71.784598 s
Sep  7 17:59:10 sn ntpd[11721]: synchronized to 10.7.100.27, stratum 2
Sep  8 05:29:11 sn ntpd[11721]: no servers reachable
Sep  8 05:34:37 sn ntpd[11721]: synchronized to 10.7.100.28, stratum 2
Sep  8 05:35:52 sn ntpd[11721]: time reset +74.977115 s
Sep  8 05:36:41 sn ntpd[11721]: synchronized to 10.7.100.28, stratum 2

It appears to have an issue nearly exactly every 12 hours, the time
difference is getting worse, 
it started at around 1-2 seconds, and has steadily increased, and how
deviate with 74 seconds.

One day I was watching the event, and saw the machine go from being +/-
4 ms out to suddenly 
becoming +/- 36000+ out, so it appears to be something specific that
causes the problem. 

The strange thing is that the servers appear to be fully synchronised
for most of that time, but 
suddenly the jitter increases dramatically, and the clocks offset change
to something very 
large, and then is reset by ntp a few cycles later - I've decreased the
poll times to be maximum 
of 256 seconds, as that seems to correct the problem faster.

The configuration of all servers (ntp.conf) is

restrict default nomodify notrap noquery
restrict 127.0.0.1
server ntp1.i.tv2.dk minpoll 1 maxpoll 8 iburst
server ntp2.i.tv2.dk minpoll 1 maxpoll 8 iburst
server ntp.i.tv2.dk  iburst

tinker huffpuff 7200
driftfile /var/lib/ntp/drift
broadcastdelay  0.008
keys            /etc/ntp/keys

The versions of ntp that's in use, varies a lot with the machine ages,
operating systems, 
and so forth, strangely enough it seems that most machines have the same
problem.    
Similiarly the systems are running suse, rh 4/5, on different kernel
versions.  My initial 
instinct was that it might have been network related.

NB: the tinker huffpuff 7200 was something that was added to see if it
had any effect 
or not,   minpoll, maxpoll values were added as they seem to improve
recovery from the 
problem, however it's treating the symptom.

As far as I've ascertained, the ntp servers are at no time unreachable,
and do not appear 
to ever loose sync with their clock sources.    Their logs indicate that
the occasionally 
- ever hour or so, that they do change between different time servers.

Any ideas are most welcome.

Michael,

Michael Nielsen wrote:
> The logs on the systems that have problems usually show the following
> pattern
> 
> 
> Sep  7 17:43:48 sn ntpd[11721]: synchronized to 10.7.100.28, stratum 2
> Sep  7 17:57:51 sn ntpd[11721]: time reset +71.784598 s
> Sep  7 17:59:10 sn ntpd[11721]: synchronized to 10.7.100.27, stratum 2
> Sep  8 05:29:11 sn ntpd[11721]: no servers reachable
> Sep  8 05:34:37 sn ntpd[11721]: synchronized to 10.7.100.28, stratum 2
> Sep  8 05:35:52 sn ntpd[11721]: time reset +74.977115 s
> Sep  8 05:36:41 sn ntpd[11721]: synchronized to 10.7.100.28, stratum 2

This looks like some other program is called in cyclic intervals (~12 hours,
run by cron?) to simply set the time to match the time of some external
time source.

After this has happened the offset and jitter figures reported by "ntpq -p"
should increase drastically, and about 15 minutes (or a certain number of
polling intervals) after the system time has been stepped by the other
program the time is stepped back by ntpd, causing the "time reset"
messages.

If the reference time source used by the other program continuously drifts
apart from the reference time source(s) used by ntpd then you should see
the amount of time reported by the "time reset"s increase or decrease
continuously.

Martin
-- 
Martin Burnicki

Meinberg Funkuhren
Bad Pyrmont
Germany

T24gV2VkLCBTZXAgOCwgMjAxMCBhdCAwNjo0MCBVVEMsIE1pY2hhZWwgTmllbHNlbiA8aGFlbEB0
djIuZGs+IHdyb3RlOgo+IEkgaGF2ZSB0d28gbnRwIHNlcnZlcnMgc2V0IHVwIGluIHRoZSBsb2Nh
bCBuZXR3b3JrLCBmcm9tIHdoaWNoIGFub3RoZXIKPiA0MCBvZGQgc2VydmVycyBzeW5jaHJvbmlz
ZS4KPgo+IFNvbWUgb2YgdGhlIHNlcnZlcnMgZG8gZnVuY3Rpb24gYXMgZXhwZWN0ZWQsIGFuZCBr
ZWVwIHRpbWUgcXVpdGUKPiBhY2N1cmF0ZWx5LCBob3dldmVyLCB0aGUKPiBtYWpvcml0eSBvZiDC
oCBzZXJ2ZXJzIHJhcHBvcnQgYW4gZXJyb3IgdGhhdCBJIGNhbm5vdCBsb2NrIGRvd24sIGFuZCBp
dCdzCj4gY3VycmVudGx5IGJhZmZsaW5nIG1lLi4uCj4gKEl0J3MgcHJvYmFibHkgwqBzb21ldGhp
bmcgb2J2aW91cyBJJ20gb3Zlcmxvb2tpbmcgb3Igc28gSSBob3BlKS4KPgo+IFRoZSBsb2dzIG9u
IHRoZSBzeXN0ZW1zIHRoYXQgaGF2ZSBwcm9ibGVtcyB1c3VhbGx5IHNob3cgdGhlIGZvbGxvd2lu
Zwo+IHBhdHRlcm4KPgo+IFNlcCDCoDcgMTc6NDM6NDggc24gbnRwZFsxMTcyMV06IHN5bmNocm9u
aXplZCB0byAxMC43LjEwMC4yOCwgc3RyYXR1bSAyCj4gU2VwIMKgNyAxNzo1Nzo1MSBzbiBudHBk
WzExNzIxXTogdGltZSByZXNldCArNzEuNzg0NTk4IHMKPiBTZXAgwqA3IDE3OjU5OjEwIHNuIG50
cGRbMTE3MjFdOiBzeW5jaHJvbml6ZWQgdG8gMTAuNy4xMDAuMjcsIHN0cmF0dW0gMgo+IFNlcCDC
oDggMDU6Mjk6MTEgc24gbnRwZFsxMTcyMV06IG5vIHNlcnZlcnMgcmVhY2hhYmxlCj4gU2VwIMKg
OCAwNTozNDozNyBzbiBudHBkWzExNzIxXTogc3luY2hyb25pemVkIHRvIDEwLjcuMTAwLjI4LCBz
dHJhdHVtIDIKPiBTZXAgwqA4IDA1OjM1OjUyIHNuIG50cGRbMTE3MjFdOiB0aW1lIHJlc2V0ICs3
NC45NzcxMTUgcwo+IFNlcCDCoDggMDU6MzY6NDEgc24gbnRwZFsxMTcyMV06IHN5bmNocm9uaXpl
ZCB0byAxMC43LjEwMC4yOCwgc3RyYXR1bSAyCgpTaW5jZSB0aGUgcHJvYmxlbSBoYXBwZW5zIG9u
IG1vc3QgYnV0IG5vdCBhbGwgTlRQIGNsaWVudHMsIHRoZSBOVFAKc2VydmVycyBjYW4gcHJvYmFi
bHkgYmUgYXNzdW1lZCB0byBiZSB3b3JraW5nIGNvcnJlY3RseS4gIEFyZSBzb21lIG9mCnRoZXNl
IDQwIE5UUCBjbGllbnRzIHZpcnR1YWwgbWFjaGluZXM/ICBBIHJlcGVhdGVkIHBhdHRlcm4gb2Yg
bG9zaW5nCnRpbWUgZHVlIHRvIGxvc3QgaW50ZXJydXB0cyBjb3VsZCBiZSBjYXVzZWQgYnkgdGhl
IHZpcnR1YWxpemF0aW9uCmVudmlyb25tZW50IGFuZCB0aGUgbG9hZCBvbiB0aGUgcmVhbCBoYXJk
d2FyZS4KCklmIHRoYXQgZG9lc24ndCBwYW4gb3V0LCBjb21wYXJlIGxvZ3MgZnJvbSBtdWx0aXBs
ZSBOVFAgY2xpZW50cwpjb3ZlcmluZyB0aGUgc2FtZSB0aW1lIHBlcmlvZCB0byBzZWUgaWYgdGhl
cmUncyBhIHBhdHRlcm4gb2Ygc2VydmVyCm1pc2JlaGF2aW9yIHRvIGJlIGRldGVjdGVkLgoKR29v
ZCBodW50aW5nLApEYXZlIEhhcnQKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX18KcXVlc3Rpb25zIG1haWxpbmcgbGlzdApxdWVzdGlvbnNAbGlzdHMubnRwLm9y
ZwpodHRwOi8vbGlzdHMubnRwLm9yZy9saXN0aW5mby9xdWVzdGlvbnM=

DQoNCi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQpGcm9tOiBEYXZlIEhhcnQgW21haWx0bzpo
YXJ0QG50cC5vcmddIA0KU2VudDogV2VkbmVzZGF5LCA4IFNlcHRlbWJlciAyMDEwIDU6MzMgUE0N
ClRvOiBNaWNoYWVsIE5pZWxzZW4NCkNjOiBxdWVzdGlvbnNAbGlzdHMubnRwLm9yZw0KU3ViamVj
dDogUmU6IFtudHA6cXVlc3Rpb25zXSBOVFBkIGxvb3NlcyBzeW5jIHJlZ3VsYXJseSAvIDEyIGhv
dXIgaW50ZXJ2YWxzLg0KDQpPbiBXZWQsIFNlcCA4LCAyMDEwIGF0IDA2OjQwIFVUQywgTWljaGFl
bCBOaWVsc2VuIDxoYWVsQHR2Mi5kaz4gd3JvdGU6DQo+IEkgaGF2ZSB0d28gbnRwIHNlcnZlcnMg
c2V0IHVwIGluIHRoZSBsb2NhbCBuZXR3b3JrLCBmcm9tIHdoaWNoIGFub3RoZXINCj4gNDAgb2Rk
IHNlcnZlcnMgc3luY2hyb25pc2UuDQo+DQo+IFNvbWUgb2YgdGhlIHNlcnZlcnMgZG8gZnVuY3Rp
b24gYXMgZXhwZWN0ZWQsIGFuZCBrZWVwIHRpbWUgcXVpdGUNCj4gYWNjdXJhdGVseSwgaG93ZXZl
ciwgdGhlDQo+IG1ham9yaXR5IG9mIMKgIHNlcnZlcnMgcmFwcG9ydCBhbiBlcnJvciB0aGF0IEkg
Y2Fubm90IGxvY2sgZG93biwgYW5kIGl0J3MNCj4gY3VycmVudGx5IGJhZmZsaW5nIG1lLi4uDQo+
IChJdCdzIHByb2JhYmx5IMKgc29tZXRoaW5nIG9idmlvdXMgSSdtIG92ZXJsb29raW5nIG9yIHNv
IEkgaG9wZSkuDQo+DQo+IFRoZSBsb2dzIG9uIHRoZSBzeXN0ZW1zIHRoYXQgaGF2ZSBwcm9ibGVt
cyB1c3VhbGx5IHNob3cgdGhlIGZvbGxvd2luZw0KPiBwYXR0ZXJuDQo+DQo+IFNlcCDCoDcgMTc6
NDM6NDggc24gbnRwZFsxMTcyMV06IHN5bmNocm9uaXplZCB0byAxMC43LjEwMC4yOCwgc3RyYXR1
bSAyDQo+IFNlcCDCoDcgMTc6NTc6NTEgc24gbnRwZFsxMTcyMV06IHRpbWUgcmVzZXQgKzcxLjc4
NDU5OCBzDQo+IFNlcCDCoDcgMTc6NTk6MTAgc24gbnRwZFsxMTcyMV06IHN5bmNocm9uaXplZCB0
byAxMC43LjEwMC4yNywgc3RyYXR1bSAyDQo+IFNlcCDCoDggMDU6Mjk6MTEgc24gbnRwZFsxMTcy
MV06IG5vIHNlcnZlcnMgcmVhY2hhYmxlDQo+IFNlcCDCoDggMDU6MzQ6Mzcgc24gbnRwZFsxMTcy
MV06IHN5bmNocm9uaXplZCB0byAxMC43LjEwMC4yOCwgc3RyYXR1bSAyDQo+IFNlcCDCoDggMDU6
MzU6NTIgc24gbnRwZFsxMTcyMV06IHRpbWUgcmVzZXQgKzc0Ljk3NzExNSBzDQo+IFNlcCDCoDgg
MDU6MzY6NDEgc24gbnRwZFsxMTcyMV06IHN5bmNocm9uaXplZCB0byAxMC43LjEwMC4yOCwgc3Ry
YXR1bSAyDQoNClNpbmNlIHRoZSBwcm9ibGVtIGhhcHBlbnMgb24gbW9zdCBidXQgbm90IGFsbCBO
VFAgY2xpZW50cywgdGhlIE5UUA0Kc2VydmVycyBjYW4gcHJvYmFibHkgYmUgYXNzdW1lZCB0byBi
ZSB3b3JraW5nIGNvcnJlY3RseS4gIEFyZSBzb21lIG9mDQp0aGVzZSA0MCBOVFAgY2xpZW50cyB2
aXJ0dWFsIG1hY2hpbmVzPyAgQSByZXBlYXRlZCBwYXR0ZXJuIG9mIGxvc2luZw0KdGltZSBkdWUg
dG8gbG9zdCBpbnRlcnJ1cHRzIGNvdWxkIGJlIGNhdXNlZCBieSB0aGUgdmlydHVhbGl6YXRpb24N
CmVudmlyb25tZW50IGFuZCB0aGUgbG9hZCBvbiB0aGUgcmVhbCBoYXJkd2FyZS4NCg0KSWYgdGhh
dCBkb2Vzbid0IHBhbiBvdXQsIGNvbXBhcmUgbG9ncyBmcm9tIG11bHRpcGxlIE5UUCBjbGllbnRz
DQpjb3ZlcmluZyB0aGUgc2FtZSB0aW1lIHBlcmlvZCB0byBzZWUgaWYgdGhlcmUncyBhIHBhdHRl
cm4gb2Ygc2VydmVyDQptaXNiZWhhdmlvciB0byBiZSBkZXRlY3RlZC4NCg0KR29vZCBodW50aW5n
LA0KRGF2ZSBIYXJ0DQoNCg0KWWVzIHRoZSBjbGllbnRzIGFyZSB2aXJ0dWFsIG1hY2hpbmVzLCBo
b3dldmVyLCB0aGV5IHRpbWUgdmFyaWF0aW9uIG9jY3VycyBpbiB2ZXJ5IHNob3J0IHdpbmRvd3Ms
IGFuZCBkb2VzIG5vdCBzbG93bHkgY3JlZXAgdXAsIHdoaWNoIHJlYWxseSBjb25mdXNlcyB0aGUg
aGVjayBvdXQgb2YgbWUuIA0KDQoNCkkgd291bGQgZXhwZWN0IHRoYXQgaWYgaXQgd2FzIGEgbG9z
cyBvZiBpbnRlcnJ1cHQsIEkgc2hvdWxkIHNlZSB0aGVtIGRyaWZ0IGFwYXJ0LCByYXRoZXIgdGhh
biBhIHNpbmdsZSBzdGVwLg0KDQpSZWdhcmRzDQogICBNaWtlLg0KX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX18KcXVlc3Rpb25zIG1haWxpbmcgbGlzdApxdWVz
dGlvbnNAbGlzdHMubnRwLm9yZwpodHRwOi8vbGlzdHMubnRwLm9yZy9saXN0aW5mby9xdWVzdGlv
bnM=

Here are some more details from the ntp logging - ntp_mon peerstats..

It's doesn't seem to be a interrupt creep, as the server is maintaining
time properly all day, except for 30 mins each day.

So I guess it's a program I need to look for, any ideas as to what I
should look for ?

Regards
   Mike.

55449 8029.459 10.7.100.27 9634 -0.002910845 0.000745249 0.014838494
0.000182297
55449 8043.473 10.7.100.28 9434 0.001379801 0.000747673 0.007061521
0.000157649
55449 8299.732 10.7.100.28 9434 0.001414736 0.000731931 0.005022575
0.000115537
55449 8555.991 10.7.100.28 9434 0.001414736 0.000731931 0.007541755
0.000068723
55449 8813.250 10.7.100.28 9434 0.001414736 0.000731931 0.009477517
0.000031755
55449 9056.494 10.7.100.27 9634 -0.002179427 0.000873918 0.014844999
0.000731418
55449 9070.508 10.7.100.28 9434 0.001414736 0.000731931 0.010334099
0.000040580
55449 9239.820 10.7.100.28 9434 0.001414736 0.000731931 0.010206775
43.473413580
55449 9498.080 10.7.100.28 9034 86.948205334 0.000737279 0.004326775
75.298106172
55449 9754.338 10.7.100.28 9034 86.948171299 0.000713359 0.003715583
61.480627575
55449 9994.579 10.7.100.27 9034 86.945547774 0.000796132 0.014833280
86.947727202
55449 10011.597 10.7.100.28 9034 86.948171299 0.000713359 0.005644196
43.473347566
55449 10029.614 10.7.100.28 9034 86.947986361 0.000411148 0.002904350
38.883661440
55449 10046.631 10.7.100.28 9034 86.947986361 0.000411148 0.001597708
35.495764148
55449 10062.648 10.7.100.28 9034 86.947984728 0.000403990 0.000880227
32.862691288
55449 10078.664 10.7.100.28 9034 86.947984728 0.000403990 0.000680422
0.000158151
55449 10093.679 10.7.100.28 9034 86.947984728 0.000403990 0.000700578
0.000124794
55449 10111.696 10.7.100.28 9034 86.947984728 0.000403990 0.000636477
0.000092856
55449 10126.712 10.7.100.28 9034 86.947984728 0.000403990 0.000652824
0.000060522
55449 10144.730 10.7.100.28 9034 86.947984728 0.000403990 0.000884035
0.000005692
55449 10160.746 10.7.100.28 9034 86.947984728 0.000403990 0.000954348
0.000006893
55449 10263.711 10.7.100.28 8034 0.000000000 0.000000000 0.000000000
4.000000000
55449 10265.714 10.7.100.28 9044 -0.000018194 0.000384368 7.937500957
0.000000954
55449 10280.728 10.7.100.28 9044 -0.000029944 0.000349054 3.937557685
0.000011750
55449 10298.746 10.7.100.28 9044 -0.000029944 0.000349054 1.937760424
0.000031646

-----Original Message-----
From: questions-bounces+hael=tv2.dk@lists.ntp.org
[mailto:questions-bounces+hael=tv2.dk@lists.ntp.org] On Behalf Of Martin
Burnicki
Sent: Wednesday, 8 September 2010 5:26 PM
To: questions@lists.ntp.org
Subject: Re: NTPd looses sync regularly / 12 hour
intervals.

Michael,

Michael Nielsen wrote:
> The logs on the systems that have problems usually show the following
> pattern
> 
> 
> Sep  7 17:43:48 sn ntpd[11721]: synchronized to 10.7.100.28, stratum 2
> Sep  7 17:57:51 sn ntpd[11721]: time reset +71.784598 s
> Sep  7 17:59:10 sn ntpd[11721]: synchronized to 10.7.100.27, stratum 2
> Sep  8 05:29:11 sn ntpd[11721]: no servers reachable
> Sep  8 05:34:37 sn ntpd[11721]: synchronized to 10.7.100.28, stratum 2
> Sep  8 05:35:52 sn ntpd[11721]: time reset +74.977115 s
> Sep  8 05:36:41 sn ntpd[11721]: synchronized to 10.7.100.28, stratum 2

This looks like some other program is called in cyclic intervals (~12
hours,
run by cron?) to simply set the time to match the time of some external
time source.

After this has happened the offset and jitter figures reported by "ntpq
-p"
should increase drastically, and about 15 minutes (or a certain number
of
polling intervals) after the system time has been stepped by the other
program the time is stepped back by ntpd, causing the "time reset"
messages.

If the reference time source used by the other program continuously
drifts
apart from the reference time source(s) used by ntpd then you should see
the amount of time reported by the "time reset"s increase or decrease
continuously.

Martin
-- 
Martin Burnicki

Meinberg Funkuhren
Bad Pyrmont
Germany

_______________________________________________
questions mailing list
questions@lists.ntp.org
http://lists.ntp.org/listinfo/questions

Michael,

sorry for the late reply.

Michael Nielsen wrote:
> Here are some more details from the ntp logging - ntp_mon peerstats..
> 
> It's doesn't seem to be a interrupt creep, as the server is maintaining
> time properly all day, except for 30 mins each day.

So once again this meets perfectly my assumption. The system time is stepped
by some program, and a few minutes later ntpd steps the system time back.

And, since your first post the offset between the time sources used by ntpd
and the reference time used by the extra program has increased from about
70 seconds to to 87 seconds ...

> So I guess it's a program I need to look for, any ideas as to what I
> should look for ?

Unfortunately not. This can be any program calling "ntpdate -u" or rdate or
whatever in some intervals.

Some time ago I had a customer who tried to set synchronize computer times
using NTP, and we found out he had a 3rd party client/server application
running where the client "synchronized" its time to its server's time in
periodic intervals, using a proprietary protocol with 1 s resolution only.
This messed up all attempts by ntpd to get the system time synchronized
properly because the maximum error was always 1 s due to the granularity of
the protocol.

Anyway, that was under Windows, but basically the OS doesn't matter for
cases like this.

Martin
-- 
Martin Burnicki

Meinberg Funkuhren
Bad Pyrmont
Germany