Hello,

We've looking to retire a nameserver in our environment. We've made
changes to the Solaris clients' /etc/resolv.conf, but some processes
are still sending DNS requests to this "old" nameserver.

We've restarted several system processes (e.g., autofs, nfs.client,
syslog, sendmail, ldap.client, etc.), but something is still
referencing the "old" nameserver.

Is there a way to determine which process is responsible for this
traffic? dtrace is unfortunately not an option, as in this case the
Solaris systems are not running Solaris 10.

Any other solutions using truss or lsof? We can of course reboot the
systems as a last resort.

Thanks in advance!

Brandon

In article 
<0fd950e2-cb8c-4933-85c7-383a50b787eb@u37g2000vbc.googlegroups.com>,
 Brandon Hutchinson <bhutch@gmail.com> wrote:

> Hello,
> 
> We've looking to retire a nameserver in our environment. We've made
> changes to the Solaris clients' /etc/resolv.conf, but some processes
> are still sending DNS requests to this "old" nameserver.
> 
> We've restarted several system processes (e.g., autofs, nfs.client,
> syslog, sendmail, ldap.client, etc.), but something is still
> referencing the "old" nameserver.
> 
> Is there a way to determine which process is responsible for this
> traffic? dtrace is unfortunately not an option, as in this case the
> Solaris systems are not running Solaris 10.
> 
> Any other solutions using truss or lsof? We can of course reboot the
> systems as a last resort.
> 
> Thanks in advance!
> 
> Brandon

I think a reboot is your fastest and best option.  Schedule it.

-- 
DeeDee, don't press that button!  DeeDee!  NO!  Dee...
[I filter all Goggle Groups posts, so any reply may be automatically by ignored]

On Dec 16, 4:03=A0pm, Brandon Hutchinson <bhu...@gmail.com> wrote:
> Hello,
>
> We've looking to retire a nameserver in our environment. We've made
> changes to the Solaris clients' /etc/resolv.conf, but some processes
> are still sending DNS requests to this "old" nameserver.
>
> We've restarted several system processes (e.g., autofs, nfs.client,
> syslog, sendmail, ldap.client, etc.), but something is still
> referencing the "old" nameserver.
>
> Is there a way to determine which process is responsible for this
> traffic? dtrace is unfortunately not an option, as in this case the
> Solaris systems are not running Solaris 10.
>
> Any other solutions using truss or lsof? We can of course reboot the
> systems as a last resort.
>
> Thanks in advance!
>
> Brandon

snoop

Brandon Hutchinson wrote:
> Hello,
> 
> We've looking to retire a nameserver in our environment. We've made
> changes to the Solaris clients' /etc/resolv.conf, but some processes
> are still sending DNS requests to this "old" nameserver.
> 
> We've restarted several system processes (e.g., autofs, nfs.client,
> syslog, sendmail, ldap.client, etc.), but something is still
> referencing the "old" nameserver.
> 
> Is there a way to determine which process is responsible for this
> traffic? dtrace is unfortunately not an option, as in this case the
> Solaris systems are not running Solaris 10.
> 
> Any other solutions using truss or lsof? We can of course reboot the
> systems as a last resort.
> 
> Thanks in advance!
> 
> Brandon

I don't think you have reboot the whole server.  See the man page for 
"kill" with special attention to "kill -1".  I think that will restart 
the application and cause it to reread any initialization files.

If worst comes to worst you can use sys-unconfig (see the man page for 
it) and redo your configuration.  That wipes the existing configuration 
and does a reboot after which you will be asked for a new configuration 
just as you were when you first installed Solaris.

Before you start be sure that you know the correct node name, IP address 
or addresses, the date, time,. . . .   Go thou and RTFM.  It's all in there.

Hello,

>
> I don't think you have reboot the whole server. =A0See the man page for
> "kill" with special attention to "kill -1". =A0I think that will restart
> the application and cause it to reread any initialization files.
>
> If worst comes to worst you can use sys-unconfig (see the man page for
> it) and redo your configuration. =A0That wipes the existing configuration
> and does a reboot after which you will be asked for a new configuration
> just as you were when you first installed Solaris.
>
> Before you start be sure that you know the correct node name, IP address
> or addresses, the date, time,. . . . =A0 Go thou and RTFM. =A0It's all in=
 there.

I may not have been clear in my original post, but I know which
systems are generating the DNS traffic to the "old" /etc/resolv.conf
nameserver entries. I want to find out which process(es) on these
systems are generating this DNS traffic so I can restart them.

We may end up just rebooting these systems, but I was wondering if it
is possible to figure this out using lsof, truss, etc.

Thanks,

Brandon

In article 
<734ee302-fb7e-4491-a2e2-720a4eb3e924@o9g2000vbj.googlegroups.com>,
 Brandon Hutchinson <bhutch@gmail.com> wrote:

> Hello,
> 
> >
> > I don't think you have reboot the whole server. �See the man page for
> > "kill" with special attention to "kill -1". �I think that will restart
> > the application and cause it to reread any initialization files.
> >
> > If worst comes to worst you can use sys-unconfig (see the man page for
> > it) and redo your configuration. �That wipes the existing configuration
> > and does a reboot after which you will be asked for a new configuration
> > just as you were when you first installed Solaris.
> >
> > Before you start be sure that you know the correct node name, IP address
> > or addresses, the date, time,. . . . � Go thou and RTFM. �It's all in there.
> 
> I may not have been clear in my original post, but I know which
> systems are generating the DNS traffic to the "old" /etc/resolv.conf
> nameserver entries. I want to find out which process(es) on these
> systems are generating this DNS traffic so I can restart them.
> 
> We may end up just rebooting these systems, but I was wondering if it
> is possible to figure this out using lsof, truss, etc.
> 
> Thanks,
> 
> Brandon

I'm sure it is possible to figure out what processes are generating the 
traffic if you use snoop or a network analyzer's capture.  You can see 
what packets came from what MAC address.  AFAIK (which isn't very much I 
admit), you could look at the open network ports on the machine 
generating the traffic with lsof and netstat, then trace what those 
processes might be that are generating the traffic.  If it's just a 
couple machines, do the research and tracing out, restart the processes 
(assuming they respond to hup -1 and will reload their initialization 
files), and you should be good.  If you miss something, you may have to 
reboot anyway, but you've learned something in the meantime.  You're 
probably gonna have to figure out how to do this, then post your results.

If you have to do this for more than 5 systems, I'd say "fuck it" and 
just reboot them.   You'll get them all in a lot less time than futzing 
with each system.  How much time do you want to spend on this project?  
60 minutes or all day?

-- 
DeeDee, don't press that button!  DeeDee!  NO!  Dee...
[I filter all Goggle Groups posts, so any reply may be automatically by ignored]

Am Wed, 16 Dec 2009 13:58:37 -0800 schrieb Brandon Hutchinson:

> Hello,
> 
>>
>> I don't think you have reboot the whole server.  See the man page for
>> "kill" with special attention to "kill -1".  I think that will restart
>> the application and cause it to reread any initialization files.
>>
>> If worst comes to worst you can use sys-unconfig (see the man page for
>> it) and redo your configuration.  That wipes the existing configuration
>> and does a reboot after which you will be asked for a new configuration
>> just as you were when you first installed Solaris.
>>
>> Before you start be sure that you know the correct node name, IP address
>> or addresses, the date, time,. . . .   Go thou and RTFM.  It's all in there.
> 
> I may not have been clear in my original post, but I know which
> systems are generating the DNS traffic to the "old" /etc/resolv.conf
> nameserver entries. I want to find out which process(es) on these
> systems are generating this DNS traffic so I can restart them.
> 
> We may end up just rebooting these systems, but I was wondering if it
> is possible to figure this out using lsof, truss, etc.
> 
> Thanks,
> 
> Brandon

look for nscd  svc:/system/name-service-cache:default

-- 
12. Gebot: Wenn Ihr eine Fahrradklingel hört, wechsele die, die links geht
nach rechts, die die rechts geht nach links, aber nicht bis zum Straßenrand.
Danach dreht Euch um, reißt Mund, Nase und Augen auf, tretet aber keinesfalls
zur Seite.

In article <pan.2009.12.17.08.11.01.187003@2010.antispam.de>,
 Horst Scheuermann <hs01@2010.antispam.de> wrote:

> Am Wed, 16 Dec 2009 13:58:37 -0800 schrieb Brandon Hutchinson:
> 
> > Hello,
> > 
> >>
> >> I don't think you have reboot the whole server. �See the man page for
> >> "kill" with special attention to "kill -1". �I think that will restart
> >> the application and cause it to reread any initialization files.
> >>
> >> If worst comes to worst you can use sys-unconfig (see the man page for
> >> it) and redo your configuration. �That wipes the existing configuration
> >> and does a reboot after which you will be asked for a new configuration
> >> just as you were when you first installed Solaris.
> >>
> >> Before you start be sure that you know the correct node name, IP address
> >> or addresses, the date, time,. . . . � Go thou and RTFM. �It's all in 
> >> there.
> > 
> > I may not have been clear in my original post, but I know which
> > systems are generating the DNS traffic to the "old" /etc/resolv.conf
> > nameserver entries. I want to find out which process(es) on these
> > systems are generating this DNS traffic so I can restart them.
> > 
> > We may end up just rebooting these systems, but I was wondering if it
> > is possible to figure this out using lsof, truss, etc.
> > 
> > Thanks,
> > 
> > Brandon
> 
> look for nscd  svc:/system/name-service-cache:default

If you're just joining this thread, he's not running Solaris 10.  He 
didn't say exactly, but it's older.  So, no svc.  nscd is a good thing 
to look for on the older systems if they're running it, though.

-- 
DeeDee, don't press that button!  DeeDee!  NO!  Dee...
[I filter all Goggle Groups posts, so any reply may be automatically by ignored]

Brandon Hutchinson <bhutch@gmail.com> wrote:
> Hello,
> 
> We've looking to retire a nameserver in our environment. We've made
> changes to the Solaris clients' /etc/resolv.conf, but some processes
> are still sending DNS requests to this "old" nameserver.
> 
> We've restarted several system processes (e.g., autofs, nfs.client,
> syslog, sendmail, ldap.client, etc.), but something is still
> referencing the "old" nameserver.
> 
> Is there a way to determine which process is responsible for this
> traffic? dtrace is unfortunately not an option, as in this case the
> Solaris systems are not running Solaris 10.
> 
> Any other solutions using truss or lsof? We can of course reboot the
> systems as a last resort.

Not really a 'how to' answer, but are you running any of Sun's messaging
products (webmail, ims, etc.)? We've found that most of them will read
resolv.conf at start-time, and then cache the information.

That would be my guess: Sun products that aren't part of a base install.
As far as I've ever seen, any part of the normal OS infrastructure
(syslog, nfs, nis, sendmail, etc.) will behave as expected, and parse
resolv.conf as used.

I assume you've scoured the logfiles. If you're willing to risk breaking
(temporarily) the mystery application, you could use IPF (most likely on
the old nameserver) to block port 53 from one host, and see what complains.
I would HOPE, at least, that the rogue applications would send up a flag
if they couldn't resolve hostnames.

Colin

Colin B. wrote:
> Brandon Hutchinson <bhutch@gmail.com> wrote:
>> Hello,
>>
>> We've looking to retire a nameserver in our environment. We've made
>> changes to the Solaris clients' /etc/resolv.conf, but some processes
>> are still sending DNS requests to this "old" nameserver.
>>
>> We've restarted several system processes (e.g., autofs, nfs.client,
>> syslog, sendmail, ldap.client, etc.), but something is still
>> referencing the "old" nameserver.
>>
>> Is there a way to determine which process is responsible for this
>> traffic? dtrace is unfortunately not an option, as in this case the
>> Solaris systems are not running Solaris 10.
>>
>> Any other solutions using truss or lsof? We can of course reboot the
>> systems as a last resort.
> 
> Not really a 'how to' answer, but are you running any of Sun's messaging
> products (webmail, ims, etc.)? We've found that most of them will read
> resolv.conf at start-time, and then cache the information.
> 
> That would be my guess: Sun products that aren't part of a base install.
> As far as I've ever seen, any part of the normal OS infrastructure
> (syslog, nfs, nis, sendmail, etc.) will behave as expected, and parse
> resolv.conf as used.
> 
> I assume you've scoured the logfiles. If you're willing to risk breaking
> (temporarily) the mystery application, you could use IPF (most likely on
> the old nameserver) to block port 53 from one host, and see what complains.
> I would HOPE, at least, that the rogue applications would send up a flag
> if they couldn't resolve hostnames.
> 

Yes, you can hope, but one misfeature implies another!  Or, where 
there's one bug, suspect an infestation!

Brandon Hutchinson <bhutch@gmail.com> wrote:
> We've looking to retire a nameserver in our environment. We've made
> changes to the Solaris clients' /etc/resolv.conf, but some processes
> are still sending DNS requests to this "old" nameserver.

Would the content of the requests give any hint as the application making
the query?  snoop should show what's being asked for.

-- 
Brandon Hume    - hume -> BOFH.Ca, http://WWW.BOFH.Ca/

On Dec 17, 8:19=A0am, "Colin B." <cbi...@somewhereelse.shaw.ca> wrote:

> Not really a 'how to' answer, but are you running any of Sun's messaging
> products (webmail, ims, etc.)? We've found that most of them will read
> resolv.conf at start-time, and then cache the information.

That's going to be true of just about any process on a machine where
nscd is not running.

> That would be my guess: Sun products that aren't part of a base install.
> As far as I've ever seen, any part of the normal OS infrastructure
> (syslog, nfs, nis, sendmail, etc.) will behave as expected, and parse
> resolv.conf as used.

I think that is only true if nscd is running before the program is
launched.

--
Darren

On Dec 17, 3:03=A0pm, Darren Dunham <darren.dun...@gmail.com> wrote:
> On Dec 17, 8:19=A0am, "Colin B." <cbi...@somewhereelse.shaw.ca> wrote:
>
> > Not really a 'how to' answer, but are you running any of Sun's messagin=
g
> > products (webmail, ims, etc.)? We've found that most of them will read
> > resolv.conf at start-time, and then cache the information.
>
> That's going to be true of just about any process on a machine where
> nscd is not running.
>
> > That would be my guess: Sun products that aren't part of a base install=
..
> > As far as I've ever seen, any part of the normal OS infrastructure
> > (syslog, nfs, nis, sendmail, etc.) will behave as expected, and parse
> > resolv.conf as used.
>
> I think that is only true if nscd is running before the program is
> launched.
>
> --
> Darren

Thanks everyone for the responses. We're not retiring this name server
for a few weeks, so I have the luxury of trying to figure this out
without a reboot. It's really just a problem I found "interesting." A
reboot makes the most sense.

The systems using the "old" name server are running Solaris 8. nscd
was one of the system processes that I looked for an restarted when
troubleshooting this problem.

Surprisingly (at least to me), restarting cron took care of most of
the remaining traffic; DNS traffic to the old name server seemed to be
generated when cron was running /usr/lib/sa/sa1 to collect sar data.

I now have pretty much everything off the old name server, and if I
can't figure out the remaining items, I'll just reboot the systems.

I was wondering if running lsof in a loop or something similar would
be able to identify these processes referencing the old name server,
but my attempts so far have failed.

Best regards,

Brandon