I have a few dozen servers at work that reboot every week. 

On one of them, sshd could not start.

Feb 28 00:04:51 WORK_SERVER_NAME sshd[5465]: Server listening on :: port 22.
Feb 28 00:04:52 WORK_SERVER_NAME sshd[5465]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Feb 28 00:04:52 WORK_SERVER_NAME sshd[5465]: Received SIGHUP; restarting.

My question is WTF, what could it be that keeps the port during boot
time?

For now, I added a command that attempts to restart sshd after 3
minutes, but I am a little concerned. 

Googling suggests that it is because ipv4 and ipv6 comes up at a
diff. time, but still, this is kind of bad. 

i

Hello,

Be aware that crossposting to multiple groups without a follow-up
reduces your chances to be read, as some servers reject such articles.

Ignoramus26630 a �crit :
> 
> Feb 28 00:04:51 WORK_SERVER_NAME sshd[5465]: Server listening on :: port 22.
> Feb 28 00:04:52 WORK_SERVER_NAME sshd[5465]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
> Feb 28 00:04:52 WORK_SERVER_NAME sshd[5465]: Received SIGHUP; restarting.
> 
> My question is WTF, what could it be that keeps the port during boot
> time?

Probably the IPv6 socket listening on :: (short form of 0:0:0:0:0:0:0:0,
the IPv6 unspecified address), that sshd opened just before. If the
socket was opened without the V6ONLY option or if sysctl
net.ipv6.bindv6only=0 (kernel default), such a socket also accepts IPv4
connections and thus prevents opening an IPv4 socket on the same port.
But it should be fine, as the IPv6 socket accepts both IPv4 and IPv6
connections. This was intended to ease transition from IPv4 to IPv6 in
applications. However I don't understand the SIGHUP.

> For now, I added a command that attempts to restart sshd after 3
> minutes, but I am a little concerned. 

What does this change ?

On 2010-02-28, Pascal Hambourg <boite-a-spam@plouf.fr.eu.org> wrote:
> Hello,
>
> Be aware that crossposting to multiple groups without a follow-up
> reduces your chances to be read, as some servers reject such articles.

Not in my experience.

> Ignoramus26630 a ?crit :
>> 
>> Feb 28 00:04:51 WORK_SERVER_NAME sshd[5465]: Server listening on :: port 22.
>> Feb 28 00:04:52 WORK_SERVER_NAME sshd[5465]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
>> Feb 28 00:04:52 WORK_SERVER_NAME sshd[5465]: Received SIGHUP; restarting.
>> 
>> My question is WTF, what could it be that keeps the port during boot
>> time?
>
> Probably the IPv6 socket listening on :: (short form of 0:0:0:0:0:0:0:0,
> the IPv6 unspecified address), that sshd opened just before. If the
> socket was opened without the V6ONLY option or if sysctl
> net.ipv6.bindv6only=0 (kernel default), such a socket also accepts IPv4
> connections and thus prevents opening an IPv4 socket on the same port.
> But it should be fine, as the IPv6 socket accepts both IPv4 and IPv6
> connections. This was intended to ease transition from IPv4 to IPv6 in
> applications. However I don't understand the SIGHUP.
>
>> For now, I added a command that attempts to restart sshd after 3
>> minutes, but I am a little concerned. 
>
> What does this change ?

I am fairly positive that it is a very temporary, transient
issue. That's why it does not happen all the time on all servers. In
only (so far) occurred once on one server out of very many ones that
reboot every week.

Starting sshd again after 3 minutes should cover me pretty well. 

However, I am disappointed that such an essential (for accessing
remote computers) function like ssh, could not initialize and gave
up.

I was lucky that I could hack (kind of) my way into this server using
its autoupdate scripts. 

If, say, httpd could not start, I would not be as disappointed,
because it can be fixed remotely, but if, say, sshd cannot start then
you end up with a dangerous brick. 

i

In news:ztKdnX7D2J0gQRfWnZ2dnUVZ_vmdnZ2d@giganews.com,
Ignoramus26630 <ignoramus26630@NOSPAM.26630.invalid> typed:


> Feb 28 00:04:52 WORK_SERVER_NAME sshd[5465]: error: Bind to port 22
> on 0.0.0.0 failed: Address already in use. Feb 28 00:04:52
....
> My question is WTF, what could it be that keeps the port during boot
> time?

A previously started process that uses port 22.

Use "lsof -i :22" or "netstat -antp | egrep ':22|:ssh' " to see what's there
akready.

Ignoramus26630 a �crit :
> On 2010-02-28, Pascal Hambourg <boite-a-spam@plouf.fr.eu.org> wrote:
>> Hello,
>>
>> Be aware that crossposting to multiple groups without a follow-up
>> reduces your chances to be read, as some servers reject such articles.
> 
> Not in my experience.

Out of the three NNTP services I use to read Usenet groups, only one
shows this thread.

>> Ignoramus26630 a ?crit :
>>> Feb 28 00:04:51 WORK_SERVER_NAME sshd[5465]: Server listening on :: port 22.
>>> Feb 28 00:04:52 WORK_SERVER_NAME sshd[5465]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
>>> Feb 28 00:04:52 WORK_SERVER_NAME sshd[5465]: Received SIGHUP; restarting.
>>>
>>> My question is WTF, what could it be that keeps the port during boot
>>> time?
>> Probably the IPv6 socket listening on :: (short form of 0:0:0:0:0:0:0:0,
>> the IPv6 unspecified address), that sshd opened just before. If the
>> socket was opened without the V6ONLY option or if sysctl
>> net.ipv6.bindv6only=0 (kernel default), such a socket also accepts IPv4
>> connections and thus prevents opening an IPv4 socket on the same port.
>> But it should be fine, as the IPv6 socket accepts both IPv4 and IPv6
>> connections. This was intended to ease transition from IPv4 to IPv6 in
>> applications. However I don't understand the SIGHUP.

The "address already in use" issue was never a problem for me (anyway it
can be solved with -4 or -6 options on the sshd commandline or
ListenAddress directives in sshd_config according to one's needs).
Actually the real problem seems to be caused by the SIGHUP, which is
totally unrelated AFAICS.

I found this Debian bug report that may be related to your problem :
<http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=502444>
A script sends a SIGHUP to sshd and creates a race condition.

Pascal Hambourg <boite-a-spam@plouf.fr.eu.org> writes:
> Probably the IPv6 socket listening on :: (short form of 0:0:0:0:0:0:0:0,
> the IPv6 unspecified address), that sshd opened just before. If the
> socket was opened without the V6ONLY option or if sysctl
> net.ipv6.bindv6only=0 (kernel default), such a socket also accepts IPv4
> connections and thus prevents opening an IPv4 socket on the same port.

OpenSSH 4.9p1 and newer is aware of this issue and sets the V6ONLY
option on the socket before binding it to the listen address (normally
0:22).  

Hardy still has 4.7p1, but that doesn't matter.  In the default
configuration, sshd will use getaddrinfo() with AF_UNSPEC and AI_PASSIVE
to obtain a list of all available addresses.  It will then attempt to
bind to each of them in order.  It will warn about those that fail, but
as long as at least one of them worked, it will eun.  Hence, the only
impact is that if 0.0.0.0 was listed first (which was clearly not the
case here), :: will fail and sshd will only listen for IPv4 connections.
In either case - whether 0.0.0.0 or :: was listed first - it will warn
about failing to bind to the second address, but it will still run.

I would concentrate on finding the cause of the SIGHUP.

DES
-- 
Dag-Erling Smørgrav - des@des.no