Hi,

I'm looking for ideas as to how I can selectively throttle
a node's traffic on a connection-by-connection basis (and,
by extension, on a client-by-client basis -- though this
poses additional design issues).

Outbound traffic can obviously be limited (size and number
of packets per unit time).  A more interesting problem is
the means and consequences of limiting *inbound* traffic
(e.g., deliberately dropping packets and/or queueing for
delayed presentation to the client).

I suspect the protocols being mangled have the most impact
on how effective these techniques might be...

Thanks,
--don

http://en.wikipedia.org/wiki/Transmission_Control_Protocol


D Yuniskis wrote:

> Hi,
> 
> I'm looking for ideas as to how I can selectively throttle
> a node's traffic on a connection-by-connection basis (and,
> by extension, on a client-by-client basis -- though this
> poses additional design issues).
> 
> Outbound traffic can obviously be limited (size and number
> of packets per unit time).  A more interesting problem is
> the means and consequences of limiting *inbound* traffic
> (e.g., deliberately dropping packets and/or queueing for
> delayed presentation to the client).
> 
> I suspect the protocols being mangled have the most impact
> on how effective these techniques might be...
> 
> Thanks,
> --don

On Mar 24, 10:12=A0pm, D Yuniskis <not.going.to...@seen.com> wrote:
> Hi,
>
> I'm looking for ideas as to how I can selectively throttle
> a node's traffic on a connection-by-connection basis (and,
> by extension, on a client-by-client basis -- though this
> poses additional design issues).
>
> Outbound traffic can obviously be limited (size and number
> of packets per unit time). =A0A more interesting problem is
> the means and consequences of limiting *inbound* traffic
> (e.g., deliberately dropping packets and/or queueing for
> delayed presentation to the client).
>
> I suspect the protocols being mangled have the most impact
> on how effective these techniques might be...


For inbound TCP traffic, you can just limit the advertized window size
and the rate at which you send acks.  Obviously this implies control
over the lower parts of the stack.

Alternatively, many firewalls, including some of the open source ones,
have traffic shaping functions (and there are some stand-alone traffic
shapers, but they're rare these days except as very high end
products).

Hi Robert,

On 3/24/2011 8:50 PM, robertwessel2@yahoo.com wrote:
> On Mar 24, 10:12 pm, D Yuniskis<not.going.to...@seen.com>  wrote:
>> I'm looking for ideas as to how I can selectively throttle
>> a node's traffic on a connection-by-connection basis (and,
>> by extension, on a client-by-client basis -- though this
>> poses additional design issues).
>>
>> Outbound traffic can obviously be limited (size and number
>> of packets per unit time).  A more interesting problem is
>> the means and consequences of limiting *inbound* traffic
>> (e.g., deliberately dropping packets and/or queueing for
>> delayed presentation to the client).
>>
>> I suspect the protocols being mangled have the most impact
>> on how effective these techniques might be...
>
> For inbound TCP traffic, you can just limit the advertized window size
> and the rate at which you send acks.  Obviously this implies control
> over the lower parts of the stack.

Yes.  But does nothing about UDP traffic.

Also, doesn't do anything about limiting unsolicited connection
attempts, etc.  (as well as misbehaving net-neighbors)

> Alternatively, many firewalls, including some of the open source ones,
> have traffic shaping functions (and there are some stand-alone traffic
> shapers, but they're rare these days except as very high end
> products).

Ah!  This is more what I need!  Something I can embed in a
switch (though I will then need a protocol by which each
client can tell the switch what it wants/"can have", etc.).

Thanks!  I'll start poking around...

--don

On Mar 25, 2:04=A0pm, D Yuniskis <not.going.to...@seen.com> wrote:
> Hi Robert,
>
> On 3/24/2011 8:50 PM, robertwess...@yahoo.com wrote:
>
>
>
>
>
> > On Mar 24, 10:12 pm, D Yuniskis<not.going.to...@seen.com> =A0wrote:
> >> I'm looking for ideas as to how I can selectively throttle
> >> a node's traffic on a connection-by-connection basis (and,
> >> by extension, on a client-by-client basis -- though this
> >> poses additional design issues).
>
> >> Outbound traffic can obviously be limited (size and number
> >> of packets per unit time). =A0A more interesting problem is
> >> the means and consequences of limiting *inbound* traffic
> >> (e.g., deliberately dropping packets and/or queueing for
> >> delayed presentation to the client).
>
> >> I suspect the protocols being mangled have the most impact
> >> on how effective these techniques might be...
>
> > For inbound TCP traffic, you can just limit the advertized window size
> > and the rate at which you send acks. =A0Obviously this implies control
> > over the lower parts of the stack.
>
> Yes. =A0But does nothing about UDP traffic.


There's only one possible thing to do with excess UPD traffic, and
that's drop it.


> Also, doesn't do anything about limiting unsolicited connection
> attempts, etc. =A0(as well as misbehaving net-neighbors)


Not that anything can do anything about those things.  You have no
control over what anyone outside your local network interface is
doing.  And having a physically separate firewall just move the
problem one box further away.  A firewall on your box, whether that's
provided by a traditional firewall package, or your own low level IP
stack, will have to deal with those things hitting your NIC.

Hi Robert,

On 3/25/2011 11:00 PM, robertwessel2@yahoo.com wrote:

>>> On Mar 24, 10:12 pm, D Yuniskis<not.going.to...@seen.com>    wrote:
>>>> I'm looking for ideas as to how I can selectively throttle
>>>> a node's traffic on a connection-by-connection basis (and,
>>>> by extension, on a client-by-client basis -- though this
>>>> poses additional design issues).
>>
>>>> Outbound traffic can obviously be limited (size and number
>>>> of packets per unit time).  A more interesting problem is
>>>> the means and consequences of limiting *inbound* traffic
>>>> (e.g., deliberately dropping packets and/or queueing for
>>>> delayed presentation to the client).
>>
>>>> I suspect the protocols being mangled have the most impact
>>>> on how effective these techniques might be...
>>
>>> For inbound TCP traffic, you can just limit the advertized window size
>>> and the rate at which you send acks.  Obviously this implies control
>>> over the lower parts of the stack.
>>
>> Yes.  But does nothing about UDP traffic.
>
> There's only one possible thing to do with excess UPD traffic, and
> that's drop it.

Yes, though it still ties up the wire.  (see below)

>> Also, doesn't do anything about limiting unsolicited connection
>> attempts, etc.  (as well as misbehaving net-neighbors)
>
> Not that anything can do anything about those things.  You have no
> control over what anyone outside your local network interface is
> doing.  And having a physically separate firewall just move the
> problem one box further away.  A firewall on your box, whether that's
> provided by a traditional firewall package, or your own low level IP
> stack, will have to deal with those things hitting your NIC.

Agreed.

But, I'm looking at the design of the switch.  I.e., if this
functionality is *in* the switch then all traffic *through*
the switch can be metered (including choked off).

The problem then becomes one of conveying the needs of each
node (clients therein) *to* the switch so that it can enforce
those restrictions.

On Fri, 25 Mar 2011 12:04:55 -0700, D Yuniskis
<not.going.to.be@seen.com> wrote:

>Hi Robert,
>
>On 3/24/2011 8:50 PM, robertwessel2@yahoo.com wrote:

>> For inbound TCP traffic, you can just limit the advertized window size
>> and the rate at which you send acks.  Obviously this implies control
>> over the lower parts of the stack.
>
>Yes.  But does nothing about UDP traffic.
>
>Also, doesn't do anything about limiting unsolicited connection
>attempts, etc.  (as well as misbehaving net-neighbors)

When listening for TCP connections, just delay accepting the
connections e.g. by 1 s and then close the socket if not appropriate.
With repeated attacks, this will fill up the listening queue (default
size 5 requests).

This will of course deny service also from legitime requests, but
embedded systems usually have other critical tasks to do than work as
a firewall :-).

On Mar 26, 1:22=A0am, D Yuniskis <not.going.to...@seen.com> wrote:
> Hi Robert,
>
> On 3/25/2011 11:00 PM, robertwess...@yahoo.com wrote:
>
>
>
>
>
> >>> On Mar 24, 10:12 pm, D Yuniskis<not.going.to...@seen.com> =A0 =A0wrot=
e:
> >>>> I'm looking for ideas as to how I can selectively throttle
> >>>> a node's traffic on a connection-by-connection basis (and,
> >>>> by extension, on a client-by-client basis -- though this
> >>>> poses additional design issues).
>
> >>>> Outbound traffic can obviously be limited (size and number
> >>>> of packets per unit time). =A0A more interesting problem is
> >>>> the means and consequences of limiting *inbound* traffic
> >>>> (e.g., deliberately dropping packets and/or queueing for
> >>>> delayed presentation to the client).
>
> >>>> I suspect the protocols being mangled have the most impact
> >>>> on how effective these techniques might be...
>
> >>> For inbound TCP traffic, you can just limit the advertized window siz=
e
> >>> and the rate at which you send acks. =A0Obviously this implies contro=
l
> >>> over the lower parts of the stack.
>
> >> Yes. =A0But does nothing about UDP traffic.
>
> > There's only one possible thing to do with excess UPD traffic, and
> > that's drop it.
>
> Yes, though it still ties up the wire. =A0(see below)
>
> >> Also, doesn't do anything about limiting unsolicited connection
> >> attempts, etc. =A0(as well as misbehaving net-neighbors)
>
> > Not that anything can do anything about those things. =A0You have no
> > control over what anyone outside your local network interface is
> > doing. =A0And having a physically separate firewall just move the
> > problem one box further away. =A0A firewall on your box, whether that's
> > provided by a traditional firewall package, or your own low level IP
> > stack, will have to deal with those things hitting your NIC.
>
> Agreed.
>
> But, I'm looking at the design of the switch. =A0I.e., if this
> functionality is *in* the switch then all traffic *through*
> the switch can be metered (including choked off).
>
> The problem then becomes one of conveying the needs of each
> node (clients therein) *to* the switch so that it can enforce
> those restrictions.- Hide quoted text -


A number of firewalls can run in "bridge mode" (sometimes called
"transparent mode", although that's sometimes a bit different - lotsa
funky stuff happens in firewalls).  At which point the firewall
appears to be just a bridge (or what we called "switches" before
marketing decided a cooler name was needed).  Not sure what you're
building, but a standalone switch with this kind of functionality
would be more a firewall (possibly in bridge mode) rather than a
conventional switch.  Or is this switch embedded in another device?

Hi Robert,

On 3/26/2011 12:26 AM, robertwessel2@yahoo.com wrote:

[much elided]

>> But, I'm looking at the design of the switch.  I.e., if this
>> functionality is *in* the switch then all traffic *through*
>> the switch can be metered (including choked off).
>>
>> The problem then becomes one of conveying the needs of each
>> node (clients therein) *to* the switch so that it can enforce
>> those restrictions.- Hide quoted text -
>
> A number of firewalls can run in "bridge mode" (sometimes called
> "transparent mode", although that's sometimes a bit different - lotsa
> funky stuff happens in firewalls).  At which point the firewall
> appears to be just a bridge (or what we called "switches" before
> marketing decided a cooler name was needed).  Not sure what you're
> building, but a standalone switch with this kind of functionality
> would be more a firewall (possibly in bridge mode) rather than a
> conventional switch.  Or is this switch embedded in another device?

[begin lengthy explanation]

The projects I'm involved with are getting physically
(geographically?) larger.  More "distributed" (in the classic
sense of the word).  As such, "people" can be "inside" the
system/device (hard to use the device for something so
physically large... but, to an insect, a PC wouldn't look
like a single "device" either!).

So, people are in a position to directly interact with the
device/system in ways that might not be benevolent.  E.g.,
would you let a small (sentient) creature wander around inside
your PC -- even if their intentions were *not* malevolent?
"Hey, where did my video go?"  "How is it that I suddenly have
more physical memory?"  "Why is the CPU locked in a tight loop
as if the bus was shorted?"

[sorry that this is obvious but sometimes the obvious needs
to be stated explicitly]

If the network fabric is a key part of the performance and
behavior of the product, then reliable operation of that
product (system/device) depends on the maintaining the
integrity of that fabric.

Network infrastructure (cabling, switches, etc.) is now
(unfortunately?) commonplace and no longer treated as
"the realm of wizards".  E.g., a 1cm dia orange cable
rarely was tampered with while a bunch of "phone" (-like)
plugs that can be plugged and unplugged effortlessly
see more (ab)use.

Someone (well intentioned or not) plugging a foreign
device into a switch that is a key part of a system/device
and causing that device to "mysteriously" malfunction
("Hey, Tom, I just got a call from Fred in Manufacturing
claiming the Frajistat3000 is doing some mighty weird things.
Can you go take a look at it?") would be unfortunate, to
put it mildly.  :>

E.g., how often, when your PC "misbehaves", do you think:
"Gee, I had better open the cover and see if someone
installed or removed something while I wasn't looking..."

Your PC -- as a *system* -- expects the components to
behave in predictable ways.  Memory should yield up its
results within N clock cycles of an address being presented
to it.  A video adapter should draw a blue pixel at screen
location (x,y) when the value <foo> is written to <bar>.

These are BASIC ASSUMPTIONS inherent in the design of that
system.  (bugs and anomalous behaviors happen when assumptions
prove to be invalid)

So, since I can't armor all of the connections to the system/device
(the equivalent of putting the PC's components into a case and
putting a lid on it), I have too protect places that it is
likely that someone could interfere (regardless of motive or
intent) with the performance of that system.

I have to design a switch, regardless (for other requirements),
so putting this functionality in the switch fortifies the
entire system.  And, if I expect that functionality to reside
outside the "nodes", then I can rethink the obligations that
the nodes, themselves, have to the system and gain some
design economies, there!  (e.g., if you're wearing "braces",
no need to also wear a belt!)

Of course, I have absolutely no idea how I can extend this
to technologies that do *not* rely on switches (e.g., "bus"
technologies like older 10Base2, wireless, etc.)  :<

On Mar 26, 1:20=A0pm, D Yuniskis <not.going.to...@seen.com> wrote:
> Hi Robert,
>
> On 3/26/2011 12:26 AM, robertwess...@yahoo.com wrote:
>
> [much elided]
>
>
>
>
>
> >> But, I'm looking at the design of the switch. =A0I.e., if this
> >> functionality is *in* the switch then all traffic *through*
> >> the switch can be metered (including choked off).
>
> >> The problem then becomes one of conveying the needs of each
> >> node (clients therein) *to* the switch so that it can enforce
> >> those restrictions.- Hide quoted text -
>
> > A number of firewalls can run in "bridge mode" (sometimes called
> > "transparent mode", although that's sometimes a bit different - lotsa
> > funky stuff happens in firewalls). =A0At which point the firewall
> > appears to be just a bridge (or what we called "switches" before
> > marketing decided a cooler name was needed). =A0Not sure what you're
> > building, but a standalone switch with this kind of functionality
> > would be more a firewall (possibly in bridge mode) rather than a
> > conventional switch. =A0Or is this switch embedded in another device?
>
> [begin lengthy explanation]
>
> The projects I'm involved with are getting physically
> (geographically?) larger. =A0More "distributed" (in the classic
> sense of the word). =A0As such, "people" can be "inside" the
> system/device (hard to use the device for something so
> physically large... but, to an insect, a PC wouldn't look
> like a single "device" either!).
>
> So, people are in a position to directly interact with the
> device/system in ways that might not be benevolent. =A0E.g.,
> would you let a small (sentient) creature wander around inside
> your PC -- even if their intentions were *not* malevolent?
> "Hey, where did my video go?" =A0"How is it that I suddenly have
> more physical memory?" =A0"Why is the CPU locked in a tight loop
> as if the bus was shorted?"
>
> [sorry that this is obvious but sometimes the obvious needs
> to be stated explicitly]
>
> If the network fabric is a key part of the performance and
> behavior of the product, then reliable operation of that
> product (system/device) depends on the maintaining the
> integrity of that fabric.
>
> Network infrastructure (cabling, switches, etc.) is now
> (unfortunately?) commonplace and no longer treated as
> "the realm of wizards". =A0E.g., a 1cm dia orange cable
> rarely was tampered with while a bunch of "phone" (-like)
> plugs that can be plugged and unplugged effortlessly
> see more (ab)use.
>
> Someone (well intentioned or not) plugging a foreign
> device into a switch that is a key part of a system/device
> and causing that device to "mysteriously" malfunction
> ("Hey, Tom, I just got a call from Fred in Manufacturing
> claiming the Frajistat3000 is doing some mighty weird things.
> Can you go take a look at it?") would be unfortunate, to
> put it mildly. =A0:>
>
> E.g., how often, when your PC "misbehaves", do you think:
> "Gee, I had better open the cover and see if someone
> installed or removed something while I wasn't looking..."
>
> Your PC -- as a *system* -- expects the components to
> behave in predictable ways. =A0Memory should yield up its
> results within N clock cycles of an address being presented
> to it. =A0A video adapter should draw a blue pixel at screen
> location (x,y) when the value <foo> is written to <bar>.
>
> These are BASIC ASSUMPTIONS inherent in the design of that
> system. =A0(bugs and anomalous behaviors happen when assumptions
> prove to be invalid)
>
> So, since I can't armor all of the connections to the system/device
> (the equivalent of putting the PC's components into a case and
> putting a lid on it), I have too protect places that it is
> likely that someone could interfere (regardless of motive or
> intent) with the performance of that system.
>
> I have to design a switch, regardless (for other requirements),
> so putting this functionality in the switch fortifies the
> entire system. =A0And, if I expect that functionality to reside
> outside the "nodes", then I can rethink the obligations that
> the nodes, themselves, have to the system and gain some
> design economies, there! =A0(e.g., if you're wearing "braces",
> no need to also wear a belt!)
>
> Of course, I have absolutely no idea how I can extend this
> to technologies that do *not* rely on switches (e.g., "bus"
> technologies like older 10Base2, wireless, etc.) =A0:<


While bus based architectures are hard - after all you can't prevent
tapping into the wire...  But it strikes me that relatively little of
that remains in place, and certainly the amount of new stuff like that
is miniscule (outside of dedicated control busses, which should not be
subject to "normal" user actions like unplugging Ethernet cables).

But two suggestions:  First, 802.1x adds authentication to switches -
thus only authenticated devices can gain access to a switch after
being plugged into a port.  More-or-less similar in principal to
putting a password on your wireless hotspot.  The older protocol had
some weaknesses (although these mostly require deliberate attempts at
circumvention), but the newest standard (-2010) resolves those.
Numerous switches support the -2004 version, and -2010 support is
rolling out (for example, Cisco is shipping a pre-standard version of
that on a number of switches).  Support (at least for the -2004
standard) is available for Windows, Linux and MacOS, as well as other
devices.  The -2010 also allows a shared segment (either truly shared
media or hub, or a conventional switch) off one port of an
authenticated switch, and multiple devices on that shared segment to
authenticate - the unauthenticated devices cannot communicate past the
authenticated switch port, but the shared segment obviously isn't
internally protected.

Second, look at VLANs, possibly in conjunction with authentication.
With dynamic configuration, you can assign specific MACs to a VLAN, so
anytime a device with one of your MACs attaches to a switch, that port
is assigned to the specific VLAN, and traffic is segregated.  Static
(per port) configuration of VLANs is possible as well (and was the
first type supported).  In that case apply a dab of red paint to the
ports in your VLAN, and take the head of anyone unauthorized touching
a red port.  Static VLANs have an advantage in ease of implementation,
but are otherwise an inferior solution compared to dynamic VLANs or
802.1x.

Of course to do any of that, you have to buy "real" switches, and not
the SOHO/small business stuff.

Hi Robert,

On 3/26/2011 12:59 PM, robertwessel2@yahoo.com wrote:

[8<]

>> Of course, I have absolutely no idea how I can extend this
>> to technologies that do *not* rely on switches (e.g., "bus"
>> technologies like older 10Base2, wireless, etc.)  :<
>
> While bus based architectures are hard - after all you can't prevent
> tapping into the wire...  But it strikes me that relatively little of
> that remains in place, and certainly the amount of new stuff like that
> is miniscule (outside of dedicated control busses, which should not be
> subject to "normal" user actions like unplugging Ethernet cables).

*Wired* "bus" topologies are obsolescent -- but *wireless*
(the "ether" can be regarded as a bus -- of sorts) are more
and more common.

[consider your comments in that light -- e.g., how incredibly
easy it would be to disrupt/corrupt a wireless protocol]

> But two suggestions:  First, 802.1x adds authentication to switches -
> thus only authenticated devices can gain access to a switch after
> being plugged into a port.  More-or-less similar in principal to
> putting a password on your wireless hotspot.  The older protocol had
> some weaknesses (although these mostly require deliberate attempts at
> circumvention), but the newest standard (-2010) resolves those.
> Numerous switches support the -2004 version, and -2010 support is
> rolling out (for example, Cisco is shipping a pre-standard version of

Yes.  But, that requires preshared keys or other means of
"tying" devices together.  I.e., this box (node) *belongs*
on this network.  It adds another configuration maintenance
issue.  E.g., imagine if you had to "pair" your DRAM DIMMs
to your PC (in addition to "plugging them in") before the
PC would "recognize them" ("consent to talk to them").

I am looking for a solution that lets boxes that *should*
"fit together" (nodes, switch, etc.) automagically do so
while still enforcing these constraints on communications.
E.g., a smart switch can be designed that still allows
"foreign" devices to talk to the "system" -- without giving
them "free reign" to interfere (even unintentionally) with
the operation of the rest of the system.

(i.e., the system can reserve certain bandwidth WITH PREDEFINED
CHARACTERISTICS for interaction with the outside world KNOWING
that this was "planned" in the original design -- not a
parasitic relationship)

> that on a number of switches).  Support (at least for the -2004
> standard) is available for Windows, Linux and MacOS, as well as other
> devices.  The -2010 also allows a shared segment (either truly shared
> media or hub, or a conventional switch) off one port of an
> authenticated switch, and multiple devices on that shared segment to
> authenticate - the unauthenticated devices cannot communicate past the
> authenticated switch port, but the shared segment obviously isn't
> internally protected.
>
> Second, look at VLANs, possibly in conjunction with authentication.

Re: authentication -- see above.

> With dynamic configuration, you can assign specific MACs to a VLAN, so
> anytime a device with one of your MACs attaches to a switch, that port
> is assigned to the specific VLAN, and traffic is segregated.  Static
> (per port) configuration of VLANs is possible as well (and was the
> first type supported).  In that case apply a dab of red paint to the
> ports in your VLAN, and take the head of anyone unauthorized touching

I frown on ex post factum remedies -- we don't design cars with
"toggle switch ignitions" and then rely on the threat of punishment
(prison) to keep folks from stealing them!  :>  And, some "errors"
can be costly (the "head taking" doesn't compensate for these
losses) -- e.g., if an error shuts down a production line for
an hour or more, no amount of "punishment" is going to make up
for the missing product, wasted salaries, lost opportunities, etc.

I'd prefer a solution that tries real hard not to be broken
(perhaps I'm too much of a Cynic to expect folks to read the
'Do Not Touch -- under penalty of Death' signs  :> )

> a red port.  Static VLANs have an advantage in ease of implementation,
> but are otherwise an inferior solution compared to dynamic VLANs or
> 802.1x.
>
> Of course to do any of that, you have to buy "real" switches, and not
> the SOHO/small business stuff.

Cost of a "real" switch isn't significant (consider the cost of
designing and producing something functionally equivalent).  And, I
expect the SOHO kit will eventually catch up as homes get more
"wired" (I think we are the only wired house in the neighborhood).
The downside to the COTS approach is it *only* addresses that
one problem.  I.e., if you look at the switch, etc. as
components in a *traditional* design ("discretes", etc.), then
you realize you have more options available to provide the
functionality you need -- beyond the *obvious* ones.  E.g.,
if you need to convert an analog value to a "digital" form,
the "A-to-D" can be performed in a wide variety of means
besides the "obvious".

I don't want to get sucked into the sort of COTS designs of
which "automated checkouts" have become the bane (lots of
boxes thrown together in a piss-poor attempt at solving what
could otherwise be accomplished more effectively if the
designers had started with a clean sheet of paper).

On 25 Mar., 05:12, D Yuniskis <not.going.to...@seen.com> wrote:
> Hi,
>
> I'm looking for ideas as to how I can selectively throttle
> a node's traffic on a connection-by-connection basis (and,
> by extension, on a client-by-client basis -- though this
> poses additional design issues).
>
> Outbound traffic can obviously be limited (size and number
> of packets per unit time). =A0A more interesting problem is
> the means and consequences of limiting *inbound* traffic
> (e.g., deliberately dropping packets and/or queueing for
> delayed presentation to the client).
>
> I suspect the protocols being mangled have the most impact
> on how effective these techniques might be...
>
> Thanks,
> --don

Hi Don

It depends on what you want to base your connection-by-connection
limitation by.

If it is only the connection number per client then it is "simple".

or

Do you want to limit:
* each individual destination port?
* a small number of groups of destination ports classified by e.g.
interactive (http,telnet) non-interactive (ftp,http-filetransfer,P2P)
etc.?
for each client.

Do you want to control each connections bandwidth?

Do you want to control each at the WAN? Cisco ASA firewalls can track
connections numbers.

http://www.netfilter.org/
Quote: "...
# stateful packet filtering (IPv4 and IPv6) [<- connection
tracking ! ]
....
# aid the tc and iproute2 systems used to build sophisticated QoS and
policy routers [ tc - traffic classification ]
# do further packet manipulation (mangling) like altering the TOS/DSCP/
ECN bits of the IP header [ ToS old QoS - and DSCP, ECN is post 2005
QoS. ]
...."

Deep packet inspection with the purpose of bandwidth management:
Application Layer Packet Classifier for Linux:
http://l7-filter.sourceforge.net/
Quote: "...
L7-filter is a classifier for Linux's Netfilter that identifies
packets based on application layer data. It can classify packets as
Kazaa, HTTP, Jabber, Citrix, Bittorrent, FTP, Gnucleus, eDonkey2000,
etc., regardless of port. It complements existing classifiers that
match on IP address, port numbers and so on.

Our intent is for l7-filter to be used in conjunction with Linux QoS
to do bandwith arbitration ("packet shaping") or traffic accounting.
...."

-

Do want to control LAN traffic?:

MultiLayer Switch (MLS) - OSI layer 3 and/or 4:
http://en.wikipedia.org/wiki/Multilayer_switch#MultiLayer_Switch_.28MLS.29_=
-_OSI_layer_3_and.2For_4

-

This might be a enormous amount of information you need to grasp - and
it touches:

Network congestion:
http://en.wikipedia.org/wiki/Network_congestion

Network capacity:
http://en.wikipedia.org/wiki/Network_congestion#Congestive_collapse

http://en.wikipedia.org/wiki/Quality_of_service
(with one or more queues, Explicit Congestion Notification (ECN))

http://en.wikipedia.org/wiki/Network_neutrality

http://en.wikipedia.org/wiki/Traffic_shaping
Traffic Classification:
http://en.wikipedia.org/wiki/Traffic_shaping#Traffic_Classification

L3 QoS:
http://en.wikipedia.org/wiki/Explicit_congestion_notification
Quote: "...
ECN allows end-to-end notification of network congestion without
dropping packets
...."
http://en.wikipedia.org/wiki/Explicit_congestion_notification#Operation_of_=
ECN_with_IP

Unfortunately ECN is disabled by default in al major OSes - idiotic?

L3 QoS: Note the 6 bits - DSCP (and last 2 bits ECN) - formerly ToS-
field:
http://en.wikipedia.org/wiki/DiffServ
Quote: "...
DiffServ is a coarse-grained, class-based mechanism for traffic
management. In contrast, IntServ is a fine-grained, flow-based
mechanism.
...."

L3 QoS: DiffServ -- The Scalable End-to-End QoS Model:
http://www.cisco.com/en/US/technologies/tk543/tk766/technologies_white_pape=
r09186a00800a3e2f.html

Please note that IntServ is seldom used because the OS/application
need to be IntServ aware - and only a very few applications are that.
And the whole network infrastructure needs to be IntServ - so you do
need to read about IntServ.

And there is also another old L2 QoS still in use - the 3 bits in
802.1p which only is mediated between (network) equipment if vlan
tagging is enabled in both ethernet port ends for each connections:
http://en.wikipedia.org/wiki/IEEE_P802.1p
Quote: "...
The QoS technique developed by the working group, also known as class
of service (CoS), is a 3-bit field called the Priority Code Point
(PCP) within an Ethernet frame header when using tagged frames on an
802.1 network. It specifies a priority value of between 0 and 7
inclusive that can be used by QoS disciplines to differentiate
traffic. Although this technique is commonly referred to as IEEE
802.1p, there is no standard or amendment by that name published by
the IEEE.
...."

Hi Glenn,

On 3/27/2011 1:23 AM, Glenn wrote:

> It depends on what you want to base your connection-by-connection
> limitation by.
>
> If it is only the connection number per client then it is "simple".
>
> or
>
> Do you want to limit:
> * each individual destination port?
> * a small number of groups of destination ports classified by e.g.
> interactive (http,telnet) non-interactive (ftp,http-filetransfer,P2P)
> etc.? for each client.
>
> Do you want to control each connections bandwidth?

Ideally, I want to control the bandwidth allocated to each
individual connection -- i.e., (from, to, port)-tuple.

However, the constraints placed on those connections may be
dynamic and/or interrelated with other connections.  E.g.,
the total bandwidth allocated to a particular *thread* -- or
process -- might be constrained and the multiple connections
open *within* that task/process might have to share that
allocation (using criteria that I define).

Or, the total bandwidth between a particular src_ip and dest_ip
might be constrained and the various connections on that particular
src_ip have to share that bandwidth (using some criteria).

Most of the traffic is proprietary RPC invocations et al.  So,
it is unlikely that a COTS switch would be able to recognize
the particular protocol, "registered ports", etc.  (though I
have to be able to tolerate -- and throttle -- traffic from
The Outside using well-known ports/protocols).

My stack has hooks that give me reasonably fine control over
outbound traffic without the client having to be involved in
this issue.  I.e., a "throttling task" can update individual
"quotas" for each connection based on whatever criteria I
impose.

My original approach was to coordinate the activities of these
"throttling tasks" among the various (cooperating) nodes in the
system.  Since all *inbound* traffic had to originate as
*outbound* traffic on some *other* node, I can control the
inbound traffic *at* at node by simply (!) telling the throttling
task(s) on each of those nodes how to throttle their own local
tasks (connections).

This won't work for "outside" connections.

One solution (which currently appears the most economical) is to
add another node that acts as a gateway to the outside world.
It's sole purpose would be to impose throttling on the "system"
side of the node -- i.e., the "out-side" looks like a normal
network node while the "in-side" is constrained to follow my rules.

But, this means any "foreign" connections to the system *must* go
through this gateway box.  Any such connections which bypass the
gateway device compromise the integrity of the system -- in much
the same way that adding a bus-mastering controller to a PC (see
my PC analogy, elsewhere) that wants to grab the physical bus for
X% of the time "steals" bandwidth from the rest of the PC.

As I mentioned elsewhere, networks are now considered somewhat
ubiquitous.  Folks think they can plug something into a switch
just because they know how a CAT5/6 connector "works".  (e.g.,
note how PoE takes pains to note that it is very likely that
folks will plug *telco* kit into PoE-enabled connections "because
the connector *seems* to fit").  I can't risk the system misbehaving
because someone did something they shouldn't have:  "Ooops!  I'm
/sorry/!"

> Do you want to control each at the WAN? Cisco ASA firewalls can track
> connections numbers.
>
> http://www.netfilter.org/
> Quote: "...
> # stateful packet filtering (IPv4 and IPv6) [<- connection
> tracking ! ]
> ....
> # aid the tc and iproute2 systems used to build sophisticated QoS and
> policy routers [ tc - traffic classification ]
> # do further packet manipulation (mangling) like altering the TOS/DSCP/
> ECN bits of the IP header [ ToS old QoS - and DSCP, ECN is post 2005
> QoS. ]
> ...."
>
> Deep packet inspection with the purpose of bandwidth management:
> Application Layer Packet Classifier for Linux:
> http://l7-filter.sourceforge.net/
> Quote: "...
> L7-filter is a classifier for Linux's Netfilter that identifies
> packets based on application layer data. It can classify packets as
> Kazaa, HTTP, Jabber, Citrix, Bittorrent, FTP, Gnucleus, eDonkey2000,
> etc., regardless of port. It complements existing classifiers that
> match on IP address, port numbers and so on.

<frown>  I'd have to see if this is driven by generic algorithms
or tailored to particular protocols (i.e., my RPC is "none of the
above" -- would I be able to configure things to recognize my
"protocol"/application?)

> Our intent is for l7-filter to be used in conjunction with Linux QoS
> to do bandwith arbitration ("packet shaping") or traffic accounting.
> ...."
>
> -
>
> Do want to control LAN traffic?:
>
> MultiLayer Switch (MLS) - OSI layer 3 and/or 4:
> http://en.wikipedia.org/wiki/Multilayer_switch#MultiLayer_Switch_.28MLS.29_-_OSI_layer_3_and.2For_4
>
> -
>
> This might be a enormous amount of information you need to grasp - and
> it touches:
>
> Network congestion:
> http://en.wikipedia.org/wiki/Network_congestion
>
> Network capacity:
> http://en.wikipedia.org/wiki/Network_congestion#Congestive_collapse
>
> http://en.wikipedia.org/wiki/Quality_of_service
> (with one or more queues, Explicit Congestion Notification (ECN))

I should be able to avoid all of these issues if I can get
the right mechanism(s) in place -- instead of being REactive,
I will be PROactive.

> http://en.wikipedia.org/wiki/Network_neutrality
>
> http://en.wikipedia.org/wiki/Traffic_shaping
> Traffic Classification:
> http://en.wikipedia.org/wiki/Traffic_shaping#Traffic_Classification
>
> L3 QoS:
> http://en.wikipedia.org/wiki/Explicit_congestion_notification
> Quote: "...
> ECN allows end-to-end notification of network congestion without
> dropping packets
> ...."
> http://en.wikipedia.org/wiki/Explicit_congestion_notification#Operation_of_ECN_with_IP
>
> Unfortunately ECN is disabled by default in al major OSes - idiotic?
>
> L3 QoS: Note the 6 bits - DSCP (and last 2 bits ECN) - formerly ToS-
> field:
> http://en.wikipedia.org/wiki/DiffServ
> Quote: "...
> DiffServ is a coarse-grained, class-based mechanism for traffic
> management. In contrast, IntServ is a fine-grained, flow-based
> mechanism.
> ...."

I'll have to chase this down.

> L3 QoS: DiffServ -- The Scalable End-to-End QoS Model:
> http://www.cisco.com/en/US/technologies/tk543/tk766/technologies_white_paper09186a00800a3e2f.html
>
> Please note that IntServ is seldom used because the OS/application
> need to be IntServ aware - and only a very few applications are that.
> And the whole network infrastructure needs to be IntServ - so you do
> need to read about IntServ.

This isn't an issue as my nodes are designed to work as
compatible "components" -- adhering to whatever rules I
want to impose.

However, I would have to see how this impacts an external
connection that isn't "IntServ-aware" (?)

> And there is also another old L2 QoS still in use - the 3 bits in
> 802.1p which only is mediated between (network) equipment if vlan
> tagging is enabled in both ethernet port ends for each connections:
> http://en.wikipedia.org/wiki/IEEE_P802.1p
> Quote: "...
> The QoS technique developed by the working group, also known as class
> of service (CoS), is a 3-bit field called the Priority Code Point
> (PCP) within an Ethernet frame header when using tagged frames on an
> 802.1 network. It specifies a priority value of between 0 and 7
> inclusive that can be used by QoS disciplines to differentiate
> traffic. Although this technique is commonly referred to as IEEE
> 802.1p, there is no standard or amendment by that name published by
> the IEEE.
> ...."

Again, for the co-operating nodes in the system, this isn't a problem
as they already *want* to cooperate.  As I mentioned elsewhere, it's
"easy" if you can put a box around the system/device and *prevent*
others from connecting/interfering.

Hmmm... I wonder if a *cheap* solution might be to repackage an existing
switch and replacing the connectors with something nonstandard.  I.e.,
so a user couldn't "casually" plug a foreign device into the switch.
And, package that "gateway node" inside this switch *with* a regular
connection so any attempt to plug a "foreign" device in would be
forced to go through that "gateway"...

[I need to think on this and see if it might be a cheaper solution...]

Try www.packetcontroller.com