f



Suggestions for custom application-layer protocol?

I need to implement a simple application layer protocol that will be used to
communicate between an embedded device (single-board computer running Linux)
and a monitoring terminal (running Windows).  There isn't alot of data being
passed around, mostly status information collected by the embedded device,
and control messages from the monitoring terminal.  Messages will consist of
between 1 and 10 fields of data.

I want the protocol to be text based rather than binary since the data
throughput is low.  I also want the protocol to be based on TCP/IP.  Does
anyone here have any suggestions on the design of a simple protocol?  Are
there simple, standard ways of formatting text messages to be sent over a
TCP socket (such as comma-separated)?  I would prefer to avoid the
complexities of XML.  Since either the embedded device or the monitoring
terminal can initiate a message, is it preferable to have TCP servers
running on both sides?  Or is it better to simply leave a TCP connection
open between client and server?  Any suggestions on ultra-simple standard
protocols that do something similar to this?

Any suggestions or pointers to further reading are much appreciated.

Thanks,
Mark


0
Mark
5/25/2005 5:01:05 PM
comp.unix.programmer 10848 articles. 0 followers. kokososo56 (349) is leader. Post Follow

56 Replies
8724 Views

Similar Articles

[PageSpeed] 44

> I want the protocol to be text based rather than binary since the data
> throughput is low.

If the throughput is low, binary would be better.

For example, "1.0000236" is four bytes in binary, but 10 bytes as text.


   I also want the protocol to be based on TCP/IP.  Does
> anyone here have any suggestions on the design of a simple protocol?  Are
> there simple, standard ways of formatting text messages to be sent over a
> TCP socket (such as comma-separated)?  

Read the HTTP protocol.  If you really want text-based, HTTP is probably 
the way to go.  My suggestion would be just to use CORBA, since it maps 
easily into most languages, has a binary representation, and is fairly 
standard.

> I would prefer to avoid the
> complexities of XML.

What sort of complexities do you wish to avoid?  Most of these 
complexities exist with all text-based messages.  With XML you have the 
advantage that you don't have to write your own parser.

> Any suggestions or pointers to further reading are much appreciated.

Use CORBA.  There are lots of good, free ORBs, including MICO, TAO, and 
ORBit.  There are also some Java ORBs.

Jon
----
Learn to program using Linux assembly language
http://www.cafeshops.com/bartlettpublish.8640017
0
Jonathan
5/25/2005 4:54:33 PM
In article <lb2le.10958$on1.6364@clgrps13>,
Mark <mark_2811nospam@excite.com> wrote:
:I need to implement a simple application layer protocol that will be used to
:communicate between an embedded device (single-board computer running Linux)
:and a monitoring terminal (running Windows).  There isn't alot of data being
:passed around, mostly status information collected by the embedded device,
:and control messages from the monitoring terminal.  Messages will consist of
:between 1 and 10 fields of data.

:I want the protocol to be text based rather than binary since the data
:throughput is low.

OK.

:I also want the protocol to be based on TCP/IP.

For the purposes you describe, it sounds as if UDP might be a better
fit. UDP has a lot fewer states to worry about. The main question,
though, would be how you want either end to react if it notices
that a packet has gone missing.

:Does
:anyone here have any suggestions on the design of a simple protocol?  Are
:there simple, standard ways of formatting text messages to be sent over a
:TCP socket (such as comma-separated)?

If the fields are consistant (e.g., if there are 3 fields then
the first 2 are exactly the same as if there were only 2 fields,
or if there are 3 fields they are always the -same- 3 fields)
then you can just list the values with some convenient delimeter character.

If the fields are not consistant (including, e.g., only transfering
data that changed) then keyword/value pairs would be typical.
TEMP3=46 STATE=stable

The protocol doesn't start to get interesting until you have
"fields" which are lists of values, or some fields may contain
arbitrary text (including your standard delimeter character.)
If it's less complex than that, just go ahead and do whatever seems
natural.

:Since either the embedded device or the monitoring
:terminal can initiate a message, is it preferable to have TCP servers
:running on both sides?  Or is it better to simply leave a TCP connection
:open between client and server?

Sounds like UDP. If you use TCP servers on both sides, then -every-
message requires the full 3-way negotiation. If you leave the
TCP connection open, you have to worry about the possibility of
the connection dropping and needing to detect and recover from that.
[e.g., suppose you have to reboot the Windows machine.]
-- 
Warning: potentially contains traces of nuts.
0
roberson
5/25/2005 5:13:29 PM

Mark wrote:
> I need to implement a simple application layer protocol that will be used to
> communicate between an embedded device (single-board computer running Linux)
> and a monitoring terminal (running Windows).  There isn't alot of data being
> passed around, mostly status information collected by the embedded device,
> and control messages from the monitoring terminal.  Messages will consist of
> between 1 and 10 fields of data.
> 
> I want the protocol to be text based rather than binary since the data
> throughput is low.  I also want the protocol to be based on TCP/IP.  Does
> anyone here have any suggestions on the design of a simple protocol?  Are
> there simple, standard ways of formatting text messages to be sent over a
> TCP socket (such as comma-separated)?  I would prefer to avoid the
> complexities of XML.  Since either the embedded device or the monitoring
> terminal can initiate a message, is it preferable to have TCP servers
> running on both sides?  Or is it better to simply leave a TCP connection
> open between client and server?  Any suggestions on ultra-simple standard
> protocols that do something similar to this?
> 
> Any suggestions or pointers to further reading are much appreciated.

    Could you use telnet, plus your own convention for
formatting the messages?

-- 
Eric.Sosman@sun.com

0
Eric
5/25/2005 5:22:27 PM
On Wed, 25 May 2005 17:01:05 +0000, Mark wrote:

> I need to implement a simple application layer protocol that will be used to
> communicate between an embedded device (single-board computer running Linux)
> and a monitoring terminal (running Windows).  There isn't alot of data being
> passed around, mostly status information collected by the embedded device,
> and control messages from the monitoring terminal.  Messages will consist of
> between 1 and 10 fields of data.
> 
> I want the protocol to be text based rather than binary since the data
> throughput is low.  I also want the protocol to be based on TCP/IP.  Does
> anyone here have any suggestions on the design of a simple protocol?  Are
> there simple, standard ways of formatting text messages to be sent over a
> TCP socket (such as comma-separated)?  I would prefer to avoid the
> complexities of XML.  Since either the embedded device or the monitoring
> terminal can initiate a message, is it preferable to have TCP servers
> running on both sides?  Or is it better to simply leave a TCP connection
> open between client and server?  Any suggestions on ultra-simple standard
> protocols that do something similar to this?


Take a look at netcat or nc.
It open a tcp port where you can send any data binary or ascii. You can
find nc for both windows and linux. 


-- 
/ Peter Andersen
/ Skanderborg

0
Peter
5/25/2005 5:43:18 PM
>I want the protocol to be text based rather than binary since the data
>throughput is low.  I also want the protocol to be based on TCP/IP.  Does
>anyone here have any suggestions on the design of a simple protocol?  Are
>there simple, standard ways of formatting text messages to be sent over a
>TCP socket (such as comma-separated)?  I would prefer to avoid the
>complexities of XML.  Since either the embedded device or the monitoring
>terminal can initiate a message, is it preferable to have TCP servers
>running on both sides?  Or is it better to simply leave a TCP connection
>open between client and server?  Any suggestions on ultra-simple standard
>protocols that do something similar to this?

Take a look at protocols like SMTP, NNTP, POP3, IMAP, etc.  Another
example to look at is the "AT" command set of modems (although this
doesn't use TCP).  A command is one line of text.  It starts with
some kind of command verb that identifies what follows.  The response
is usually one line of text, and something at the beginning of the
line identifies what kind of response it is (status code) and may
identify whether it's the last line or not.  The status codes are
in groups so a client doesn't have to be aware of all the status
codes (e.g. 4NN is a temporary failure of some kind, 5NN is a
permanent failure).

Spontaneous responses make the protocol a bit more complicated.
You need to be able to identify a spontaneous response (status code)
and perhaps the spontaneous response only says there IS data, and
the client needs to ask for it to actually get it sent.  IMAP
uses tags on requests and replies so you can match them up, and
deals with sponteneous responses.

Use any kind of field separator that's appropriate for the data you
are sending.  POP3 and IMAP use spaces.  Commas might be more
appropriate for certain kinds of data.

Testing is generally easy:  telnet to the appropriate port and
manually type commands.  Read the responses.

						Gordon L. Burditt
0
gordonb
5/25/2005 6:03:17 PM
Jonathan Bartlett <johnnyb@eskimo.com> writes:

>> I want the protocol to be text based rather than binary since the data
>> throughput is low.
>
> If the throughput is low, binary would be better.
>
> For example, "1.0000236" is four bytes in binary, but 10 bytes as text.

I think the OP means that the amount of data is small, so the larger
size of a text based protocol doesn't matter.

>> I also want the protocol to be based on TCP/IP.  Does anyone here
>> have any suggestions on the design of a simple protocol?  Are there
>> simple, standard ways of formatting text messages to be sent over a
>> TCP socket (such as comma-separated)?
>
> Read the HTTP protocol.  If you really want text-based, HTTP is
> probably the way to go.

HTTP is a document retrieval protocol, not a two-way message passing
protocol, despite it's frequent abuse as such.

> My suggestion would be just to use CORBA, since it maps easily into
> most languages, has a binary representation, and is fairly standard.

CORBA is also difficult to use and debug, has a large overhead in
code, and is largely incompatible with itself.  Avoid at all cost.

>> I would prefer to avoid the complexities of XML.
>
> What sort of complexities do you wish to avoid?  Most of these
> complexities exist with all text-based messages.

Have you ever read the XML specification, and all the other required
documents (DTD spec, etc.)?  Do so, and let's discuss complexities
afterwards.

> With XML you have the advantage that you don't have to write your
> own parser.

With XML you have the disadvantage that you have to write a
complicated parser, or integrate a huge library, for little gain.

>> Any suggestions or pointers to further reading are much appreciated.
>
> Use CORBA.  There are lots of good, free ORBs, including MICO, TAO,
> and ORBit.  There are also some Java ORBs.

Just keep in mind that most of these are incompatible, often even
between different versions of the same one.

-- 
M�ns Rullg�rd
mru@inprovide.com
0
iso
5/25/2005 6:33:33 PM
"Mark" <mark_2811nospam@excite.com> writes:

> I need to implement a simple application layer protocol that will be used to
> communicate between an embedded device (single-board computer running Linux)
> and a monitoring terminal (running Windows).  There isn't alot of data being
> passed around, mostly status information collected by the embedded device,
> and control messages from the monitoring terminal.  Messages will consist of
> between 1 and 10 fields of data.
> 
> I want the protocol to be text based rather than binary since the data
> throughput is low.  I also want the protocol to be based on TCP/IP.  Does
> anyone here have any suggestions on the design of a simple protocol?  Are
> there simple, standard ways of formatting text messages to be sent over a
> TCP socket (such as comma-separated)?  I would prefer to avoid the
> complexities of XML.  Since either the embedded device or the monitoring
> terminal can initiate a message, is it preferable to have TCP servers
> running on both sides?  Or is it better to simply leave a TCP connection
> open between client and server?  Any suggestions on ultra-simple standard
> protocols that do something similar to this?
> 
> Any suggestions or pointers to further reading are much appreciated.

One option to consider is to embed a web interface into your
application. This has the advantage that you can use any web
browser as the client side of the interface.

There are a variety of very small web servers that are suitable
for embedding. One such that I have used successfully is "pserv".
From the FreeBSD ports description:

| pServ is a small, portable HTTP server. It is written in pure C for speed and
| portability. It runs as a standalone program and does not require inetd.
| It should be small enough to be used in a mobile computer or to be run on your
| obsolete workstation you have somewhere.
| 
| This port contains a patched version to support the execution of php scripts.
| 
| WWW: http://sourceforge.net/projects/pserv/

-SEan


0
Sean
5/25/2005 7:21:02 PM
gordonb.ir1o9@burditt.org (Gordon Burditt) writes:

>>I want the protocol to be text based rather than binary since the data
>>throughput is low.  I also want the protocol to be based on TCP/IP.  Does
>>anyone here have any suggestions on the design of a simple protocol?  Are
>>there simple, standard ways of formatting text messages to be sent over a
>>TCP socket (such as comma-separated)?  I would prefer to avoid the
>>complexities of XML.  Since either the embedded device or the monitoring
>>terminal can initiate a message, is it preferable to have TCP servers
>>running on both sides?  Or is it better to simply leave a TCP connection
>>open between client and server?  Any suggestions on ultra-simple standard
>>protocols that do something similar to this?
>
> Take a look at protocols like SMTP, NNTP, POP3, IMAP, etc. 

This is the best advice.  The design of Internet protocols follows
good principles, and leads to simple implementations.


More complex protocols such as HTTP are needed only when your requests
cannot hold on one command with a few arguments.  Then we format
commands as email headers, with several lines of 'field: value' ended
with an empty line.

Read: http://www.rfc-editor.org 

> Another
> example to look at is the "AT" command set of modems (although this
> doesn't use TCP).  

Note that AT commands are cryptic (not HELP command like in the
Internet protocols, and no mnemonic commands). AT came for the unique
bit pattern A and T have in ASCII which helped the modem to detect and
adjust automatically the bit rate.  You don't need that for a network
protocol.

> A command is one line of text.  It starts with
> some kind of command verb that identifies what follows.  The response
> is usually one line of text, and something at the beginning of the
> line identifies what kind of response it is (status code) and may
> identify whether it's the last line or not.  The status codes are
> in groups so a client doesn't have to be aware of all the status
> codes (e.g. 4NN is a temporary failure of some kind, 5NN is a
> permanent failure).
>
> Spontaneous responses make the protocol a bit more complicated.
> You need to be able to identify a spontaneous response (status code)
> and perhaps the spontaneous response only says there IS data, and
> the client needs to ask for it to actually get it sent.  IMAP
> uses tags on requests and replies so you can match them up, and
> deals with sponteneous responses.
>
> Use any kind of field separator that's appropriate for the data you
> are sending.  POP3 and IMAP use spaces.  Commas might be more
> appropriate for certain kinds of data.
>
> Testing is generally easy:  telnet to the appropriate port and
> manually type commands.  Read the responses.
>
> 						Gordon L. Burditt

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/

Nobody can fix the economy.  Nobody can be trusted with their finger
on the button.  Nobody's perfect.  VOTE FOR NOBODY.
0
Pascal
5/25/2005 7:33:15 PM
"Mark" <mark_2811nospam@excite.com> wrote in message
news:lb2le.10958$on1.6364@clgrps13...
> I need to implement a simple application layer protocol that will be used
to
> communicate between an embedded device (single-board computer running
Linux)
> and a monitoring terminal (running Windows).  There isn't alot of data
being
> passed around, mostly status information collected by the embedded device,
> and control messages from the monitoring terminal.  Messages will consist
of
> between 1 and 10 fields of data.
>
> I want the protocol to be text based rather than binary since the data
> throughput is low.  I also want the protocol to be based on TCP/IP.  Does
> anyone here have any suggestions on the design of a simple protocol?  Are
> there simple, standard ways of formatting text messages to be sent over a
> TCP socket (such as comma-separated)?  I would prefer to avoid the
> complexities of XML.  Since either the embedded device or the monitoring
> terminal can initiate a message, is it preferable to have TCP servers
> running on both sides?  Or is it better to simply leave a TCP connection
> open between client and server?  Any suggestions on ultra-simple standard
> protocols that do something similar to this?
>
> Any suggestions or pointers to further reading are much appreciated.
>
> Thanks,
> Mark
>
>

Thanks for all the excellent suggestions so far.  I'll read through the RFCs
for various Internet application protocols.  I've also found some good
information in Eric S. Raymond's online book "Art of Unix Programming".  It
would be nice to be able to use a simple standard protocol of some kind.  It
seems strange to have to come up with ways of formatting text fields,
delimiting messages etc since I'm sure this has been done already many times
before.
Thanks,
Mark


0
Mark
5/25/2005 7:57:48 PM
"Mark" <mark_2811nospam@excite.com> wrote in message 
news:0N4le.10977$on1.1525@clgrps13...

> Thanks for all the excellent suggestions so far.  I'll read through the 
> RFCs
> for various Internet application protocols.  I've also found some good
> information in Eric S. Raymond's online book "Art of Unix Programming". 
> It
> would be nice to be able to use a simple standard protocol of some kind. 
> It
> seems strange to have to come up with ways of formatting text fields,
> delimiting messages etc since I'm sure this has been done already many 
> times
> before.

    You can take a page from the URL encoding scheme and use a list of 
<param_name>=<value> pairs delimited by & signs. Encode all less than signs, 
greater than signs, & signs, = signs, percent signs, whitespace, control 
characters, and non-ascii characters as '%xy' where x and y are the first 
and second hexadecimal digit of the ASCII value.

    So a query might be:

query=get&parameter=uptime<newline>

    And a reply might be:

query=get&parameter=uptime&value=1%20hour,%205%20minutes<newline>

    Specify that each end ignore any parameters they don't understand 
(rather than considering it an error), and your protocol is extensible.

    DS


0
David
5/25/2005 8:07:30 PM
On 2005-05-25, Mark <mark_2811nospam@excite.com> wrote:

> I need to implement a simple application layer protocol that will be used to
> communicate between an embedded device (single-board computer running Linux)
> and a monitoring terminal (running Windows).

Not sure what you mean by "monitoring terminal"?  

Is the data intended for use by man or by machine?  If the
former, then why not just use the telnet protocol and telnet
into your SBC?

If by machine, then simple ASCII newline terminated messages
are often the easiest to impliment and debug.

> I want the protocol to be text based rather than binary since
> the data throughput is low.  I also want the protocol to be
> based on TCP/IP.  Does anyone here have any suggestions on the
> design of a simple protocol?

Telnet, SMTP, IMAP, NNTP, etc. are all worth looking at.

> Are there simple, standard ways of formatting text messages to
> be sent over a TCP socket (such as comma-separated)?

No, there aren't really any overall standards.  I'd stick with
something plain ASCII and line-oriented.

> I would prefer to avoid the complexities of XML.

I would hope so. :)

> Since either the embedded device or the monitoring terminal
> can initiate a message, is it preferable to have TCP servers
> running on both sides? Or is it better to simply leave a TCP
> connection open between client and server?

Who can initiate a message and who can initiate a connection
are orthogonal issues. We have no way of knowing what approach
will work best for your application.

> Any suggestions on ultra-simple standard protocols that do
> something similar to this?

You haven't given enough details for us to know what might be
similar and what wouldn't.

> Any suggestions or pointers to further reading are much
> appreciated.

Whatever you pick, my main recommendation would be to do
something you can test/debug by using a telnet client and
netcat.

-- 
Grant Edwards                   grante             Yow!  OVER the
                                  at               underpass! UNDER the
                               visi.com            overpass! Around the FUTURE
                                                   and BEYOND REPAIR!!
0
Grant
5/25/2005 8:36:55 PM
In article <0N4le.10977$on1.1525@clgrps13>,
Mark <mark_2811nospam@excite.com> wrote:
:It
:would be nice to be able to use a simple standard protocol of some kind.  It
:seems strange to have to come up with ways of formatting text fields,
:delimiting messages etc since I'm sure this has been done already many times
:before.

Well you could use XDR, but changes are that is overkill.

There are lots of different standards you could use, but if the
standard is flexible then you end up putting in a lot more logic
than you need for your particular application.

For example, a flexible standard needs to be able to account
for nested lists of values, including the possibility that
the nesting is not the same at all points. Then there are
sparse representations to be handled. And it doesn't take
much in a flexible standard before "something comes up" and you
need to refer to a previously transmitted value in order to
make sense of what follows...
-- 
Studies show that the average reader ignores 106% of all statistics
they see in .signatures.
0
roberson
5/25/2005 10:20:32 PM
In article <d72lu4$pn$1@nntp.webmaster.com>,
David Schwartz <davids@webmaster.com> wrote:
:    You can take a page from the URL encoding scheme and use a list of 
:<param_name>=<value> pairs delimited by [...]

:    Specify that each end ignore any parameters they don't understand 
:(rather than considering it an error), and your protocol is extensible.

I disagree. If you ignore parameters you don't understand,
then you may well be ignorning something that changes the meaning
of something else that you thought you understood.

When I last did this kind of protocol definition work, the first
thing I had the two sides exchange is their respective protocol
revision numbers. Each side would have a range of versions over
which it was able to support backwards compatability; if
the two sides were too far apart, then either end could say
"Sorry, I don't know how to talk to you."

You could offer "ignore what you don't understand" extensibility
within a protocol release provided that the extensions don't
change the meaning of any established parameter, and the
extensions don't command anything critical. For example,
if one added an arrow-shape parameter to a graph, then
likely it would be considered acceptable if the other end
ignored the shape parameter -- but a new parameter that 
indicated (e.g.) "Turn off motor #7!" is probably too important
to ignore. Any time there is a notable semantic change or
critical command change, the version number should get a boost.


Note to the original poster:

One of the other issues that I ran into, which might or might not
be relevant to you, is that some of my operations could take
a long time. I needed command response timeouts; and I needed
keep-alives so the one end would know the other end was
working instead of hung up; and I needed a way to interrupt
executing commands. [My code was single threaded and
non-preemptive, so I had to put in co-operative multitasking
hooks.] There's no clean equivilent to ^C over a network connection ;-)
-- 
  The rule of thumb for speed is:

  1. If it doesn't work then speed doesn't matter.  -- Christian Bau
0
roberson
5/25/2005 10:40:48 PM
"Walter Roberson" <roberson@ibd.nrc-cnrc.gc.ca> wrote in message 
news:d72utg$2f4$1@canopus.cc.umanitoba.ca...

> In article <d72lu4$pn$1@nntp.webmaster.com>,

> David Schwartz <davids@webmaster.com> wrote:
> :    You can take a page from the URL encoding scheme and use a list of
> :<param_name>=<value> pairs delimited by [...]
>
> :    Specify that each end ignore any parameters they don't understand
> :(rather than considering it an error), and your protocol is extensible.
>
> I disagree. If you ignore parameters you don't understand,
> then you may well be ignorning something that changes the meaning
> of something else that you thought you understood.

    That's not a problem. Just have a parameter like 'minimum-version' that 
goes in each request. If the minimum version of the query is higher than the 
version you are coded to, you know you don't understand the protocol. You 
can also use ':' instead of '=' for "critical" parameters, that is, ones 
that should cause you to reject the request if you don't understand them.

> When I last did this kind of protocol definition work, the first
> thing I had the two sides exchange is their respective protocol
> revision numbers. Each side would have a range of versions over
> which it was able to support backwards compatability; if
> the two sides were too far apart, then either end could say
> "Sorry, I don't know how to talk to you."

    The only problem with this is that you can't easily have multiple 
overlapping changes. For example, suppose one person adds a compression 
option to the protocol while independently someone adds a security option. 
No implementation exists with both options because they're temporarily (or 
maybe even fundamentally) incompatible.

    This is a really hard problem. I've seen many "negotiation" 
specifications that didn't actually do anything more than announce 
capabilities. That's not negotiation.

> You could offer "ignore what you don't understand" extensibility
> within a protocol release provided that the extensions don't
> change the meaning of any established parameter, and the
> extensions don't command anything critical. For example,
> if one added an arrow-shape parameter to a graph, then
> likely it would be considered acceptable if the other end
> ignored the shape parameter -- but a new parameter that
> indicated (e.g.) "Turn off motor #7!" is probably too important
> to ignore. Any time there is a notable semantic change or
> critical command change, the version number should get a boost.

    That works fine if and only if one person controls all implementations 
of and changes to the protocol.

> Note to the original poster:
>
> One of the other issues that I ran into, which might or might not
> be relevant to you, is that some of my operations could take
> a long time. I needed command response timeouts; and I needed
> keep-alives so the one end would know the other end was
> working instead of hung up; and I needed a way to interrupt
> executing commands. [My code was single threaded and
> non-preemptive, so I had to put in co-operative multitasking
> hooks.] There's no clean equivilent to ^C over a network connection ;-)

    Yeah, we can make this as hard as we want to. ;)

    DS


0
David
5/25/2005 11:15:52 PM
In article <d730vd$674$2@nntp.webmaster.com>,
 "David Schwartz" <davids@webmaster.com> wrote:

> "Walter Roberson" <roberson@ibd.nrc-cnrc.gc.ca> wrote in message 
> news:d72utg$2f4$1@canopus.cc.umanitoba.ca...
> > When I last did this kind of protocol definition work, the first
> > thing I had the two sides exchange is their respective protocol
> > revision numbers. Each side would have a range of versions over
> > which it was able to support backwards compatability; if
> > the two sides were too far apart, then either end could say
> > "Sorry, I don't know how to talk to you."
> 
>     The only problem with this is that you can't easily have multiple 
> overlapping changes. For example, suppose one person adds a compression 
> option to the protocol while independently someone adds a security option. 
> No implementation exists with both options because they're temporarily (or 
> maybe even fundamentally) incompatible.

Right.  Look at HTTP as an example of a protocol that has evolved pretty 
gracefully by making use of the "ignore what you don't understand" 
philosophy.  Many of the features that eventually became codified in 
HTTP/1.1 were originally done as nonstandard extensions in HTTP/1.0.  
For instance, the "Host:" header, which is used by virtual servers, was 
nonstandard in 1.0; it would have slowed down growth of the web 
significantly if we had to wait for a new version of the standard to be 
published before browsers and servers could start using this feature.

What you have to do when you start creating nonstandard extensions, 
though, is ensure that they're not critical to operation of the 
protocol.  The protocol has to operate properly if the recipient ignores 
them.  You need to bump up the protocol version number, and make use of 
some form of negotiation, when you introduce new required features.  An 
example in HTTP/1.1 is "chunked" encoding -- if a server sends this and 
the client doesn't understand it, the document will look all wrong and 
the client and server will get out of sync in communication (the 
recipient expects the sender to close the connection when it's done).

-- 
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
0
Barry
5/26/2005 12:03:33 AM
In article <d730vd$674$2@nntp.webmaster.com>,
David Schwartz <davids@webmaster.com> wrote:
:    The only problem with this is that you can't easily have multiple 
:overlapping changes. For example, suppose one person adds a compression 
:option to the protocol while independently someone adds a security option. 
:No implementation exists with both options because they're temporarily (or 
:maybe even fundamentally) incompatible.

:    This is a really hard problem. I've seen many "negotiation" 
:specifications that didn't actually do anything more than announce 
:capabilities. That's not negotiation.

This discussion reminds me of the old VMS options processing
setup. It was table driven, but it had the flexibility to know
what options took what kind of parameter, and to know what options
were not valid in combination with other options.
-- 
Warning: potentially contains traces of nuts.
0
roberson
5/26/2005 12:51:58 AM
Sean Burke wrote:

> "Mark" <mark_2811nospam@excite.com> writes:
> 
> 
>>I need to implement a simple application layer protocol that will be used to
>>communicate between an embedded device (single-board computer running Linux)
>>and a monitoring terminal (running Windows).  There isn't alot of data being
>>passed around, mostly status information collected by the embedded device,
>>and control messages from the monitoring terminal.  Messages will consist of
>>between 1 and 10 fields of data.
>>
>>I want the protocol to be text based rather than binary since the data
>>throughput is low.  I also want the protocol to be based on TCP/IP.  Does
>>anyone here have any suggestions on the design of a simple protocol?  Are
>>there simple, standard ways of formatting text messages to be sent over a
>>TCP socket (such as comma-separated)?  I would prefer to avoid the
>>complexities of XML.  Since either the embedded device or the monitoring
>>terminal can initiate a message, is it preferable to have TCP servers
>>running on both sides?  Or is it better to simply leave a TCP connection
>>open between client and server?  Any suggestions on ultra-simple standard
>>protocols that do something similar to this?
>>
>>Any suggestions or pointers to further reading are much appreciated.
> 
> 
> One option to consider is to embed a web interface into your
> application. This has the advantage that you can use any web
> browser as the client side of the interface.
> 
> There are a variety of very small web servers that are suitable
> for embedding. One such that I have used successfully is "pserv".
> From the FreeBSD ports description:
> 
> | pServ is a small, portable HTTP server. It is written in pure C for speed and
> | portability. It runs as a standalone program and does not require inetd.
> | It should be small enough to be used in a mobile computer or to be run on your
> | obsolete workstation you have somewhere.
> | 
> | This port contains a patched version to support the execution of php scripts.
> | 
> | WWW: http://sourceforge.net/projects/pserv/
> 
I don't think this link is even exist.

D

> -SEan
> 
> 
0
Dave
5/26/2005 5:11:22 AM
In comp.os.linux.embedded Mark <mark_2811nospam@excite.com> wrote:
> I need to implement a simple application layer protocol that will be used to
> communicate between an embedded device (single-board computer running Linux)
> and a monitoring terminal (running Windows).  There isn't alot of data being
> passed around, mostly status information collected by the embedded device,
> and control messages from the monitoring terminal.  Messages will consist of
> between 1 and 10 fields of data.

> I want the protocol to be text based rather than binary since the data
> throughput is low.  I also want the protocol to be based on TCP/IP.  Does
> anyone here have any suggestions on the design of a simple protocol?  Are
> there simple, standard ways of formatting text messages to be sent over a
> TCP socket (such as comma-separated)?  I would prefer to avoid the
> complexities of XML.  Since either the embedded device or the monitoring
> terminal can initiate a message, is it preferable to have TCP servers
> running on both sides?  Or is it better to simply leave a TCP connection
> open between client and server?  Any suggestions on ultra-simple standard
> protocols that do something similar to this?

> Any suggestions or pointers to further reading are much appreciated.

> Thanks,
> Mark


Hi Mark,

You seem to be implying that you have a network connection to your SBC,
but don't confirm it!

For simplicity of implementation, I'd recommend implementing a message
layer over the TCP socket - something as simple as a two byte binary header
to indicate the length of the message to follow.

This means you can then read a whole message - text based or otherwise,
and the process it accordingly.  You don't need to mess about with parsing
lines and stuff like that.  It drops out to:

 1. Read Header:
 2. Read Data:
 3    Process Request
Repeat (ad infinitum)

I'd suggest just running the server on the SBC.  Its always possible you
might need to run multiple monitoring terminals in other places, and there
is no need for the SBC to connect to the monitoring terminal if the software
for the monitoring is not running (by the sound of it)

After that, the format of your payload could be a simple text header indicating the
message type with the specific format of data to follow.  CSV for the data for example

Hope this helps,

Glyn
-- 

------------------------------------------------------------------------
Glyn Davies / gryn@plig.net / www.technobobbins.com / Insert quote here?
------------------------------------------------------------------------
0
Glyn
5/26/2005 8:09:15 AM
On Wed, 25 May 2005, Mark wrote:

> I need to implement a simple application layer protocol that will be 
> used to communicate between an embedded device (single-board computer 
> running Linux) and a monitoring terminal (running Windows).  There isn't 
> alot of data being passed around, mostly status information collected by 
> the embedded device, and control messages from the monitoring terminal. 
> Messages will consist of between 1 and 10 fields of data.
>
> I want the protocol to be text based rather than binary since the data 
> throughput is low.  I also want the protocol to be based on TCP/IP. 
> Does anyone here have any suggestions on the design of a simple 
> protocol?  Are there simple, standard ways of formatting text messages 
> to be sent over a TCP socket (such as comma-separated)?  I would prefer 
> to avoid the complexities of XML.

I've written a bunch of little protocols like this. I think it really is 
as simple as it sounds - use TCP, frame messages by terminating with a 
line break (i'd tolerate CR, LF or CR+LF, and consistently emit CR + LF, i 
think), split each message into fields with tab characters, use the first 
field for a command or status code, and put your data in the other fields. 
If you want to be able to have multiple commands in flight at once, use 
the first field for a transaction tag (as in IMAP), and the second field 
for command or status.

> Since either the embedded device or the monitoring terminal can initiate 
> a message, is it preferable to have TCP servers running on both sides? 
> Or is it better to simply leave a TCP connection open between client and 
> server?

The latter. Engineer the protocol to be stateless, though, so if you lose 
the connection, you can just throw up a new one and carry on.

As for versioning, if you think implementations of this might persist in 
the wild for several years, or if anyone else is going to implement it, by 
all means do some version negotiation, but otherwise, if you control both 
ends, i'd forget about it.

Add a no-op command so you can do keepalives. Maybe add ping and pong 
commands, so you can see if the far end is still alive. If you have 
long-running transactions and are paranoid, go with the tagged command 
approach and add a 'query transaction status' command and appropriate 
responses.

If you want to encode complex data structures, use JSON:

http://www.crockford.com/JSON/index.html

JSON encodings are valid javascript and python source, which makes parsing 
them pretty easy!

> Any suggestions on ultra-simple standard protocols that do something 
> similar to this?

Ultra-simple and standard: pick one!

And, while i'm slinging aphorisms: make it work, then make it right, then 
make it fast. Oh, and do the simplest thing that could possibly work.

tom

-- 
Punk's not sexual, it's just aggression.
0
Tom
5/26/2005 2:51:23 PM
In article <d7407b$2e4$1@magenta.plig.net>,
Glyn Davies  <gryn@riffraff.plig.net> wrote:
:For simplicity of implementation, I'd recommend implementing a message
:layer over the TCP socket - something as simple as a two byte binary header
:to indicate the length of the message to follow.

:This means you can then read a whole message - text based or otherwise,
:and the process it accordingly.  You don't need to mess about with parsing
:lines and stuff like that.

You still need to "mess about" if your data might not fit in a single
packet. Recall that when you read() from a socket you are not certain
to get all of the data unless the data length is no greater than
your system's atomicity limits [PIPE_BUF]. The POSIX minimum
value for PIPE_BUF is 512 -- so in the general case a write() of
more than 512 bytes may require multiple read() statements.
-- 
Ceci, ce n'est pas une id�e.
0
roberson
5/26/2005 6:11:14 PM
"Walter Roberson" <roberson@ibd.nrc-cnrc.gc.ca> wrote in message 
news:d753g2$lim$1@canopus.cc.umanitoba.ca...

> You still need to "mess about" if your data might not fit in a single
> packet.

    And even if it does.

> Recall that when you read() from a socket you are not certain
> to get all of the data unless the data length is no greater than
> your system's atomicity limits [PIPE_BUF]. The POSIX minimum
> value for PIPE_BUF is 512 -- so in the general case a write() of
> more than 512 bytes may require multiple read() statements.

    These atomicity limits don't apply to TCP connections, which have their 
own segmentation rules.

    DS


0
David
5/26/2005 6:13:38 PM
On 2005-05-26, Walter Roberson <roberson@ibd.nrc-cnrc.gc.ca> wrote:

>:For simplicity of implementation, I'd recommend implementing a message
>:layer over the TCP socket - something as simple as a two byte binary header
>:to indicate the length of the message to follow.
>
>:This means you can then read a whole message - text based or otherwise,
>:and the process it accordingly.  You don't need to mess about with parsing
>:lines and stuff like that.
>
> You still need to "mess about" if your data might not fit in a single
> packet.

And even if it does.

> Recall that when you read() from a socket you are not certain
> to get all of the data unless the data length is no greater
> than your system's atomicity limits [PIPE_BUF]. The POSIX
> minimum value for PIPE_BUF is 512 -- so in the general case a
> write() of more than 512 bytes may require multiple read()
> statements.

The data may have passed through a bridge or router that may
have broken it up even further.  It's a bad idea to make any
assumptions about any correspondance between write() and read()
calls on a TCP connection.  It's just a byte stream: you
oughtn't depend on any sort of datagram behavior whatsoever.

A 500 byte "message" may have been sent using a single call to
write(), but the receiver had better be prepared to call read()
anywhere from 1 to 500 times.

-- 
Grant Edwards                   grante             Yow!  Two with FLUFFO,
                                  at               hold th' BEETS...side of
                               visi.com            SOYETTES!
0
Grant
5/26/2005 6:22:18 PM
"Mark" <mark_2811nospam@excite.com> wrote in message
news:lb2le.10958$on1.6364@clgrps13...
> I need to implement a simple application layer protocol that will be used
to
> communicate between an embedded device (single-board computer running
Linux)
> and a monitoring terminal (running Windows).  There isn't alot of data
being
> passed around, mostly status information collected by the embedded device,
> and control messages from the monitoring terminal.  Messages will consist
of
> between 1 and 10 fields of data.
>
> I want the protocol to be text based rather than binary since the data
> throughput is low.  I also want the protocol to be based on TCP/IP.  Does
> anyone here have any suggestions on the design of a simple protocol?  Are
> there simple, standard ways of formatting text messages to be sent over a
> TCP socket (such as comma-separated)?  I would prefer to avoid the
> complexities of XML.  Since either the embedded device or the monitoring
> terminal can initiate a message, is it preferable to have TCP servers
> running on both sides?  Or is it better to simply leave a TCP connection
> open between client and server?  Any suggestions on ultra-simple standard
> protocols that do something similar to this?
>
> Any suggestions or pointers to further reading are much appreciated.
>
> Thanks,
> Mark
>
>

Thanks alot for all the responses, there are some excellent ideas here which
I will investigate further.  I looked briefly at the BEEP protocol but it
seems to support alot more features than I need.  I control both ends of the
protocol, so don't need to overdo it on versioning robustness, etc.  To
clarify a few responders questions ...  by "monitoring terminals" I meant
Windows-based computers with a simple GUI that displays the data and allows
users to control various aspects of the embedded device from a remote
location.  I expect I will end up with a text-based protocol, with a header
that indicates the number of bytes in the payload.  The payloads will be
composed of field=value pairs.

Thanks guys I really appreciate it, this discussion has helped me out a
great deal!

Mark


0
Mark
5/26/2005 10:36:23 PM
On 2005-05-26, Mark <mark_2811nospam@excite.com> wrote:

> I expect I will end up with a text-based protocol, with a header
> that indicates the number of bytes in the payload.

If you're using a text-based protocol, you don't nead a header
with a payload count to tell you where the message ends.  Use
an end-of-message character or string.  The cr/lf pair is used
as the end-of-message delimiter by a lot of text-based
runs-on-top of TCP protocols.  That approach works wonderfully
with line-oriented high-level text-handling libraries and
utilties -- you can do all sorts of useful stuff using nothing
but a few utilities like netcat, expect, and so on.

I've also seen protocols that use STX/ETX to frame messages,
and that works nicely as well, but isn't as compatible with
pre-existing utilities.

> The payloads will be composed of field=value pairs.

That Sounds like a good plan.

> Thanks guys I really appreciate it, this discussion has helped
> me out a great deal!

-- 
Grant Edwards                   grante             Yow!  Hey, LOOK!! A pair of
                                  at               SIZE 9 CAPRI PANTS!! They
                               visi.com            probably belong to SAMMY
                                                   DAVIS, JR.!!
0
Grant
5/27/2005 12:04:16 AM
On Wed, 25 May 2005 19:21:02 +0000, Sean Burke wrote:

> One option to consider is to embed a web interface into your
> application. This has the advantage that you can use any web
> browser as the client side of the interface.

 Not a terrible idea, a simple HTTP/1.0 server can be pretty small esp. if
you don't mind stopping as soon as something works (the very basics can be
done in < 20 lines of C).

> There are a variety of very small web servers that are suitable
> for embedding. One such that I have used successfully is "pserv".
> From the FreeBSD ports description:

 Yeh, pretty good ... I stopped looking after seeing 2 major
vulnerabilities on the first google page.
 The code also looked promising ... for more exploits.


 Writing a custom "simple protocol" is likely to be much easier, using
netstrings is probably more likely to make you do the right thing ... but
a simple "CMD arg1 arg2" type telnet/SMTP/NNTP like protocol isn't hard to
get right.

-- 
James Antill -- james@and.org
http://www.and.org/vstr/httpd

0
James
5/27/2005 5:04:17 AM
On Fri, 27 May 2005 00:04:16 +0000, Grant Edwards wrote:

> On 2005-05-26, Mark <mark_2811nospam@excite.com> wrote:
> 
>> I expect I will end up with a text-based protocol, with a header
>> that indicates the number of bytes in the payload.
> 
> If you're using a text-based protocol, you don't nead a header
> with a payload count to tell you where the message ends.  Use
> an end-of-message character or string.  The cr/lf pair is used
> as the end-of-message delimiter by a lot of text-based
> runs-on-top of TCP protocols.  That approach works wonderfully
> with line-oriented high-level text-handling libraries and
> utilties -- you can do all sorts of useful stuff using nothing
> but a few utilities like netcat, expect, and so on.

 It interacts well with telnet, which is about the best thing that can be
said for it. There are large problems with how you limit "too long" lines,
and what happens when you hit a CR or LF on it's own. Esp. if you put
something like a perl regexp directly on the socket.
 The HTTP response splitting problem is entirely the fault of pretending
CRLF terminated lines are a good idea.

-- 
James Antill -- james@and.org
http://www.and.org/vstr/httpd

0
James
5/27/2005 5:12:46 AM
On Fri, 27 May 2005 01:12:46 -0400, James Antill wrote:

> On Fri, 27 May 2005 00:04:16 +0000, Grant Edwards wrote:
> 
>> On 2005-05-26, Mark <mark_2811nospam@excite.com> wrote:
>> 
>>> I expect I will end up with a text-based protocol, with a header
>>> that indicates the number of bytes in the payload.
>> 
>> If you're using a text-based protocol, you don't nead a header
>> with a payload count to tell you where the message ends.  Use
>> an end-of-message character or string.  The cr/lf pair is used
>> as the end-of-message delimiter by a lot of text-based
>> runs-on-top of TCP protocols.  That approach works wonderfully
>> with line-oriented high-level text-handling libraries and
>> utilties -- you can do all sorts of useful stuff using nothing
>> but a few utilities like netcat, expect, and so on.
> 
>  It interacts well with telnet, which is about the best thing that can be
> said for it. There are large problems with how you limit "too long" lines,
> and what happens when you hit a CR or LF on it's own. Esp. if you put
> something like a perl regexp directly on the socket.
>  The HTTP response splitting problem is entirely the fault of pretending
> CRLF terminated lines are a good idea.

This is the main reason I'm leaning towards a byte count to delimit the
message.  But I think I'll end up with some concept of special characters
in the protocol (= , crlf).  Is there a standard way to escape these
characters if they appear in the body of the message?  I suppose I could
use nested netstrings to encode each individual field in the message. 
Thoughts?


0
Mark
5/27/2005 7:34:32 AM
In comp.os.linux.embedded Walter Roberson <roberson@ibd.nrc-cnrc.gc.ca> wrote:
> In article <d7407b$2e4$1@magenta.plig.net>,
> Glyn Davies  <gryn@riffraff.plig.net> wrote:
> :For simplicity of implementation, I'd recommend implementing a message
> :layer over the TCP socket - something as simple as a two byte binary header
> :to indicate the length of the message to follow.

> :This means you can then read a whole message - text based or otherwise,
> :and the process it accordingly.  You don't need to mess about with parsing
> :lines and stuff like that.

> You still need to "mess about" if your data might not fit in a single
> packet. Recall that when you read() from a socket you are not certain
> to get all of the data unless the data length is no greater than
> your system's atomicity limits [PIPE_BUF]. The POSIX minimum
> value for PIPE_BUF is 512 -- so in the general case a write() of
> more than 512 bytes may require multiple read() statements.

Yes - but you do know how much you are getting.
You can handle all this stuff in a comms layer, then concentrate on your
application logic.

I've done countless systems using this technique.  Works well, and is simple.

Glyn

-- 

------------------------------------------------------------------------
Glyn Davies / gryn@plig.net / www.technobobbins.com / Insert quote here?
------------------------------------------------------------------------
0
Glyn
5/27/2005 10:52:20 AM
In comp.os.linux.embedded Grant Edwards <grante@visi.com> wrote:
> On 2005-05-26, Mark <mark_2811nospam@excite.com> wrote:

>> I expect I will end up with a text-based protocol, with a header
>> that indicates the number of bytes in the payload.

> If you're using a text-based protocol, you don't nead a header
> with a payload count to tell you where the message ends.  Use
> an end-of-message character or string.  The cr/lf pair is used
> as the end-of-message delimiter by a lot of text-based
> runs-on-top of TCP protocols.  That approach works wonderfully
> with line-oriented high-level text-handling libraries and
> utilties -- you can do all sorts of useful stuff using nothing
> but a few utilities like netcat, expect, and so on.

> I've also seen protocols that use STX/ETX to frame messages,
> and that works nicely as well, but isn't as compatible with
> pre-existing utilities.

<snip>

Most STX/ETX stuff I have seen was over serial comms.
Not seen it in use over TCP - except where there was an annex box
feeding out a bunch of serial lines converted to TCP.

Glyn

-- 

------------------------------------------------------------------------
Glyn Davies / gryn@plig.net / www.technobobbins.com / Insert quote here?
------------------------------------------------------------------------
0
Glyn
5/27/2005 10:55:40 AM
In comp.os.linux.embedded Mark <mark_2811nospam@excite.com> wrote:
> On Fri, 27 May 2005 01:12:46 -0400, James Antill wrote:

<snip>

> This is the main reason I'm leaning towards a byte count to delimit the
> message.  But I think I'll end up with some concept of special characters
> in the protocol (= , crlf).  Is there a standard way to escape these
> characters if they appear in the body of the message?  I suppose I could
> use nested netstrings to encode each individual field in the message. 
> Thoughts?

Standard way of escaping a 'special character' is to put it in twice.

ie, if % was your special character, to have a '%' in your message you'd
put %%

If you do put a byte count in, don't put it in as text.  It might make
testing easier, but I doubt it.

Being able to telnet in and test things is the only advantage of CR/LF
protocols.  Its not hard to knock up test apps which can send a message
though once you have done your basic comms library (if using a binary header)

Glyn



-- 

------------------------------------------------------------------------
Glyn Davies / gryn@plig.net / www.technobobbins.com / Insert quote here?
------------------------------------------------------------------------
0
Glyn
5/27/2005 11:06:12 AM
Glyn Davies <gryn@riffraff.plig.net> writes:

>> I've also seen protocols that use STX/ETX to frame messages,
>> and that works nicely as well, but isn't as compatible with
>> pre-existing utilities.
>
> Most STX/ETX stuff I have seen was over serial comms.
> Not seen it in use over TCP - except where there was an annex box
> feeding out a bunch of serial lines converted to TCP.

The protocol used by Micros Fidelio point of sale terminals use
STX/ETX framing when talking to the property management system.  The
reason for this is probably that these devices historically were
connected with serial links.

-- 
M�ns Rullg�rd
mru@inprovide.com
0
iso
5/27/2005 11:12:34 AM
In comp.os.linux.embedded M?ns Rullg?rd <mru@inprovide.com> wrote:
> Glyn Davies <gryn@riffraff.plig.net> writes:

>>> I've also seen protocols that use STX/ETX to frame messages,
>>> and that works nicely as well, but isn't as compatible with
>>> pre-existing utilities.
>>
>> Most STX/ETX stuff I have seen was over serial comms.
>> Not seen it in use over TCP - except where there was an annex box
>> feeding out a bunch of serial lines converted to TCP.

> The protocol used by Micros Fidelio point of sale terminals use
> STX/ETX framing when talking to the property management system.  The
> reason for this is probably that these devices historically were
> connected with serial links.


There is a multi-drop (I think thats the term, my brain is a bit frazzled today)
which was used to connect many terminals to a host controller.

ISO/ASYNC I think it was called.  Relied on a host polling each terminal
in turn, at which point they were allowed to speak.

Was used extensively with serial driven ATMs - most are TCP/IP these days
I believe.

But we digress...

Glyn
-- 

------------------------------------------------------------------------
Glyn Davies / gryn@plig.net / www.technobobbins.com / Insert quote here?
------------------------------------------------------------------------
0
Glyn
5/27/2005 1:37:44 PM
Glyn Davies <gryn@riffraff.plig.net> writes:

> In comp.os.linux.embedded M?ns Rullg?rd <mru@inprovide.com> wrote:
>> Glyn Davies <gryn@riffraff.plig.net> writes:
>
>>>> I've also seen protocols that use STX/ETX to frame messages,
>>>> and that works nicely as well, but isn't as compatible with
>>>> pre-existing utilities.
>>>
>>> Most STX/ETX stuff I have seen was over serial comms.
>>> Not seen it in use over TCP - except where there was an annex box
>>> feeding out a bunch of serial lines converted to TCP.
>
>> The protocol used by Micros Fidelio point of sale terminals use
>> STX/ETX framing when talking to the property management system.  The
>> reason for this is probably that these devices historically were
>> connected with serial links.
>
> There is a multi-drop (I think thats the term, my brain is a bit
> frazzled today) which was used to connect many terminals to a host
> controller.
>
> ISO/ASYNC I think it was called.  Relied on a host polling each terminal
> in turn, at which point they were allowed to speak.

I've never had the misfortune of having to deal with such a thing.

> Was used extensively with serial driven ATMs - most are TCP/IP these
> days I believe.

I've once seen an ATM display an mswindows dialog box with a DHCP
related error message.

-- 
M�ns Rullg�rd
mru@inprovide.com
0
iso
5/27/2005 1:53:13 PM
<snip>

>> There is a multi-drop (I think thats the term, my brain is a bit
>> frazzled today) which was used to connect many terminals to a host
>> controller.
>>
>> ISO/ASYNC I think it was called.  Relied on a host polling each terminal
>> in turn, at which point they were allowed to speak.

> I've never had the misfortune of having to deal with such a thing.

:-)
By the time I got to it we were moving people off them to dedicated lines.
Protocol stayed though as it worked fine.

>> Was used extensively with serial driven ATMs - most are TCP/IP these
>> days I believe.

> I've once seen an ATM display an mswindows dialog box with a DHCP
> related error message.

Yeah - they obviously didn't have the clever* stuff we knocked up
to stop that kind of thing happening.

Glyn
-- 

------------------------------------------------------------------------
Glyn Davies / gryn@plig.net / www.technobobbins.com / Insert quote here?
------------------------------------------------------------------------
0
Glyn
5/27/2005 2:38:13 PM
On 2005-05-27, James Antill <james-netnews@and.org> wrote:

>> If you're using a text-based protocol, you don't nead a header
>> with a payload count to tell you where the message ends.  Use
>> an end-of-message character or string.  The cr/lf pair is used
>> as the end-of-message delimiter by a lot of text-based
>> runs-on-top of TCP protocols.  That approach works wonderfully
>> with line-oriented high-level text-handling libraries and
>> utilties -- you can do all sorts of useful stuff using nothing
>> but a few utilities like netcat, expect, and so on.
>
> It interacts well with telnet,

If you handle IAC seqeunces -- at least well enough to ignore
them.

> which is about the best thing that can be said for it.

And it's a pretty big thing, in my experience.

> There are large problems with how you limit "too long" lines,

What problems?  If a message has to be X bytes long, it's X
bytes long regardless of how you choose to delimit it.
Handling long messages is a general issue and not specific to
the delimiter charater chosen.

> and what happens when you hit a CR or LF on it's own.

Yes, that can be a problem if you have to be able to include
arbitrary strings in the message body.  I was under the
impression that this wasn't the case for the OP's application.
I could be wrong.

> Esp. if you put something like a perl regexp directly on the
> socket. The HTTP response splitting problem is entirely the
> fault of pretending CRLF terminated lines are a good idea.

My other suggestion to use STX/ETX is pretty simple and is
quite robust if you impliment an escape mechanism for when
STX,ETX, or the escape character occur in the message body.

-- 
Grant Edwards                   grante             Yow!  Sometime in 1993
                                  at               NANCY SINATRA will lead a
                               visi.com            BLOODLESS COUP on GUAM!!
0
Grant
5/27/2005 2:59:49 PM
On 2005-05-27, Mark <mark_2811nospam@excite.com> wrote:

>>  It interacts well with telnet, which is about the best thing that can be
>> said for it. There are large problems with how you limit "too long" lines,
>> and what happens when you hit a CR or LF on it's own. Esp. if you put
>> something like a perl regexp directly on the socket.
>>  The HTTP response splitting problem is entirely the fault of pretending
>> CRLF terminated lines are a good idea.
>
> This is the main reason I'm leaning towards a byte count to delimit the
> message.  But I think I'll end up with some concept of special characters
> in the protocol (= , crlf).  Is there a standard way to escape these
> characters if they appear in the body of the message?

Assuming you're using a 7-bit character set, one of the easiest
schemes I've seen is on transmit to convert <special char> to 

   [Escape] <special char>|0x80

Remember to include the [Escape] character in your set of
special characters that get escaped:

When sending a message body,

  STX becomes ESC,STX|0x80
  ETX becomes ESC,ETX|0x80
  ESC becomes ESC,ESC|0x80

When receiving a message body

  ESC,<whatever> becomes <whatever>&0x7f

-- 
Grant Edwards                   grante             Yow!  Yow! Are you the
                                  at               self-frying president?
                               visi.com            
0
Grant
5/27/2005 3:04:53 PM
On 2005-05-27, Glyn Davies <gryn@riffraff.plig.net> wrote:

>> I've also seen protocols that use STX/ETX to frame messages,
>> and that works nicely as well, but isn't as compatible with
>> pre-existing utilities.
>
><snip>
>
> Most STX/ETX stuff I have seen was over serial comms.
> Not seen it in use over TCP - except where there was an annex box
> feeding out a bunch of serial lines converted to TCP.

A byte stream is a byte stream.  The serial (as in RS-232) byte
stream isn't reliable, but I cann't see any difference between a
serial comm link and a TCP link when it comes to message framing.

-- 
Grant Edwards                   grante             Yow!  How do I get HOME?
                                  at               
                               visi.com            
0
Grant
5/27/2005 3:07:15 PM
In comp.os.linux.embedded Grant Edwards <grante@visi.com> wrote:
> On 2005-05-27, Glyn Davies <gryn@riffraff.plig.net> wrote:

>>> I've also seen protocols that use STX/ETX to frame messages,
>>> and that works nicely as well, but isn't as compatible with
>>> pre-existing utilities.
>>
>><snip>
>>
>> Most STX/ETX stuff I have seen was over serial comms.
>> Not seen it in use over TCP - except where there was an annex box
>> feeding out a bunch of serial lines converted to TCP.

> A byte stream is a byte stream.  The serial (as in RS-232) byte
> stream isn't reliable, but I cann't see any difference between a
> serial comm link and a TCP link when it comes to message framing.


Nope, there is no general difference...
The serial streams were generally 7bit though, which would stop you
using a binary length header.

Also, as you point out the serial stream is unreliable.
That makes a difference in the protocol you choose.  With STX/ETX framing if you get
garbage you can resync at the next STX. If you are using length headers
and you get some noise on the line, you are lost and have no way to
resync.

Glyn

-- 

------------------------------------------------------------------------
Glyn Davies / gryn@plig.net / www.technobobbins.com / Insert quote here?
------------------------------------------------------------------------
0
Glyn
5/27/2005 3:15:33 PM
On Fri, 27 May 2005 11:06:12 +0000, Glyn Davies wrote:

> In comp.os.linux.embedded Mark <mark_2811nospam@excite.com> wrote:
>> On Fri, 27 May 2005 01:12:46 -0400, James Antill wrote:
> 
> <snip>
> 
>> This is the main reason I'm leaning towards a byte count to delimit the
>> message.  But I think I'll end up with some concept of special characters
>> in the protocol (= , crlf).  Is there a standard way to escape these
>> characters if they appear in the body of the message?  I suppose I could
>> use nested netstrings to encode each individual field in the message. 
>> Thoughts?

 Generally I'd say one of the major advantages of having leading byte
counts is that you don't have to escape anything. Where you need multiple
arguments just enclose them all in a single netstring. Ie. for abcd = xyz
do[1]

13:4:abcd,3:xyz,,

....if you want to get really fancy (and allow easier exensibility) you can
do key/value pairs[2].

> Standard way of escaping a 'special character' is to put it in twice.
> 
> ie, if % was your special character, to have a '%' in your message you'd
> put %%

 That works "ok" if you will only ever have a single character that needs
escaping, like IAC escaping in telnet ... it works much worse if you have
to do more than one (CSV is the obvious example here). Then doing
html/C/url numbered escaping works better (but again, if you can avoid it
I'd do so).

> If you do put a byte count in, don't put it in as text.  It might make
> testing easier, but I doubt it.

 Putting a byte count in as a binary encoding should always be a last
resort. You save little, and you are then tied to that byte size[3].


[1] http://www.and.org/vstr/ex_netstr.c
[2] http://www.and.org/vstr/ex_dir_list.c
[3] http://www.and.org/texts/ascii_binary_data

-- 
James Antill -- james@and.org
http://www.and.org/vstr/httpd

0
James
5/27/2005 4:25:40 PM
On Fri, 27 May 2005 14:59:49 +0000, Grant Edwards wrote:

> On 2005-05-27, James Antill <james-netnews@and.org> wrote:
> 
>>> If you're using a text-based protocol, you don't nead a header
>>> with a payload count to tell you where the message ends.  Use
>>> an end-of-message character or string.  The cr/lf pair is used
>>> as the end-of-message delimiter by a lot of text-based
>>> runs-on-top of TCP protocols.  That approach works wonderfully
>>> with line-oriented high-level text-handling libraries and
>>> utilties -- you can do all sorts of useful stuff using nothing
>>> but a few utilities like netcat, expect, and so on.
>>
>> It interacts well with telnet,
> 
> If you handle IAC seqeunces -- at least well enough to ignore
> them.

 I don't know of any HTTP, SMTP, NNTP, POP3, IMAP, etc. server that does
anything with IAC sequences. Generally servers that care initiate by
sending IAC commands EOR, NAWS and TTYPE etc to the client.

>> which is about the best thing that can be said for it.
> 
> And it's a pretty big thing, in my experience.

 It's "big" because it's there already so noone has to write a
simple client to test a service with. If SMTP had been written
using netstrings, then a telnetstr command would be available
everywhere and this "big" thing would be worth nothing.
 A similar thing is happening with HTTP, it's not "good"[1] but there are
a significant number of tools that understand a HTTP stream ... so people
hack it into places it shouldn't be so they can leverage those tools.

>> There are large problems with how you limit "too long" lines,
> 
> What problems?

 Say I connect to an SMTP service and keep sending 'a', the other end has
to keep accepting data and parsing it upto the amount it limits itself to
accepting for a single line. With something like a netstring the remote
end can decide within 10 characters if it's going to just drop the
connection/message or handle it.

>> and what happens when you hit a CR or LF on it's own.
> 
> Yes, that can be a problem if you have to be able to include arbitrary
> strings in the message body.  I was under the impression that this
> wasn't the case for the OP's application. I could be wrong.

 It isn't just a problem of arbitrary data, but of different
clients/servers parsing the same data in different ways. This is
obviously a bigger problem the more implementations of clients and servers
you have, and how compatible you want to be ... but even with just one
client and one server it wouldn't be unique to have silent bugs where
someone typed \n\r or r\\n or just \n instead of \r\n at one point in the
code.
 At which point a third application implementing the protocol, or a
supposedly compatible change to either the server or client can bring out
bugs (often in the edge cases) ... something like netstrings is much
less likely to have this kind of problem.

[1] http://www.and.org/texts/server-http

-- 
James Antill -- james@and.org
http://www.and.org/vstr/httpd

0
James
5/27/2005 5:04:09 PM
On Fri, 27 May 2005 15:07:15 -0000, Grant Edwards <grante@visi.com>
wrote:

>A byte stream is a byte stream.  The serial (as in RS-232) byte
>stream isn't reliable, but I cann't see any difference between a
>serial comm link and a TCP link when it comes to message framing.

Except for Modbus RTU style framing, in which the time gaps between
bytes _are_ the actual frame delimiters. Maintaining these over a
TCP/IP link would be a bit problematic :-).

Paul

0
Paul
5/27/2005 6:41:34 PM
On 2005-05-27, James Antill <james-netnews@and.org> wrote:

>> If you handle IAC seqeunces -- at least well enough to ignore
>> them.
>
>  I don't know of any HTTP, SMTP, NNTP, POP3, IMAP, etc. server that does
> anything with IAC sequences. Generally servers that care initiate by
> sending IAC commands EOR, NAWS and TTYPE etc to the client.

I've run into a few telnet clients that aren't as well behaved
as the Unix ones.  The Unix ones typically won't initiate any
negotiation if they're connecting to a non-telnet port.
Several of the windows clients I've tried aren't as polite.

-- 
Grant Edwards                   grante             Yow!  Psychoanalysis?? I
                                  at               thought this was a nude
                               visi.com            rap session!!!
0
Grant
5/28/2005 12:02:40 AM
On 2005-05-27, Paul Keinanen <keinanen@sci.fi> wrote:
> On Fri, 27 May 2005 15:07:15 -0000, Grant Edwards <grante@visi.com>
> wrote:
>
>>A byte stream is a byte stream.  The serial (as in RS-232) byte
>>stream isn't reliable, but I cann't see any difference between a
>>serial comm link and a TCP link when it comes to message framing.
>
> Except for Modbus RTU style framing, in which the time gaps between
> bytes _are_ the actual frame delimiters. Maintaining these over a
> TCP/IP link would be a bit problematic :-).

True.  Modbus RTU's 3.5 byte time delimiter sucks.  You're
screwed even if all you want to do is use the RX FIFO in a
UART.

-- 
Grant Edwards                   grante             Yow!  It's a lot of fun
                                  at               being alive... I wonder if
                               visi.com            my bed is made?!?
0
Grant
5/28/2005 12:04:16 AM
Grant Edwards wrote:
> On 2005-05-27, Paul Keinanen <keinanen@sci.fi> wrote:
> 
>>On Fri, 27 May 2005 15:07:15 -0000, Grant Edwards <grante@visi.com>
>>wrote:
>>
>>
>>>A byte stream is a byte stream.  The serial (as in RS-232) byte
>>>stream isn't reliable, but I cann't see any difference between a
>>>serial comm link and a TCP link when it comes to message framing.
>>
>>Except for Modbus RTU style framing, in which the time gaps between
>>bytes _are_ the actual frame delimiters. Maintaining these over a
>>TCP/IP link would be a bit problematic :-).
> 
> 
> True.  Modbus RTU's 3.5 byte time delimiter sucks.  You're
> screwed even if all you want to do is use the RX FIFO in a
> UART.
> 

Hardly any of the Modbus/RTU programs on PC:s handle the
frame timing correctly.

Modbus framing is one of the best examples how framing
should not be done. The only thing that competes with
it is the idea of tunneling Modbus datagrams with TCP
instead of UDP.

-- 

Tauno Voipio
tauno voipio (at) iki fi

0
Tauno
5/28/2005 9:34:53 AM
On 2005-05-28, Tauno Voipio <tauno.voipio@iki.fi.NOSPAM.invalid> wrote:

>>>Except for Modbus RTU style framing, in which the time gaps
>>>between bytes _are_ the actual frame delimiters. Maintaining
>>>these over a TCP/IP link would be a bit problematic :-).
>>
>> True.  Modbus RTU's 3.5 byte time delimiter sucks.  You're
>> screwed even if all you want to do is use the RX FIFO in a
>> UART.
>
> Hardly any of the Modbus/RTU programs on PC:s handle the
> frame timing correctly.

As long as they're the master, or it's a full-duplex bus, they
can get away with it.  Being a slave on a half-duplex bus
(everybody sees both commands and responses) is where the
problems usually happen.

I once talked to somebody who used an interesting scheme to
detect Modbus RTU messages.  He ignored timing completely (so
he could use HW FIFOs), and decided that he would just monitor
the receive bytestream for any block of data that started with
his address and had the correct CRC at the location indicated
by the bytecount.  It meant that he had to have a receive
buffer twice as long as the max message and keep multiple
partial CRCs running, but I guess it worked.

> Modbus framing is one of the best examples how framing should
> not be done.

I'd have to agree that Modbus RTU's framing was a horrible
mistake. ASCII mode was fine since it had unique
start-of-message and end-of-message delimiters.

> The only thing that competes with it is the idea of tunneling
> Modbus datagrams with TCP instead of UDP.

I don't even want to know how they did message delimiting in
Modubs over TCP...

-- 
Grant Edwards                   grante             Yow!  What GOOD is a
                                  at               CARDBOARD suitcase ANYWAY?
                               visi.com            
0
Grant
5/28/2005 2:08:17 PM
On Sat, 28 May 2005 14:08:17 -0000, Grant Edwards <grante@visi.com>
wrote:

>On 2005-05-28, Tauno Voipio <tauno.voipio@iki.fi.NOSPAM.invalid> wrote:

>> Hardly any of the Modbus/RTU programs on PC:s handle the
>> frame timing correctly.
>
>As long as they're the master, or it's a full-duplex bus, they
>can get away with it.  Being a slave on a half-duplex bus
>(everybody sees both commands and responses) is where the
>problems usually happen.

Being a slave on a multidrop network is the problem, a half-duplex
point to point connection is not critical.

>I once talked to somebody who used an interesting scheme to
>detect Modbus RTU messages.  He ignored timing completely (so
>he could use HW FIFOs), and decided that he would just monitor
>the receive bytestream for any block of data that started with
>his address and had the correct CRC at the location indicated
>by the bytecount.  It meant that he had to have a receive
>buffer twice as long as the max message and keep multiple
>partial CRCs running, but I guess it worked.

As long as you are not using broadcast messages, it should be
sufficient for a multidrop slave to have just one CRC running (and
check it after each byte received). This calculation should be done on
all frames, not just those addressed to you.

From time to time, a message frame may be corrupted when the master is
communicating with an other slave and after that, your slave does no
longer make any sense of the incomming bytes, it might as well ignore
them and set a timeout that is shorter than the master retransmission
timeout.

As long as the master communicates with other slaves, your slave does
not understand anything what is going on. When the master addresses
your node, the first request will be lost and master is waiting for a
response, until the retransmission timeout expires.

The slave timeout will expire before this, synchronisation is regained
and your slave is now eagerly waiting for the second attempt of the
request from the master. If there are multiple slaves that have lost
synchronisation, they all will regain synchronisation, when one
addressed slave fails to respond to the first request. 

If the bus is so badly corrupted, that all slaves get a bad frame, the
communication will timeout anyway, thus all slaves will regain
synchronisation immediately. Only in situations (e.g. badly terminated
bus) that the master and the actively communicating slave does not get
an CRC error, but your slave will detect a CRC error, your slave will
be out of sync, until the master addresses your slave and the timeout
occurs.

Thus, the only harm is really that broadcasts can not be used, as all
out of sync slaves would loose them.

>> The only thing that competes with it is the idea of tunneling
>> Modbus datagrams with TCP instead of UDP.
>
>I don't even want to know how they did message delimiting in
>Modubs over TCP...

Since Modbus over TCP is really point to point, the situation is
similar to the serial point to point case.

Some converter boxes also convert Modbus RTU to Modbus/TCP before
sending it over the net. The Modbus/TCP protocol contains a fixed
header (including a byte count) and the variable length RTU frame
(without CRC).

The strange thing is why they created Modbus/TCP for running the
Modbus protocol over Ethernet instead of Modbus/UDP. 

However, googling for "Modbus/UDP" gives quite a lot of hits, so quite
a few vendors have implemented some kind of Modbus over UDP protocols
in their products in addition to Modbus/TCP.

Paul

0
Paul
5/28/2005 8:18:37 PM
On 2005-05-28, Paul Keinanen <keinanen@sci.fi> wrote:

>>> Hardly any of the Modbus/RTU programs on PC:s handle the
>>> frame timing correctly.
>>
>>As long as they're the master, or it's a full-duplex bus, they
>>can get away with it.  Being a slave on a half-duplex bus
>>(everybody sees both commands and responses) is where the
>>problems usually happen.
>
> Being a slave on a multidrop network is the problem, a
> half-duplex point to point connection is not critical.

In my experience the problems usually occur only on a
half-duplex network.  If the slave doesn't see the other slaves
responses, it has no problem finding the start of commands from
the host.

>>I once talked to somebody who used an interesting scheme to
>>detect Modbus RTU messages.  He ignored timing completely (so
>>he could use HW FIFOs), and decided that he would just monitor
>>the receive bytestream for any block of data that started with
>>his address and had the correct CRC at the location indicated
>>by the bytecount.  It meant that he had to have a receive
>>buffer twice as long as the max message and keep multiple
>>partial CRCs running, but I guess it worked.
>
> As long as you are not using broadcast messages, it should be
> sufficient for a multidrop slave to have just one CRC running
> (and check it after each byte received).

How so?  If you don't know where the frame started because you
can't detect timing gaps, you have to have a seperate CRC
running for each possible frame starting point, where each
"starting point" is any byte matching your address (or the
broadcast address).

> This calculation should be done on all frames, not just those
> addressed to you.

The problem is that you don't kow where the frames start so the
phrase "on all frames" isn't useful.

>>I don't even want to know how they did message delimiting in
>>Modubs over TCP...
>
> Since Modbus over TCP is really point to point, the situation is
> similar to the serial point to point case.

Ah.

> Some converter boxes also convert Modbus RTU to Modbus/TCP
> before sending it over the net. The Modbus/TCP protocol
> contains a fixed header (including a byte count) and the
> variable length RTU frame (without CRC).
>
> The strange thing is why they created Modbus/TCP for running
> the Modbus protocol over Ethernet instead of Modbus/UDP. 

Using UDP would seem to be a lot more analgous to a typical
multidrop

> However, googling for "Modbus/UDP" gives quite a lot of hits, so quite
> a few vendors have implemented some kind of Modbus over UDP protocols
> in their products in addition to Modbus/TCP.

Interesting.

-- 
Grant Edwards                   grante             Yow!  What a
                                  at               COINCIDENCE! I'm an
                               visi.com            authorized "SNOOTS OF THE
                                                   STARS" dealer!!
0
Grant
5/29/2005 3:11:42 PM
On Sun, 29 May 2005 15:11:42 -0000, Grant Edwards <grante@visi.com>
wrote:


>>>I once talked to somebody who used an interesting scheme to
>>>detect Modbus RTU messages.  He ignored timing completely (so
>>>he could use HW FIFOs), and decided that he would just monitor
>>>the receive bytestream for any block of data that started with
>>>his address and had the correct CRC at the location indicated
>>>by the bytecount.  It meant that he had to have a receive
>>>buffer twice as long as the max message and keep multiple
>>>partial CRCs running, but I guess it worked.

<some reordering>

>> This calculation should be done on all frames, not just those
>> addressed to you.
>
>The problem is that you don't kow where the frames start so the
>phrase "on all frames" isn't useful.

If the master starts after all slaves are ready waiting for the first
message, they all know were the first message starts. When the correct
CRC is detected, the end of the first frame is known and it can now be
assumed that the next byte received will start the next frame :-).
This continues from frame to frame.

Everything works well as long as there are no transfer errors on the
bus or any premature "CRC matches" within the data part of a message.
If the sync is lost, unless broadcasting is used, is there any need to
regain sync until the master addresses your slave ? The master will
not get an answer to the first request, but it will resend the request
after the resend timeout period. The slave only needs to be able to
detect this timeout period, which would usually be a few hundred byte
transfer times.

If the slave pops up into an active bus, it will regain
synchronisation, when the master will address it first time and it
will resend the command after a timeout period, since it did not get a
reply. The slave just needs to detect the long resend timeout.

>> As long as you are not using broadcast messages, it should be
>> sufficient for a multidrop slave to have just one CRC running
>> (and check it after each byte received).
>
>How so?  If you don't know where the frame started because you
>can't detect timing gaps, you have to have a seperate CRC
>running for each possible frame starting point, where each
>"starting point" is any byte matching your address (or the
>broadcast address).

A shift, bit test and xor operation is needed for each data bit
received to calculate the CRC into the CRC "accumulator". This
calculation can be done for all received bytes when the end of frame
gap is detected or eight bits can be calculated each time a new byte
is received (actually calculate with the byte received two bytes since
the current byte). 

From a computational point of view, both methods are nearly equal. The
only difference is that in the first case, the CRC accumulator is
compared with the last two bytes in the frame (in correct byte order)
only after the gap was detected, however in the latter case, the
updated CRC accumulator must be compared with the two most recently
received bytes each time a byte is received. Thus, the computational
difference is only two byte compares for each received byte.

Thus a single CRC accumulator is sufficient, if no broadcasts are
expected.

However, if it is required to receive broadcasts, then it might be
justified to use about 260 parallel CRC accumulators (if the maximum
size frames are expected). Each accumulator starts calculating from a
different byte in the buffer. Each time a new character is received,
the last two bytes (the CRC candidates) are compared with all 260 CRC
accumulators and once a match is found, the start of that frame is
also known and synchronisation is regained. After synchronisation is
regained, one CRC accumulator is sufficient for keeping track of the
frame gaps.

Since the Modbus CRC is only 16 bits long, relying on solely the CRC
calculations can cause premature CRC detections with the likelihood of
1/65536 or about 16 ppm, so additional message header checks should be
employed to detect these, before declaring a true CRC end of frame.

Paul

0
Paul
5/29/2005 5:14:19 PM
On 2005-05-29, Paul Keinanen <keinanen@sci.fi> wrote:

> Everything works well as long as there are no transfer errors on the
> bus or any premature "CRC matches" within the data part of a message.
> If the sync is lost, unless broadcasting is used, is there any need to
> regain sync until the master addresses your slave ? The master will
> not get an answer to the first request, but it will resend the request
> after the resend timeout period. The slave only needs to be able to
> detect this timeout period, which would usually be a few hundred byte
> transfer times.

Ah.  Right.  I should have thought of that.  [I never used the
scheme in question -- I always had an interrupt for each rx
byte, and did the gap detection according to the spec.]

> If the slave pops up into an active bus, it will regain
> synchronisation, when the master will address it first time
> and it will resend the command after a timeout period, since
> it did not get a reply. The slave just needs to detect the
> long resend timeout.

Yup.  And in most of the control systems I've run into,
timeouts are usually on the order of a second or two, so
detecting that isn't a problem -- even with a FIFO.

> However, if it is required to receive broadcasts, then it
> might be justified to use about 260 parallel CRC accumulators
> (if the maximum size frames are expected). Each accumulator
> starts calculating from a different byte in the buffer. Each
> time a new character is received, the last two bytes (the CRC
> candidates) are compared with all 260 CRC accumulators and
> once a match is found, the start of that frame is also known
> and synchronisation is regained. After synchronisation is
> regained, one CRC accumulator is sufficient for keeping track
> of the frame gaps.

Hmm. I seem to have forgotten how broadcast messages work in
the spec [I haven't done Modbus for a few years].  In our
systems, we implimented a non-standard broadcast scheme by
reserving address 0 as a broadcast address. One would only
needed to start a CRC when a byte was seen that was 0 or that
matched the slave address.

> Since the Modbus CRC is only 16 bits long, relying on solely the CRC
> calculations can cause premature CRC detections with the likelihood of
> 1/65536 or about 16 ppm, so additional message header checks should be
> employed to detect these, before declaring a true CRC end of frame.

Good point.  Thanks for the detailed explination.

-- 
Grant Edwards                   grante             Yow!  This ASEXUAL
                                  at               PIG really BOILS
                               visi.com            my BLOOD... He's
                                                   so... so... URGENT!!
0
Grant
5/29/2005 5:49:54 PM
James Antill <james-netnews@and.org> writes:

> On Wed, 25 May 2005 19:21:02 +0000, Sean Burke wrote:
> 
> > One option to consider is to embed a web interface into your
> > application. This has the advantage that you can use any web
> > browser as the client side of the interface.
> 
>  Not a terrible idea, a simple HTTP/1.0 server can be pretty small esp. if
> you don't mind stopping as soon as something works (the very basics can be
> done in < 20 lines of C).
> 
> > There are a variety of very small web servers that are suitable
> > for embedding. One such that I have used successfully is "pserv".
> > From the FreeBSD ports description:
> 
>  Yeh, pretty good ... I stopped looking after seeing 2 major
> vulnerabilities on the first google page.
>  The code also looked promising ... for more exploits.

Are you commenting on anything beyond the obvious fact 
that the code uses strcpy and sprintf?  
 
>  Writing a custom "simple protocol" is likely to be much easier, using
> netstrings is probably more likely to make you do the right thing ... but
> a simple "CMD arg1 arg2" type telnet/SMTP/NNTP like protocol isn't hard to
> get right.

None of these are inherently simpler than HTTP, and you
don't get the advantages that a web browser's sophisticated
support for HTML confers.

-SEan








0
Sean
5/29/2005 8:45:42 PM
Sean Burke wrote:

> None of these are inherently simpler than HTTP, and you
> don't get the advantages that a web browser's sophisticated
> support for HTML confers.

Speaking as one who has done it, adapting HTTP instead of using a custom 
protocol has many advantages besides the above. Think of all the proxies 
and filters out there, the tools that snoop the wire, making a nice 
graphical display sorted into request and response sequences (e.g. 
HTTPlook). Consider the existence of client-side libraries ready to use 
in any language (libwww or libcurl for C, java.net.* or Jakarta 
HttpClient for Java, lots of Perl modules, etc). None of this is 
available to a custom protocol, however easy to implement.

-- 
Henry Townsend
0
Henry
5/30/2005 2:45:34 AM
Glyn Davies wrote:

> 
> Most STX/ETX stuff I have seen was over serial comms.
> 

STX/ETX over serial is used to steady the line noise 
that occurs in rs485 communication, because when slave
switches it's transmit line ON, it generates noise on the line
as a sideffect that could be missineterpred, so it's a good
practice for serial messages (bin/text) to start with several 
STX chars and end with several ETX chars - message itself should
have some crc check...

best regards,
  Mario

0
Mile
5/30/2005 12:15:58 PM
I didn't read all the answers, just butting in, but take a look at NMEA0183.
That is a simple text based protocol to send formatted data, used by marine
equipement (GPS for instance). Is is more or less one way but you can simply
add the other way if you like. Use it over UDP and add some acknowledge
messages for instance.

John


0
John
5/30/2005 4:03:14 PM
On Mon, 30 May 2005 14:15:58 +0200, Mile Blenton
<nomejl@blentronix.com> wrote:

>Glyn Davies wrote:
>
>> 
>> Most STX/ETX stuff I have seen was over serial comms.
>> 
>
>STX/ETX over serial is used to steady the line noise 
>that occurs in rs485 communication, because when slave
>switches it's transmit line ON, it generates noise on the line
>as a sideffect that could be missineterpred, 

Why should turning on the transmitter cause any noise ? In any
properly terminated RS-485 system, the line is drawn by resistors to
the Mark (idle) state when no transceiver is active driving the bus.
Turning the transmitter on in the low impedance Mark state does not
change the voltage levels. The voltages change when the transmitter
start to send the start bit (Space).

However, if the RS-485 line in a noisy environment that is used
infrequently, i.e. there are long idle periods between messages, the
line is more prone to random errors due to the high impedance idle
state than the low impedance active Mark or Space state. The noise
will often cause false start bit triggerings (often seen as 0xFF bytes
in the UART).

When the protocol frame always starts with a known character (such as
STX), it is quite easy to ignore any noise received during the idle
period. 

In fact this also applies to actively driven RS-232/422/20 mA lines in
very noisy environments if there are long pauses between messages.

However, the STX detection fails, if there is a Space noise pulse less
than 10 bit times ahead of the STX (at 8N1 characters). The Space is
assumed to be a start bit and the UART starts counting bits. While the
UART is still counting data bits, the start bit for the true STX
character is received, but it is interpreted as a data bit by the
UART. When the UART is finally ready to receive the stop bit, actually
some of the middle bits of the STX will be received. If this bit
happens to be in the Mark state, the UART is satisfied with the stop
bit and is waiting for the next Mark to Space transition, which is
interpreted as the next start bit. However, if a Space data bit is
received when the UART expects the stop bit, the framing error occurs.

In both cases, the STX character will not be detected and usually the
whole frame will be lost.

BTW, the Modbus RTU specification (which does not use STX) specifies,
that the transmitter should be turned on at least 1.5 bit times before
the start of the transmission. This assumes that while errors may
occur in the passively maintained Mark state, the actively driven low
impedance Mark state will keep the line clean for at least 15 bit
times. Thus, there should be no false start bits too close to the
actual message, thus the first byte is always decoded correctly.

>so it's a good
>practice for serial messages (bin/text) to start with several 
>STX chars 

Using multiple STX characters makes sense only if there are more than
10 bit actively driven Mark (idle) between the STX characters. Even if
the first STX is lost due to the false start bit, the second will be
reliably detected. Sending multiple STX characters without a time gap
would just cause a few framing errors, but it is unlikely to regain
synchronisation. In fact, it would make more sense to send a few 0xFF
bytes, so if the first is lost due to a premature start bit, the UART
would get a few bits in the Mark state and assume that the line is
idle and correctly detect the start bit of the next 0xFF byte.
  
>and end with several ETX chars 

A known end character (such as ETX) is used, since a simple interrupt
service routine can independently receive characters until the end
character is detected and the whole message can be passed to a higher
level routine as a single entity.

How would multiple ETX characters help ? The end of message is already
detected. 

Paul

0
Paul
5/30/2005 8:20:19 PM
On Sun, 29 May 2005 20:45:42 +0000, Sean Burke wrote:

> 
> James Antill <james-netnews@and.org> writes:
> 
>> On Wed, 25 May 2005 19:21:02 +0000, Sean Burke wrote:
>> 
>> > There are a variety of very small web servers that are suitable
>> > for embedding. One such that I have used successfully is "pserv".
>> > From the FreeBSD ports description:
>> 
>>  Yeh, pretty good ... I stopped looking after seeing 2 major
>> vulnerabilities on the first google page.
>>  The code also looked promising ... for more exploits.
> 
> Are you commenting on anything beyond the obvious fact 
> that the code uses strcpy and sprintf?  

 That isn't enough? See: http://www.and.org/vstr/security
 Programmers _cannot_ get this right.

 If, when I look outside, there is water falling from the sky I do not
need to walk outside to know I'm going to get wet ... and if you are
arguing that the drops of water are small and have large gaps between
them, I am still not going to feel compelled to walk outside to see if
I get wet.

>>  Writing a custom "simple protocol" is likely to be much easier, using
>> netstrings is probably more likely to make you do the right thing ... but
>> a simple "CMD arg1 arg2" type telnet/SMTP/NNTP like protocol isn't hard to
>> get right.
> 
> None of these are inherently simpler than HTTP, and you
> don't get the advantages that a web browser's sophisticated
> support for HTML confers.

 Which web browser? As I said, in theory you can get something "simplish"
that looks like a HTTP/1.0 server from the right angle ... and it might
even work with mozilla (as that client is very forgiving), but making it a
real HTTP/1.0 server isn't trivial and supporting HTTP/1.1 is very hard.
 Also if you need state to cross message boundaries you'll have to
implement a lot more code on the server side.

-- 
James Antill -- james@and.org
http://www.and.org/vstr/httpd

0
James
5/30/2005 9:41:26 PM
Hi,  Too much answers,seemingly, but one more why not. try to see the
RFC 3117 and the protocol BEEP (the old BXXP) www.beepcore.org It is
XML based but I think it can help you. good luck

0
ivaylo
6/8/2005 12:53:09 AM
Reply: