f



bundling efficiency, too good to be true?

Everyone has seen that sending one big file works much more
efficiently than many small files.  The effect quite astounding, many
orders of magnitude. It just occurred to me that I don't think I can
account for that huge difference.  Where is all the time going?

It then occurred to me, that any sort of technique to reduce the
difference could have a huge effect on the Internet as a whole.
-- 
Roedy Green Canadian Mind Products
http://mindprod.com

If you tell a computer the same fact in more than one place, unless you have an automated mechanism to ensure           they stay in sync, the versions of the fact will eventually get out of sync.
0
Roedy
3/31/2010 9:23:04 PM
comp.lang.java.programmer 52633 articles. 0 followers. Post Follow

6 Replies
351 Views

Similar Articles

[PageSpeed] 14

On 31-03-2010 17:23, Roedy Green wrote:
> Everyone has seen that sending one big file works much more
> efficiently than many small files.  The effect quite astounding, many
> orders of magnitude. It just occurred to me that I don't think I can
> account for that huge difference.  Where is all the time going?

Given the lack of context, then one can only guess:
- file open and file creation are rather expensive operations
   so many small files have huge overhead
- per file protocol overhead
- really small files can not be compressed as efficiently
   as larger files

Arne
0
ISO
3/31/2010 10:38:15 PM
On Wed, 31 Mar 2010, Roedy Green wrote:

> Everyone has seen that sending one big file works much more efficiently 
> than many small files.  The effect quite astounding, many orders of 
> magnitude. It just occurred to me that I don't think I can account for 
> that huge difference.  Where is all the time going?

TCP handshake, TCP slow start (look that one up if you don't know it), 
roundtrips for control packets at the start of the connection. Losing a 
bit of time can have a huge impact on throughput - it's all about the 
bandwidth-delay product, which on today's long, fat networks is huge.

> It then occurred to me, that any sort of technique to reduce the 
> difference could have a huge effect on the Internet as a whole.

Yes. It's called pipelining, and it's been in HTTP since 1999.

Although it's not that widely used by browsers, because of worries about 
compatibility with servers, which seems a bit of a waste.

tom

-- 
I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth. -- Umberto Eco
0
Tom
3/31/2010 10:51:27 PM
Tom Anderson wrote:
> On Wed, 31 Mar 2010, Roedy Green wrote:
> 
>> Everyone has seen that sending one big file works much more 
>> efficiently than many small files.  The effect quite astounding, many 
>> orders of magnitude. It just occurred to me that I don't think I can 
>> account for that huge difference.  Where is all the time going?
> 
> TCP handshake, TCP slow start (look that one up if you don't know it), 
> roundtrips for control packets at the start of the connection. Losing a 
> bit of time can have a huge impact on throughput - it's all about the 
> bandwidth-delay product, which on today's long, fat networks is huge.

"back in the day" some serial protocols had asynchronous
ACK and resend on packets.

    BugBear
0
bugbear
4/1/2010 7:46:34 AM
In article <alpine.DEB.1.10.1003312344430.13579@urchin.earth.li>,
 Tom Anderson <twic@urchin.earth.li> wrote:

> On Wed, 31 Mar 2010, Roedy Green wrote:
> 
> > Everyone has seen that sending one big file works much more efficiently 
> > than many small files.  The effect quite astounding, many orders of 
> > magnitude. It just occurred to me that I don't think I can account for 
> > that huge difference.  Where is all the time going?
> 
> TCP handshake, TCP slow start (look that one up if you don't know it), 
> roundtrips for control packets at the start of the connection. Losing a 
> bit of time can have a huge impact on throughput - it's all about the 
> bandwidth-delay product, which on today's long, fat networks is huge.
> 
> > It then occurred to me, that any sort of technique to reduce the 
> > difference could have a huge effect on the Internet as a whole.
> 
> Yes. It's called pipelining, and it's been in HTTP since 1999.
> 
> Although it's not that widely used by browsers, because of worries about 
> compatibility with servers, which seems a bit of a waste.
> 
> tom

Browsers don't support pipelining because the multiplexer/demultiplexer 
is too complicated for the average software engineer.  Out-of-order 
response processing requires forcing preceding responses in the pipeline 
into memory.  That's tricky, but not too bad.  Now do that and rebuild 
the pipeline when the connection closes or drops.  Ugly!

At least in the old Innovation HTTPClient, that results in multiple lock 
grabs on components of linked list that's prone to failure.  The code is 
convoluted and it's looking like a total rewrite might be easier.

Last I heard, Microsoft and Apache clients can't pipeline; WebKit can 
but it's an experimental feature.
-- 
I won't see Google Groups replies because I must filter them as spam
0
Kevin
4/1/2010 7:47:54 AM
Kevin McMurtrie wrote:
> In article <alpine.DEB.1.10.1003312344430.13579@urchin.earth.li>,
>  Tom Anderson <twic@urchin.earth.li> wrote:
> 
>> On Wed, 31 Mar 2010, Roedy Green wrote:
>>
>>> Everyone has seen that sending one big file works much more efficiently 
>>> than many small files.  The effect quite astounding, many orders of 
>>> magnitude. It just occurred to me that I don't think I can account for 
>>> that huge difference.  Where is all the time going?
>> TCP handshake, TCP slow start (look that one up if you don't know it), 
>> roundtrips for control packets at the start of the connection. Losing a 
>> bit of time can have a huge impact on throughput - it's all about the 
>> bandwidth-delay product, which on today's long, fat networks is huge.
>>
>>> It then occurred to me, that any sort of technique to reduce the 
>>> difference could have a huge effect on the Internet as a whole.
>> Yes. It's called pipelining, and it's been in HTTP since 1999.
>>
>> Although it's not that widely used by browsers, because of worries about 
>> compatibility with servers, which seems a bit of a waste.
>>
>> tom
> 
> Browsers don't support pipelining because the multiplexer/demultiplexer 
> is too complicated for the average software engineer.
[ SNIP ]
AHS

Simple solution - don't use average software engineers to work on this 
code. There are how many significant web browsers again? Surely it's not 
so tough to ensure that the 2 or 3 people per important browser, maybe a 
dozen people for the planet overall, are above-average.

Average software engineers (programmers in countries where you can't 
legally call a programmer an engineer, or in the US where it's usually a 
misnomer) can't reliably code chunked transfer encoding either, yet 
significant web browsers have that feature.

If 10% of all programmers are competent to work on problems like this 
(that might be generous, it might be 3 or 5 percent) then you've still 
got hundreds of thousands of people that can do it.

AHS
0
Arved
4/1/2010 9:24:31 AM
On Thu, 1 Apr 2010, Kevin McMurtrie wrote:

> In article <alpine.DEB.1.10.1003312344430.13579@urchin.earth.li>,
> Tom Anderson <twic@urchin.earth.li> wrote:
>
>> On Wed, 31 Mar 2010, Roedy Green wrote:
>>
>>> Everyone has seen that sending one big file works much more efficiently
>>> than many small files.  The effect quite astounding, many orders of
>>> magnitude. It just occurred to me that I don't think I can account for
>>> that huge difference.  Where is all the time going?
>>
>> TCP handshake, TCP slow start (look that one up if you don't know it),
>> roundtrips for control packets at the start of the connection. Losing a
>> bit of time can have a huge impact on throughput - it's all about the
>> bandwidth-delay product, which on today's long, fat networks is huge.
>>
>>> It then occurred to me, that any sort of technique to reduce the
>>> difference could have a huge effect on the Internet as a whole.
>>
>> Yes. It's called pipelining, and it's been in HTTP since 1999.
>>
>> Although it's not that widely used by browsers, because of worries about
>> compatibility with servers, which seems a bit of a waste.
>
> Browsers don't support pipelining because the multiplexer/demultiplexer
> is too complicated for the average software engineer.

Opera - uses pipelining
Firefox - supports pipelining, not turned on by default
Konqueror - supports pipelining, not turned on by default
IE - does not support pipelining
Safari - does not support pipelining

I guess Opera, Mozilla and KDE all have above-average engineers.

> Last I heard, Microsoft and Apache clients can't pipeline;

No surprise about MS. What's the Apache client? I'm not aware of a browser 
made by Apache - do you mean some non-browser client? HttpClient, perhaps?

tom

-- 
The question of whether computers can think is just like the question
of whether submarines can swim. -- Edsger W. Dijkstra
0
Tom
4/1/2010 3:02:05 PM
Reply:

Similar Artilces:

java.lang.ExceptionInInitializerError: java.lang.ArrayIndexOutOfBoundsException
Hi, I am new to DB2. I am getting this error while loading the DB2Driver. I don't have any idea about where i might have gone wrong. please help me. Below is the stack trace. Stack Trace: java.lang.ExceptionInInitializerError: java.lang.ArrayIndexOutOfBoundsException at COM.ibm.db2.jdbc.app.DB2Driver.SQLAllocEnv(Native Method) at COM.ibm.db2.jdbc.app.DB2Driver.<init>(DB2Driver.java:245) at COM.ibm.db2.jdbc.app.DB2Driver.<clinit>(DB2Driver.java:130) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Cla...

java.lang.StringIndexOutOfBoundsException: String index out of range: 23 at java.lang.String.charAt(String.java:460)
Hi All I am having the torque3.1.jar and postgresql-7.4. I have compiled the new jdbc driver called as postgresql.jar and have placed it in the lib directory from where the ant scripts catch the jars. Whenever i try to access through torque gestList = BaseGestlistPeer.doSelect(new Criteria()); this error arises java.lang.StringIndexOutOfBoundsException: String index out of range: 23 at java.lang.String.charAt(String.java:460) at org.postgresql.jdbc2.ResultSet.toTimestamp(ResultSet.java:1653) at org.postgresql.jdbc2.ResultSet.getTimestamp(ResultSet.java...

java.lang.Set with elements of type java.lang.Set
Roughly I do something along the lines of: Set set = new HashSet(); Set elem = new HashSet(); set.add(elem); // now we change the elem and add it again to the set elem.add(some object here); set.add(elem); I found out the hard way that 'set' may now contain 'elem' either once or twice, the reason being that 'elem.add()' changes the hashCode of elem such that it is not noticed that it is in 'set' already on the 2nd 'set.add()'. Question: What I would actually want is an IdentityHashSet() set = new IdentityHashSet() but this does not...

to use import java.lang.* or import java.lang.Math or none at all?
Hi guys, i knew that by default all java.lang classes will be imported by the compiler during compilation. but, to make it easier for the computer, should i specify which class i really will be using? does this action will boost the performance during compilation and runtime or not a matter at all? the answer to this post will definitely affect my programming style in the future when i'm considering "to import or not to import"... hmm,,, thanks in advance. JPractitioner wrote: > i knew that by default all java.lang classes will be imported by the > compiler during com...

Error occurred during intialization of VM java/lang/NoClassDefFoundError: java/lang/Object
Good day to all, I have installed the j2se/netbeans binary bundle on red hat 9. I can run everything perfectly as root but when I try to compile with any other user I get: Error occurred during intialization of VM java/lang/NoClassDefFoundError: java/lang/Object When I saw this it seemed like a permissions problem but I checked the permissions and everything seemed fine. All users have execute permissions of javac and java. I have read other threads dealing with the same or similar problem but have not reached any solution yet. I would appreciate if anyone that has run into this type o...

Error occurred during initialization of VM java/lang/NoClassDefFoundError: java/lang/Object
I downloaded jdk-6u7-solaris-sparcv9.tar.Z and installed it by these commands: # zcat jdk-6u7-solaris-sparc.tar.Z | tar -xf - # pkgadd -d . SUNWj6rtx SUNWj6dvx SUNWj6dmx # /usr/jdk/instances/jdk1.6.0/bin/sparcv9/java -version Error occurred during initialization of VM java/lang/NoClassDefFoundError: java/lang/Object # ls /usr/jdk/instances/ jdk1.5.0 jdk1.6.0 # uname -a SunOS sun1 5.10 Generic sun4u sparc SUNW,Sun-Blade-2500 Please help to fix the error. Thanks. TsanChung wrote: > I downloaded jdk-6u7-solaris-sparcv9.tar.Z and installed it by these > commands: > # zcat jdk-6u7-so...

2008 computers.lang for java.programmer
jwehnf iuwhhttp://www.freewebs.com/thuiss/http://indianfriendfinder.com/go/g906725-pmem ashik wrote:Sub: 2008 computers.lang for java.programmerYou have nothing of value to anybody here. Pleasestop wasting our bandwidth.Further, please don't become the latest in a long line of perennial spamming f*ckw*t morons who infestthis usenet newsgroup.-- Andrew Thompsonhttp://www.physci.org/Message posted via JavaKB.comhttp://www.javakb.com/Uwe/Forums.aspx/java-general/200711/1 Andrew Thompson wrote:> ashik wrote:> > Sub: 2008 computers.lang for java.programmer> > You have nothing of...

java.lang.NoClassDefFoundError: java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory
Hi, I'm trying to use the httpclient within Jython (see http://jakarta.apache.org/commons/httpclient/ for more information on the httpclient). My Jython version is: Jython 2.1 on java1.4.2_04 (JIT: null) My Java version is: java version "1.4.2_04" Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_04-b05) Java HotSpot(TM) Client VM (build 1.4.2_04-b05, mixed mode) My CLASSPATH includes: jdom.jar xerces.jar jython.jar jt400.jar log4j-1.2.8.jar commons-httpclient-2.0.jar When I just try to perform the import statements from example code I get the error pasted below....

CheckBox in Column of JTable: Exception: java.lang.String cannot be cast to java.lang.Boolean
Hello, I have discovered a hidden error. My project was working for awhile, but then I started to get the below error. My error comes from the fact that I'm using a checkbox in a jtable, and I'm using the below "getColumnClass". Thank you, compile: run: Exception in thread "AWT-EventQueue-0" java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Boolean at javax.swing.JTable$BooleanRenderer.getTableCellRendererComponent(JTable.java:5412) at javax.swing.JTable.prepareRenderer(JTable.java:5735) at javax.swing.plaf.basic.BasicTableU...

Programmers, Programmers, Programmers, ...
As Steve Balmer correctly stated, while making his monkey dance, it is applications and hence programmers that make a platform. The fact though is that if you want to do professional programming, then Linux is the platform for you. I know that this statement will get the heckels up on a lot of trolls in C.O.L.A, but I have a recent experience that proves this. I am currently working for a Windows only house producing a system that receives and transmits around 1000 telegrams per second in each direction on a UDP socket, translates them into a different format and creates a log entry for each ...

Error occurred during initialization of VM java/lang/NoClassDefFoundError: java/lang/Object (ant/tomcat/spring)
Hi, I've been trying to get Spring working with ant and tomcat. Ant was building just fine, but I came in today and tried to build it and got this: Error occurred during initialization of VM java/lang/NoClassDefFoundError: java/lang/Object ?!?! Makes no sense to me. There _is_ an older version of java installed on my machine; but JAVA_HOME and ANT_HOME are set to the correct paths, and <which java>and <java -version> produce the correct version. Any help would greatly alleviate my frustration! Thanks in advance... Courtney ...

Error occurred during initialization of VM java/lang/NoClassDefFoundError: java/lang/Object (ant/tomcat/spring)
Hi, I've been trying to get Spring working with ant and tomcat. Ant was building just fine, but I came in today and tried to build it and got this: Error occurred during initialization of VM java/lang/NoClassDefFoundError: java/lang/Object ?!?! Makes no sense to me. There _is_ an older version of java installed on my machine; but JAVA_HOME and ANT_HOME are set to the correct paths, and <which java>and <java -version> produce the correct version. Any help would greatly alleviate my frustration! Thanks in advance... Courtney ...

A good article on java.lang.String best practices
Here is good article on Strings in Java, of you are a Java developer then this article could be helpful to write good and optimized code. http://www.groupkt.com/post/5bcd8475/javalangstring-tips-and-tricks:-best-java- coding-practices.htm On 7/28/2014 11:50 PM, im.manoj.tripathi@gmail.com wrote: > Here is good article on Strings in Java, of you are a Java developer then this article could be helpful to write good and optimized code. > > http://www.groupkt.com/post/5bcd8475/javalangstring-tips-and-tricks:-best-java- > coding-practices.htm Did you write the article ...

Maybe C is the perfect language for really good systems programmers, but unfortunately not-so-good systems and applications programmers are using it and they shouldn�t be.
I found the quote from "Coders at Work" where Bernie Cossel is interviewed, page 560. I don?t want to say that C has outlived its usefulness, but I think it was used by too many good programmers so that now not-good-enough programmers are using it to build applications and the bottom line is they?re not good enough and they can?t. Maybe C is the perfect language for really good systems programmers, but unfortunately not-so-good systems and applications programmers are using it and they shouldn?t be. -- Regards, Casey On Nov 3, 9:09=A0pm, Casey Hawthorne <caseyhHAMMER_T...@ista...

Web resources about - bundling efficiency, too good to be true? - comp.lang.java.programmer

Thermal efficiency - Wikipedia, the free encyclopedia
In thermodynamics , the thermal efficiency ( ) is a dimensionless performance measure of a device that uses thermal energy , such as an internal ...

ACT Greens support reforms to 'clunky' energy efficiency ratings system
Greens MLA Shane Rattenbury has backed calls from Canberra architects for a bolstered energy efficiency ratings system in the ACT.

Example of Visualizing Process Cycle Efficiency with LEGO
... and culture with the Real Agility Program In-depth article here: Using Lego[sic] to capture raw data for Cycle Time and Process Cycle Efficiency ...

Balancing Fairness and Efficiency
The trick is managing the trade-offs. One common criticism of economics is that it focuses too much on efficiency, and not enough on things like ...

Panasonic Launching Larger High-Efficiency Solar PV HIT Modules
Originally published on Sustainnovate. By Henry Lindon Larger high-efficiency solar photovoltaic HIT module offerings from Panasonic — including ...

IDG Contributor Network: IT efficiency is a moving target; here's how to hit the bull's-eye
Higher education is experiencing many changes in both academic and operational processes. New learning and course delivery models are aimed to ...

Inventec Appliances hikes efficiency, cuts labor via automation
Smartphone and smart hand-held device ODM Inventec Appliances has increased overall production efficiency by over 20% and reduced the workforce ...

Electronic Health Records Foster Increased Efficiency
Although many challenges still exist, measuring the impact of the EHR on treatment and clinical areas is happening among health care innovators. ...

CSX: Lower Fuel Expenses And Efficiency Improvements To Partially Offset Impact Of Lower Shipments
... CSX reporting a 32.5% year-over-year decline in coal carloads in Q4. However, a decline in fuel expenses and improvements in operating efficiency ...

Tax efficiency alone is not enough
Boards must not let possible savings override industrial logic

Resources last updated: 2/2/2016 9:15:26 PM