bundling efficiency, too good to be true?

  • Permalink
  • submit to reddit
  • Email
  • Follow


Everyone has seen that sending one big file works much more
efficiently than many small files.  The effect quite astounding, many
orders of magnitude. It just occurred to me that I don't think I can
account for that huge difference.  Where is all the time going?

It then occurred to me, that any sort of technique to reduce the
difference could have a huge effect on the Internet as a whole.
-- 
Roedy Green Canadian Mind Products
http://mindprod.com

If you tell a computer the same fact in more than one place, unless you have an automated mechanism to ensure           they stay in sync, the versions of the fact will eventually get out of sync.
0
Reply Roedy 3/31/2010 9:23:04 PM

See related articles to this posting


On 31-03-2010 17:23, Roedy Green wrote:
> Everyone has seen that sending one big file works much more
> efficiently than many small files.  The effect quite astounding, many
> orders of magnitude. It just occurred to me that I don't think I can
> account for that huge difference.  Where is all the time going?

Given the lack of context, then one can only guess:
- file open and file creation are rather expensive operations
   so many small files have huge overhead
- per file protocol overhead
- really small files can not be compressed as efficiently
   as larger files

Arne
0
Reply ISO 3/31/2010 10:38:15 PM

On Wed, 31 Mar 2010, Roedy Green wrote:

> Everyone has seen that sending one big file works much more efficiently 
> than many small files.  The effect quite astounding, many orders of 
> magnitude. It just occurred to me that I don't think I can account for 
> that huge difference.  Where is all the time going?

TCP handshake, TCP slow start (look that one up if you don't know it), 
roundtrips for control packets at the start of the connection. Losing a 
bit of time can have a huge impact on throughput - it's all about the 
bandwidth-delay product, which on today's long, fat networks is huge.

> It then occurred to me, that any sort of technique to reduce the 
> difference could have a huge effect on the Internet as a whole.

Yes. It's called pipelining, and it's been in HTTP since 1999.

Although it's not that widely used by browsers, because of worries about 
compatibility with servers, which seems a bit of a waste.

tom

-- 
I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth. -- Umberto Eco
0
Reply Tom 3/31/2010 10:51:27 PM

Tom Anderson wrote:
> On Wed, 31 Mar 2010, Roedy Green wrote:
> 
>> Everyone has seen that sending one big file works much more 
>> efficiently than many small files.  The effect quite astounding, many 
>> orders of magnitude. It just occurred to me that I don't think I can 
>> account for that huge difference.  Where is all the time going?
> 
> TCP handshake, TCP slow start (look that one up if you don't know it), 
> roundtrips for control packets at the start of the connection. Losing a 
> bit of time can have a huge impact on throughput - it's all about the 
> bandwidth-delay product, which on today's long, fat networks is huge.

"back in the day" some serial protocols had asynchronous
ACK and resend on packets.

    BugBear
0
Reply bugbear 4/1/2010 7:46:34 AM

In article <alpine.DEB.1.10.1003312344430.13579@urchin.earth.li>,
 Tom Anderson <twic@urchin.earth.li> wrote:

> On Wed, 31 Mar 2010, Roedy Green wrote:
> 
> > Everyone has seen that sending one big file works much more efficiently 
> > than many small files.  The effect quite astounding, many orders of 
> > magnitude. It just occurred to me that I don't think I can account for 
> > that huge difference.  Where is all the time going?
> 
> TCP handshake, TCP slow start (look that one up if you don't know it), 
> roundtrips for control packets at the start of the connection. Losing a 
> bit of time can have a huge impact on throughput - it's all about the 
> bandwidth-delay product, which on today's long, fat networks is huge.
> 
> > It then occurred to me, that any sort of technique to reduce the 
> > difference could have a huge effect on the Internet as a whole.
> 
> Yes. It's called pipelining, and it's been in HTTP since 1999.
> 
> Although it's not that widely used by browsers, because of worries about 
> compatibility with servers, which seems a bit of a waste.
> 
> tom

Browsers don't support pipelining because the multiplexer/demultiplexer 
is too complicated for the average software engineer.  Out-of-order 
response processing requires forcing preceding responses in the pipeline 
into memory.  That's tricky, but not too bad.  Now do that and rebuild 
the pipeline when the connection closes or drops.  Ugly!

At least in the old Innovation HTTPClient, that results in multiple lock 
grabs on components of linked list that's prone to failure.  The code is 
convoluted and it's looking like a total rewrite might be easier.

Last I heard, Microsoft and Apache clients can't pipeline; WebKit can 
but it's an experimental feature.
-- 
I won't see Google Groups replies because I must filter them as spam
0
Reply Kevin 4/1/2010 7:47:54 AM

Kevin McMurtrie wrote:
> In article <alpine.DEB.1.10.1003312344430.13579@urchin.earth.li>,
>  Tom Anderson <twic@urchin.earth.li> wrote:
> 
>> On Wed, 31 Mar 2010, Roedy Green wrote:
>>
>>> Everyone has seen that sending one big file works much more efficiently 
>>> than many small files.  The effect quite astounding, many orders of 
>>> magnitude. It just occurred to me that I don't think I can account for 
>>> that huge difference.  Where is all the time going?
>> TCP handshake, TCP slow start (look that one up if you don't know it), 
>> roundtrips for control packets at the start of the connection. Losing a 
>> bit of time can have a huge impact on throughput - it's all about the 
>> bandwidth-delay product, which on today's long, fat networks is huge.
>>
>>> It then occurred to me, that any sort of technique to reduce the 
>>> difference could have a huge effect on the Internet as a whole.
>> Yes. It's called pipelining, and it's been in HTTP since 1999.
>>
>> Although it's not that widely used by browsers, because of worries about 
>> compatibility with servers, which seems a bit of a waste.
>>
>> tom
> 
> Browsers don't support pipelining because the multiplexer/demultiplexer 
> is too complicated for the average software engineer.
[ SNIP ]
AHS

Simple solution - don't use average software engineers to work on this 
code. There are how many significant web browsers again? Surely it's not 
so tough to ensure that the 2 or 3 people per important browser, maybe a 
dozen people for the planet overall, are above-average.

Average software engineers (programmers in countries where you can't 
legally call a programmer an engineer, or in the US where it's usually a 
misnomer) can't reliably code chunked transfer encoding either, yet 
significant web browsers have that feature.

If 10% of all programmers are competent to work on problems like this 
(that might be generous, it might be 3 or 5 percent) then you've still 
got hundreds of thousands of people that can do it.

AHS
0
Reply Arved 4/1/2010 9:24:31 AM

On Thu, 1 Apr 2010, Kevin McMurtrie wrote:

> In article <alpine.DEB.1.10.1003312344430.13579@urchin.earth.li>,
> Tom Anderson <twic@urchin.earth.li> wrote:
>
>> On Wed, 31 Mar 2010, Roedy Green wrote:
>>
>>> Everyone has seen that sending one big file works much more efficiently
>>> than many small files.  The effect quite astounding, many orders of
>>> magnitude. It just occurred to me that I don't think I can account for
>>> that huge difference.  Where is all the time going?
>>
>> TCP handshake, TCP slow start (look that one up if you don't know it),
>> roundtrips for control packets at the start of the connection. Losing a
>> bit of time can have a huge impact on throughput - it's all about the
>> bandwidth-delay product, which on today's long, fat networks is huge.
>>
>>> It then occurred to me, that any sort of technique to reduce the
>>> difference could have a huge effect on the Internet as a whole.
>>
>> Yes. It's called pipelining, and it's been in HTTP since 1999.
>>
>> Although it's not that widely used by browsers, because of worries about
>> compatibility with servers, which seems a bit of a waste.
>
> Browsers don't support pipelining because the multiplexer/demultiplexer
> is too complicated for the average software engineer.

Opera - uses pipelining
Firefox - supports pipelining, not turned on by default
Konqueror - supports pipelining, not turned on by default
IE - does not support pipelining
Safari - does not support pipelining

I guess Opera, Mozilla and KDE all have above-average engineers.

> Last I heard, Microsoft and Apache clients can't pipeline;

No surprise about MS. What's the Apache client? I'm not aware of a browser 
made by Apache - do you mean some non-browser client? HttpClient, perhaps?

tom

-- 
The question of whether computers can think is just like the question
of whether submarines can swim. -- Edsger W. Dijkstra
0
Reply Tom 4/1/2010 3:02:05 PM
comp.lang.java.programmer 51809 articles. 38 followers. Post

6 Replies
209 Views

Similar Articles

[PageSpeed] 25


  • Permalink
  • submit to reddit
  • Email
  • Follow


Reply:

Similar Artilces:

Too good to be true?
I found this website today from an email a friend sent me: http://www.famousmaverickmoneymakers.com I watched the videos on the site and he makes it look so easy and simple. Anyone tried this? I'd love to know! ...

Ubuntu: Too Good to be True?
Quote: ----------------- A Curmudgeonly Review by a Very Irritable Webmaster By James R. Hood ConsumerAffairs.Com ....OK, enough preaching. Let's get down to practical matters. I decided a few months ago, when I set up a new Dell desktop machine, that Microsoft and I were about to part company. That happened when I was "invited" to "authenticate" my copy of Microsoft XP to be sure it was "genuine."... It always makes me sad to see all the old computers that are tossed in landfills and left to rot in attics and garages. Every one of them could probably be ru...

Is Linux too good to be true?
Is Linux too good to be true? Michael Pollitt Aug 25 2004 ... Windows belongs to Microsoft, lock, stock and security vulnerability. Apple computers (just over a million Google results) are proprietary too: a great choice for Windows refugees but not a radical one. For real change, there is only one answer - Linux. It's open source, generally free to use and share and is not controlled by a single company. Its penguin mascot is cute too. [...] Linux didn't like my new flat-panel screen and insisted on unusable low resolutions. Much wasted time and online answer seeking (back in Windo...

Is the xbox 360 too good to be true???
Nope... Especially when you can get one free.. Hurry!!!!! www.customsoftwarezone.com/free It's time for you to get your free iPod.. What are you waiting for. It's not a scam.. You hear about it EVERYWHERE.. It's not a trick.. I just got a 20GB iPod and sold it on ebay for $250 in CASH !!! I've also got a Sony Vaio Laptop and a Panasonic 42" PLASMA TV!!! (I'll be keeping them.. Thanks :-) Right now, I'm working on a new 60GB iPod, a 1GB iPod Shuffle, and OH, I almost forgot, a new X-Box 360 Premium Edition (the $399 version, not the cheapo one...) This stuff R...

Efficiency Issue
This is supposed to be a very general question. Consider the following 2 code samples: Code Sample 1: for (int i=0; i<n; ++i) { for (int j=0; j<n; ++j) { /* Do something here */ } } Code Sample 2: (Now we declare j in advance rather than each time inside the i loop as in code sample 1) int j; for (int i=0; i<n; ++i) { for (j=0; j<n; ++j) // Notice the differnce here, now { /* Do something here */ } } Is code sample 2 faster than code sample 1? Plz specify the reasons too. rahul.batra@gmail.com wrote: > This is supposed to be a v...

Raspberry Pi
I have just received the following from Farrell They have slipped the original delivery date they gave me by a month. <===================================================================> Further to notification from our supplier we write to advise that the expected delivery of items listed in bold has been amended. .... Description Qty Ordered Qty to Follow Est. Delivery SBC, RASPBERRY PI, MODEL B 1 1 W/C 14/05/2012 Please be assured that we are working closely with our suppliers to ensure that every effort is being made to satisfy your requir...

ALMOST SOUNDS TOO GOOD TO BE TRUE.........
Subject: Make Money Real Fast ( THIS IS NO SCAM IT IS FOR REAL) Message: GREAT WAY TO MAKE MONEY-READ THIS!!!!!! HOW TO TURN $6 INTO $6,000!!!!!! READING THIS COULD CHANGE YOUR LIFE! I found this on a bulletin board and decided to try it. A little while back, I was browsing through newsgroups, just like you are now, and came across an article similar to this that said you could make thousands of dollars within weeks with only an initial investment of $6.00! So I thought, Yeah right, this must be a scam,but like most of us, I was curious, so I kept reading. Anyway, it said that you send $1.00 t...

Did I write a good (efficient) program?
I haven't written COBOL in a number of years and am quite rusty. Recently someone asked my help to write some code to display an IP V6 address in human-readable code. I wrote something, but am not sure if I used the best available techniques. Please have a look at what I wrote and feel free to comment. Background. An IP V6 address is 4 words (128-bit) long. It is in binary format. For human-readable purposes it is converted into 8 "octets" separated by colons. leading zeroes in each field are suppressed. To compact the address, one or more adjacent octets containing all zeroes c...

Sequel to ICO
Ico is not just a game, it's an amazing work of art and probably my favorite game of all time (bought a playstation2 just so I could play it). Here's hoping.. http://www.1up.com/do/newsStory?cId=3134530 "MrFlash" <not myre@laddress.com> wrote in message news:5527d.49554$Ot3.39870@twister.nyc.rr.com... > Ico is not just a game, it's an amazing work of art and probably my > favorite > game of all time (bought a playstation2 just so I could play it). > > Here's hoping.. > > http://www.1up.com/do/newsStory?cId=3134530 ...

Battlefield Bad Company: Too good to be true?
When BF2 Came out people were expecting destructible envirements which we got in a very limited capacity. I Was hoping things like vehicles bridges would stay destroyed and not dissapear . All these things are promised with the new frostbite engine for BAD COMPANY. If this turns out to be true it will change all the ways we look and the BF Franchise as it will add to realism like nothing any game has seen as of yet...except for maybe mercinaries. It might also add to problems as too much realism can block bridges such as destroyed tanks etc... but we'll wait and see how this is dealt wit...

knapsack algorithm that looks to good to be true
I have an idea for solving the knapsack problem, but it looks too good to be true. I would like someone to explain potential problems with this approach. I'll give an example: I want to find a subset of {2,7,11} which has elements that sum to 13. Here is the algorithm for solving this: In binary notation, 2=0010, 7=0111, 11=1011, 13=1101. Suppose that 2x +7y+11z=13, where x,y,z are binary. Then y+z=1 (mod 2), since the last bits of 7, 11, and 13 are 1 and the last bit of 2 is 0. And (y+z-1)+2*(x+y+z)=0 (mod 4), since the second to last bits of 2, 7, and 11 are 1 and the second to last b...

Re: Did I write a good (efficient) program?
>>> On 3/31/2008 at 8:40 PM, in message <c463v3p7pdl8dc6dl5seo7j6u0mfrev3qr@4ax.com>, Robert<no@e.mail> wrote: > In 1973 everything (except files) was fixed length. We used fixed length > strings, numbers, > arrays, records and blocks in files. Now, outside Cobol, most things are > variable length. > The next Cobol standard proposes adding 'any length' data items and > arrays. The Old Guard > are opposed. Is anyone here a member of this Old Guard that is opposed to variable length strings? I certainly am not. I would love them. Frank F...

Review: Webaroo Service too Good to be True
By ANICK JESDANUN, AP Internet Writer 4 minutes ago It sounds too good to be true: the highlights of the Web squeezed into a data file small enough to fit on your laptop or mobile phone, letting you browse even when you don't have a live Internet connection. After giving Webaroo a whirl, I found that it is, indeed, too good to be true. Webaroo, downloadable as a free "beta" test from Webaroo Inc., promises to make it "simple for you to take the Web with you - and find what you are looking for anywhere, anytime." On your PC or phone, it stores the Web sites ...

Xilinx S3DSP + EDK Board, too good to be true?
Hi All, I was looking at the Xilinx website today and saw the new Spartan 3 DSP EDK board. It seems like a pretty good deal, as the board by itself is 295, the usb JTAG cable is 199 and the EDK, well, I couldn't work out how much that normally costs ($130?!?) , but still thats at least $100 off the price assuming the EDK is free, which is extremely tempting. Am I missing something. Or is this just a good initial offering to get me "hooked" Thanks Joel joel.pigdon@gmail.com wrote: > Hi All, > I was looking at the Xilinx website today and saw the new Spartan 3 > DSP E...

Multiple dispatch using templates
Greetings, I too have succumbed to the perhaps foolish urge to write a video game, and I have been struggling with the implementation of multiple dispatch. I read through "More Effective C++" by Scott Meyers, and though I am impressed with his implementation, I wanted to find a way to use multiple dispatch in a way that was less complicated. Forgive me if the result, below, has been posted before or written of before - I haven't read the "Purple Book" at all, so maybe it's all been done before. What I'd like to know is what I'm missing - this compiles on ...

almost too good to be true!
to get an ipod for free all you need to do is complet 3 simple steps! Complete 1 offer get 5 friends to complete 1 offer recieve your ipod! you can choose from any colur ipod mini, the 20gb photo ipod or the new ipod nano. please go to the site via this link http://ipods.freepay.com/?r=21425216 In article <1127403805.174175.200130@g43g2000cwa.googlegroups.com>, badmanmark@hotmail.co.uk wrote: > to get an ipod for free all you need to do is Get spam for the rest of your natural life! Be a pyramid scamming idiot! -- "No urban night is like the night [in NYC]...here is o...

HOMM 5 for 19.99 WAS too good to be true.
Just got this from Target: "Greetings from Target.com. We regret to inform you of an error that has resulted in the display of incorrect pricing of the video game "Heroes of Might & Magic 5" you have ordered. Regrettably, we are unable to offer this item for the incorrect price. The correct price is $49.99. We have canceled your order for this item. Please accept our sincerest apology for this error. If you would like to order this item at the correct price, please feel free to visit our Web site and reorder. Despite our best efforts, a small number of items on o...

Microsoft/Cisco PAID Internships: Too good to be true?
Microsoft/Cisco PAID Internships: Too good to be true? http://www.networkworld.com/community/node/30483 Sincerely, Brad Reese - Network World http://www.networkworld.com/topics/lan-wan.html ...

DVD disks at low price
Hi ! There's currenty a sale at my local staples. Among the items in promotion there's Verbatim DVD-R 8x 50 disk cylinder for $20 (cdn), more than half the regular price. In another store, I see the same Verbatim disks in 5 unit box for.. $1 (reg. price: $7.49) Is there something "technoligically related" hidden behind these specials ? (I mean : other than a huge marketing pitch) ? Thanks Michel S. <NoSpam@msn.com> wrote: > There's currenty a sale at my local staples. Among the items in promotion > there's Verbatim DVD-R...

Good figures w/ my compression technique: If true, I have an issue.
I am still working diligently with file compression and am recieving impres= sive results. In fact, I seem to beat 7zip in non-solid mode! :D Now, I = need to implement the decoder and debug the package. But before I do that,= I'm looking for information on DOS/Win9x device drivers. I have Open Watc= om 1.9 and Digital Mars C. Can these produce device drivers? I really wan= t to compress a Win98SE FAT32 hard drive--even if I'm the only one to benef= it from it. I could also use links to information on Win32 console app-bui= lding, creating an interface to folders and ar...

Not an advert: Too good to be true
Anybody seen/know about these - 16-Port Gigabit Switch - for only ~$300. http://www.cnet.com.tw/pgswitch.html# Sounds too good to be true. Any gotchas? Anybody have one? tkx bk ...

Ambrosia Software holiday bundles are damn good value IMO.
<http://www.ambrosiasw.com/promos/xmas09/> in case anyone has missed this. Ambrosia have always been my favourite shareware game company (Escape Velocity series is my favourite), but they also do some top notch utilities. Are there any other Mac software sales going for the holidays? Share them here please! Regards, Jamie Kahn Genet -- If you're not part of the solution, you're part of the precipitate. On 01/12/09 02:01, Jamie Kahn Genet wrote: > Are there any other Mac software sales going for the holidays? Share > them here please! You've already missed all the...

where can I find good samples for efficient computation of matrix multiplication?
Dear all, I am facing a problem about matrix multiplication using VHDL. The target is to compute ZZ=XX*A' (8x8 matrix multiplication, XX is the input). This is a 1-D IDCT. Can you tell me where can I find good samples for efficient computation of this kind of matrix multiplication, using minium resources? Thank you very much, -Walala ------------------------------------------------------------ -- A'= -- 91 91 91 91 91 91 91 91 -- 126 106 71 25 -25 -71 -106 -126 -- 118 49 -49 -118 -118 -49 49 118 -- 106 -25 -126 -71 ...

what'ch think of my fraction program, is it correct and good and efficient?
// CHO, JOHN #include<iostream> class fracpri{ int whole; int numer; int denom; public: // constructors: fracpri(); fracpri(int w, int n, int d); fracpri(float f); // float to class constructor // member functions to show output and get input void getFraction(); void showFraction(); // function and operators to do artimathetic fracpri addfracts(fracpri &obj1, fracpri &obj2); fracpri operator+(fracpri &obj2); fracpri operator-(fracpri &obj2); fracpri operator*(fracpri &obj2); fracpri operator/(fracpri &obj2); // adding with constants frien...