COMPGROUPS.NET | Search | Post Question | Groups | Stream | About | Register

### Zero padding and the Shannon information content

• Email
• Follow

Hello all,

I am confused about how zero-filling / windowing a signal alters its
information content and I'm hoping someone here can help.

I have a continuous signal (time domain) and I Fourier transform it to
get a band-limited spectrum (frequency domain).  From Shannon &
Weaver, I can compute the information content of the time domain
signal from the signal variance and, given the linearity of a DFT, I
expect the information content of the frequency domain signal is the
same.

The DFT code that I have has the option of zero-filling/zero-padding
the original signal as a means of interpolating in the frequency
domain.  This doesn't change the location/frequency of the peaks in
the spectrum but does make it easier to accurately locate the maximum
of each peak.  So, does zero-filling/zero-padding change the
information content of the signal?

Many thanks,
Dave


 0

See related articles to this posting

"Dave" <Confused.Scientist@gmail.com> writes:

> Hello all,
>
> I am confused about how zero-filling / windowing a signal alters its
> information content and I'm hoping someone here can help.
>
> I have a continuous signal (time domain) and I Fourier transform it to
> get a band-limited spectrum (frequency domain).  From Shannon &
> Weaver, I can compute the information content of the time domain
> signal from the signal variance and, given the linearity of a DFT, I
> expect the information content of the frequency domain signal is the
> same.
>
> The DFT code that I have has the option of zero-filling/zero-padding
> the original signal as a means of interpolating in the frequency
> domain.  This doesn't change the location/frequency of the peaks in
> the spectrum but does make it easier to accurately locate the maximum
> of each peak.  So, does zero-filling/zero-padding change the
> information content of the signal?

Intuitively, no. How to show analytically? Perhaps you could use the
Data Processing theorem?
--
%  Randy Yates                  % "Maybe one day I'll feel her cold embrace,
%% Fuquay-Varina, NC            %                    and kiss her interface,
%%% 919-577-9882                %            til then, I'll leave her alone."
%%%% <yates@ieee.org>           %        'Yours Truly, 2095', *Time*, ELO

 0
Reply yates (3949) 3/16/2007 4:51:27 AM

> > So, does zero-filling/zero-padding change the
> > information content of the signal?
>
> Intuitively, no. How to show analytically? Perhaps you could use the
> Data Processing theorem?

Indeed! intuitively, I expect the information content after zero
padding to be the same, but I'm at a total loss as to how to prove
it.  To me, the data processing theorem suggests the opposite as the
act of interpolating has altered the signal and hence, destroyed the
original information.  I suspect that in the early days of information
theory someone worked out an analytic proof showing that tapering/
windowing and interpolation does not change the information content of
a signal.


 0

On Mar 16, 10:00 am, "Dave" <Confused.Scient...@gmail.com> wrote:
> > > So, does zero-filling/zero-padding change the
> > > information content of the signal?
>
> > Intuitively, no. How to show analytically? Perhaps you could use the
> > Data Processing theorem?
>
> Indeed! intuitively, I expect the information content after zero
> padding to be the same, but I'm at a total loss as to how to prove
> it.  To me, the data processing theorem suggests the opposite as the
> act of interpolating has altered the signal and hence, destroyed the
> original information.  I suspect that in the early days of information
> theory someone worked out an analytic proof showing that tapering/
> windowing and interpolation does not change the information content of
> a signal.

I should add that I'm only interested in linear systems and the signal
noise is a Gaussian process model.  The FFT is a linear transformation
of the original signal and by virtue of its inverse (i.e. the IFFT
transform) it does not add/destroy information.  This suggests that
any linear and invertible transformation does not change the
information content of a signal.

Can anyone recommend a good book on signal processing that considers
the information content of the input/output of each process?


 0

Dave wrote:
>>> So, does zero-filling/zero-padding change the
>>> information content of the signal?
>> Intuitively, no. How to show analytically? Perhaps you could use the
>> Data Processing theorem?
>
> Indeed! intuitively, I expect the information content after zero
> padding to be the same, but I'm at a total loss as to how to prove
> it.  To me, the data processing theorem suggests the opposite as the
> act of interpolating has altered the signal and hence, destroyed the
> original information.  I suspect that in the early days of information
> theory someone worked out an analytic proof showing that tapering/
> windowing and interpolation does not change the information content of
> a signal.

no new information, but if you can tell the added zeros from original
data, it removes or subtracts nothing either.

What would you consider an analytic proof? Do encrypted and clear
versions of the same message carry the same information?

Jerry
--
Engineering is the art of making what you want from things you can get.
¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯

 0
Reply jya (12872) 3/16/2007 5:23:46 PM

Dave wrote:
> On Mar 16, 10:00 am, "Dave" <Confused.Scient...@gmail.com> wrote:
>>>> So, does zero-filling/zero-padding change the
>>>> information content of the signal?
>>> Intuitively, no. How to show analytically? Perhaps you could use the
>>> Data Processing theorem?
>> Indeed! intuitively, I expect the information content after zero
>> padding to be the same, but I'm at a total loss as to how to prove
>> it.  To me, the data processing theorem suggests the opposite as the
>> act of interpolating has altered the signal and hence, destroyed the
>> original information.  I suspect that in the early days of information
>> theory someone worked out an analytic proof showing that tapering/
>> windowing and interpolation does not change the information content of
>> a signal.
>
> I should add that I'm only interested in linear systems and the signal
> noise is a Gaussian process model.  The FFT is a linear transformation
> of the original signal and by virtue of its inverse (i.e. the IFFT
> transform) it does not add/destroy information.  This suggests that
> any linear and invertible transformation does not change the
> information content of a signal.
>
> Can anyone recommend a good book on signal processing that considers
> the information content of the input/output of each process?

Any entry-level book on signal processing will discuss the properties of
direct and inverse Fourier transforms in enough depth to resolve the
issues you raise. Like many here, I like Lyons: "Understanding Digital
Signal Processing". Smith's "The Scientist and Engineer's Guide to
Digital Signal Processing" is on line at http://www.dspguide.com/. There
may be material for you in Chapter 8 http://www.dspguide.com/ch8.htm .

Zero padding has profound implications processing, aside from the
obvious interpolation that makes curves easier to read. DFTs come
conceptually from periodic signals, and the math behaves as if it is
dealing with complete period(s). Artifacts occur when that's not the
case; windowing helps to suppress those at some cost. The inherent
periodicity of the math can cause an overlap of the beginning and end
data, confusing the result often to the point of uselessness. Zero
padding can render the overlap harmless; think of it as a guard band for
the real data. The sequence FFT - Multiply by a mask - IFFT achieves
convolution, often more quickly than classical convolution. Usually,
linear convolution is wanted. Without a sufficient zero-padded guard
band, circular convolution results. http://www.dspguide.com/ch7.htm

Jerry
--
Engineering is the art of making what you want from things you can get.
¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯

 0
Reply jya (12872) 3/16/2007 5:46:18 PM

On Mar 16, 1:23 pm, Jerry Avins <j...@ieee.org> wrote:
> Dave wrote:
> >>> So, does zero-filling/zero-padding change the
> >>> information content of the signal?
> >> Intuitively, no. How to show analytically? Perhaps you could use the
> >> Data Processing theorem?
>
> > Indeed! intuitively, I expect the information content after zero
> > padding to be the same, but I'm at a total loss as to how to prove
> > it.  To me, the data processing theorem suggests the opposite as the
> > act of interpolating has altered the signal and hence, destroyed the
> > original information.  I suspect that in the early days of information
> > theory someone worked out an analytic proof showing that tapering/
> > windowing and interpolation does not change the information content of
> > a signal.
>
> no new information, but if you can tell the added zeros from original
> data, it removes or subtracts nothing either.

Sigh, I should have re-read that before posting it.  What I intended
to write is that interpolation should not destroy any information but
it may (superficially at least) appear to increase it.  I was thinking
about the data processing theorem which states that the system's
output can not contain more information than present in its input.  A
Fourier transform recasts the input onto a different basis but it
doesn't alter the information content.  After a little more thought, I
think that zero-filling/interpolating/windowing are all methods that
result in signal enhancement.  In the case of zero-padding, the
increase in information content is observed as an improved
representation of each spectra line.

> What would you consider an analytic proof?

Ideally, I'd like to show is that the information content of a signal,
H(s), is equal to or less than the information content of a zero-
padded signal, H(s'), or a windowed signal, H(s'').  In practice, I
think it may be easier to show this with a matlab simulation.

Thanks,
Dave


 0

On Mar 16, 11:26 am, "Dave" <Confused.Scient...@gmail.com> wrote:
> On Mar 16, 1:23 pm, Jerry Avins <j...@ieee.org> wrote:...
....
>   I was thinking
> about the data processing theorem which states that the system's
> output can not contain more information than present in its input.  A
> Fourier transform recasts the input onto a different basis but it
> doesn't alter the information content.  After a little more thought, I
> think that zero-filling/interpolating/windowing are all methods that
> result in signal enhancement.  In the case of zero-padding, the
> increase in information content is observed as an improved
> representation of each spectra line.
>
> Dave

A zero-padded (Discrete) Fourier transform casts the input onto a
larger set of different bases but it doesn't alter the information
content. The new bases are not independent and do not contain any new
information. The larger set of bases allows a more accurate estimation
of the peak by a peak picking process. The 'improved representation'
is only a matter finding a better estimate of a different basis:
frequency peaks with amplitudes, than the set that the non-extended
DFT provides by peak picking. A variety of interpolation methods allow
the same 'improved representation' to be determined.

I use quotes around 'improved representation' because it is 'improved'
for a specific purpose that is not an information content criteria but
a parameter estimation criteria.

Dale B. Dalrymple
http://dbdimages.com


 0
Reply dbd1 (1036) 3/16/2007 6:53:42 PM

On Mar 15, 11:06 pm, "Dave" <Confused.Scient...@gmail.com> wrote:
[snip]
> of each peak.  So, does zero-filling/zero-padding change the
> information content of the signal?
>
> Many thanks,
> Dave

No.  I think you're confusing "information content" and
"convenience of representation."

It's like when you receive a noisy version of a PAM signal.
Quantizing it makes it look nicer and many people then say
bits.

Hope that helps a bit,
Julius


 0
Reply juliusk (652) 3/16/2007 7:27:17 PM

On Mar 16, 2:26 pm, "Dave" <Confused.Scient...@gmail.com> wrote:
> On Mar 16, 1:23 pm, Jerry Avins <j...@ieee.org> wrote:
>
>
>
> > Dave wrote:
> > >>> So, does zero-filling/zero-padding change the
> > >>> information content of the signal?
> > >> Intuitively, no. How to show analytically? Perhaps you could use the
> > >> Data Processing theorem?
>
> > > Indeed! intuitively, I expect the information content after zero
> > > padding to be the same, but I'm at a total loss as to how to prove
> > > it.  To me, the data processing theorem suggests the opposite as the
> > > act of interpolating has altered the signal and hence, destroyed the
> > > original information.  I suspect that in the early days of information
> > > theory someone worked out an analytic proof showing that tapering/
> > > windowing and interpolation does not change the information content of
> > > a signal.
>
> > no new information, but if you can tell the added zeros from original
> > data, it removes or subtracts nothing either.
>
> Sigh, I should have re-read that before posting it.  What I intended
> to write is that interpolation should not destroy any information but
> it may (superficially at least) appear to increase it.  I was thinking
> about the data processing theorem which states that the system's
> output can not contain more information than present in its input.  A
> Fourier transform recasts the input onto a different basis but it
> doesn't alter the information content.  After a little more thought, I
> think that zero-filling/interpolating/windowing are all methods that
> result in signal enhancement.  In the case of zero-padding, the
> increase in information content is observed as an improved
> representation of each spectra line.
>
> > What would you consider an analytic proof?
>
> Ideally, I'd like to show is that the information content of a signal,
> H(s), is equal to or less than the information content of a zero-
> padded signal, H(s'), or a windowed signal, H(s'').  In practice, I
> think it may be easier to show this with a matlab simulation.
>
> Thanks,
> Dave

Your intuition seems to be at odds with what you said previously. You
had (correctly, I believe) pointed out that since the FFT has an
inverse that can be used to exactly reconstruct the FFT's input, the
FFT operation doesn't add or destroy any information content. The same
exact inverse operations that yield the original signal, so they can't
possibly change the information content.

Jason


 0
Reply cincydsp (353) 3/16/2007 7:52:44 PM

On Mar 15, 8:06 pm, "Dave" <Confused.Scient...@gmail.com> wrote:
> Hello all,
>
> I am confused about how zero-filling / windowing a signal alters its
> information content and I'm hoping someone here can help.
>
> I have a continuous signal (time domain) and I Fourier transform it to
> get a band-limited spectrum (frequency domain).  From Shannon &
> Weaver, I can compute the information content of the time domain
> signal from the signal variance and, given the linearity of a DFT, I
> expect the information content of the frequency domain signal is the
> same.
>
> The DFT code that I have has the option of zero-filling/zero-padding
> the original signal as a means of interpolating in the frequency
> domain.  This doesn't change the location/frequency of the peaks in
> the spectrum but does make it easier to accurately locate the maximum
> of each peak.  So, does zero-filling/zero-padding change the
> information content of the signal?

domain is identical to sinc interpolation in the time domain.  The
result is just a bunch of weighted averages of the existing
information
content in your original signal vector (minus rounding/quantization
noise).

Windowing might destroy information, especially if any portion
of the window is zero, or small relative to the quantization level.

IMHO. YMMV.
--
rhn A.T nicholson d.0.t C-o-M


 0
Reply rhnlogic (1111) 3/16/2007 9:39:10 PM

10 Replies
36 Views

Similar Articles

12/11/2013 6:55:31 PM
page loaded in 62445 ms. (0)

Similar Artilces:

FAQ 4.33 How do I pad a string with blanks or pad a number with zeroes? #16
This is an excerpt from the latest version perlfaq4.pod, which comes with the standard Perl distribution. These postings aim to reduce the number of repeated questions as well as allow the community to review and update the answers. The latest version of the complete perlfaq is at http://faq.perl.org . -------------------------------------------------------------------- 4.33: How do I pad a string with blanks or pad a number with zeroes? In the following examples, $pad_len is the length to which you wish to pad the string,$text or $num contains the string to be padded, and$pa...

FAQ 4.33 How do I pad a string with blanks or pad a number with zeroes? #23
This is an excerpt from the latest version perlfaq4.pod, which comes with the standard Perl distribution. These postings aim to reduce the number of repeated questions as well as allow the community to review and update the answers. The latest version of the complete perlfaq is at http://faq.perl.org . -------------------------------------------------------------------- 4.33: How do I pad a string with blanks or pad a number with zeroes? In the following examples, $pad_len is the length to which you wish to pad the string,$text or $num contains the string to be padded, and$pa...

FAQ 4.33 How do I pad a string with blanks or pad a number with zeroes? #7 554663
This is an excerpt from the latest version perlfaq4.pod, which comes with the standard Perl distribution. These postings aim to reduce the number of repeated questions as well as allow the community to review and update the answers. The latest version of the complete perlfaq is at http://faq.perl.org . -------------------------------------------------------------------- 4.33: How do I pad a string with blanks or pad a number with zeroes? In the following examples, $pad_len is the length to which you wish to pad the string,$text or $num contains the string to be padded, and$pa...

FAQ 4.33 How do I pad a string with blanks or pad a number with zeroes? #2 324899
This message is one of several periodic postings to comp.lang.perl.misc intended to make it easier for perl programmers to find answers to common questions. The core of this message represents an excerpt from the documentation provided with Perl. -------------------------------------------------------------------- 4.33: How do I pad a string with blanks or pad a number with zeroes? In the following examples, $pad_len is the length to which you wish to pad the string,$text or $num contains the string to be padded, and$pad_char contains the padding character. You can use a singl...

I have 5 elements so my size is 5x1 ex. 1 2 3 4 5 Now I wanted to padded with 5 zeros in the beginning first. so it would like this: 0 0 0 0 0 1 2 3 4 5. My size is now 10x1. How would I do this without reshape because it would make my matrix to be 1x10 which is not what I want to do. On 11-01-07 01:23 PM, John Stane wrote: > I have 5 elements so my size is 5x1 > ex. > 1 > 2 > 3 > 4 > 5 > > Now I wanted to padded with 5 zeros in the beginning first. so it would like > this: > 0 > 0 > 0 > 0 > 0 > 1 >...

Dear All, &nbsp; I would like to pad an image with different amount of zeros depend on the size of the image. The size of the image is determined by the user choice so it has to be a variable. &nbsp; I can do it using the 'Insert into array' function but not with a variable size of zeros. Does anyone knows how to do it? &nbsp; &nbsp; Cheers in advance, &nbsp; &nbsp; Eldad ...

Hi all, i want to design a zero padding circuit. The problem is the number of zero depends on the number of input data. Let's say i have 2000 data input, the number of output data should be divided by factor of 600. So number of zero padding = 400 --> (2000+400)/600 = integer value Remember that number of data input could vary, so it will change the number of zero padding Can you give me suggestion, how to implement zero padding circuit in hardware? *I plan to design the circuit 10 port parallely, so for above example, it will need 200 clock to input all data. ...

Does isomorphism preserv information content?
Does isomorphism preserv information content of a "structure" ... If so, does anyone have any idea how to prove that? I'm almost sure this statement is true, and I think the proof should be somehow similar with Solomonoff Invariance Theorem, but I do not have any idea how to start it. ...

Hello, I try to make a cross correlation when using the FFT to speed up (in contrast to the normal O(n^2) method). First I did not zero-pad because changes from the pictures are small. It worked. Then I added zero-padding around my pictures (they are about 600x400 pixels so now with zero-padding they are about 1200x800 pixels). Then all the results are wrong because the images from the digital camera I take have a high DC offset (around value of 1000). So the padded areas are totally dark and when there is the picture then the values becomes suddenly around 1000-4000. So I get very wrong resu...

Hello all, Sorry for starting this topic all over again. I had seen some previous discussions on zero padding and when I started reading them I completely got lost. When I am provided with samples which are not of power of 2 and I need to implement FFT, I zero pad the samples to the next highest power of 2. I do FFT using Cooley Tukey algo and observe that the frequency componets in several bin - ie there is spectral leakage even though the signal given to me had a single tone. I did think of using some of the algos given to interpolate and find the exact signal frequency. But then this ...

Information content of natural language sentences
Hi, In the following the term 'information content' (IC) of a sentence is mainly to be understood in the conventional, everyday-life, sense and not in the sense employed strictly in the scientific discipline of information theory (cf. Kolmogorov complexity, Shannon's entropy, etc.). It refers to the amount of 'new' knowlegde/information which a person acquires, when he reads or hears a sentence in a natural language. [Anyone who is inspired to lead deep and lengthy discussions on topics of theoretical computer science is sincerely requested to initiate an indepen...

Doubles and zero/negative zero
How does one determine whether a double is equal to -0.0000...? Will fabs() from math.h return 0.000... on -0.0000...? -- Christopher Benson-Manica | I *should* know what I'm talking about - if I ataru(at)cyberspace.org | don't, I need to know. Flames welcome. "Christopher Benson-Manica" <ataru@nospam.cyberspace.org> wrote in message news:cbv0n3$k0e$1@chessie.cirr.com... > How does one determine whether a double is equal to -0.0000...? You have to use a C99 function such as signbit or copysign to see the sign bit, because -0 == +0. > ...

<div style="width: 100px; padding-left: 10px; border: 1px solid Black;">padding</div><br> <div style="width: 100px; border: 1px black solid;">no padding</div> firefox increases the width of the div to 110px. ie includes the padding in the 100. is there an easier work-around than reducing the width by 10px if firefox? Well, 1st off, whenever you start any CSS block, must have as 1st line: * {margin: 0;padding: 0} to avoid user-agent defaults, which is I think the case here, from there on, override, what you think is padding, is pr...

Information... :)
CALGARY (CP) -- Researchers at the University of Calgary have found that nerve cells grown on a microchip can learn and memorize information which can be communicated to the brain. http://cnews.canoe.ca/CNEWS/TechNews/2004/02/19/353584-cp.html ...

CJK contents disappears, ASCII content stays
Dear all, I ran into a riddle concerning the CJK package. I have data containing inline commands, these inline commands then expand to what I thought is ordinary text. If this 'ordinary' text is plain ASCII stuff, everything works as expected, if the text contains CJK material, the material disappears. The following sample file contains a line \zeile with some data, some of which is text, some of which is the command \dian. When digesting \zeile with \digest, the \dian stuff expands as long as it is ASCII (uncomment first definition of \dian to see), but CJK contents vanishes while...