COMPGROUPS.NET | Search | Post Question | Groups | Stream | About | Register

### time series correlation / autocorrelation / xcorr

• Email
• Follow

```Hi, I need a pointer on how to achieve what the time series correlation (autocorrelation) GUI does in tstool,  but from within a custom script instead.

A look at the help file revealed:

"When computing the cross-correlation of two vector-valued time series x and y, Time Series Tools uses an algorithm that is functionally equivalent to calling the Signal Processing Toolbox xcorr function from with the 'biased' option, after the time series means have been removed"

I have used xcorr on my data and it outputs values greater than 1. I have no idea what these could mean. I  was excepting r-statistics for each Y(ii) vs X(ii) combination at each lag. Furthermore, when I compare zero-lag xcorr values to those from pairwise correlation they are all different. What could I be doing wrong?
```
 0
Reply Matthias 9/20/2010 10:48:03 AM

See related articles to this posting

```"Matthias " <matthew.goddard@bcf.uni-freiburg.de> wrote in message <i77e53\$buk\$1@fred.mathworks.com>...
> Hi, I need a pointer on how to achieve what the time series correlation (autocorrelation) GUI does in tstool,  but from within a custom script instead.
>
> A look at the help file revealed:
>
> "When computing the cross-correlation of two vector-valued time series x and y, Time Series Tools uses an algorithm that is functionally equivalent to calling the Signal Processing Toolbox xcorr function from with the 'biased' option, after the time series means have been removed"
>
> I have used xcorr on my data and it outputs values greater than 1. I have no idea what these could mean. I  was excepting r-statistics for each Y(ii) vs X(ii) combination at each lag. Furthermore, when I compare zero-lag xcorr values to those from pairwise correlation they are all different. What could I be doing wrong?

Hi Matthias, you just need to call xcorr with the 'coeff' option. That will produce output between [-1,1].  Also, note that xcorr returns the negative and positive lags. That behavior is documented. If you just want the nonnegative lags, then use only those. For example:

x = randn(100,1);
[c,lags] = xcorr(x,'coeff');
h = stem(lags(100:end),c(100:end));
set(h,'markerfacecolor',[0 0 1]);

Hope that helps,
Wayne
```
 0
Reply Wayne 9/20/2010 11:10:07 AM

```"Wayne King" <spamthis@mailinator.com> wrote in message <i77fef\$3bd\$1@fred.mathworks.com>...
> "Matthias " <matthew.goddard@bcf.uni-freiburg.de> wrote in message <i77e53\$buk\$1@fred.mathworks.com>...
> > Hi, I need a pointer on how to achieve what the time series correlation (autocorrelation) GUI does in tstool,  but from within a custom script instead.
> >
> > A look at the help file revealed:
> >
> > "When computing the cross-correlation of two vector-valued time series x and y, Time Series Tools uses an algorithm that is functionally equivalent to calling the Signal Processing Toolbox xcorr function from with the 'biased' option, after the time series means have been removed"
> >
> > I have used xcorr on my data and it outputs values greater than 1. I have no idea what these could mean. I  was excepting r-statistics for each Y(ii) vs X(ii) combination at each lag. Furthermore, when I compare zero-lag xcorr values to those from pairwise correlation they are all different. What could I be doing wrong?
>
> Hi Matthias, you just need to call xcorr with the 'coeff' option. That will produce output between [-1,1].  Also, note that xcorr returns the negative and positive lags. That behavior is documented. If you just want the nonnegative lags, then use only those. For example:
>
>  x = randn(100,1);
> [c,lags] = xcorr(x,'coeff');
> h = stem(lags(100:end),c(100:end));
> set(h,'markerfacecolor',[0 0 1]);
>
> Hope that helps,
> Wayne

Hi Wayne, thanks for your quick response.

I have tried the following syntax:

---------------------------
xcorr(X,Y,200); where X & Y are two univariate vectors

The last 5 values it returns are:
1.0427
1.0423
1.0419
1.0415
1.0411

---------------------------
xcorr(X,Y,200,'unbiased'); where X & Y are two univariate vectors

The last 5 values it returns are:
1.2127
1.2124
1.2121
1.2118
1.2114

See what I mean? These cannot be r-values. I don't know what they are....
```
 0
Reply Matthias 9/20/2010 11:28:04 AM

```"Matthias " <matthew.goddard@bcf.uni-freiburg.de> wrote in message <i77gg4\$9dj\$1@fred.mathworks.com>...
> "Wayne King" <spamthis@mailinator.com> wrote in message <i77fef\$3bd\$1@fred.mathworks.com>...
> > "Matthias " <matthew.goddard@bcf.uni-freiburg.de> wrote in message <i77e53\$buk\$1@fred.mathworks.com>...
> > > Hi, I need a pointer on how to achieve what the time series correlation (autocorrelation) GUI does in tstool,  but from within a custom script instead.
> > >
> > > A look at the help file revealed:
> > >
> > > "When computing the cross-correlation of two vector-valued time series x and y, Time Series Tools uses an algorithm that is functionally equivalent to calling the Signal Processing Toolbox xcorr function from with the 'biased' option, after the time series means have been removed"
> > >
> > > I have used xcorr on my data and it outputs values greater than 1. I have no idea what these could mean. I  was excepting r-statistics for each Y(ii) vs X(ii) combination at each lag. Furthermore, when I compare zero-lag xcorr values to those from pairwise correlation they are all different. What could I be doing wrong?
> >
> > Hi Matthias, you just need to call xcorr with the 'coeff' option. That will produce output between [-1,1].  Also, note that xcorr returns the negative and positive lags. That behavior is documented. If you just want the nonnegative lags, then use only those. For example:
> >
> >  x = randn(100,1);
> > [c,lags] = xcorr(x,'coeff');
> > h = stem(lags(100:end),c(100:end));
> > set(h,'markerfacecolor',[0 0 1]);
> >
> > Hope that helps,
> > Wayne
>
>
> Hi Wayne, thanks for your quick response.
>
> I have tried the following syntax:
>
> ---------------------------
> xcorr(X,Y,200); where X & Y are two univariate vectors
>
> The last 5 values it returns are:
> 1.0427
> 1.0423
> 1.0419
> 1.0415
> 1.0411
>
> ---------------------------
> xcorr(X,Y,200,'unbiased'); where X & Y are two univariate vectors
>
> The last 5 values it returns are:
> 1.2127
> 1.2124
> 1.2121
> 1.2118
> 1.2114
>
> See what I mean? These cannot be r-values. I don't know what they are....

Hi Matthias, You need to use the 'coeff' option to return what you are calling r-values. If you read the documentation

>>doc xcorr

It tells you exactly what 'biased' and 'unbiased' mean in this situation. There are many reasons to define such estimates of the autocorrelation. For example, the Fourier transform of the biased autocorrelation estimate is the periodogram, which is a nonparametric estimate of the power spectral density of a random process.

Wayne
```
 0
Reply Wayne 9/20/2010 11:39:04 AM

```Hi again,

> Hi Matthias, You need to use the 'coeff' option to return what you are calling r-values. If you read the documentation

Forgive me, but  I found the reference in question to be a little ambiguous:

"coeff': Normalizes the sequence so the autocorrelations at zero lag are identically 1.0."

I did assume that what was meant were r-values. But what confuses me is that if I compare the zero-lag r-value from xcorr with 'coeff' called to the r-value from a) unadjusted pairwise correlation (corrcoef),  and b) adjusted pairwise correlation (corr2), the results I obtain are, respectively: 0.6511, -0.0396, and -0.0396.

The latter two r-values are not surprising, since I normalized the data set anyway; but  xcorr managing to return 0.65 at zero lag is mystifying (range across all lags: 0.6534 to 0.6488)

I checked cross correlation in R on the same data and it gave -0.040 at zero lag, so from this I know something is not right with xcorr. But I just cannot see the problem; my syntax is:
xcorr(D(:,1),D(:,2),10,'coeff'); % where D contains two univariate vectors
```
 0
Reply Matthias 9/20/2010 12:39:04 PM

```"Matthias " <matthew.goddard@bcf.uni-freiburg.de> wrote in message <i77kl8\$28d\$1@fred.mathworks.com>...
> Hi again,
>
> > Hi Matthias, You need to use the 'coeff' option to return what you are calling r-values. If you read the documentation
>
> Forgive me, but  I found the reference in question to be a little ambiguous:
>
> "coeff': Normalizes the sequence so the autocorrelations at zero lag are identically 1.0."
>
> I did assume that what was meant were r-values. But what confuses me is that if I compare the zero-lag r-value from xcorr with 'coeff' called to the r-value from a) unadjusted pairwise correlation (corrcoef),  and b) adjusted pairwise correlation (corr2), the results I obtain are, respectively: 0.6511, -0.0396, and -0.0396.
>
> The latter two r-values are not surprising, since I normalized the data set anyway; but  xcorr managing to return 0.65 at zero lag is mystifying (range across all lags: 0.6534 to 0.6488)
>
> I checked cross correlation in R on the same data and it gave -0.040 at zero lag, so from this I know something is not right with xcorr. But I just cannot see the problem; my syntax is:
> xcorr(D(:,1),D(:,2),10,'coeff'); % where D contains two univariate vectors

Hi, Can you send me the data? You can send it either as .RData files, or as a .mat file. It looks like from above you had the data in a matrix D. If that is the case, then just save that as a .mat file and mail it to my gmail account. I will post back what I find here on the newsgroup so others may benefit as well.

Wayne
```
 0
Reply Wayne 9/20/2010 12:58:03 PM

```"Wayne King" <wmkingty@gmail.com> wrote in message <i77lor\$cmc\$1@fred.mathworks.com>...
> "Matthias " <matthew.goddard@bcf.uni-freiburg.de> wrote in message <i77kl8\$28d\$1@fred.mathworks.com>...
> > Hi again,
> >
> > > Hi Matthias, You need to use the 'coeff' option to return what you are calling r-values. If you read the documentation
> >
> > Forgive me, but  I found the reference in question to be a little ambiguous:
> >
> > "coeff': Normalizes the sequence so the autocorrelations at zero lag are identically 1.0."
> >
> > I did assume that what was meant were r-values. But what confuses me is that if I compare the zero-lag r-value from xcorr with 'coeff' called to the r-value from a) unadjusted pairwise correlation (corrcoef),  and b) adjusted pairwise correlation (corr2), the results I obtain are, respectively: 0.6511, -0.0396, and -0.0396.
> >
> > The latter two r-values are not surprising, since I normalized the data set anyway; but  xcorr managing to return 0.65 at zero lag is mystifying (range across all lags: 0.6534 to 0.6488)
> >
> > I checked cross correlation in R on the same data and it gave -0.040 at zero lag, so from this I know something is not right with xcorr. But I just cannot see the problem; my syntax is:
> > xcorr(D(:,1),D(:,2),10,'coeff'); % where D contains two univariate vectors
>
> Hi, Can you send me the data? You can send it either as .RData files, or as a .mat file. It looks like from above you had the data in a matrix D. If that is the case, then just save that as a .mat file and mail it to my gmail account. I will post back what I find here on the newsgroup so others may benefit as well.
>
> Wayne

Hi Wayne,

Thanks again for your help. I looked into this issue a little further with a colleague and we both can see from the xcov.m file that what 'cross-covariance' with 'coeff' actually computes is cross correlation. This explains why zero-lag cross-covariance is equal to the Pearson product moment coefficient (r-value).

Here's an example to illustrate exactly what I mean:

X = [1:1:100]';
A = normpdf(X,50,4);%generate first Gaussian curve
B = normpdf(X,30,4);% generate second Gaussian curve with mean = 20 less
c = xcov(A,B,30,'coeff');
c(31);% to find the zero lag coefficient

Now, compare to:
corr2(A,B)

And we can see that c(31) = Pearson product moment for A ~ B

So, if xcov is really cross-correlation, then what is xcorr?

In the xcorr.m file it does a weird normalization for the output variable, like this:
c = c./scale;
But I  have not tracked down yet how c is calculated before this final calculation.

Franky I'm a little annoyed Matlab calls cross correlation 'cross covariance' ; and uses 'cross correlation' to describe some unknown statistical test. Finding this out has cost me three hours of work.....
```
 0
Reply Matthias 9/20/2010 3:46:04 PM

```"Matthias " <matthew.goddard@bcf.uni-freiburg.de> wrote in message <i77vjs\$sqo\$1@fred.mathworks.com>...
> "Wayne King" <wmkingty@gmail.com> wrote in message <i77lor\$cmc\$1@fred.mathworks.com>...
> > "Matthias " <matthew.goddard@bcf.uni-freiburg.de> wrote in message <i77kl8\$28d\$1@fred.mathworks.com>...
> > > Hi again,
> > >
> > > > Hi Matthias, You need to use the 'coeff' option to return what you are calling r-values. If you read the documentation
> > >
> > > Forgive me, but  I found the reference in question to be a little ambiguous:
> > >
> > > "coeff': Normalizes the sequence so the autocorrelations at zero lag are identically 1.0."
> > >
> > > I did assume that what was meant were r-values. But what confuses me is that if I compare the zero-lag r-value from xcorr with 'coeff' called to the r-value from a) unadjusted pairwise correlation (corrcoef),  and b) adjusted pairwise correlation (corr2), the results I obtain are, respectively: 0.6511, -0.0396, and -0.0396.
> > >
> > > The latter two r-values are not surprising, since I normalized the data set anyway; but  xcorr managing to return 0.65 at zero lag is mystifying (range across all lags: 0.6534 to 0.6488)
> > >
> > > I checked cross correlation in R on the same data and it gave -0.040 at zero lag, so from this I know something is not right with xcorr. But I just cannot see the problem; my syntax is:
> > > xcorr(D(:,1),D(:,2),10,'coeff'); % where D contains two univariate vectors
> >
> > Hi, Can you send me the data? You can send it either as .RData files, or as a .mat file. It looks like from above you had the data in a matrix D. If that is the case, then just save that as a .mat file and mail it to my gmail account. I will post back what I find here on the newsgroup so others may benefit as well.
> >
> > Wayne
>
>
> Hi Wayne,
>
> Thanks again for your help. I looked into this issue a little further with a colleague and we both can see from the xcov.m file that what 'cross-covariance' with 'coeff' actually computes is cross correlation. This explains why zero-lag cross-covariance is equal to the Pearson product moment coefficient (r-value).
>
> Here's an example to illustrate exactly what I mean:
>
> X = [1:1:100]';
> A = normpdf(X,50,4);%generate first Gaussian curve
> B = normpdf(X,30,4);% generate second Gaussian curve with mean = 20 less
> c = xcov(A,B,30,'coeff');
> c(31);% to find the zero lag coefficient
>
> Now, compare to:
> corr2(A,B)
>
> And we can see that c(31) = Pearson product moment for A ~ B
>
> So, if xcov is really cross-correlation, then what is xcorr?
>
> In the xcorr.m file it does a weird normalization for the output variable, like this:
> c = c./scale;
> But I  have not tracked down yet how c is calculated before this final calculation.
>
> Franky I'm a little annoyed Matlab calls cross correlation 'cross covariance' ; and uses 'cross correlation' to describe some unknown statistical test. Finding this out has cost me three hours of work.....

Hi Matthias, don't be annoyed :) The difference between xcov() and xcorr() is documented. There is no "weird" normalization. The difference is that xcov() first subtracts the mean from each input vector. xcorr() does not.

Try this:

c = xcorr(A-mean(A),B-mean(B),30,'coeff');
c(31)

Hope that helps,
Wayne
```
 0
Reply Wayne 9/20/2010 4:14:05 PM

```"Wayne King" <wmkingty@gmail.com> wrote in message
> Hi Matthias, don't be annoyed :) The difference between xcov() and xcorr() is documented. There is no "weird" normalization. The difference is that xcov() first subtracts the mean from each input vector. xcorr() does not.
>
> Try this:
>
> c = xcorr(A-mean(A),B-mean(B),30,'coeff');
> c(31)
>
> Hope that helps,
> Wayne

Ok, I understand that xcorr calculates the product moments between A and B at each lag, and this is different to subtracting the mean from each input vector first. I guess my confusion stemmed from me mistakingly thinking that xcorr(A,B,lag,'coeff') should return r-values, as in the 'r-values' in returned in classical correlation. My bad....biologists are just simple creatures after all ;)
```
 0
Reply Matthias 9/21/2010 9:20:06 AM

8 Replies
534 Views

Similar Articles

12/7/2013 12:40:29 PM
page loaded in 218960 ms. (0)