Every now and then, links to my remote offices are lost due to T1
failures.  The symptoms of the malfunction line involve extended pings
showing dropped packets.  Other times, no traffic will pass through the
link at all; the T1 module interface status will either be UpDown or
DownDown.  The topology is straight-forward, containing only
point-to-point IP addressing between the Cisco routers, which include
various combinations of the following models: 7206, 3640, 2621, 1721.

I have found that physically unplugging the T1 cable from the interface
module and reinserting it will bring the connection back to normal.
Reseating the telecom company's Pairgain card or simply reloading the
router will also work.  Telecom has only ever seen clean circuits after
the problem is resolved.

Part of my troubleshooting has been to replace  every piece of
hardware, including swapping WIC modules, routers, and Pairgain cards.

I need a technical explanation for what is occurring.  Is this an issue
I should bring to my telecommunications provider?  Or is this an issue
with the Cisco hardware?

Greetings,

> Every now and then, links to my remote offices are lost due to T1
> failures.  The symptoms of the malfunction line involve extended pings
> showing dropped packets.  Other times, no traffic will pass through the
> link at all; the T1 module interface status will either be UpDown or
> DownDown.  The topology is straight-forward, containing only
> point-to-point IP addressing between the Cisco routers, which include
> various combinations of the following models: 7206, 3640, 2621, 1721.

My initial thoughts would tend to head towards some obscure 
configuration issue, possibly revolving around a Clocking problem 
(such as a clock not locked to a stable source), but then you talk 
about -

> Reseating the telecom company's Pairgain card 

and at that point I would back off, because "pair gain" apperatus can 
be really dicey to work with. If it means the same thing in your part 
of the world as my part of the world (New Zealand), then Pair Gain is 
deadly to good Data communications and is avoided at all costs. 
Basically, our version of Pair Gain uses a Provider supplied device 
that Frequency Division Multiplexes 2 data streams down one copper 
pair (effectively providing 2 copper pairs over 1 cable pair). The 
Frequency shifting means the transport must use 100% analog 
technology, and we haven't see that around here for 10+ years now, its
full Digital Data Services end to end here. But then your Pair Gain 
may be different to ours. Here we can refuse a Data Service if the 
Telco attempts to use Pair Gain H/W, and we haven't seen one used for 
over 10 years now. If it does use FDM then clocking can be a serious 
issue, as you are 100% reliant on the provider network for the Clock 
source in ALL cases...

All the rest of your message strongly suggests your Provider has a 
configuration issue and is not providing stable clocking to your 
devices, but I say this thinking about how its done in this country, 
and things may be different at your end.

Good luck...............pk.


-- 
Peter from Auckland.

Peter wrote:

> Greetings,
>
> > Every now and then, links to my remote offices are lost due to T1
> > failures.  The symptoms of the malfunction line involve extended pings
> > showing dropped packets.  Other times, no traffic will pass through the
> > link at all; the T1 module interface status will either be UpDown or
> > DownDown.  The topology is straight-forward, containing only
> > point-to-point IP addressing between the Cisco routers, which include
> > various combinations of the following models: 7206, 3640, 2621, 1721.
>
> My initial thoughts would tend to head towards some obscure
> configuration issue, possibly revolving around a Clocking problem
> (such as a clock not locked to a stable source), but then you talk
> about -
>
> > Reseating the telecom company's Pairgain card
>
> and at that point I would back off, because "pair gain" apperatus can
> be really dicey to work with. If it means the same thing in your part
> of the world as my part of the world (New Zealand), then Pair Gain is
> deadly to good Data communications and is avoided at all costs.
> Basically, our version of Pair Gain uses a Provider supplied device
> that Frequency Division Multiplexes 2 data streams down one copper
> pair (effectively providing 2 copper pairs over 1 cable pair). The
> Frequency shifting means the transport must use 100% analog
> technology, and we haven't see that around here for 10+ years now, its
> full Digital Data Services end to end here. But then your Pair Gain
> may be different to ours. Here we can refuse a Data Service if the
> Telco attempts to use Pair Gain H/W, and we haven't seen one used for
> over 10 years now. If it does use FDM then clocking can be a serious
> issue, as you are 100% reliant on the provider network for the Clock
> source in ALL cases...
>
> All the rest of your message strongly suggests your Provider has a
> configuration issue and is not providing stable clocking to your
> devices, but I say this thinking about how its done in this country,
> and things may be different at your end.

My experience suggests (limited E1/T1, extensive Cisco)
suggests that the routers hardware is unlikely to be an issue.

If you clocking is configured correctly (and even of it is not
configured correctly, in my experience all you see are a
few line errors) then it must be a telco issue.

There are several docs on troubleshooting T1 on CCO.

sh controllers t1 x/x/x

displays comprehensive error reports for the last 96
15 minute periods. 96 x 900s = 24 hours.

Zero errors are the number to expect really.


T1 has extensive integrated diagnostics and problem
reports are sent from one end to the other so that both
ends have a view on the behaviour of the link.
I am not an expert but remote alarms for example
are problem indications that have been sent from the
other end.

Post the controllers and serial interface sections of the config
and also a "sh network-clocks" and a few sections of
a "sh controllers t1 x/x/x".

You could ask the Telco to do an end to end test.

Bod43@hotmail.co.uk wrote:
> Peter wrote:
> 
>> Greetings,
>>
>>> Every now and then, links to my remote offices are lost due to T1
>>> failures.  The symptoms of the malfunction line involve extended pings
>>> showing dropped packets.  Other times, no traffic will pass through the
>>> link at all; the T1 module interface status will either be UpDown or
>>> DownDown.  The topology is straight-forward, containing only
>>> point-to-point IP addressing between the Cisco routers, which include
>>> various combinations of the following models: 7206, 3640, 2621, 1721.
>> My initial thoughts would tend to head towards some obscure
>> configuration issue, possibly revolving around a Clocking problem
>> (such as a clock not locked to a stable source), but then you talk
>> about -
>>
>>> Reseating the telecom company's Pairgain card
>> and at that point I would back off, because "pair gain" apperatus can
>> be really dicey to work with. If it means the same thing in your part
>> of the world as my part of the world (New Zealand), then Pair Gain is
>> deadly to good Data communications and is avoided at all costs.
>> Basically, our version of Pair Gain uses a Provider supplied device
>> that Frequency Division Multiplexes 2 data streams down one copper
>> pair (effectively providing 2 copper pairs over 1 cable pair). The
>> Frequency shifting means the transport must use 100% analog
>> technology, and we haven't see that around here for 10+ years now, its
>> full Digital Data Services end to end here. But then your Pair Gain
>> may be different to ours. Here we can refuse a Data Service if the
>> Telco attempts to use Pair Gain H/W, and we haven't seen one used for
>> over 10 years now. If it does use FDM then clocking can be a serious
>> issue, as you are 100% reliant on the provider network for the Clock
>> source in ALL cases...
>>
>> All the rest of your message strongly suggests your Provider has a
>> configuration issue and is not providing stable clocking to your
>> devices, but I say this thinking about how its done in this country,
>> and things may be different at your end.
> 
> My experience suggests (limited E1/T1, extensive Cisco)
> suggests that the routers hardware is unlikely to be an issue.
> 
> If you clocking is configured correctly (and even of it is not
> configured correctly, in my experience all you see are a
> few line errors) then it must be a telco issue.
> 
> There are several docs on troubleshooting T1 on CCO.
> 
> sh controllers t1 x/x/x
> 
> displays comprehensive error reports for the last 96
> 15 minute periods. 96 x 900s = 24 hours.
> 
> Zero errors are the number to expect really.
> 
> 
> T1 has extensive integrated diagnostics and problem
> reports are sent from one end to the other so that both
> ends have a view on the behaviour of the link.
> I am not an expert but remote alarms for example
> are problem indications that have been sent from the
> other end.
> 
> Post the controllers and serial interface sections of the config
> and also a "sh network-clocks" and a few sections of
> a "sh controllers t1 x/x/x".
> 
> You could ask the Telco to do an end to end test.
> 
There are many issues with a local loop these days, from coroding 
connections from decades old installations along the streets, to failing 
NIU ,etc.

Most of the issues I have seen are from the NIU (or smart jacks) that 
terminate the telco connection. It provides loop back diagnostic 
functions as well as the telco interface to the customer DS1 interface.
Once these have been in service for a while the contacts on the relays 
can become intermittent/noisy. the telco will loop this device, causing 
the relay contacts to switch to put the circuit in a loop back to the 
telco and thus cleaning the connects and correcting the problem, will 
run clean and the telco will claim there is no issues with their circuit.
If this happens two times in a row I request a replacement.

Also, two wire HDSL t1 extended lines from the telco can have issues 
with some services such as frame relay, etc. This is similar to DSL as 
the service is a modulated data stream over just one copper pair and not 
the bi-polar data stream over the seperate transmit and receive pairs in 
traditional t1 circuits which would require repeters every mile whereas 
the HDSL lines do not so telcos are trying to put the HDSL lines in as 
much as possible, and saves on pairs. However any noise on the modulated 
signal can cause frame relay and other protocol services to drop and 
have to regain connectivity which can be several seconds on never can id 
too noisy whereas most point-to-point circuits would not care as much. 
The HDSL is more sensitive to noise just like other DSL services than 
the traditional 4-wire T1 repeated lines.

Well, I'm pretty sure that I don't have much of a choice as far as the
Pair Gain cards are concerned.  Technology is slowly picking up in the
middle of the Pacific.  I will look into an upgrade from my telecom and
ask for more tests to be done on their end.

Because I am using WIC T1 DSUs, I do not have the "show controller t1"
command.  However, I've included the "show controller serial" command
in my post.

Also, I could not execute the "show network-clocks" command on the
remote end.

All the help is much appreciated!
Brian

LOCAL_2621>sh ver
Cisco IOS Software, C2600 Software (C2600-ADVIPSERVICESK9-M), Version
12.4(8a), RELEASE SOFTWARE (fc2)
LOCAL_2621 uptime is 14 weeks, 4 days, 4 hours, 37 minutes
Cisco 2621XM (MPC860P) processor (revision 3.1) with 127308K/3764K
bytes of memory.
Processor board ID FOC08251CWQ
M860 processor: part number 5, mask 2
2 FastEthernet interfaces
2 Serial interfaces
32K bytes of NVRAM.
32768K bytes of processor board System flash (Read/Write)

LOCAL_2621#sh network-clocks
  Network Clock Configuration
  ---------------------------
  Priority      Clock Source    Clock State     Clock Type

     5          Backplane       NOT AVAILABLE   PLL

  Current Primary Clock Source
  ---------------------------
  Priority      Clock Source    Clock State     Clock Type

     5          Backplane       NOT AVAILABLE   PLL

LOCAL_2621#show service-module
Module type is T1/fractional
    Hardware revision is 0.88, Software revision is 20050811-0.3,
    Image checksum is 0x80ACD1B3, Protocol revision is 0.1
Receiver has no alarms.
Framing is ESF, Line Code is B8ZS, Current clock source is line,
Fraction has 24 timeslots (64 Kbits/sec each), Net bandwidth is 1536
Kbits/sec.
Last module self-test (done at startup): Passed
Last clearing of alarm counters 1w1d
    loss of signal        :    1, last occurred 1w1d
    loss of frame         :    2, last occurred 19:46:22
    AIS alarm             :    1, last occurred 19:46:22
    Remote alarm          :    1, last occurred 1w1d
    Module access errors  :    0,
Total Data (last 96 15 minute intervals):
    0 Line Code Violations, 2036 Path Code Violations
    0 Slip Secs, 50 Fr Loss Secs, 0 Line Err Secs, 681 Degraded Mins
    1336 Errored Secs, 67 Bursty Err Secs, 0 Severely Err Secs, 50
Unavail Secs
Data in current interval (480 seconds elapsed):
    0 Line Code Violations, 13 Path Code Violations
    0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 7 Degraded Mins
    13 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail
Secs


LOCAL_2621#sh controller s0/1
Interface Serial0/1
Hardware is PowerQUICC MPC860 with Integrated FT1 CSU/DSU module
 TX and RX clocks detected.
PowerQUICC SCC specific errors:
31321 input aborts on receiving flag sequence
0 throttles, 0 enables
0 overruns
0 transmitter underruns
0 transmitter CTS losts
11910 aborted short frames




Bod43@hotmail.co.uk wrote:
> Peter wrote:
>
> > Greetings,
> >
> > > Every now and then, links to my remote offices are lost due to T1
> > > failures.  The symptoms of the malfunction line involve extended pings
> > > showing dropped packets.  Other times, no traffic will pass through the
> > > link at all; the T1 module interface status will either be UpDown or
> > > DownDown.  The topology is straight-forward, containing only
> > > point-to-point IP addressing between the Cisco routers, which include
> > > various combinations of the following models: 7206, 3640, 2621, 1721.
> >
> > My initial thoughts would tend to head towards some obscure
> > configuration issue, possibly revolving around a Clocking problem
> > (such as a clock not locked to a stable source), but then you talk
> > about -
> >
> > > Reseating the telecom company's Pairgain card
> >
> > and at that point I would back off, because "pair gain" apperatus can
> > be really dicey to work with. If it means the same thing in your part
> > of the world as my part of the world (New Zealand), then Pair Gain is
> > deadly to good Data communications and is avoided at all costs.
> > Basically, our version of Pair Gain uses a Provider supplied device
> > that Frequency Division Multiplexes 2 data streams down one copper
> > pair (effectively providing 2 copper pairs over 1 cable pair). The
> > Frequency shifting means the transport must use 100% analog
> > technology, and we haven't see that around here for 10+ years now, its
> > full Digital Data Services end to end here. But then your Pair Gain
> > may be different to ours. Here we can refuse a Data Service if the
> > Telco attempts to use Pair Gain H/W, and we haven't seen one used for
> > over 10 years now. If it does use FDM then clocking can be a serious
> > issue, as you are 100% reliant on the provider network for the Clock
> > source in ALL cases...
> >
> > All the rest of your message strongly suggests your Provider has a
> > configuration issue and is not providing stable clocking to your
> > devices, but I say this thinking about how its done in this country,
> > and things may be different at your end.
>
> My experience suggests (limited E1/T1, extensive Cisco)
> suggests that the routers hardware is unlikely to be an issue.
>
> If you clocking is configured correctly (and even of it is not
> configured correctly, in my experience all you see are a
> few line errors) then it must be a telco issue.
>
> There are several docs on troubleshooting T1 on CCO.
>
> sh controllers t1 x/x/x
>
> displays comprehensive error reports for the last 96
> 15 minute periods. 96 x 900s = 24 hours.
>
> Zero errors are the number to expect really.
>
>
> T1 has extensive integrated diagnostics and problem
> reports are sent from one end to the other so that both
> ends have a view on the behaviour of the link.
> I am not an expert but remote alarms for example
> are problem indications that have been sent from the
> other end.
>
> Post the controllers and serial interface sections of the config
> and also a "sh network-clocks" and a few sections of
> a "sh controllers t1 x/x/x".
> 
> You could ask the Telco to do an end to end test.

Greetings,

Note: Please see my final comments before acting on any part of the 
following...

> LOCAL_2621#show service-module
> Module type is T1/fractional
>     Hardware revision is 0.88, Software revision is 20050811-0.3,
>     Image checksum is 0x80ACD1B3, Protocol revision is 0.1
> Receiver has no alarms.
> Framing is ESF, Line Code is B8ZS, Current clock source is line,

This all looks pretty "normal" to me, the Framing and Line code are 
what I would expect to see, the Clock source is fine. Hopefuly the 
other end of these fractional Circuits should look pretty much the 
same. Interestingly, B8ZS suggests that NRZI encoding is being used 
(which is good) as that can help with recovery of clocking in older 
tranmissions systems. In modern transmission systems NRZI encoding is 
not usually required these days, although most still support it.

> Last clearing of alarm counters 1w1d
>     loss of signal        :    1, last occurred 1w1d
>     loss of frame         :    2, last occurred 19:46:22
>     AIS alarm             :    1, last occurred 19:46:22
>     Remote alarm          :    1, last occurred 1w1d

This suggests that the bearer is stable (loss of signal = 1w1d = 
uptime) and is not being broken, however the "loss of frame" is my 
guess as your main concern. 1 loss in over 1 week might be acceptable,
but 2 in that time, the last less tan 1 day would start to raise my 
concern. It may be a once off event, but 4 in 2 weeks I would 
definately start to worry about...

>     Module access errors  :    0,
> Total Data (last 96 15 minute intervals):
>     0 Line Code Violations, 2036 Path Code Violations

The line coding is fine (zero), but I am not sure what the PATH code 
error might mean. Whatever it is I would say thats a high reading that
your provider may be interested in

>     0 Slip Secs, 50 Fr Loss Secs, 0 Line Err Secs, 681 Degraded Mins
>     1336 Errored Secs, 67 Bursty Err Secs, 0 Severely Err Secs, 50

Are you able to confirm that the other ends of these Fractional links 
have no obvious setting issues? Assuming that they are set the same 
(B8ZS and clocking external), then whatever the issue is, it looks to 
be 100% within the provider area of resolution to me, and NOT in the 
Routers at either end. 

> Unavail Secs
> Data in current interval (480 seconds elapsed):
>     0 Line Code Violations, 13 Path Code Violations
>     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 7 Degraded Mins
>     13 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail
> Secs

Having said all that, the circuit looks like it is "basically" 
working, with a high count of intermittent issues. While not 
preventing its overall use, the "random events" would provide periods 
of degraded operation.


> LOCAL_2621#sh controller s0/1
> Interface Serial0/1
> Hardware is PowerQUICC MPC860 with Integrated FT1 CSU/DSU module
>  TX and RX clocks detected.
> PowerQUICC SCC specific errors:
> 31321 input aborts on receiving flag sequence

Now THAT is a good indication of what the provider may need to look 
at, circuit framing (no received Flag due to ABORT) is failing at a 
fairly high rate. This is likely to only be an end user noticable 
event if the user traffic is moderate to high. At lower traffic 
volumes it would pass almost unnoticed (to an end-user).

> 0 throttles, 0 enables
> 0 overruns
> 0 transmitter underruns
> 0 transmitter CTS losts
> 11910 aborted short frames

And thats the cause of the frame errors, an early ABORT. An ABORT can 
be a local/remote site issue but in this case its more likely to be a 
Provider issue.

Do you have any other Fractional Circuits from this provider? Do you 
know how many of those they have set up before?

Cheers..............pk.


-- 
Peter from Auckland.

I appreciate the help once again! The analysis of the errors is very
helpful, and I intend to bring this issue to my telecommunications
provider.  Yes, I do have several other T1 circuits from the same
company, who does have a decent amount of experience in this area.

Brian


Peter wrote:
> Greetings,
>
> Note: Please see my final comments before acting on any part of the
> following...
>
> > LOCAL_2621#show service-module
> > Module type is T1/fractional
> >     Hardware revision is 0.88, Software revision is 20050811-0.3,
> >     Image checksum is 0x80ACD1B3, Protocol revision is 0.1
> > Receiver has no alarms.
> > Framing is ESF, Line Code is B8ZS, Current clock source is line,
>
> This all looks pretty "normal" to me, the Framing and Line code are
> what I would expect to see, the Clock source is fine. Hopefuly the
> other end of these fractional Circuits should look pretty much the
> same. Interestingly, B8ZS suggests that NRZI encoding is being used
> (which is good) as that can help with recovery of clocking in older
> tranmissions systems. In modern transmission systems NRZI encoding is
> not usually required these days, although most still support it.
>
> > Last clearing of alarm counters 1w1d
> >     loss of signal        :    1, last occurred 1w1d
> >     loss of frame         :    2, last occurred 19:46:22
> >     AIS alarm             :    1, last occurred 19:46:22
> >     Remote alarm          :    1, last occurred 1w1d
>
> This suggests that the bearer is stable (loss of signal = 1w1d =
> uptime) and is not being broken, however the "loss of frame" is my
> guess as your main concern. 1 loss in over 1 week might be acceptable,
> but 2 in that time, the last less tan 1 day would start to raise my
> concern. It may be a once off event, but 4 in 2 weeks I would
> definately start to worry about...
>
> >     Module access errors  :    0,
> > Total Data (last 96 15 minute intervals):
> >     0 Line Code Violations, 2036 Path Code Violations
>
> The line coding is fine (zero), but I am not sure what the PATH code
> error might mean. Whatever it is I would say thats a high reading that
> your provider may be interested in
>
> >     0 Slip Secs, 50 Fr Loss Secs, 0 Line Err Secs, 681 Degraded Mins
> >     1336 Errored Secs, 67 Bursty Err Secs, 0 Severely Err Secs, 50
>
> Are you able to confirm that the other ends of these Fractional links
> have no obvious setting issues? Assuming that they are set the same
> (B8ZS and clocking external), then whatever the issue is, it looks to
> be 100% within the provider area of resolution to me, and NOT in the
> Routers at either end.
>
> > Unavail Secs
> > Data in current interval (480 seconds elapsed):
> >     0 Line Code Violations, 13 Path Code Violations
> >     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 7 Degraded Mins
> >     13 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail
> > Secs
>
> Having said all that, the circuit looks like it is "basically"
> working, with a high count of intermittent issues. While not
> preventing its overall use, the "random events" would provide periods
> of degraded operation.
>
>
> > LOCAL_2621#sh controller s0/1
> > Interface Serial0/1
> > Hardware is PowerQUICC MPC860 with Integrated FT1 CSU/DSU module
> >  TX and RX clocks detected.
> > PowerQUICC SCC specific errors:
> > 31321 input aborts on receiving flag sequence
>
> Now THAT is a good indication of what the provider may need to look
> at, circuit framing (no received Flag due to ABORT) is failing at a
> fairly high rate. This is likely to only be an end user noticable
> event if the user traffic is moderate to high. At lower traffic
> volumes it would pass almost unnoticed (to an end-user).
>
> > 0 throttles, 0 enables
> > 0 overruns
> > 0 transmitter underruns
> > 0 transmitter CTS losts
> > 11910 aborted short frames
>
> And thats the cause of the frame errors, an early ABORT. An ABORT can
> be a local/remote site issue but in this case its more likely to be a
> Provider issue.
>
> Do you have any other Fractional Circuits from this provider? Do you
> know how many of those they have set up before?
> 
> Cheers..............pk.
> 
> 
> -- 
> Peter from Auckland.