So what's your point?

  -----Original Message-----
  From: Lee [mailto:lytmah@telusplanet.net]
  Sent: Friday, June 04, 2004 5:43 PM
  To: Info-VAX@Mvb.Saic.Com
  Subject: Computer foul-up has bank customers fuming
  
  
  Here some interesting news.
  Another black eye for computer technology
  
  http://www.canada.com/edmonton/edmontonjournal/news/story.html?id=
  1b790ee7-570f-480b-af26-069bbe7035e5
  
  ---
  Incoming mail is certified Virus Free.
  Checked by AVG anti-virus system (http://www.grisoft.com).
  Version: 6.0.680 / Virus Database: 442 - Release Date: 5/9/2004
  
---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.680 / Virus Database: 442 - Release Date: 5/9/2004

Tom Linden wrote:
> 
> So what's your point?

No matter how fancy a disaster recovery plan you may have at the OS and disk
level, when your application screws up, you're up the creek perhaps even more
if it takes time to realise the foul up.

The Royal Bank  had one of the most stable and mature IT departments. For as
much as they were arrogant towards customers, when they introduced a service,
it was usually well done and very reliable.

For them to have a foul up lasting a whole week, preventing direct deposits
(depriving customers of their money), blocking  debit transactions because
pension/paychecks were not deposited into customer's accounts etc etc is a
HUGE problem for a bank. This isn't an outage of a few hours, it is something
lasting almost a whole week.

It is also about IT change management. Banks in canada have stringent change
management procedures that include rollback procedures in case a change
doesn't go as planned. In this case, this clearly failed.

Is this part of a trend where windows weenies are also invading banks and are
destroying all the "red tape" that was change management ? (yes, to
programmers and system managers, change management procedures and committess
are seen as red tape, but they are a necessary evil to ensure stability of systems).

Banks' only product is information. And they live and die with confidence from
customers. Their reliance on IT is thus far greater than manufacturers,
restaurants etc. 

There have been many examples of banks failing inside a week after consumers
lost confidence in the bank. (including the Mercantile Bank in canada in the
1980s). Should the Royal Bank cease to exist in a week or two (probably not
likely, but still a possibility), who will be held responsible ? Will the IT
director be held responsible ?

When an organisation relies so heavily on IT, taking chances on procedures or
low quality systems can jeoperdize its own survival.

"JF Mezei" <jfmezei.spamnot@teksavvy.com> wrote in message
news:c8ce6db5859fc3d14b4bdc724d3a36cf@news.teranews.com...
> Tom Linden wrote:
> >
> > So what's your point?
>
> No matter how fancy a disaster recovery plan you may have at the OS and
disk
> level, when your application screws up, you're up the creek perhaps even
more
> if it takes time to realise the foul up.
>
> The Royal Bank  had one of the most stable and mature IT departments. For
as
> much as they were arrogant towards customers, when they introduced a
service,
> it was usually well done and very reliable.
>
> For them to have a foul up lasting a whole week, preventing direct
deposits
> (depriving customers of their money), blocking  debit transactions because
> pension/paychecks were not deposited into customer's accounts etc etc is a
> HUGE problem for a bank. This isn't an outage of a few hours, it is
something
> lasting almost a whole week.
>
> It is also about IT change management. Banks in canada have stringent
change
> management procedures that include rollback procedures in case a change
> doesn't go as planned. In this case, this clearly failed.
>
> Is this part of a trend where windows weenies are also invading banks and
are
> destroying all the "red tape" that was change management ? (yes, to
> programmers and system managers, change management procedures and
committess
> are seen as red tape, but they are a necessary evil to ensure stability of
systems).
>
> Banks' only product is information. And they live and die with confidence
from
> customers. Their reliance on IT is thus far greater than manufacturers,
> restaurants etc.
>
> There have been many examples of banks failing inside a week after
consumers
> lost confidence in the bank. (including the Mercantile Bank in canada in
the
> 1980s). Should the Royal Bank cease to exist in a week or two (probably
not
> likely, but still a possibility), who will be held responsible ? Will the
IT
> director be held responsible ?
>
> When an organisation relies so heavily on IT, taking chances on procedures
or
> low quality systems can jeoperdize its own survival.


JF has a good point here:

This situation is about process more than anything else, and confidence in
the process. With bad QC, and in most organizations the headlong rush to
hurry changes, cut costs, and lower head counts, anything can happen in this
regard.

What the article(s) don't say is whether the 'staff' which made the changes
are permanent full-time, contract, or off-shore, it could be any of these.
And while one might be tempted to say that code reviews and regression
testing will catch this sort of thing, often it is not because the really
good people are often not tempted to stick around given the bureacracy and
rigid heirarchy.

This particular organization has many inititatives underway, some with major
off-shore coding companies for key business systems. I have seen examples of
the work done by these particular companies in the past in other shops and
none came in on-time, on-budget, or performed as specifed when first
implemented.

Another major initiative also underway with this bank is a complete re-write
of a critical business system on Tandem, this being done in-house with the
addition of some contract staff. The bank's IT staff on this project (at the
time the project was initated) were, as explained to me by the development
manager in charge, " 2/3rd's of my staff have been here less than 6 months
and virtually none of them have experience in the line of business this
application is used for."  They had decided that the best way to manage this
project was to hire a bean counter 'project manager' first as opposed to
doing a detailed business analysis first and possibly hiring an experienced
line-of-business-specific project manager at the same time.

But this is not uncommon these days in many organizations. Lower head
counts, a reluctance to spend the right amount of money on the right people,
the loss of 'institutional knowledge', the desire to 'pump the earnings
numbers' by hiring lesser qualified consultants base on cost issues rather
than getting the best, all contribute to situations where these sorts of
things can, and do, occur. Just because they are banks or other 'reputable'
institutions does not mean that they don't have their fair share of PHB's
(pointy-hair bosses) [ see www.dilbert.com ]

This bank has a public relations disaster of major proportions on its hands.
I will grant that newspapers tend to focus on the lurid details, but this
story is somewhat indicative of the discussion going on
http://www.thestar.com/NASApp/cs/ContentServer?pagename=thestar/Layout/Artic
le_Type1&call_pageid=971358637177&c=Article&cid=1086300609453. This bank has
60,000+ employees of its own as well as tens of thousands of other
companies, representing millions of individuals, who are affected by this.
Aside from direct costs, including such things as full-page ads in
newspapers across the country apologizing for this foul-up, other costs and
risks which are harder to quantify are the uncertainty in cash management
balances for the bank's own day-to-day money market operations and the loss
of goodwill and client defections that may arise as a result of this
debacle.

John Smith wrote:

<snipped>

> JF has a good point here:
> 
> This situation is about process more than anything else, and confidence in
> the process. With bad QC, and in most organizations the headlong rush to
> hurry changes, cut costs, and lower head counts, anything can happen in this
> regard.
> 
> What the article(s) don't say is whether the 'staff' which made the changes
> are permanent full-time, contract, or off-shore, it could be any of these.
> And while one might be tempted to say that code reviews and regression
> testing will catch this sort of thing, often it is not because the really
> good people are often not tempted to stick around given the bureacracy and
> rigid heirarchy.
> 
> This particular organization has many inititatives underway, some with major
> off-shore coding companies for key business systems. I have seen examples of
> the work done by these particular companies in the past in other shops and
> none came in on-time, on-budget, or performed as specifed when first
> implemented.
> 
> Another major initiative also underway with this bank is a complete re-write
> of a critical business system on Tandem, this being done in-house with the
> addition of some contract staff. The bank's IT staff on this project (at the
> time the project was initated) were, as explained to me by the development
> manager in charge, " 2/3rd's of my staff have been here less than 6 months
> and virtually none of them have experience in the line of business this
> application is used for."  They had decided that the best way to manage this
> project was to hire a bean counter 'project manager' first as opposed to
> doing a detailed business analysis first and possibly hiring an experienced
> line-of-business-specific project manager at the same time.
> 
> But this is not uncommon these days in many organizations. Lower head
> counts, a reluctance to spend the right amount of money on the right people,
> the loss of 'institutional knowledge', the desire to 'pump the earnings
> numbers' by hiring lesser qualified consultants base on cost issues rather
> than getting the best, all contribute to situations where these sorts of
> things can, and do, occur. Just because they are banks or other 'reputable'
> institutions does not mean that they don't have their fair share of PHB's
> (pointy-hair bosses) [ see www.dilbert.com ]
> 

<more stuff snipped>

I recently had a conversation with a well educated mechanical engineer - 
owns his own company and knows nothing about IT -- concerning some of 
the idiotic IT "business decisions" being made by major companies. His 
reply was to the effect "you know, heads of companies force their 
management staff to read all of these books on how to succeed and these 
books all say, to one degree or another, to surround yourself and hire 
only the best, most talented people that will almost certainly guarantee 
success.  But when you look at how skilled, experienced people are 
tossed aside so they can make their bottom line look good for a single 
quarter is absolute lunacy." (this was also in reference to off-shore 
programming for critical business systems and applications where the 
idea is to get 2 or 3 OSP for the price of one domestic programmer)

I guess that means in todays market, hire the most talented staff you 
can get for a $1.00 an hour and hope they don't screw it up...

And then they wonder why there are blunders like happend at this bank... 
  I have also seen manglers make "business decisions" to not downgrade 
or upgrade storage firmware that the vendor said "the version you are 
running will cause controller crashes or hangs" -- and even after 
several months and having several of these events, fired the technical 
resource because that resource continued to warn them to allow him to 
fix the problem - not the symptom.  And then look for a guy with only 
2-3 years experience to replace that senior, experienced resource.  And 
this is the recipe for success??? (that's a rhetorical question...)

M.

JF Mezei wrote:
>...
> For them to have a foul up lasting a whole week, preventing direct
> deposits (depriving customers of their money), blocking  debit
> transactions because pension/paychecks were not deposited into
> customer's accounts etc etc is a HUGE problem for a bank. This isn't
> an outage of a few hours, it is something lasting almost a whole week.
>...

Those are just the points they are admitting to publicly. Not only was
my mortgage payment processed three days late, a credit card payment
showed correctly on Tuesday and Wednesday, then it appeared twice for
one day before it was corrected again on Friday. Two days last week they
did not publish their Mutual Funds prices by their usual time and one
day the Mutual Funds section was not available at all. Tuesday, June 1
they sent out a message through their Internet Banking system but that
message had a date of July 3, 2008. The next day the message was deleted
from my read box.

-- 
Peter Weaver
Weaver Consulting Services Inc.
Canadian VAR for CHARON-VAX
www.weaverconsulting.ca

"Peter Weaver" <WeaverConsultingServices@sympatico.ca> wrote in message
news:2ijj3rFo06o7U1@uni-berlin.de...
> Those are just the points they are admitting to publicly. Not only was
> my mortgage payment processed three days late, a credit card payment
> showed correctly on Tuesday and Wednesday, then it appeared twice for
> one day before it was corrected again on Friday.
>
Cascading failures tend to be career-limiting situations for IT managers and
CIOs.  Now the race will be to see if the people who fixed the problem are
shown the door before the people above them are given the opportunity to
explore new career objectives.  Sounds like a major breakdown in daily
operations and no contingency plans in place.
  Jack Peacock