jerryhung

I actually think IBM are at fault.

They are the experts who were brought in to advise on, and implement the system.

They should have understood the configuration options, what the software could do out of the box, and which features would require customization. They should also have run stringent testing cycles to ensure that user requirements were being met.

I can't believe IBM got away with this **** show tbh.

jerryhung

Yes I've seen bits and bobs but it could also be my own lack of attention to it. "Phoenix systems" is not a very catchy headliner for me.

Incidentally I googled it after this thread, and apparently some union reps are asking the Federal Government to compensate them for pain and stress associated with this system.

I think the real issue here is that the contract with IBM was poorly worded and the Harper Government and successive Governments (Liberal Trudeau now) don't know how to set up contracts like this and now don't know how to get out of this mess.

Too bad because competent professionals should have been able to do it for them within a six month timeline.

Not sure if they are still employing IBM and if so, then IBM is just milking this for all its worth.

jerryhung

OK I've read more about this issue:

IBM contract cost for failure-plagued Phoenix payroll system jumped to total $185M - Ottawa - CBC News

https://ipolitics.ca/2017/11/20/ags-...-system-fixed/

Phoenix creators helped build failed pay system's business case - Ottawa - CBC News

IBM doesn't take a hit at all - unbelievable.

They recommended the off the shelf software and that this was the optimal solution for the Government.

The 2007 IBM study pointed out that custom, off-the-shelf software systems are "consistently more cost effective and enable higher quality and efficiency, when implemented and sustained properly."

This was exactly the kind of system that IBM was hired to implement for Phoenix.

IBM also noted the government's old system was at risk of failure. Its report warned that these "payroll errors can have significant consequences for both the financial picture of the organization and talent retention."

And yet they didn't provide the options for the client to consider, outline the implications of those options, take the client through the different phases of testing which would have calibrated, and highlighted the impacts of the business requirements/decisions.

And now a $5M contract has blown up to >$150M and can go up to $1BN.

Wow, the incompetence is amazing and the fact that IBM have gotten away with this scott free and with a significant revenue boost is (INSERT WORD)

jerryhung

Also - to continue my RANT:

How did the users sign off user acceptance testing?

With go-live and the conversion of the old system to the new, why weren't these problems detected?

How in the world did IBM get away with turning off the old system, with the state of the new system being in such a mess

As to that silly excuse "training issue" it is always the vendor's responsibility to train the users competently so there is no further reliance on the consultant.

I think IBM have taken Canada for a wild ride and no-one seems to have the experience or expertise to know any better.

Gosh, IBM must love Canadians.

BristolUK

Canadians or government and its officials?

Just wondering because I'm familiar with this sort of cock up in the UK - seemingly just with government departments as I don't recall hearing about such things in the private sector.

My old dept gave a contract to an IT company who already failed badly with another government dept and it seemed that every reorganisation or change in benefits, a new IT contract was needed and there were always big failures.

The powers that be were repeatedly told that what they wanted to do needed more/different/better etc or it wouldn't work but they always blundered on regardless, trying to get things done on the cheap and paying for the mistakes in the long run but never learning from the mistakes.

One can only assume they got nice little semi retirement jobs for the companies they awarded contracts to.

dave_j

Quote from Debi Daviau, a union president of the Professional Institute of the Public Service of Canada who is also a computer systems analyst who had private-sector experience before going to work for the government.

“When they say IBM is adhering to the terms of the contract, it’s true,the government just didn’t scope out the contract properly.”

At a parliamentary committee hearing in September, Ms. Foote said the Conservatives had skimped on training for the 500 workers hired to run Phoenix, “versus actually buying into what IBM had advocated as the amount of training that really needed to take place.”

also worth a read https://www.itworldcanada.com/articl...roblems/396407

jerryhung

Latter might be a bit worse but things can be a bit backdated here, I find. I doubt they know what the system was about, which is why IBM got away with what they did IMHO.

jerryhung

Thanks. Dave. I read somewhere (I think) the Govt still has hundreds of payroll processing rules which is why so many things need to be manually processed.

THIS itself is a huge (or huuuugggggeee) indicator that this was never going to be a "simple vanilla" implementation.

IBM misled the buyers who had no clue what they were getting into, nor how to reign in this software group.

As to the training, 500 to run the software (haha) sorry I can't get over this fleece job. In a typical case, training should have been throughout the project - advancing the skills, knowledge and capabilities of in house staff who could also later train others.

The general all user & other training is obviously important but not more important than an effective and easy to understand/intuitive system, advanced users/administrators of the system trained over many months of the project, simplified rules/processes and effective training/processing manuals.

dave_j

Imagine you want to update your bathroom. You instruct your plumber, I want everything replaced with new. You go away for a week and when you return you find that you have a bright blue bath instead of the shiny white one you wanted. You argue with the plumber.. 'But', he points out reasonably, 'you didn't tell me what colour you wanted and I like blue.'

The issue is that the job wasn't 'scoped properly'.

A company I worked for knew that all jobs it undertook would generally cost double for the client because they were so poor at describing what they wanted the contractor to do. They always left out important details and the phrase 'but I thought that was in the contract..' was often heard only to find that it wasn't.. with one notable exception.

We obtained a specification for a large project from China. The specification was so detailed that it encompassed several large volumes and we couldn't fault it. If they could do it why not others?

Don't blame the contractor if the client employs people who are so out of their depth that they shouldn't be doing the job. The fact that IBM was the only company to quote for the job speaks volumes for the manner in which the task was approached in the first place.

The political imperitives to rush the system into operation against the advice of IBM, the main contractor, and then to ignore their advice as to training should be a matter for criminal negligence.

'Ah' I hear you repeat, 'IBM were the experts here and recommended the system'. This may be so but like the plumber they only do what's included in the contract and there was a qualification that they attached to their 2007 study pointing out that custom, off-the-shelf software systems are "consistently more cost effective and enable higher quality and efficiency, when implemented and sustained properly." No doubt IBM can argue that rushing the system into use for political reasons when problems were still present and ignoring their recommendations as to training was not implementing or sustaining the system properly.

The resulting chaos has been blamed on training, data input, system failures, IBM etc etc and this has created an enormous backlog of pay issues and this was against a background of a political decision to sack 2700 people who ran the old system in order to demonstrate a saving thereby removing any safety net.

I worked in a brand new plant incorporating a novel system. It replaced an older version that worked well but was considered more efficient. A decision had been made not to incorporate a backup to ensure that the new system would have to be made to work. It didn't and the end result was a complete destruction of the plant. Plus ca change.....

Former Lancastrian

Very well put Dave J. I think when it comes to the Canadian Govt and contracts for large, medium or small projects is not always the greatest. I think the Brits are still pissing themselves over the subs that they sold them.
Have we got our new planes yet? The Aussies are trying their best to suppress their laughter as to selling them their 2nd hand fighter jets.
Occasionally we get things right though.

dave_j

I'm not running down the canadian government here. It's always the client who decides what the contractor is and is not responsible and being paid to do and most get it wrong. Reading between the lines here IBM was not responsible for training and the government made some serious mistakes in how and when it brought the system into operation.

To my mind IBM itself probably underestimated the difficulty involved. I doubt that the initial study was sufficiently deep or detailed enough to describe the complexity of the pay structures involved and I doubt those in charge on the governmental side knew enough about the task to advise or warn IBM during the contract negotiations. It no doubt come as an unpleasant surprise to everyone involved.

Former Lancastrian

I am Look its the Government and most bidders/contractors know that if you land a contract with them the sky is the limit for them. I wonder when the last contract ever came in below or at least cost and ran smoothly? They cannot usually sole source out a contract. Even the changing of our uniforms was a complete boondoggle and its still just as bad although somewhat better than a few years ago. OK time to order a pair of pants put in size 36" waist for example. Receive said pants then it was a toss up if you could actually manage to get them on or you could fit 2 of you in them. Quick look at the label Made in Vietnam.
Shirts don't even get me started on those things.

dave_j

It shouldn't be..... Where do I apply to supply shirts.... "Got a lockup full of khaki ones, got a few 'oles in 'em and they're a bit scratchy but once deloused they'll do a treat. Save me 'avin to feed 'em to the pigs. Worf a pony for a good contact"

jerryhung

A well thought out post as usual Dave Thanks.

Let me add my thoughts to this, and I will point out that I have been involved in these systems before so I am somewhat biased.

And this is why it is up to the expert - not a plumber - but a professional services company with the expertise to do so - the basic outline of the system.

i.e. It is also up to IBM to have produced a decent scope document too - by speaking to the client, reviewing basic documentation etc in order to understand that this system would NEVER have been a basic vanilla implementation. Retaining hundreds of different rules immediately means and meant that there is no way it is a candidate for an off the shelf type of implementation.

In your scenario, they would actually ask the client, "do you want the bathroom in a specific color?" or say "oohhh you want 500 different colors all in square blocks along the bathroom - looks like we are not talking about the basic Behr project I quoted you for"


Having said that YES! I have seen here in Canada where professional firms seem to get away with this type of incompetence. I see your reasoning, but perhaps I am just not used to them getting away with blaming the lack of knowledge/ability to scope/specify as a client issue.

i.e. To be fair to the Government, the skillset of large scale systems implementations is a niche and specific skillset. I've seen firms here where firms have very little clue about IT related matters and IT partners fleece firms like plumbers can do to me

More seriously, it is very specialized and you can't (IMO) expect the client to know how to translate their understanding of business processes to an initial scoping document.

Exactly. And your firm knew that and kept accepting dumbed down specifications - knowing that they would double their money down the road. Not the sort of firm I support but in this case with IBM, as a professional services firm which is not only selling software but the expertise and professional advice around it, clients should be getting more. And they do in other countries.

Disagree. ERP is a specialized skillset. Having said that, AGREE that the Feds should have hired people off the market with expertise to help them but I guess they just trusted IBM haha.

Yes this was an interesting one. Typically firms jump all over government contracts. I wonder why they didn't. This could be indicative of what you say below:

The political imperitives to rush the system into operation against the advice of IBM, the main contractor, and then to ignore their advice as to training should be a matter for criminal negligence.

Can agree on the political imperatives part. I think it was the Liberals who made the decision, is that right?

No I don't buy this. The system still has 800+ rules. That in itself shows it is in NO WAY SHAPE OR FORM a off the shelf implementation - nor is it the candidate for it. The fact is IBM misled the Government strongly on this one. Clients which implement it "vanilla" are ones which basically adopt ALL the processes of the system - and change their businesses etc to fit the system. IF the client wants to keep existing processes and rules etc and the rules are as voluminous and complex as the above, this is immediately a system which is by definition non-vanilla and/or defective by recommendation.

Stupid Harper initiative then. They just saw the $ and didn't understand what they were getting into, and got fleeced into thinking that this was a slam dunk business case. Pathetic really. ERP systems are more complex than slam dunking a not-fit-for-purpose, overly complex, manual, inefficient, ineffective system and expecting 2 weeks of training for 500 people at the end of the day is the answer to it.

Most likely the problems are a combination of factors and "training, data input, system issues" are all accurate - however I maintain that they received the wrong advice and wrong marketing spiel from IBM. Where the Feds take responsibility is their naivety and lack of knowledge/expertise - so again agreeing with you on the political drivers and the "Wish" to save $ - result: thousands of workers out of pocket and stressed and IBM profiting like crazy over a botched up job (not for them of course)

Yes of course but again this comes down to expertise and professionalism. If IBM were hired to implement the system - as they were - they were not hired to be brainless code monkeys - they were hired for their professional expertise, knowledge, experience and competence to help the client deliver a proper product. From the initial sales job they botched it up - for the client. And the result is not JUST the Government's fault, it's IBM's too without a doubt. They should have known - and they did so they were either too scared to tell the Government what they didn't want to hear, and/or didn't have the real expertise/skillset/experience/knowledge to advise and implement properly and/or knew but didn't care as they knew this would increase their revenue stream substantially down the road. Either one is an issue IMO and IBM are culpable here.

YMMV

jerryhung

Probably because IBM sold it as some simple, effective, efficient solution with a compelling business case (2700 Headcount down to 500) - and the silly Feds thought it was easy to train. again bad advice from IBM.

Yes agree sounds like IBM didn't scope at all to be honest. Even a cursory questionnaire or interview would have demonstrated that their proposal was a joke.