- 注册
- 2002-10-07
- 消息
- 402,176
- 荣誉分数
- 76
- 声望点数
- 0
Liseanne Forand really didn’t know what to expect. On Aug. 4, 2011 she was making her way on foot to Phase III of the Gatineau office complex known as Place du Portage.
A career bureaucrat, Forand had, the day before, been appointed president of Shared Services Canada — a new federal department that would manage the government’s email, data centres and telecommunications.
Her mandate — representing about one-third of the federal government’s $5-billion-a-year technology services budget — was breathtaking in its scope and complexity. She was to simultaneously streamline and modernize the government’s electronic backbone, and keep the old gear running. Sixty-three email systems would be collapsed into one. More than 500 data centres were to be decommissioned, to be replaced by a mere handful. Fifty telecommunications networks connecting 3,500 federal buildings were to be upgraded.
Forand was to do this with staff cobbled together from 43 different departments. But on this day, she had no office, and the paperwork had been approved for just 1,200 of an eventual 6,000 employees.
Rona Ambrose, former Public Services Minister, was present at the agency’s launch.
As she entered Phase III, she was surprised, seemingly touched, to see that a small army of government workers had gathered in the main foyer to greet her.
After introductions, she was escorted to the 17th floor. There, Forand found a temporary office. She sat down and began making lists of things to do.
As Forand tapped out her notes, Public Services minister Rona Ambrose and Treasury Board president Tony Clement were in the National Press Theatre, directly across from Parliament’s venerable West Block.
Ambrose said she and Clement were there to introduce “new measures that will improve the efficiency of information technology services.” Shared Services would lead this effort, Ambrose said, and generate “substantial savings” through economies of scale.
It was up to Forand to flesh out a strategy that had little substance at launch.
Not that a more detailed plan would have mattered.
What Forand didn’t realize was that the seeds of failure were already being sown. Prime Minister Stephen Harper was preparing a government-wide spending strategy that would hobble Shared Services before it could even get going.
Related
It’s rare that issues involving information technology intrude on Ottawa’s political agenda — rarer still that they do so with the force displayed in 2016.
All of Shared Services’ core projects are behind schedule, substantially so, in the case of email transformation. Not only have promised savings not materialized, the agency is spending tens of millions of dollars more than forecast maintaining networks that should by now have been decommissioned.
But it’s not just Shared Services that’s struggled. Federal departments — from Agriculture and Agri-Food Canada to Veterans Affairs Canada — are upgrading software and IT infrastructure that falls outside the Shared Services mandate.
Some, such as the $300-million project to modernize the government-wide pay system known as Phoenix, are fumbling badly.
At one level, Shared Services and Phoenix are very different. The first is a new department with a specific mandate; the second is a single project — albeit a large one — being managed by Public Services. Indeed, Phoenix is one of nearly two-dozen major IT projects underway across government.
But Shared Services and Phoenix have something in common — a botched introduction caused, it appears, by deep flaws in how government operates. In both cases, cabinet ministers and bureaucrats underestimated complexity and risk. In this, they were hardly unique — it was the scale of the misjudgment that set the federal IT agenda apart.
Standish Group, a Boston-based consulting firm, has been tracking the performance of IT projects since the mid-1990s — with surprisingly little variation in results. The consultants last year examined 50,000 projects worldwide, including government and private sector. About 30 per cent of these efforts succeeded — that is, they were on time, on budget and produced a payoff. Roughly half the projects ran into difficulty and nearly 20 per cent failed outright. The larger and more complex the project, the higher the rate of failure.
Carol Bellringer, the auditor general for the B.C. government, last month offered three key reasons why IT projects fail: Government departments, she said, lack in-house expertise; they attempt “overly ambitious” programs; they justify the latter through “incomplete” business cases.
All three elements were present at the launch of Shared Services. Most of the responsible bureaucrats were not trained in IT, yet were tasked with remaking on the country’s electronic infrastructure. Many also lacked experience in project management with a heavy IT component.
“I don’t know how many times I heard from deputy ministers that they didn’t understand information technology,” said a senior Shared Services official, “They didn’t like IT and they hoped never to see anything to do with IT for the rest of their career.”
Yet it is a group of deputy ministers — the ones in charge of the most IT-intensive departments — who determine the shape and scope of large IT projects. And when it came to launching Shared Services — the centrepiece of the government’s online renewal — the already high risks were exacerbated by a political agenda that stripped it of the capital necessary to get the job done.
It will likely end up costing taxpayers a fortune to set things right again.
Detail from one of the many legacy data centres in the National Capital Region.
It had seemed so simple in the beginning. The idea for Shared Services emerged from the Conservatives’ fifth budget, tabled March 4, 2010. The themes were clear: The economic recession was over; it was time to regain control of government spending.
One aspect of the strategy — little noticed at the time — was the launch of a “comprehensive review” of government spending on administration and overhead expenses. This should have offered easy pickings: Federal government employment was near high tide; and most departments and agencies had expanded rapidly.
Daniel Jean, the deputy secretary to the cabinet of the Privy Council Office, was picked to run the review, making it a big deal. The PCO is home to 950 bureaucrats who provide advice to cabinet and the Prime Minister’s Office, and oversee the development of the civil service.
Jean reported directly to Wayne Wouters (pronounced “Waters”) — the clerk of the privy council and the government’s top bureaucrat. Among the members of the review helping out Jean were Benoit Long, a senior manager seconded from the office of the government’s chief technology officer, and Liseanne Forand, then chief operating officer of Service Canada. The latter department offers Canadians online access to pensions and employment insurance.
Wayne Wouters, Former Clerk of the Privy Council, instrumental in Shared Services’ creation.
The administrative services review was carried out in secret, typical PCO modus operandi. Its members roamed the bureaucracy, collecting information and searching for ways to consolidate or standardize how things were done. Some departments were already moving down this path.
The Department of National Defence, for instance, calculated it was spending $30 million annually more than necessary because its 16,000 Ottawa-area employees were inefficiently spread across more than 40 facilities. Late in 2010, the government bought the former Nortel campus to allow for a major consolidation.
Information technology offered an even richer vein of potential savings. For half a century, computer networks and software applications had multiplied willy-nilly as individual departments and agencies looked after their own needs. The result was a patchwork of incompatible, higher-cost systems. Standardizing common, basic technologies such as email, data storage and telecommunications seemed logical.
It had been tried before. But attempts to centralize the buying of high-tech gear and services had failed, largely because federal departments were allowed to opt out. Most did so. They did want to give up control of their IT networks to a central agency.
The PCO determined this time would be different. The prime minister had the authority to create a new federal department through a simple cabinet approval known as an order-in-council. Most departments, including a reluctant Canada Revenue Agency and Department of National Defence, would be forced to carve out a significant portion of their IT groups and budgets — about 40 per cent on average — and hand them over to Shared Services.
Crucially, the move would not be subject to scrutiny by Parliament. And so Shared Services was born on Aug. 3, 2011.
Speed was demanded of the agency from the start. Minutes of meetings involving senior Shared Services staff are studded with references to “tight schedules” and the “urgency” of getting projects done.
Part of that had to do with the sheer age of the government’s infrastructure. The hardware was in danger of breaking down and the underlying software for many applications was so old that suppliers such as Microsoft, PeopleSoft and Adobe had stopped supporting it.
The faster Shared Services could install new networks, the less money it would be forced to throw at solving the problems caused by older technology.
But that wasn’t the only reason Shared Services was pressed for time. Senior Shared Services officials said the Conservatives were eager to see cost savings, and impressed upon them the importance of securing an early win.
The PCO framed the upgrade in simple terms: Consolidate, modernize and reap the savings. And Shared Services would have nearly a decade to get it done. By 2020, the thinking went, the government of Canada would be able to offer its citizens secure, online services that would be the envy of the world; and Shared Services would be a magnet for attracting the best and the brightest employees in government.
But cabinet — and to some extent the PCO — failed to account for the complexity. They were proposing to create a new organization using bits and pieces from other departments and loading it up with a series of mandates on a tight schedule. The entire production was fraught with risk.
“They had articulated the problem and come up with an organizational response (in the form of Shared Services),” an independent adviser to the PCO said in an interview, “but they completely underestimated the scale.”
Start with the logistics. Forand would not receive the bulk of her new workforce until mid-November in 2011. And it wasn’t just a matter of transferring 4,000-plus workers from 43 departments. Shared Services didn’t immediately have any place to put them. The employees in question were in more than 500 locations across the country — and worked out of 80 buildings in the national capital region alone. They would remain in place for the moment.
Liseanne Forand, the first president of Shared Services, was tripped up by significant and early budget cuts.
Another problem for Shared Services was that it had little control over just who would join it. That decision was left up to the departments. During her testimony earlier this year before a House of Commons committee, Forand pointed out that one of the larger departments sent more than 1,000 IT staffers but included just three executives. “If that’s true for IT staff, you can imagine what it was for administrative staff,” she said. It was difficult to imagine “a single organization ever transferring a really crackerjack finance person or human resources or auditor.”
Indeed, in order to fill Shared Services’ executive ranks, Forand would later have to advertise for at least a dozen senior managers.
Shifting Shared Services employees into new facilities was a miserable, 17-step process that required nearly a year for each move. The agency reckoned it would take as long as five years to complete the transfer of bodies and responsibilities.
Nor did Forand have access to her chief operating officer right away. Because the formation of Shared Services had been shrouded in such secrecy, the hiring of experienced outsiders happened last minute. Grant Westcott would join Shared Services as COO in mid-September.
Sixty-three years old at the time, Wescott had served as assistant deputy minister of Public Services in the late 1990s, when he managed the federal government’s telecommunications networks. In that role, he oversaw the successful effort to prepare systems for the year 2000.
More relevant to his new position at Shared Services, Westcott had also worked for nearly a decade at CIBC. As executive vice-president, he had been instrumental in overhauling the bank’s IT infrastructure. Twenty-two of the Canadian Imperial Bank of Commerce’s data centres were shrunk into just two, saving the bank more than $40 million annually in operating costs. Among other things, Westcott and his team consolidated 15 global communications networks into a single voice and data system. More savings resulted.
It wasn’t without glitches. CIBC was forced on several occasions to apologize to customers for a breakdown in certain banking services.
Nevertheless, Westcott’s experience suggested this was not rocket science, at least not in the private sector. Streamlining and modernizing IT networks could be done under the right conditions. But, just weeks after he started, Shared Services was jolted by its first big shock.
It came in the form of a letter from Treasury Board, the cabinet committee responsible for managing the government’s finances. The October 2011 note outlined how the Conservative government was planning significant spending cuts in the upcoming federal budget. Shared Services was expected to contribute its share.
That provoked astonishment among Shared Services’ executives; afterall, the agency wasn’t even close to being fully formed.
Less than one-fifth of Shared Services’ expected workforce had been formally transferred on paper, though they continued working out of their old offices. Forand and her immediate support staff were working out of temporary headquarters, and the agency’s financial books were not yet in order. Upon its formation, Shared Services had been allocated an annual base budget — about $1.5 billion. But it was being asked to trim this in stages by five per cent, 7.5 per cent, then 10 per cent over the next three years. Treasury Board was seeking $150 million in total, with the proceeds to help reduce the federal government’s deficit.
Forand, according to access-to-information documents, found early savings by renegotiating telecommunications contracts, eliminating telephone lines through consolidation, and leaving open unfilled staff positions. She also pencilled in savings that would result from the consolidation of government email systems and data centres — these would be achieved starting in the third year of the Conservatives’ deficit reduction program.
The agency knew it was just buying time, however. Forand, Westcott and other top managers had assumed prior to reading that Treasury Board letter that any savings realized by consolidating Shared Services’ electronic infrastructure could be directed toward modernization efforts. In other words, the potential savings were not only an incentive to be efficient, they would serve as seed capital for renewal.
Instead, the Conservatives grabbed the money.
“We did convey strongly at a high level that reductions of that magnitude would create increased pressure and risk,” said one official familiar with the Treasury Board request, “We couldn’t quite believe they would take the whole 10 per cent and were shocked when they did.”
From that moment, Shared Services was forced into an excruciating series of financial tradeoffs.
The agency no longer had the capital to simultaneously modernize its networks and keep the older (legacy) stuff running. The legacy gear got short shrift with inevitable results.
In recent months, the government’s networks have been plagued by a series of outages that have left thousands of government employees without access to online services and applications, and brought down government websites. A major culprit: power distribution units that have been pushed well beyond their normal lifespan.
The Conservatives, meanwhile, wanted their early win, so they convinced Shared Services that the quickest and easiest project would be to replace the government’s massive email network. While email technology can be complicated, the service is considered something of a utility in the IT world. It shouldn’t have been a huge problem. Instead, things would go badly wrong.
Shared Services Canada Organizational Chart
Minister responsible for Shared Services: Judy Foote, Minister of Public Services and Procurement
President: Ron Parker
Chief Operating Officer: John Glowacki
Assistant deputy minister Cyber & IT Security: Raj Thuppal
ADM, Data Centre Services: Patrice Rondeau
ADM, Network and End Users: Pankaj Sehgal
When government procurement officials invited potential email suppliers for a technical briefing on June 12, 2012, the project appeared daunting but doable. The date proved significant: Shared Services would be granted the authority to do its own buying a couple of weeks later. But because the email competition was already underway, Public Services — the government’s traditional procurement authority — would handle it.
The 43 federal departments linked by Shared Services’ networks collectively were using dozens of separate email systems — most of them using Microsoft technology. These were to be collapsed onto a single system, also based on Microsoft software. The new setup would require fewer people to operate it, contributing to $50 million per year in savings when it was up and running.
The winning bidder in this contest would be responsible for transferring more than 635,000 electronic mailboxes on behalf of 378,000 employees to a new, outsourced data centre.
When Bell Canada and CGI Group (using Microsoft email) won the competition in 2013, Public Services minister Rona Ambrose said the new $400-million system would be in place by March 31, 2015.
Behind the scenes, there was a much different view. Losing contenders were stunned by the differences in the submitted bids.
Bell said it could develop and run the initial part of the government’s single email system for $245 million. IBM was the next lowest bidder ($282 million), followed by HP ($368 million) and Dell ($571 million).
Bid evaluators from Public Services had given technical merit a 30 per cent weighting, compared to 70 per cent for price. Thus Dell, which had the best-ranked technical proposal, really didn’t have a chance.
The bitterness of the finalists can be seen in debriefing notes obtained by the Citizen. In them, officials from HP complain that their company’s “governance does not allow for under-bidding.” HP and Dell had included generous amounts in their proposals to take account of risks and contingencies.
Losing bidders warned they would monitor the single email project closely to make sure Bell met its obligations at the promised price. But if they believed Bell’s potential difficulties would somehow escape public scrutiny, they needn’t have worried.
It didn’t take long for frictions to surface within Bell’s engineering group. By August — just two months after the awarding of the contract — Bell replaced its entire system for establishing the identities and credentials of employees using its email system.
The technology — that, among other things, allowed workers to manage their email accounts and passwords — had been implemented three years earlier on a separate federal government contract. Bell had adapted this technology for its winning bid. Now it was proposing to put in place a new approach. It’s not clear why, though it appears to have been an internal decision, not one forced upon Bell by Shared Services — according to a programmer familiar with the re-engineering effort. Bell declined comment on this aspect of the email project.
Bell wasn’t the only member of the winning team to experience technical difficulties. It was soon apparent that Microsoft’s latest email software — Outlook 2013 — had an unexpected limitation. According to a memo prepared for Forand — and obtained under an access-to-information request — Outlook 2013 could support a maximum of just 10,000 public email folders. Data gathered by Shared Services had revealed more than 800,000 folders in use across government.
The limitation would be dealt with by using an older version of Outlook.
It was clear within months of the launch of the email project that Bell wouldn’t be able to shift, or migrate, the government email addresses as promised to its new data centre in Buckingham, Que. The original plan was to spend six months working out details of the overall design. The migration of employee mailboxes was to have been done in three waves, starting in January 2014 and ending in March 2015.
But Shared Services granted Bell a 12-week extension in December 2013 for the start of the transition. Two months later, Bell asked for and received an additional four weeks’ grace “to address all outstanding design issues and to ensure a robust, secure and integrated solution.” This, again, according to documents obtained through access-to-information requests.
By November 2014, the email project had been flagged “red” by Shared Services. That meant it was in serious difficulty of meeting its final deadline of March 31, 2015.
Indeed, a year ago, when the project was six months late, Bell had successfully moved fewer than 15 per cent of the government’s mailboxes to the company facility in Buckingham. Bell at that time was forced to halt all migrations after identifying what it called a “hardware” issue (which has since been resolved, according to Shared Services). Shared Services says Bell must resolve “outstanding contracted issues” before shifting the other government mailboxes.
“All elements of the ETI (email) platform have been running smoothly for departments already migrated,” Bell spokeswoman Jacqueline Michelis said this week in response to a Citizen query. “We look forward to resuming migrations as soon as possible.”
It’s not clear when this will be.
“We’re continuing to discuss issues with Bell,” Shared Services president Ron Parker said this week.
“There’s no end date,” added Parker, who took over from Forand in July 2015.
Ron Parker took over as president of Shared Services Canada in July 2015. He was surprised at the extent of the budget shortfall.
While Shared Services is paying Bell only for the mailboxes it has already moved, the agency is still responsible for keeping the lights on at the government’s legacy data centres — where most of the electronic mailboxes, and much of the government’s software applications, for that matter, are currently stored.
These jumbles of computers, servers, wires and cooling systems are weighed down by far too much history — and it took Shared Services a long time to get its arms around it.
“We were not created with a transformation plan already in place because to do that, we needed to know what we had,” Forand explained to members of Parliament. “So we spent a year counting things.”
Shared Services didn’t have to start from scratch. Prior to the agency’s 2011 launch, the PCO’s administrative review tried to estimate the number of data centres in government by interviewing the IT czars for each of the departments. Symptomatic of the far-flung nature of the government’s IT networks — and how loosely these were managed — the PCO didn’t even come close.
“When I was at Privy Council, we thought there might be about 200 data centres,” Benoit Long, senior assistant deputy minister of Shared Services told the House of Commons committee last May, “After a year, we had counted 495 and I am still discovering others today.”
Most of these data centres were small — less than 1,000 square feet — often set up by government scientists who didn’t bother telling their superiors. Just 22 were larger than 5,000 square feet — and most of these were in the National Capital Region. These were where the bulk of the government’s data and software applications were stored.
In 2012, Grant Westcott organized a six-week tour of these data centres to see for himself what Shared Services was dealing with. According to an agency employee familiar with the tour’s findings, Westcott came away profoundly shocked.
Nearly all the large data centres were decrepit, undercapitalized and inefficient users of energy. Yet just one was in the process of being replaced. This was a Heron Road facility with a leaking roof, considered key because it housed Canada Revenue Agency data.
Bell Canada had been awarded a contract to build and operate a new data centre in Buckingham. The CRA data would be transferred there in the fall of 2013. A separate part of that facility is also meant to host the government’s email system.
The remaining data centres offered Westcott a depressing snapshot.
Most of the gear was housed in office towers, rather than facilities tailormade for heavy-duty electronics. With each new generation of computer or servers came requirements for more power and more robust cooling systems. The entire system was approaching a nervous breakdown.
Not only were the data centres inherited by Shared Services running out of space, the arrangements for emergency power bordered on makeshift. One Defence Department data centre, for instance, relied a backup generator powered by a relatively ancient jet engine. While unconventional, there was nothing unique about pressing military hardware into this type of service. The risk had to do with with training — the older the jet engine, the fewer employees there were who knew how to fix it.
The system for backing up data wasn’t much better. At Statistics Canada — where Westcott’s group was actually required to swear an oath to keep the data confidential before entering the storage rooms — the backup procedure called for storing copies of the data in a separate but nearby facility. While that’s OK for most disasters, the relative proximity still meant Statcan’s repository of historical data was vulnerable to a bombing or major earthquake.
In the wake of Westcott’s tour, Shared Services developed a plan for refurbishing or building new data centres with plenty of distance between them. These include facilities at Buckingham, Ottawa’s international airport, Barrie, Borden and Dorval, Que.
Nevertheless, the Buckingham data centre wouldn’t be ready until late in 2013 and the contract to set up the Barrie facility, run by IBM, wasn’t awarded until the fall of 2014. A public-private partnership to build a large data centre in Borden was signed last May. A separate data centre focused on weather forecasting and other science-related data will be based near Dorval.
These facilities require careful calibration — the gear and supporting wiring and cooling systems have to be capable of handling the software applications and data coming their way.
Collectively, the 43 federal departments run 14,000 software applications, covering everything from human resources software to weather forecasting. These are all meant to migrate from the legacy data centres by 2020. But that milestone very likely will not be met.
Shared Services has had to spend much more than expected to keep legacy data centres going while new ones are built and equipped.
Minutes from a meeting of Shared Services’ senior managers held Sept. 29, 2015 give a flavour of the difficulties. Patrice Rondeau, in charge of the agency’s data centre efforts, told his colleagues that he had received “a large number of exemption requests” from federal departments that wanted to upgrade their legacy data centres rather than transfer software and data to the new ones operated by Shared Services.
This speaks volumes considering the age of the legacy data centres. The Citizen this week toured one of the larger ones. Located near the ByWard Market, the data centre was launched in 1985 and currently houses data and software on behalf of Elections Canada, Canada Mortgage and Housing Corp., Employment and Social Development Canada, among others.
The facility is clean enough, appears to have adequate backup generators and extra space for additional gear if necessary. But a closer look reveals the weak points — it relies on 11 units for distributing power, and most have been pushed beyond their average life expectancy. A couple broke down recently, temporarily disabling the facility. The other units, who knows? “They’re only now starting to fail,” explained one data centre employee.
The limiting factor for storage is not the physical space; it’s the amount of power available to feed the servers. And power is near its maximum.
The data centre continues to accept new workloads (data, software applications) from the departments it serves but there is now a quid pro quo — the departments have to get rid old data before facility will add the new stuff. “We’re pretty much full,” said the employee.
There is some hope on the horizon, albeit distant. The Shared Services half of Bell’s data centre in Buckingham appears to be successfully hosting data from several federal departments. The IBM facility in Barrie is finally accepting workloads — though still representing less than five per cent of the data centre’s capacity. Construction of a new data centre in Borden is well underway and a new science-and-weather focused facility is planned for near Dorval — though bids have yet to be solicited.
Yet, even now, the legacy data centres are the preferred option for many of the government’s IT managers. Many are still unimpressed by the untried agency.
The auditor general pointed out early this year that Shared Services had failed to establish the level of service departments should be able to expect. For instance, in the event of outages, how quickly would Shared Services get the systems running again, and how many hours per day was such assistance available?
Nor, the auditor general reported, did Shared Services give reports on the overall health of its networks. A Shared Services executive acknowledged recently customer service is improving are “on a very slow trajectory.”
This much is evident on a large schematic posted on a wall on the seventh floor of the Shared Services headquarters at 99 Metcalfe St. in downtown Ottawa. It maps out feedback from the departments it serves on a quarterly basis. Green is good. Orange and Red are bad. Thirteen are green, 18 orange and red, with the remaining departments neutral. Overall, level of satisfaction shows little change over the past year.
Westcott left Shared Services in the spring of 2015, having extended his three-year stint by seven months. He returned to his vineyard operation just west of St. Catharines. Forand followed him out of the door a few months later into retirement. Aside from their joint testimony last May, the former top guns at Shared Services have not been public about their experiences. Privately, with colleagues, they have expressed frustration with the slow pace of getting things done in government, often related to multiple tiers of oversight by Treasury Board, Public Services, the Privy Council Office and other departments affected by Shared Services’ plans.
But the loss of so much of their annual budget upfront undeniably hurt the agency’s drive to modernize. When queried about whether he had second thoughts about trimming Shared Services’ early budgets, former Conservative Treasury Board president Tony Clement told the Citizen “I wasn’t the minister (responsible) for Shared Services. And all the mistakes were made by the Liberals in any event.”
Treasury Board did play an important role in establishing Shared Services and influencing its early rollout. Rona Ambrose, the former Conservative minister of Public Services — and the minister directly responsible — declined to comment.
The Liberals were more than a little surprised at the state of affairs at Shared Services. Nevertheless they couldn’t lay all the blame at the foot of the Conservatives.
Investing in information technology at the federal level had not been a priority for decades — the government typically spends less than two per cent of its annual revenues on it, compared to more than 10 per cent for large, information-intensive organizations such as Canada’s big banks.
Quite simply, the bill for under investing is now coming due.
Parker, Shared Services’ new president, found out as much during his first briefings, when he learned his department’s “projected expenses were much greater than expected.” Indeed, a July 2015 briefing note obtained under access-to-information requests warned him the data centres branch alone was facing a $200-million shortfall.
Finance Minister Bill Morneau in his first budget added nearly $400 million to Shared Services’ coffers. It’s meant to keep the older gear going years longer than planned.
Finance Minister Bill Morneau early this year allocated a two-year lump sum of nearly $400 million in his first budget to help Shared Services keep the legacy networks operating until the new data centres are ready. Any pretense that transforming the government’s electronic networks would produce immediate savings is gone.
Will the investment be enough? Probably not. The problems are compounding. Delays in the construction of the new electronic backbone have pushed back the rollout of hundreds of software programs now overdue for replacement. Indeed, over the next few years, thousands of applications are scheduled to migrate from legacy data centres to Shared Services’ new sites. It may prove the most difficult phase of the government’s IT planned transformation.
Consultants say departments are impatient at the thought of further delay, and have been pressing for the right to move more of their applications to cloud-based (third party) services such as those offered by Amazon Web Services. Some, such as those involving websites used by the public are already hosted in the United States (a Treasury Board project). But sensitive information and software — for reasons of security and data sovereignty policy — should be hosted from data centres or cloud services supported from Canadian soil.
Just how this will play out should be made clear in coming weeks, as Shared Services prepares a revised plan for approval by cabinet.
With any luck, it will incorporate the many lessons learned to date — above all, that you cannot book savings first, then build the system.
Rebuilding electronic infrastructure requires serious upfront investment, followed by careful attention to detail. In a complex project, it is better to take small steps, be sure of your ground, then move on. Because trying to fix something on the fly costs far more than getting it right from the beginning.
A Shared Service executive said of the journey to date. “The private sector is really good at understanding you need to fix the infrastructure first,” he said. “The (software) applications that ride on that infrastructure, that’s really hard to do. But if you try to do both at the same time, it’s overwhelming.”
So it proved.
April 6, 2010: That review is to be carried out in secret by bureaucrats operating out of the Privy Council Office – which provides advice to cabinet and the prime ministers’ office. The team handling the administrative services review, led by Daniel Jean, reports directly to Clerk of the Privy Council Wayne Wouters.
Aug., 2011: Public Works Minister Rona Ambrose and Treasury Board President Tony Clement announce the launch Shared Services Canada. This new federal agency will take over government-wide email service, data centre operations and telecommunications, with a view towards significant savings through consolidating operations. The first 1,200 plus employees are transferred from department of Public Works’ information technology group.
Aug. 15, 2011: Deloitte awarded $20 million contract to advise on trimming operational expenses across government.
Nov. 15, 2011: Cabinet approves the transfer to Shared Services of IT infrastructure and related budgets from 43 federal departments. About 5,000 employees will eventually be moved.
March 29, 2012: Federal budget trims $75 million (5 per cent) from Shared Services annual budget. The plan calls for further cuts of 7 per cent and 10 per cent in the following two years.
April 1, 2012: Shared Services becomes a standalone department.
June 29, 2012: Shared Services Canada Act becomes law, giving the agency responsibility for buying information technology on behalf of most federal departments.
June 25, 2013: Bell and CGI Group win a $245 million, 7-year contract with 1-year option to consolidate 63 email systems across government into one. The new system is to be running by March 31, 2015.
November 2013: Shared Services establishes an enterprise data centre in Gatineau at a facility owned and operated by Bell Canada. The data centre is to be one of a handful meant to take over from more than 500 aging data centres across government.
April 2014: Second data centre is established, in Base Borden. It is government owned and operated.
June 2014: Citizen reports Bell’s single email project eight months behind schedule.
Oct. 7, 2014: Third data centre to be set up, in IBM’s Barrie facility.
February 2015: Internal Shared Services documents show the following projects with “yellow” status, indicating delays and/or technical problems: Gatineau data centre, Borden data centre, Internet data communications project, Borden data centre expansion, Barrie data centre, legacy data centre closures. Among the projects with ‘red’ status – indicating serious issues – are the single email project.
February 2015: Bell acknowledges it will be unable to meet the March 31, 2015 deadline for delivering the new email system.
May 11, 2015: Allstream and Telus selected to provide Internet data communications for 43 departments. Contract, which could extend to 2025, expected to exceed $100 million.
July 2015: Ron Parker takes over as president from a retiring Liseanne Forand
Oct. 19, 2015: federal election
November 2015: Bell Canada halts migration of email boxes to new system while it addresses a “hardware issue” involving one of its subcontractors. Deadline for new email system is shifted to September 2016.
Feb. 2, 2016: Auditor General Michael Ferguson excoriates Shared Services for its sloppy operations and casual disregard for meeting the expectations of the federal departments it’s meant to serve. Shared Services says it will let departments know by Dec. 31, 2016 exactly how it will deliver information technology services.
March 22, 2016: Finance Minister Bill Morneau allocates an extra $383.8 million to Shared Services to help agency keep its aging infrastructure operating while it copes with delays in modernization program. Morneau also added $77.4 million to bolster computer security at Shared Services.
May 24, 2016: Shared Services awards $322 million, 25-year contract to expand and run Borden data centre. Winning consortium led by Forum Equity Partners, Walsh Contractors and Balfour Beatty.
June 11, 2016: Treasury Board president Scott Brison orders an independent review of Shared Services, to be completed by March 31, 2017.
Dec. 1, 2016: Shared Services to make public a revised strategy for upgrading the government’s electronic backbone.
查看原文...
A career bureaucrat, Forand had, the day before, been appointed president of Shared Services Canada — a new federal department that would manage the government’s email, data centres and telecommunications.
Her mandate — representing about one-third of the federal government’s $5-billion-a-year technology services budget — was breathtaking in its scope and complexity. She was to simultaneously streamline and modernize the government’s electronic backbone, and keep the old gear running. Sixty-three email systems would be collapsed into one. More than 500 data centres were to be decommissioned, to be replaced by a mere handful. Fifty telecommunications networks connecting 3,500 federal buildings were to be upgraded.
Forand was to do this with staff cobbled together from 43 different departments. But on this day, she had no office, and the paperwork had been approved for just 1,200 of an eventual 6,000 employees.
Rona Ambrose, former Public Services Minister, was present at the agency’s launch.
As she entered Phase III, she was surprised, seemingly touched, to see that a small army of government workers had gathered in the main foyer to greet her.
After introductions, she was escorted to the 17th floor. There, Forand found a temporary office. She sat down and began making lists of things to do.
•
As Forand tapped out her notes, Public Services minister Rona Ambrose and Treasury Board president Tony Clement were in the National Press Theatre, directly across from Parliament’s venerable West Block.
Ambrose said she and Clement were there to introduce “new measures that will improve the efficiency of information technology services.” Shared Services would lead this effort, Ambrose said, and generate “substantial savings” through economies of scale.
It was up to Forand to flesh out a strategy that had little substance at launch.
Not that a more detailed plan would have mattered.
What Forand didn’t realize was that the seeds of failure were already being sown. Prime Minister Stephen Harper was preparing a government-wide spending strategy that would hobble Shared Services before it could even get going.
Related
- Bagnall: Older government systems showing age, revealing the price of email delay
- May: Shared Services Canada was a battle chief statistician couldn't win
- Phoenix pay problems 'unacceptable', auditor general says
•
It’s rare that issues involving information technology intrude on Ottawa’s political agenda — rarer still that they do so with the force displayed in 2016.
All of Shared Services’ core projects are behind schedule, substantially so, in the case of email transformation. Not only have promised savings not materialized, the agency is spending tens of millions of dollars more than forecast maintaining networks that should by now have been decommissioned.
But it’s not just Shared Services that’s struggled. Federal departments — from Agriculture and Agri-Food Canada to Veterans Affairs Canada — are upgrading software and IT infrastructure that falls outside the Shared Services mandate.
Some, such as the $300-million project to modernize the government-wide pay system known as Phoenix, are fumbling badly.
At one level, Shared Services and Phoenix are very different. The first is a new department with a specific mandate; the second is a single project — albeit a large one — being managed by Public Services. Indeed, Phoenix is one of nearly two-dozen major IT projects underway across government.
But Shared Services and Phoenix have something in common — a botched introduction caused, it appears, by deep flaws in how government operates. In both cases, cabinet ministers and bureaucrats underestimated complexity and risk. In this, they were hardly unique — it was the scale of the misjudgment that set the federal IT agenda apart.
Standish Group, a Boston-based consulting firm, has been tracking the performance of IT projects since the mid-1990s — with surprisingly little variation in results. The consultants last year examined 50,000 projects worldwide, including government and private sector. About 30 per cent of these efforts succeeded — that is, they were on time, on budget and produced a payoff. Roughly half the projects ran into difficulty and nearly 20 per cent failed outright. The larger and more complex the project, the higher the rate of failure.
Carol Bellringer, the auditor general for the B.C. government, last month offered three key reasons why IT projects fail: Government departments, she said, lack in-house expertise; they attempt “overly ambitious” programs; they justify the latter through “incomplete” business cases.
All three elements were present at the launch of Shared Services. Most of the responsible bureaucrats were not trained in IT, yet were tasked with remaking on the country’s electronic infrastructure. Many also lacked experience in project management with a heavy IT component.
“I don’t know how many times I heard from deputy ministers that they didn’t understand information technology,” said a senior Shared Services official, “They didn’t like IT and they hoped never to see anything to do with IT for the rest of their career.”
Yet it is a group of deputy ministers — the ones in charge of the most IT-intensive departments — who determine the shape and scope of large IT projects. And when it came to launching Shared Services — the centrepiece of the government’s online renewal — the already high risks were exacerbated by a political agenda that stripped it of the capital necessary to get the job done.
It will likely end up costing taxpayers a fortune to set things right again.
Detail from one of the many legacy data centres in the National Capital Region.
•
It had seemed so simple in the beginning. The idea for Shared Services emerged from the Conservatives’ fifth budget, tabled March 4, 2010. The themes were clear: The economic recession was over; it was time to regain control of government spending.
One aspect of the strategy — little noticed at the time — was the launch of a “comprehensive review” of government spending on administration and overhead expenses. This should have offered easy pickings: Federal government employment was near high tide; and most departments and agencies had expanded rapidly.
Daniel Jean, the deputy secretary to the cabinet of the Privy Council Office, was picked to run the review, making it a big deal. The PCO is home to 950 bureaucrats who provide advice to cabinet and the Prime Minister’s Office, and oversee the development of the civil service.
Jean reported directly to Wayne Wouters (pronounced “Waters”) — the clerk of the privy council and the government’s top bureaucrat. Among the members of the review helping out Jean were Benoit Long, a senior manager seconded from the office of the government’s chief technology officer, and Liseanne Forand, then chief operating officer of Service Canada. The latter department offers Canadians online access to pensions and employment insurance.
Wayne Wouters, Former Clerk of the Privy Council, instrumental in Shared Services’ creation.
The administrative services review was carried out in secret, typical PCO modus operandi. Its members roamed the bureaucracy, collecting information and searching for ways to consolidate or standardize how things were done. Some departments were already moving down this path.
The Department of National Defence, for instance, calculated it was spending $30 million annually more than necessary because its 16,000 Ottawa-area employees were inefficiently spread across more than 40 facilities. Late in 2010, the government bought the former Nortel campus to allow for a major consolidation.
Information technology offered an even richer vein of potential savings. For half a century, computer networks and software applications had multiplied willy-nilly as individual departments and agencies looked after their own needs. The result was a patchwork of incompatible, higher-cost systems. Standardizing common, basic technologies such as email, data storage and telecommunications seemed logical.
It had been tried before. But attempts to centralize the buying of high-tech gear and services had failed, largely because federal departments were allowed to opt out. Most did so. They did want to give up control of their IT networks to a central agency.
The PCO determined this time would be different. The prime minister had the authority to create a new federal department through a simple cabinet approval known as an order-in-council. Most departments, including a reluctant Canada Revenue Agency and Department of National Defence, would be forced to carve out a significant portion of their IT groups and budgets — about 40 per cent on average — and hand them over to Shared Services.
Crucially, the move would not be subject to scrutiny by Parliament. And so Shared Services was born on Aug. 3, 2011.
•
Speed was demanded of the agency from the start. Minutes of meetings involving senior Shared Services staff are studded with references to “tight schedules” and the “urgency” of getting projects done.
Part of that had to do with the sheer age of the government’s infrastructure. The hardware was in danger of breaking down and the underlying software for many applications was so old that suppliers such as Microsoft, PeopleSoft and Adobe had stopped supporting it.
The faster Shared Services could install new networks, the less money it would be forced to throw at solving the problems caused by older technology.
But that wasn’t the only reason Shared Services was pressed for time. Senior Shared Services officials said the Conservatives were eager to see cost savings, and impressed upon them the importance of securing an early win.
The PCO framed the upgrade in simple terms: Consolidate, modernize and reap the savings. And Shared Services would have nearly a decade to get it done. By 2020, the thinking went, the government of Canada would be able to offer its citizens secure, online services that would be the envy of the world; and Shared Services would be a magnet for attracting the best and the brightest employees in government.
But cabinet — and to some extent the PCO — failed to account for the complexity. They were proposing to create a new organization using bits and pieces from other departments and loading it up with a series of mandates on a tight schedule. The entire production was fraught with risk.
“They had articulated the problem and come up with an organizational response (in the form of Shared Services),” an independent adviser to the PCO said in an interview, “but they completely underestimated the scale.”
Start with the logistics. Forand would not receive the bulk of her new workforce until mid-November in 2011. And it wasn’t just a matter of transferring 4,000-plus workers from 43 departments. Shared Services didn’t immediately have any place to put them. The employees in question were in more than 500 locations across the country — and worked out of 80 buildings in the national capital region alone. They would remain in place for the moment.
Liseanne Forand, the first president of Shared Services, was tripped up by significant and early budget cuts.
Another problem for Shared Services was that it had little control over just who would join it. That decision was left up to the departments. During her testimony earlier this year before a House of Commons committee, Forand pointed out that one of the larger departments sent more than 1,000 IT staffers but included just three executives. “If that’s true for IT staff, you can imagine what it was for administrative staff,” she said. It was difficult to imagine “a single organization ever transferring a really crackerjack finance person or human resources or auditor.”
Indeed, in order to fill Shared Services’ executive ranks, Forand would later have to advertise for at least a dozen senior managers.
Shifting Shared Services employees into new facilities was a miserable, 17-step process that required nearly a year for each move. The agency reckoned it would take as long as five years to complete the transfer of bodies and responsibilities.
Nor did Forand have access to her chief operating officer right away. Because the formation of Shared Services had been shrouded in such secrecy, the hiring of experienced outsiders happened last minute. Grant Westcott would join Shared Services as COO in mid-September.
Sixty-three years old at the time, Wescott had served as assistant deputy minister of Public Services in the late 1990s, when he managed the federal government’s telecommunications networks. In that role, he oversaw the successful effort to prepare systems for the year 2000.
More relevant to his new position at Shared Services, Westcott had also worked for nearly a decade at CIBC. As executive vice-president, he had been instrumental in overhauling the bank’s IT infrastructure. Twenty-two of the Canadian Imperial Bank of Commerce’s data centres were shrunk into just two, saving the bank more than $40 million annually in operating costs. Among other things, Westcott and his team consolidated 15 global communications networks into a single voice and data system. More savings resulted.
It wasn’t without glitches. CIBC was forced on several occasions to apologize to customers for a breakdown in certain banking services.
Nevertheless, Westcott’s experience suggested this was not rocket science, at least not in the private sector. Streamlining and modernizing IT networks could be done under the right conditions. But, just weeks after he started, Shared Services was jolted by its first big shock.
•
It came in the form of a letter from Treasury Board, the cabinet committee responsible for managing the government’s finances. The October 2011 note outlined how the Conservative government was planning significant spending cuts in the upcoming federal budget. Shared Services was expected to contribute its share.
That provoked astonishment among Shared Services’ executives; afterall, the agency wasn’t even close to being fully formed.
Less than one-fifth of Shared Services’ expected workforce had been formally transferred on paper, though they continued working out of their old offices. Forand and her immediate support staff were working out of temporary headquarters, and the agency’s financial books were not yet in order. Upon its formation, Shared Services had been allocated an annual base budget — about $1.5 billion. But it was being asked to trim this in stages by five per cent, 7.5 per cent, then 10 per cent over the next three years. Treasury Board was seeking $150 million in total, with the proceeds to help reduce the federal government’s deficit.
Forand, according to access-to-information documents, found early savings by renegotiating telecommunications contracts, eliminating telephone lines through consolidation, and leaving open unfilled staff positions. She also pencilled in savings that would result from the consolidation of government email systems and data centres — these would be achieved starting in the third year of the Conservatives’ deficit reduction program.
The agency knew it was just buying time, however. Forand, Westcott and other top managers had assumed prior to reading that Treasury Board letter that any savings realized by consolidating Shared Services’ electronic infrastructure could be directed toward modernization efforts. In other words, the potential savings were not only an incentive to be efficient, they would serve as seed capital for renewal.
Instead, the Conservatives grabbed the money.
“We did convey strongly at a high level that reductions of that magnitude would create increased pressure and risk,” said one official familiar with the Treasury Board request, “We couldn’t quite believe they would take the whole 10 per cent and were shocked when they did.”
From that moment, Shared Services was forced into an excruciating series of financial tradeoffs.
The agency no longer had the capital to simultaneously modernize its networks and keep the older (legacy) stuff running. The legacy gear got short shrift with inevitable results.
In recent months, the government’s networks have been plagued by a series of outages that have left thousands of government employees without access to online services and applications, and brought down government websites. A major culprit: power distribution units that have been pushed well beyond their normal lifespan.
The Conservatives, meanwhile, wanted their early win, so they convinced Shared Services that the quickest and easiest project would be to replace the government’s massive email network. While email technology can be complicated, the service is considered something of a utility in the IT world. It shouldn’t have been a huge problem. Instead, things would go badly wrong.
Shared Services Canada Organizational Chart
Minister responsible for Shared Services: Judy Foote, Minister of Public Services and Procurement
President: Ron Parker
Chief Operating Officer: John Glowacki
Assistant deputy minister Cyber & IT Security: Raj Thuppal
ADM, Data Centre Services: Patrice Rondeau
ADM, Network and End Users: Pankaj Sehgal
•
When government procurement officials invited potential email suppliers for a technical briefing on June 12, 2012, the project appeared daunting but doable. The date proved significant: Shared Services would be granted the authority to do its own buying a couple of weeks later. But because the email competition was already underway, Public Services — the government’s traditional procurement authority — would handle it.
The 43 federal departments linked by Shared Services’ networks collectively were using dozens of separate email systems — most of them using Microsoft technology. These were to be collapsed onto a single system, also based on Microsoft software. The new setup would require fewer people to operate it, contributing to $50 million per year in savings when it was up and running.
The winning bidder in this contest would be responsible for transferring more than 635,000 electronic mailboxes on behalf of 378,000 employees to a new, outsourced data centre.
When Bell Canada and CGI Group (using Microsoft email) won the competition in 2013, Public Services minister Rona Ambrose said the new $400-million system would be in place by March 31, 2015.
Behind the scenes, there was a much different view. Losing contenders were stunned by the differences in the submitted bids.
Bell said it could develop and run the initial part of the government’s single email system for $245 million. IBM was the next lowest bidder ($282 million), followed by HP ($368 million) and Dell ($571 million).
Bid evaluators from Public Services had given technical merit a 30 per cent weighting, compared to 70 per cent for price. Thus Dell, which had the best-ranked technical proposal, really didn’t have a chance.
The bitterness of the finalists can be seen in debriefing notes obtained by the Citizen. In them, officials from HP complain that their company’s “governance does not allow for under-bidding.” HP and Dell had included generous amounts in their proposals to take account of risks and contingencies.
Losing bidders warned they would monitor the single email project closely to make sure Bell met its obligations at the promised price. But if they believed Bell’s potential difficulties would somehow escape public scrutiny, they needn’t have worried.
•
It didn’t take long for frictions to surface within Bell’s engineering group. By August — just two months after the awarding of the contract — Bell replaced its entire system for establishing the identities and credentials of employees using its email system.
The technology — that, among other things, allowed workers to manage their email accounts and passwords — had been implemented three years earlier on a separate federal government contract. Bell had adapted this technology for its winning bid. Now it was proposing to put in place a new approach. It’s not clear why, though it appears to have been an internal decision, not one forced upon Bell by Shared Services — according to a programmer familiar with the re-engineering effort. Bell declined comment on this aspect of the email project.
Bell wasn’t the only member of the winning team to experience technical difficulties. It was soon apparent that Microsoft’s latest email software — Outlook 2013 — had an unexpected limitation. According to a memo prepared for Forand — and obtained under an access-to-information request — Outlook 2013 could support a maximum of just 10,000 public email folders. Data gathered by Shared Services had revealed more than 800,000 folders in use across government.
The limitation would be dealt with by using an older version of Outlook.
It was clear within months of the launch of the email project that Bell wouldn’t be able to shift, or migrate, the government email addresses as promised to its new data centre in Buckingham, Que. The original plan was to spend six months working out details of the overall design. The migration of employee mailboxes was to have been done in three waves, starting in January 2014 and ending in March 2015.
But Shared Services granted Bell a 12-week extension in December 2013 for the start of the transition. Two months later, Bell asked for and received an additional four weeks’ grace “to address all outstanding design issues and to ensure a robust, secure and integrated solution.” This, again, according to documents obtained through access-to-information requests.
•
By November 2014, the email project had been flagged “red” by Shared Services. That meant it was in serious difficulty of meeting its final deadline of March 31, 2015.
Indeed, a year ago, when the project was six months late, Bell had successfully moved fewer than 15 per cent of the government’s mailboxes to the company facility in Buckingham. Bell at that time was forced to halt all migrations after identifying what it called a “hardware” issue (which has since been resolved, according to Shared Services). Shared Services says Bell must resolve “outstanding contracted issues” before shifting the other government mailboxes.
“All elements of the ETI (email) platform have been running smoothly for departments already migrated,” Bell spokeswoman Jacqueline Michelis said this week in response to a Citizen query. “We look forward to resuming migrations as soon as possible.”
It’s not clear when this will be.
“We’re continuing to discuss issues with Bell,” Shared Services president Ron Parker said this week.
“There’s no end date,” added Parker, who took over from Forand in July 2015.
Ron Parker took over as president of Shared Services Canada in July 2015. He was surprised at the extent of the budget shortfall.
While Shared Services is paying Bell only for the mailboxes it has already moved, the agency is still responsible for keeping the lights on at the government’s legacy data centres — where most of the electronic mailboxes, and much of the government’s software applications, for that matter, are currently stored.
These jumbles of computers, servers, wires and cooling systems are weighed down by far too much history — and it took Shared Services a long time to get its arms around it.
“We were not created with a transformation plan already in place because to do that, we needed to know what we had,” Forand explained to members of Parliament. “So we spent a year counting things.”
Shared Services didn’t have to start from scratch. Prior to the agency’s 2011 launch, the PCO’s administrative review tried to estimate the number of data centres in government by interviewing the IT czars for each of the departments. Symptomatic of the far-flung nature of the government’s IT networks — and how loosely these were managed — the PCO didn’t even come close.
“When I was at Privy Council, we thought there might be about 200 data centres,” Benoit Long, senior assistant deputy minister of Shared Services told the House of Commons committee last May, “After a year, we had counted 495 and I am still discovering others today.”
Most of these data centres were small — less than 1,000 square feet — often set up by government scientists who didn’t bother telling their superiors. Just 22 were larger than 5,000 square feet — and most of these were in the National Capital Region. These were where the bulk of the government’s data and software applications were stored.
In 2012, Grant Westcott organized a six-week tour of these data centres to see for himself what Shared Services was dealing with. According to an agency employee familiar with the tour’s findings, Westcott came away profoundly shocked.
•
Nearly all the large data centres were decrepit, undercapitalized and inefficient users of energy. Yet just one was in the process of being replaced. This was a Heron Road facility with a leaking roof, considered key because it housed Canada Revenue Agency data.
Bell Canada had been awarded a contract to build and operate a new data centre in Buckingham. The CRA data would be transferred there in the fall of 2013. A separate part of that facility is also meant to host the government’s email system.
The remaining data centres offered Westcott a depressing snapshot.
Most of the gear was housed in office towers, rather than facilities tailormade for heavy-duty electronics. With each new generation of computer or servers came requirements for more power and more robust cooling systems. The entire system was approaching a nervous breakdown.
Not only were the data centres inherited by Shared Services running out of space, the arrangements for emergency power bordered on makeshift. One Defence Department data centre, for instance, relied a backup generator powered by a relatively ancient jet engine. While unconventional, there was nothing unique about pressing military hardware into this type of service. The risk had to do with with training — the older the jet engine, the fewer employees there were who knew how to fix it.
The system for backing up data wasn’t much better. At Statistics Canada — where Westcott’s group was actually required to swear an oath to keep the data confidential before entering the storage rooms — the backup procedure called for storing copies of the data in a separate but nearby facility. While that’s OK for most disasters, the relative proximity still meant Statcan’s repository of historical data was vulnerable to a bombing or major earthquake.
In the wake of Westcott’s tour, Shared Services developed a plan for refurbishing or building new data centres with plenty of distance between them. These include facilities at Buckingham, Ottawa’s international airport, Barrie, Borden and Dorval, Que.
Nevertheless, the Buckingham data centre wouldn’t be ready until late in 2013 and the contract to set up the Barrie facility, run by IBM, wasn’t awarded until the fall of 2014. A public-private partnership to build a large data centre in Borden was signed last May. A separate data centre focused on weather forecasting and other science-related data will be based near Dorval.
These facilities require careful calibration — the gear and supporting wiring and cooling systems have to be capable of handling the software applications and data coming their way.
Collectively, the 43 federal departments run 14,000 software applications, covering everything from human resources software to weather forecasting. These are all meant to migrate from the legacy data centres by 2020. But that milestone very likely will not be met.
Shared Services has had to spend much more than expected to keep legacy data centres going while new ones are built and equipped.
•
Minutes from a meeting of Shared Services’ senior managers held Sept. 29, 2015 give a flavour of the difficulties. Patrice Rondeau, in charge of the agency’s data centre efforts, told his colleagues that he had received “a large number of exemption requests” from federal departments that wanted to upgrade their legacy data centres rather than transfer software and data to the new ones operated by Shared Services.
This speaks volumes considering the age of the legacy data centres. The Citizen this week toured one of the larger ones. Located near the ByWard Market, the data centre was launched in 1985 and currently houses data and software on behalf of Elections Canada, Canada Mortgage and Housing Corp., Employment and Social Development Canada, among others.
The facility is clean enough, appears to have adequate backup generators and extra space for additional gear if necessary. But a closer look reveals the weak points — it relies on 11 units for distributing power, and most have been pushed beyond their average life expectancy. A couple broke down recently, temporarily disabling the facility. The other units, who knows? “They’re only now starting to fail,” explained one data centre employee.
The limiting factor for storage is not the physical space; it’s the amount of power available to feed the servers. And power is near its maximum.
The data centre continues to accept new workloads (data, software applications) from the departments it serves but there is now a quid pro quo — the departments have to get rid old data before facility will add the new stuff. “We’re pretty much full,” said the employee.
There is some hope on the horizon, albeit distant. The Shared Services half of Bell’s data centre in Buckingham appears to be successfully hosting data from several federal departments. The IBM facility in Barrie is finally accepting workloads — though still representing less than five per cent of the data centre’s capacity. Construction of a new data centre in Borden is well underway and a new science-and-weather focused facility is planned for near Dorval — though bids have yet to be solicited.
Yet, even now, the legacy data centres are the preferred option for many of the government’s IT managers. Many are still unimpressed by the untried agency.
The auditor general pointed out early this year that Shared Services had failed to establish the level of service departments should be able to expect. For instance, in the event of outages, how quickly would Shared Services get the systems running again, and how many hours per day was such assistance available?
Nor, the auditor general reported, did Shared Services give reports on the overall health of its networks. A Shared Services executive acknowledged recently customer service is improving are “on a very slow trajectory.”
This much is evident on a large schematic posted on a wall on the seventh floor of the Shared Services headquarters at 99 Metcalfe St. in downtown Ottawa. It maps out feedback from the departments it serves on a quarterly basis. Green is good. Orange and Red are bad. Thirteen are green, 18 orange and red, with the remaining departments neutral. Overall, level of satisfaction shows little change over the past year.
•
Westcott left Shared Services in the spring of 2015, having extended his three-year stint by seven months. He returned to his vineyard operation just west of St. Catharines. Forand followed him out of the door a few months later into retirement. Aside from their joint testimony last May, the former top guns at Shared Services have not been public about their experiences. Privately, with colleagues, they have expressed frustration with the slow pace of getting things done in government, often related to multiple tiers of oversight by Treasury Board, Public Services, the Privy Council Office and other departments affected by Shared Services’ plans.
But the loss of so much of their annual budget upfront undeniably hurt the agency’s drive to modernize. When queried about whether he had second thoughts about trimming Shared Services’ early budgets, former Conservative Treasury Board president Tony Clement told the Citizen “I wasn’t the minister (responsible) for Shared Services. And all the mistakes were made by the Liberals in any event.”
Treasury Board did play an important role in establishing Shared Services and influencing its early rollout. Rona Ambrose, the former Conservative minister of Public Services — and the minister directly responsible — declined to comment.
The Liberals were more than a little surprised at the state of affairs at Shared Services. Nevertheless they couldn’t lay all the blame at the foot of the Conservatives.
Investing in information technology at the federal level had not been a priority for decades — the government typically spends less than two per cent of its annual revenues on it, compared to more than 10 per cent for large, information-intensive organizations such as Canada’s big banks.
Quite simply, the bill for under investing is now coming due.
Parker, Shared Services’ new president, found out as much during his first briefings, when he learned his department’s “projected expenses were much greater than expected.” Indeed, a July 2015 briefing note obtained under access-to-information requests warned him the data centres branch alone was facing a $200-million shortfall.
Finance Minister Bill Morneau in his first budget added nearly $400 million to Shared Services’ coffers. It’s meant to keep the older gear going years longer than planned.
Finance Minister Bill Morneau early this year allocated a two-year lump sum of nearly $400 million in his first budget to help Shared Services keep the legacy networks operating until the new data centres are ready. Any pretense that transforming the government’s electronic networks would produce immediate savings is gone.
Will the investment be enough? Probably not. The problems are compounding. Delays in the construction of the new electronic backbone have pushed back the rollout of hundreds of software programs now overdue for replacement. Indeed, over the next few years, thousands of applications are scheduled to migrate from legacy data centres to Shared Services’ new sites. It may prove the most difficult phase of the government’s IT planned transformation.
Consultants say departments are impatient at the thought of further delay, and have been pressing for the right to move more of their applications to cloud-based (third party) services such as those offered by Amazon Web Services. Some, such as those involving websites used by the public are already hosted in the United States (a Treasury Board project). But sensitive information and software — for reasons of security and data sovereignty policy — should be hosted from data centres or cloud services supported from Canadian soil.
Just how this will play out should be made clear in coming weeks, as Shared Services prepares a revised plan for approval by cabinet.
With any luck, it will incorporate the many lessons learned to date — above all, that you cannot book savings first, then build the system.
Rebuilding electronic infrastructure requires serious upfront investment, followed by careful attention to detail. In a complex project, it is better to take small steps, be sure of your ground, then move on. Because trying to fix something on the fly costs far more than getting it right from the beginning.
A Shared Service executive said of the journey to date. “The private sector is really good at understanding you need to fix the infrastructure first,” he said. “The (software) applications that ride on that infrastructure, that’s really hard to do. But if you try to do both at the same time, it’s overwhelming.”
So it proved.
Shared Services Canada Timeline
March 4, 2010: Federal budget outlines three-part plan to return to surplus by 2015. Included is a comprehensive review of government spending on administration.
April 6, 2010: That review is to be carried out in secret by bureaucrats operating out of the Privy Council Office – which provides advice to cabinet and the prime ministers’ office. The team handling the administrative services review, led by Daniel Jean, reports directly to Clerk of the Privy Council Wayne Wouters.
Aug., 2011: Public Works Minister Rona Ambrose and Treasury Board President Tony Clement announce the launch Shared Services Canada. This new federal agency will take over government-wide email service, data centre operations and telecommunications, with a view towards significant savings through consolidating operations. The first 1,200 plus employees are transferred from department of Public Works’ information technology group.
Aug. 15, 2011: Deloitte awarded $20 million contract to advise on trimming operational expenses across government.
Nov. 15, 2011: Cabinet approves the transfer to Shared Services of IT infrastructure and related budgets from 43 federal departments. About 5,000 employees will eventually be moved.
March 29, 2012: Federal budget trims $75 million (5 per cent) from Shared Services annual budget. The plan calls for further cuts of 7 per cent and 10 per cent in the following two years.
April 1, 2012: Shared Services becomes a standalone department.
June 29, 2012: Shared Services Canada Act becomes law, giving the agency responsibility for buying information technology on behalf of most federal departments.
June 25, 2013: Bell and CGI Group win a $245 million, 7-year contract with 1-year option to consolidate 63 email systems across government into one. The new system is to be running by March 31, 2015.
November 2013: Shared Services establishes an enterprise data centre in Gatineau at a facility owned and operated by Bell Canada. The data centre is to be one of a handful meant to take over from more than 500 aging data centres across government.
April 2014: Second data centre is established, in Base Borden. It is government owned and operated.
June 2014: Citizen reports Bell’s single email project eight months behind schedule.
Oct. 7, 2014: Third data centre to be set up, in IBM’s Barrie facility.
February 2015: Internal Shared Services documents show the following projects with “yellow” status, indicating delays and/or technical problems: Gatineau data centre, Borden data centre, Internet data communications project, Borden data centre expansion, Barrie data centre, legacy data centre closures. Among the projects with ‘red’ status – indicating serious issues – are the single email project.
February 2015: Bell acknowledges it will be unable to meet the March 31, 2015 deadline for delivering the new email system.
May 11, 2015: Allstream and Telus selected to provide Internet data communications for 43 departments. Contract, which could extend to 2025, expected to exceed $100 million.
July 2015: Ron Parker takes over as president from a retiring Liseanne Forand
Oct. 19, 2015: federal election
November 2015: Bell Canada halts migration of email boxes to new system while it addresses a “hardware issue” involving one of its subcontractors. Deadline for new email system is shifted to September 2016.
Feb. 2, 2016: Auditor General Michael Ferguson excoriates Shared Services for its sloppy operations and casual disregard for meeting the expectations of the federal departments it’s meant to serve. Shared Services says it will let departments know by Dec. 31, 2016 exactly how it will deliver information technology services.
March 22, 2016: Finance Minister Bill Morneau allocates an extra $383.8 million to Shared Services to help agency keep its aging infrastructure operating while it copes with delays in modernization program. Morneau also added $77.4 million to bolster computer security at Shared Services.
May 24, 2016: Shared Services awards $322 million, 25-year contract to expand and run Borden data centre. Winning consortium led by Forum Equity Partners, Walsh Contractors and Balfour Beatty.
June 11, 2016: Treasury Board president Scott Brison orders an independent review of Shared Services, to be completed by March 31, 2017.
Dec. 1, 2016: Shared Services to make public a revised strategy for upgrading the government’s electronic backbone.
查看原文...
