Thursday’s Microsoft Big Data event featured a panel of Canadian service providers and customers, who offered some interesting insights about the state of Big Data in Canada.

MISSISSAUGA – At Thursday’s well attended Big Data Debate at Microsoft’s Mississauga Ontario offices, both Microsoft Canada president Janet Kennedy and a panel of other Big Data service providers and customers assessed the state of Big Data in Canada. While featured speaker Tiffany Wissner, Microsoft’s Senior Director, Data Platform, emphasized Microsoft’s momentum in the Big Data market, the Canadians acknowledged the Big Data situation in Canada is not quite as advanced. Even so, significant numbers of Canadian companies are dipping their toes in the Big Data waters, notwithstanding that Canada is less far along the path than some of Microsoft’s global enterprise customers.

Microsoft Canada president Kennedy reviewed the state of technology in Canada, seeing both pluses and minuses. On the negative side, she cited World Economic Forum data ranking Canada 26^th globally in private sector innovation, 22^nd in R&D spending in the private sector, and 17^th in business productivity – the latter 30 per cent below the U.S. None of those numbers are particularly inspiring.

On the other hand, Kennedy saw significant positive signs.

“We have 500 new companies starting in Kitchener right now every year,” she said, indicating Canada is doing a good job at keeping many tech graduates from the University of Waterloo at home. She also had a bullish view of Big Data in Canada

“IDC Canada says Big Data is big — $1.5 billion this year in Canada,” Kennedy said. And that’s just software alone, with the number increasing once Big Data hardware, infrastructure and services are added in. “Technology will increasingly be driven by analytics, big data and adaptive learning.”

While Canada is lagging the U.S. somewhat in Big Data adoption, the gap is narrowing, said Utsav Arora, Senior Research Analyst, Enterprise Applications, at IDC.

“The conventional approach has been to state that Canadian adoption of Big Data technologies has been 12-18 months behind the U.S.,” he said. “I think this is too conservative a view of the Canadian market, and that we’re now more in the 6-12 month range.” The growing start-up culture, emphasis on innovation, and entrance of several niche Canadian Big Data vendors have closed the gap.

Arora also emphasized that it is important to take into account the differential in size between the U.S. and Canadian economies.

“The majority of multi-national large enterprises in Canada are U.S.-based companies which have branches or extended business units in Canada,” he said. “Therefore Big Data and Analytics projects are driven by headquarters in the U.S. Adoption and implementation of Big Data technologies often takes place in the U.S. first and then extends over to the Canadian business units of U.S. based enterprises.”

Arora indicated as well that Big Data investment in Canada has been somewhat uneven.

“Companies realize that if Big Data is driven forward, they will get a competitive edge, but many are investing on a very ad hoc basis,” he said. “While Big Data is slowly shifting to line of business units, companies with a chief analytics officer or chief data officer to act as a centre point are better able to establish an enterprise-wide strategy. This is particularly the case in large organizations. SMBs have much more consolidated hierarchies where this role might not be as suitable.”

The panelists also emphasized that while Big Data has massive value, the ability to maximize that value today is still somewhat limited in many cases.

“The challenge that exists today is that it’s not always evident what you can do with the data,” said Mohannad El-Barachi, founder and CEO at SweetIQ, a Montreal-based startup which offers retailers insights from local search using Big Data analysis. “Do you actually know what you are looking for? People often don’t know local search can capture data to deal with a problem, and that there’s an ROI they can capture from it. Yet you will almost always find things you didn’t think you would find. Organizations aren’t giving enough leeway yet to data scientists on Big Data.”

George Hamin, Director of eBusiness & Information Systems, at Subaru Canada, also emphasized that the data still needs to be used more effectively.

“We have a lot that we aren’t making very good use of right now,” Hamin said. “The problem is that the data is always backward looking. I can tell almost to the minute what our best selling colour of car is, but in terms of predicting behavior in the future, we really can’t do that. We know red may be selling faster than white, but we can’t tell what it will be like next season. Social media can be better for telling us that, but we can’t do that with it right now.”

Better tools are still needed for this, Hamin said.

“We need new tools,” he said. “For me, that’s what Big Data is – the set of tools that allows you to do things with that rapidly growing set of data. There are also not a lot of good case studies to explain to stakeholders why we need it.”

Hamin cited his own personal experience on this point.

“Two years ago, I made a mistake, in calling this Big Data in a presentation to our VPs,” he said. “What I should have said was that it’s just the next stage of BI, an opportunity using machine learning and predictive analytics. That’s something they would have understood, and then the funds would have come rolling in and I could have done some bigger stuff. Instead, they said no to me then.”

Richard Boire, Founding Partner of Boire Filler Group, a data mining company, thinks that data science is so new, that organizations often fail to grasp it, and finding people with the skills to do it is hard in any event.

“The debate isn’t as much about Big Data as much as it is about DATA, he said. “Forget Big. Companies still aren’t dealing with data adequately. The technology has advanced so quickly in the last 20 years, but people haven’t gone as far in understanding how to derive insights from it. Data mining and data science are still in their infancy, with the credit card companies being the pioneers.”

Boire also emphasized his belief that there is a tremendous lack of skills of young people coming up in this area.

“This is where the universities and colleges need to pick up the ball,” he said. “What is needed is not just somebody who can program, but who also has the ability to understand the results, to understand the business. It’s a very different skillset from the old days, when you had traditional IT and traditional statisticians, and they were separate. Now they have morphed together.”

“If you don’t deal with people and processes, it doesn’t matter how great the technology is,” said Gayle Ramsay, VP Customer Analytics, Marketing & Strategy, at BMO. “The issue is what business problem are you solving?”

Ramsay emphasized the importance at being able to get at unstructured data, especially from social media.

“Assembling all the data you have on customers is really important,” she said. “There are so many pieces of social media you can leverage now to have better insight into the customer challenge. The problem is that there is so much of it, how do you assemble it and use it quickly?”

Boire, on the other hand, valued more traditional data more.

“Do I need to go into social media when I have data on spending and transaction behavior that tell me what people have actually done,” he asked. “That’s what I want to know, not what they are tweeting. As data scientists, we are always open to new uses of data, but we need to understand when to use it.

“For instance, if I’m a CIBC customer, do I want CIBC knowing my Facebook behavior,” Boire said. “Are we respecting the customer when we do something just because we can do it? It depends what you are using the data for. If it’s being used on an aggregate level, that’s fine. If you are linking it on an individual level, the question then becomes, is that right, and do companies want that?”

Ramsay, who emphasized the importance of being able to get at clickstreams and to be able to use that information, acknowledged there are limits.

“You have to be respectful,” she said. “I work for a bank, and I can’t break laws. Even so the bank’s Privacy Section is always pushing me because I’m usually pushing the limits.”

IDC’s Arora thinks those limits are still pretty broad.

“It hasn’t reached a point today where anything is questionable,” he said. “The reality is that the world we live in today is digitization, from wearable smart watches to social media. That will naturally produce a lot of data, unstructured and structured, and the issue is how to use it. When hotels do that so that they know what kind of room a guest prefers, even what floor they like, customers like that. They expect that kind of service.”

Arora also contrasted the Big Data case studies of the organizations like NASDAQ and ThyssenKrupp presented by Wissner with the more modest efforts discussed by the Canadians.

“Wissner showed use cases of organizations that have attained a very high level of maturity as far as Big Data goes, to show the ultimate end point of Big Data capabilities,” he said. “On the panel, we got some great insights. They are trying to reach that same ultimate stage of maturity, but are just at different stages in reaching that maturity level.”