Technology is improving Siri, powering driverless cars, improving cancer treatment and even being called Big Brother. But “big data” is what makes it possible, and why it’s so important.
Big data refers to the analytic algorithms applied to vast amounts of data across several different places, or simply the math and computer formulas used to sift through massive amounts of data and analyze the results to answer questions and solve problems. The edge big data has over traditional analytics is its ability to include data types that aren’t organized in tabular formats, including written documents, images and video.
Big data has become a meme, shorthand for advancing trends in technology that can shed light on understanding the world and inform better decisions. And, it comes at a time when the amounts of data being created is exploding — growing at 50 percent a year, according to IDC estimates.
The powerful concept may revolutionize the planet with tools that can combat poverty, illness and crime while fueling unexpected and explosive innovation. Here are five reasons big data is already a big deal and affecting everyday life.
1. Big Data is Big in Business
The data increase isn’t just more information on the same stuff, but entirely new data streams generated by countless digital sensors on nearly everything from shipping crates of goods produced thousands of miles away to location data stored in a teenager’s smartphone.
Linking these sensors and their corresponding data to computing intelligence helps businesses better track shipments, define most efficient truck routes for goods, and target advertising to potential consumers.
Speakers at the MIT Sloan CIO Symposium earlier this month predicted big data can create a significant competitive advantage for companies in just about every kind of industry.
Research last year by Professor Erik Brynjolfsson and two other colleagues, recently reported in the New York Times, supports this, suggesting data-guided management is spreading across corporate America and starting to pay off.
The trio studied 179 large companies and found that those adopting “data-driven decision making” achieved productivity gains that were 5 percent to 6 percent higher than other factors could explain.
In turn, the rise in productivity attributed to big data is also translating into a demand for employees with these skills. A report last year by the McKinsey Global Institute projected that the U.S. needs 140,000 to 190,000 more workers with deep analytical expertise and 1.5 million more data-literate managers, whether retrained or hired.
The trend is ushering in the new job title of data scientist, which requires employees skilled in programming, statistics, and business analysis. This type of employee won’t necessarily originate in the ranks of IT departments, highlighting how this data discovery and decision-making employee is part of a new breed.
“It’s a revolution,” says Gary King, director of Harvard’s Institute for Quantitative Social Science. “We’re really just getting under way. But the march of quantification, made possible by enormous new sources of data, will sweep through academia, business and government. There is no area that is going to be untouched.”
Right now, the novelty of the technology is slowing efforts to create best business practices and many large organizations are still experimenting with this growing business opportunity, which is making its way into consumers’ hands more immediately in terms of new products.
2. Big Data Fuels Culture
The computer tools for gleaning knowledge and insights from the Internet’s data are advancing artificial intelligence techniques like natural-language processing, pattern recognition and machine learning which are making their way into consumer products.
Apple’s coveted virtual smartphone assistant, Siri, is evolving in part because of machine-learning algorithms. In preparation for her debut last fall on the iPhone 4S, Apple fed Siri reams of data so she could answer a variety of questions about weather, restaurants and directions.
As her universe of questions expands, thanks to big data, Siri is aiming to become the main mistress of the coming “smart home” by controlling thermostats, lights and home theater systems.
Another example is Google’s experimental robot cars, which use several artificial-intelligence and decision-making tricks to auto-pilot the driving experience.
For those who are tech averse and think heaven is a day outside at the national pastime’s ballpark, watch out: big data has been there, too.
Michael Lewis’ 2003 book, “Moneyball,” which detailed success in massaging data and baseball statistics to produce a powerhouse team, is now a primer for Major League management. “Moneyball” also hinted at big data’s predictive quality, another reason the trend is a big deal.
3. Big Data Can Predict The Future
The predictive power of Big Data shows promise in many realms. Different agencies are looking to big data from places like social media as it becomes the digital age’s “water cooler,” or the place to gauge the populations’ opinions and reactions.
The Federal Reserve Bank of New York is recognizing social media’s potential to shape public opinion, and requested monitoring and analysis of Facebook and Twitter conversations to better respond to public opinion. The real-time data will be categorized as “positive, negative or neutral,” so the agency can reportedly monitor public perception as it moves forward.
Wall Street now analyzes this data to inform investment decisions, and political parties are jumping on the trend, too. President Obama’s Twitter town hall drew nearly 170,000 questions and comments, whose content is analyzed for voter sentiment.
Researchers keeping an ear to the Twitter feed, which can as a barometer for many things, can get a heads-up on vaccination rates, and by reviewing spiking Google searches for phrases like “flu symptoms,” and “flu treatments,” big data can expect an increase in flu patients hitting emergency rooms weeks in advance.
The huge data mine of social network information reflects collective online behavior and tracking it and other datasets can paint a thorough picture. For example, Google data and image queries for housing-related issues is a more accurate predictor of housing sales, besting forecasts of real estate economists.
“I look for hot spots in the data, an outbreak of activity that I need to understand,” says Jon Kleinberg, a professor at Cornell. “It’s something you can only do with Big Data.”
4. Big Data Could Fight Cancer
Continuing the work of identifying potential health concerns in a population, big data is making strides in treating medical conditions.
Healthcare providers can significantly improve medical outcomes by detecting patterns of unhealthy behavior exhibited by patients and using that data to educate and influence behavior toward preventive medicine and home care.
And, collecting and manipulating enormous amounts of data will play a vital role in research and delivery of cancer treatment.
Last week, UC-Santa Cruz researchers announced plans with the National Cancer Institute to create the world’s largest depository for cancer genomes. The Cancer Genomics Hub provides researchers with a huge and growing database of biomedical information used in “personalized” or “precision” care, whereby the treatment targets specific genetic changes found in an individual patient’s cancer cells.
“Big data collection and computing is allowing us for the first time to get a complete molecular characterization of cancer,” said David Haussler, director of the Center for Biomolecular Science and Engineering at UC-Santa Cruz.
“I think you’re going to start to see this sort of big data effort on several fronts — partly because of supercomputing capabilities that we haven’t had until recently and also because of wireless devices that are increasingly being used to transmit data,” Haussler said.
The $10.5 million genome project will incorporate genetic information from 10,000 cancer patients fighting 20 kinds of adult cancer and five childhood cancers. The project’s concentration of previously unheard of amounts of data has the potential to change the way doctors treat cancer.
“Cancer is incredibly complex, with different kinds of tumors within the same kids of cancer,” Haussler said. “By having the ability to examine so many different cancers, we hope to move toward the idea of precision or personalized medicine, treatment tailored to your specific molecular makeup.”
The funding for this one project is part of $200 million six federal agencies have committed to the “Big Data Research and Development Initiative,” and its results are expected to cross-over into other research, like that applying to stem cells and other diseases.
These projects join another federal project, the two-day Health Datapalooza next month in Washington D.C., which will examine new methods for collecting and using big data.
5. Is Big Data Big Brother?
Big Data has its pitfalls, too. In addition to the queasy feeling that all our online activity is being scrutinized, there is the potential for an increase of “false discoveries.” The notion of diving in to a massive pile of data and finding the one true interpretation has its challenges.
There is also the temptation for some data scientists to begin their search with a set of biased “facts” and mining to find results that align with that presumption.
Also, big data is a model of research, and while the results can offer an understanding, they are often subject to over-simplification, resulting in correlations that are unfair or discriminatory. This type of misuse can influence the types of products, bank loans or health insurance a person is offered. Privacy advocates point to these potential drawbacks when calling for caution in the advance of big data.
Big data is to this time what earlier measurements, like the microscope, telephone surveys, and early computer data were in the past: a way to see and measure things as never before.
Google searches, Facebook posts, YouTube videos, Pinterest posts and Twitter messages, for example, make it possible to measure behavior and inclinations in fine detail, as it happens, and as a basis to predict future events. While the fledgling field still has to overcome privacy obstacles and establish best practices, there is massive potential for it revolutionize decision-making and innovation through decisions on data and analysis.
Though other methods of critical thinking, like experience and intuition, will still have a place in the world, Big Data is working to make the world much more scientific — for better or worse.