....Teaming with a particularly ambitious group of computer scientists from the University of California at Berkeley, Sriharsha is installing a new data crunching platform inside the massive data centers that drive Yahoo's still enormous online empire. This software platform is called Spark, and according to those who built it and use it, it's about 100 times faster than the mighty Hadoop — and could very well replace Hadoop as the stuff that fuels the modern web.
'The goal is to build a new generation of data analytics software, to be used across academia and industry.'
— Ion Stoica
"The goal is to build a new generation of data analytics software, to be used across academia and industry," says Berkeley professor Ion Stoica, part of the team behind Spark.
Little more than three years old, Spark is very much a fledgling technology. But as Yahoo takes the plunge, according to researchers at Berkeley, Amazon is kicking the tires on the platform. Chip maker Intel is helping expand and improve the project at a lab in China that typically feeds larger Chinese websites like Baidu and Tencent. And Facebook, another key force behind Hadoop, says it's exploring the use of related software in the tools that help drive its everyday operations.
Part of the trick is Spark can store data in the memory subsystems of the thousand of servers it pulls together. Hadoop stores its data on good old fashioned hard disks, and grabbing data from memory requires far less time. But Spark also is what you might call a Swiss Army knife of Big Data analytics tools, says Reynold Xin, one of the Berkeley researchers who works on the project. Hadoop is often used in tandem with sister data analysis tools — tools that let you rapidly examine "real-time" data such as Tweets or ask questions of data via the familiar SQL query language — but Spark lets you do all this from a single piece of software.
"It works in a wide variety of ways," Xin says, "and in some cases, it works better than systems optimized just for a specific task."
http://www.wired.com/wiredenterprise/2013/06/yahoo-amazon-amplab-spark/?cid=8992684~A.
No comments:
Post a Comment