Apache Spark Graduates from Incubator
The Apache Foundation has voted to elevate the Apache Spark project from the Apache incubator and give it full status as an Apache top-level project. Apache Spark process defines itself as "… a fast and general engine for large-scale data processing." The tool has recently come to prominence as a fast-processing analytics framework for Hadoop. According to the Spark project website, Spark can "run programs up to 100 times fast than Hadoop MapReduce in memory or 10 times faster on disk."
The recent attention to Big Data technologies has raised the visibility of Spark, as developers look for ways to speed up processing of large and complex queries. The project claims to have had 5,000 commits in the past 6 months, and other promising data-analysis projects, such as Shark for SQL, MLib, and GraphX, are based on Spark.
Related content
-
Apache Storm
We take you through the installation of a Storm cluster and discuss how to create your own topologies. -
Detecting security threats with Apache Spot
Security vulnerabilities often remain unknown when the data they reveal is buried in the depths of logfiles. Apache Spot uses big data and machine learning technologies to sniff out known and unknown IT security threats. -
The New Hadoop
Hadoop version 2 expands Hadoop beyond MapReduce and opens the door to MPI applications operating on large parallel data stores.
- Databricks Fully Open Sources Delta Lake
-
Big data tools for midcaps and others
Hadoop 2.x and its associated tools promise to deliver big data solutions not just to the IT-heavy big players, but to anyone with unstructured data and the need for multidimensional data analysis.