Lead Image © Ivan Trifonenko, 123RF.com
Where Is Your Big Data?
Welcome
You'd think that massive amounts of data wouldn't have the opportunity to be elusive, but we know this isn't true from the sheer number of data breaches in the past couple of years. Big data seems to be one of the greatest sources of pain for enterprises and online businesses alike. But where does all that data come from, where does it go, and why is it so hard to maintain? At first glance, the answers seem simple. Upon further inspection, the answers are still pretty simple.
Where does big data come from? This somewhat inappropriately configured question's answer is logfiles. Logfiles are by far the biggest culprits in big data generators. Every device on your network generates some type of logfile. Those logfiles either are kept on the local systems that produce them or they're sent to some type of log aggregator for further processing. Or not – meaning that someone might collect them but never bother parsing them. Preserving logfiles simply for posterity is a waste of bandwidth and disk space. If you collect logs, then you should parse, scrape, and process them for relevant and actionable information, including security breach data.
Where does the data go? The answer to this question shouldn't be much of a mystery because of how logfiles are saved or sent to another system for processing. Unfortunately, logfiles are often forgotten. Someone once called logfiles our digital exhaust. The moniker is accurate enough, because once we've jettisoned those logfiles, they're out of sight and out of mind. For a lot of us, their fate falls into the "good riddance" category. "No one looks at those stupid logfiles anyway" goes the swan song of many well-meaning but shortsighted system administrators. If you're not looking at your logfiles with some sort of aggregator and alerting system, then ignoring your big data is destined to become your biggest mistake – a mistake because you're missing security information, performance data, and
...
Buy this article as PDF
(incl. VAT)
Buy ADMIN Magazine
US / Canada
UK / Australia
Related content
-
Manage logs with logrotate
Take charge of your installation's logfiles with logrotate. -
A modern logging solution
As systems grow more complex and distributed, managing and making sense of logs used for monitoring, debugging, and troubleshooting can become a daunting task. Fluentd and its lighter counterpart Fluent Bit can help you unify data collection and consumption to make sense of logging data. -
Security is Everyone's Problem
I attended a security seminar a few weeks ago, and one of the slides read, "Security is not an IT problem." I laughed when I saw it and gave a smirk to our Security Manager at my new job. -
Detecting security threats with Apache Spot
Security vulnerabilities often remain unknown when the data they reveal is buried in the depths of logfiles. Apache Spot uses big data and machine learning technologies to sniff out known and unknown IT security threats. -
Welcome
Heisenberg's Uncertainty Principle states that you can know either the velocity or the direction of an electron, but not both at the same time – at least not in a precise way, you can't. The Hess Uncertainty Principle states that you can't know (precisely) both the speed and the direction of computing. I thought of this during my recent trip to Austin, Texas, where I attended Dell World 2014. Dell turned 30 years old this year, and I never would have predicted, with any accuracy, the new direction that Dell has taken. Nor would I have predicted its current velocity. How could I have known both simultaneously? The short answer is that I couldn't. I'm not even sure that Michael Dell could have predicted, as little as five years ago, the speed and the direction of his company today.
Subscribe to our ADMIN Newsletters
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Most Popular
Support Our Work
ADMIN content is made possible with support from readers like you. Please consider contributing when you've found an article to be beneficial.
