Streams ‘R’​ Us: Distributed Data Processing in the 21st Century

Plenty is being said about the amount of data that exists and the amount that will be created in our world. More is implied than said about how the processing of data has fundamentally changed over the years and how application architectures and hardware infrastructure have evolved.

The combination of fast and affordable computing and pervasive connectivity has turned data processing on its head. The approach to processing has gone from store-and-process to process-and-store. Couple this with scale-out architectures for processing and we’re cooking…cooking data like we’ve never done before.

Pervasive Connectivity and Computing

Looking back, the blending of connectivity and computing that underlies both client-server and cloud-mobile opened up a plethora of options to combine compute and network to distribute intelligent processing. We are now able to optimize for costs and responsiveness and realize low-latency processing at the best cost.

We put these choices to work when we choose to develop a native mobile app vs. web/HTML5 mobile app, or when we decide how much processing Amazon Echo does at home vs. in the cloud.

Scaled Out Stream Processing

Taking advantage of this compute-network merger meant we could scale-out the processing required, not just across distributed data centers and cloud, but also to the edges where end-points like wearables, phones, and Alexa exist. Further, we are not processing data in batch, but streaming data (“data-in-motion”) before it is stored.

Our infrastructure now processes streaming data like humans do, on-the-fly. Not after the data is parked in some flash storage equivalent to our sub-conscious.

The net result of stream processing on a scaled-out infrastructure looks like this. We are used to calling it IoT.

Stream Data Processing in a Connected World

If distributed artificial and human intelligence across a labyrinth of connected devices is IoT, so be it.

However, when at my command, Alexa streams Bach’s Cello Concerto performed by Yo Yo Ma, I won’t be thinking IoT. I’ll be thinking Streams ‘R’ Us. That is, if a comparison between biological information processing and computers enters my mind at all!

Hari Harikrishnan

Hari Harikrishnan

Digital Business Transformer
As a digital business executive, Hari is passionate about creating and modernizing businesses by infusing data and technology in product and service portfolios. Hari has led businesses from startups to enterprise incubations to Fortune 100 business divisions.

Hari has commercialized numerous products and services in industries ranging from healthcare to high-tech to manufacturing. His career spans leadership roles in general management, product, marketing, services, and strategy.

Leave a Comment

Your email address will not be published. Required fields are marked *

Download Whitepaper

Get a whitepaper on how the lifecycle of a Thing (product or asset) has changed with digital technologies.

Scroll to Top