Contact Us
SaaS Log InXRay Login
Big Data

How Small Data Is Fulfilling The Big Data Promise

Nastel Technologies®
April 6, 2022

Big data is a funny phrase. It’s obvious that we produce a massive amount of data: 4.9 billion people use the internet for an average of seven hours per day. Hundreds of billions of sensors, located on everything from smartphones to satellites, stream data 24/7.

 

You would think that big data software gives us the ability to discover all the insights in the world. At the very least, we should be able to pull data on demand. In reality, the process has been slow and limited. Data lakes and warehouses require teams to log in, strip data from back-end databases, clean it, integrate it and run analytics pipelines. Insights are limited by the size of your IT infrastructure, which mirrors what you can afford.

 

The current generation of data lakes and warehouses can’t access the ocean of available data. This can only happen through something I call “small data.”

 

Small data fits data packets with secure and interoperable wrappers and remains decentralized. It fulfills the original big data promise that warehouses and lakes couldn’t meet. With small data, developers can build more effective AI models, researchers better medication and CEOs more competitive insights.

 

In other words, the big data promise of the 2010s is coming to fruition but not in the original architecture. As blockchains, verifiable credentials and other Web3 technologies seep into common usage, small data is becoming the architecture of choice.

 

You Already Use Small Data

If you’ve opened up a digital vaccination certificate to gain entry to an event, that’s small data. So is scanning a QR code to see a restaurant menu, contactless checkout, the Clear pass at airport security and anything that fits into Apple Wallet or Google Pay.

 

Small data refers to interoperable, purpose-specific data containers. They can exist anywhere, from the cloud to an SMS message to a DMV database.

 

The idea sounds new because it is. In the mid-2000s, when digital architects realized the value of capturing online data, the natural progression involved stashing it somewhere for further consideration—the data lake—then structuring and filtering it for further use in the warehouse. Cloud computing helped make the architecture affordable. YouTube, Amazon, ride-hailing apps, LinkedIn and the digitized daily activities of Web 2.0 became possible.

 

When it comes to trust and the ability to collect remote, real-time data on demand, however, the architecture falls flat. The only way to (mostly) trust data is to receive it through a trusted pipeline or mechanism: a login or a firewalled data lake. Access is limited to data specialists. With Web3 on the way, the old data architecture is ramping up for a big change.

 

An Age Of Exponential Growth

We live in an age of such rapid growth that the pace of innovation is hard to track. If you’d told someone in 1976 how, thanks to Moore’s Law, the people of 2022 would have powerful, pocket-sized smartphones that act as mini computers, running on cheap software and entertaining us by way of on-demand cat videos, how many would have believed you?

 

Another wave of change is on the way. Moore’s Law gave us complex, powerful computers at an affordable price. Web 2.0 digitized many aspects of daily life. Today, signs of Web3, which is defined by machine-to-machine communications, are creeping into our lives.

 

Web3 demands that data break free from its fences and carries its own trust and verification so that everyone can use it and new kinds of innovations can take flight. Smart thermostats, 5G towers, bitcoin, NFTs and Apple Wallet are initial signs of a much bigger transformation. Nonlinear growth in global data and the exponential growth of AI promise to propel us into jaw-dropping realms.

 

Expect similarly fast, awe-inspiring innovations in the near future. They’ll be driven by AI, autonomous machines and decentralized small data.

 

Why Isn’t Small Data Everywhere Already?

If you’ve used a new application and scratched your head at how easy the process was, there’s a good chance small data is behind it. Examples include online mortgages, real-time diversified portfolio tracking software and decentralized creator platforms.

 

Why doesn’t every website use small data? Two technological advances have to happen to make small data possible.

 

1. It must be backed by trust and verification. Users must be confident that the data hasn’t been tampered with. Blockchain ledgers, decentralized IDs, verifiable credentials or some other cryptographic technology must verify the origin and movements of a piece of data. An NFT, for example, loses all its value if it’s been altered. A college transcript sent through a verifiable credential loses meaning if someone went in and changed the GPA. Small data guarantees the identity of the originator and that the data hasn’t been changed along the way.

 

2. Small data must be self-describing. It must contain a shared vocabulary, meaning it’s interoperable. For example, the boarding pass from your Apple Wallet is interoperable with an airline’s scanning system. You know what the boarding pass is and so do the computers that scan it. Every container of small data must have similar qualities. Right now, most data isn’t interoperable. We query data by being logged into something and taking hours to integrate whatever data that we can get our hands on.

 

A combination of upstart innovators and well-heeled existing companies are taking the leap to small data, but it’s not yet a widespread movement. The concept itself is still gaining traction—heading toward a tipping point.

 

Small Data, Big Change

Innovation often lags behind the hype. Small data fulfills the old promises of big data in the same way 2010s e-commerce fulfilled the promises of Webvan in the 1990s and Tesla fulfilled the promise of the 1980s-era electric car.

 

Just as those who jumped on Web 2.0 data-driven architectures early stood to gain the most (think Amazon and Facebook), those who step into small data will find a wide playing field for new market ownership. And although the volume of data remains huge, the real revolution happens when the data is small.

 

This article originally appeared on forbes.com, to read the full article, click here. 

You can read more about XRay, Nastel’s solution for real-time analytics of big and small data here.

Nastel Technologies is the global leader in Integration Infrastructure Management (i2M). It helps companies achieve flawless delivery of digital services powered by integration infrastructure by delivering tools for Middleware Management, Monitoring, Tracking, and Analytics to detect anomalies, accelerate decisions, and enable customers to constantly innovate, to answer business-centric questions, and provide actionable guidance for decision-makers. It is particularly focused on IBM MQ, Apache Kafka, Solace, TIBCO EMS, ACE/IIB and also supports RabbitMQ, ActiveMQ, Blockchain, IOT, DataPower, MFT, IBM Cloud Pak for Integration and many more.

 

The Nastel i2M Platform provides:

Comments

Write a comment
Leave a Reply
Your email address will not be published. Required fields are marked *
Comment * This field is required!
First name * This field is required!
Email * Please, enter valid email address!

Schedule your Meeting

 


Schedule your Meeting


Subscribe

Schedule a Meeting to Learn More