Redefining the Meaning of Big Data and Scalability with the Internet of Things
“The complexity for minimum component costs has increased at a rate of roughly a factor of two per year. Certainly over the short term this rate can be expected to continue, if not to increase. Over the longer term, the rate of increase is a bit more uncertain, although there is no reason to believe it will not remain nearly constant for at least 10 years.”
G. Moore, 1965
Moore’s law and its derivatives are widely applicable in a context much broader than transistors, their cost and density, as originally stated by then Director of Research and Development (R&D) at Fairchild Semiconductor, now co-Founder and Chairman Emeritus of Intel Corporation. The pace in which IoT is moving and the nature of technologies that make up IoT is not an exception. Ever decreasing hardware costs for sensors, microcontrollers and industrial computers, their increasing versatility and performance, as well as availability and bandwidth of data networks allow us to capture lots of physical parameters of remote systems: oil well machinery, truck telemetry, diesel generator operational parameters or the state of your home boiler or furnace. The availability of data presents a great opportunity for industrial and home use. Having access to streams of data in real time, as well as historical data makes our systems actually smart, allowing to respond to complex changes in behavior as well as analyze historical events.
"The pace in which IoT is moving and the nature of technologies that make up IoT is not an exception."
To fully embrace the new opportunity we need to learn how to deal with operational data and in order to do this, we need to drop our old beliefs of looking at our IoT systems the same way we look at consumer web or e-commerce applications or enterprise systems, where instead of users interacting with the system we have devices sending their data and polling for instructions. The biggest difference is that an IoT system needs to be more responsive and much more scalable. The good news is that there are a number of system design tradeoffs and paradigm shifts that make it possible. The closest consumer internet application that comes to mind is Twitter and one can think of IoT as a Twitter for devices. While Twitter is one in it’s kind there are dozens of unique IoT segments and potential applications. So what is there in IoT that makes it look like a Twitter and what IoT can learn from it? The similarity is short messages coming in at a very high rate; most of them matter in the real time or in a short period of time, a micro batch, which afterwards can be stored for later browsing, search and analysis. The same is true for IoT: we want our systems to respond to short device status updates and we want to analyze historical data, creating reports and adjusting our real-time models. An example could be a vibration sensor attached to an industrial motor, pump or some part of an HVAC system: it’s parameters are accumulated in real-time, and measured against historical averages for normal operation in given environment, triggering alerts when system parameters are outside of dynamically calculated range for normal operation. Twitter system re-engineering for scalability has been widely publicized http://www.wired.com/2013/09/the-second-coming-of-java/ and is out there for all of us to learn how to build message-oriented systems that scale.
The following is the most important architecture and infrastructure qualities the system should have that will help you evaluate your IoT systems or services provided by third party platforms. Those are greatly inspired by Reactive Manifesto (http://www.reactivemanifesto.org) and are now reinforced by increasing demands for reliable IoT.
• Responsive — focus on providing rapid and consistent response times, establishing reliable upper bounds so they deliver a consistent quality of service. This becomes extremely important when data is being pushed by sensor networks or other constrained equipment functioning autonomously allowing little room for error handling.
• Resilient — The system stays responsive in the face of failure. Failures are contained within each component, isolating components from each other and thereby ensuring that parts of the system can fail and recover without compromising the system as a whole. IoT systems are complex by nature and having resiliency supported at the very core of the system reassures that operational expenses of running the system under heavy load without downtime are going to stay manageable.
• Elastic — The system stays responsive under varying workload. Reactive Systems can react to changes in the input rate by increasing or decreasing the resources allocated to service these inputs. Very much like Twitter, there are bursts in the activity of IoT systems that are happening around certain times of day or events, maintaining big idle infrastructure as well as failure to respond to a spike in demand is equally damaging for business and operations.
• Message Driven — reactive systems rely on asynchronous message-passing to establish a boundary between components that ensures loose coupling, isolation, location transparency and provides the means to delegate errors as messages. Employing explicit message-passing enables load management, elasticity and flow control by shaping and monitoring the message queues in the system. Messages are in the very heart of the IoT, and backed systems should be built around that. As much as practical, the system rely on immutable records, so the state can be reconstructed for any point in time: a smart meter’s present value is all of it’s historical events applied to it. The same is true for system configuration: there should be recipe on how to automatically build, configure and provision each component of the system from ground up.
Similar to the early days of the Internet, the killer app for IoT is yet to be found, in the meantime, it’s important to make sure the components of IoT systems are reliable and built right based on the relevant knowledge derived from successful Internet services.