But IoT's real benefit is in industry and services. The much-quoted Rolls-Royce jet engine that uses IoT sensors is just an example. Another is the use of such sensors to detect the number of plates that remain at a cafeteria line (and that trigger an event that causes the plates to be refilled if they run low).
Sensor data is irrelevant, however, if it isn't processed or at least stored for later processing, however. And depending on the device in question, the data stream coming from such a device can be immense, as in the jet engine (multiple terabytes per hour of flight).
Currently, the companies offering cloud computing (i.e. Microsoft, Google, Amazon and some outliers) are pushing for IoT data to be directly pumped into their respective clouds to be used for analysis. This can be problematic, however. Sticking with the example of the jet engine, the amount of data generated is not only extreme, it is also very difficult to route that information quickly enough into - in this case Microsoft's - cloud. The only live connection available to a jetliner in flight is a satellite uplink - very expensive and potentially not fast enough.
Bring in what IBM terms "Edge Computing", while Cisco prefers the term "Fog Computing" and Microsoft speaks of the "intelligent Cloud". Whichever terminus finally catches on is probably yet to be determined, but my preference is that of IBM's, as the term "Fog" implies "being in the cloud and not realizing it" to me.
It is actually quite surprising to me that this topic is seemingly sinking into the various cloud firms just now. Look at it this way: the human visual system doesn't pump raw sensor data (via the optic nerve) into the brain, as this would flood the brain with information, rendering it useless for other tasks (such as writing articles like this). Nature has realized long ago, that raw sensor data needs to be pre-processed before being handed off to the brain for interpretation, resulting in the development of visual cortex in all animals with complex vision systems.
Subsequently, it only seems reasonable that there is no need to dump terabytes of raw data into the "brain" of a cloud without reducing it to sensible batches of concentrated goodness first. Bring on the AI!
Some time ago, I wrote about an exciting new sensor being developed, that uses input from various sources (sound, light, movement) to determine whether Ann has left the water running at the kitchen sink (again) or the dog is crapping on the living room floor (again). This is "edge computing" at its finest - and compactest, as all the sensors are not strewn about the house, feeding data into the AI processing all that input, but rather they all sit on one, compact logic board which in turn feeds intelligent information to whatever backend system you have (like an app), such as "The water is running in the kitchen sink and no one is there".
Going back to the jet engine example, this is clearly a case where the consolidation of raw data into at least semi-intelligent output is an absolute imperative. My guess - to be honest - is that the story of a jet engine pumping terabytes of data into Azure per hour is a case of journalistic copy-catting. That's the same effect that caused half of all Formula-1 interested Germans to call the best slot at the start "Pool Position" (instead of Pole Position): some well-known journalist had fudged while writing up a report on some race that was "stolen" multiple times by other journalists not bothering to write their own report and just a short time later, you heard "Pool Position" not only from your friends but also from race commentators on TV!
It is unlikely that engineers at Rolls Royce put together a system that generates so much data, it can't be analyzed as it happens (which is the main idea behind pumping it into the cloud). Going by this article from 2016 there are 25 sensors feeding data such as fuel flow, temperature, pressure, etc. from various parts of the engine into the data stream.
However, wether the data stream is terabytes or megabytes per hour, the idea of feeding the raw data into the cloud just doesn't make sense. AI is more than capable of analyzing even the data from the 25 sensors mentioned in the article in a deep learning system and feeding more concentrated information into the cloud for final analysis. The reason for going to these lengths on a jet engine, though it will be the same for a car or a high-speed passenger train or your house, is to save energy and enable predictive maintenance.
The solution probably lies in multiple deep learning modules analyzing a subset of sensors for key indicators that can be relayed to the cloud for individual analysis. Even more important, of course, is to use aggregated data from as many jet engines, cars, trains and houses as possible to feed an AI that can make decisions based on the pre-chewed data from an entire airplane fleet, for example. This is where a cloud-based system "shines", though more and more of this "fleet analysis" activity will likely be run in small deep learning centers of specialized companies.
Nest's newest home surveillance camera, the Nest Cam IQ, not only comes with a whopping 4k sensor, but also with new "brains" to analyze the video and audio stream. These "brains" are not built into the camera, of course. We are felt lightyears away from having that sort of capability in a package this small. The AI doing the video analysis runs on the Nest Aware service available as a subscription. You can bet your booty that Nest benefits from Google's new TPU 2.0 hardware, as analyzing video and audio content simultaneously and live takes a lot of processing power.
Anyone (like me) that runs a home surveillance system based on simple pixel change algorithms knows the benefit that intelligent analysis of video streams would bring. To give you an idea of the difference, allow me to explain how current systems (with some exceptions) work. The goal of surveillance recording is not to net the most hours of video material. The goal is to single out security issues, either so that it is easy to find the relevant footage after the fact or - ideally - to get a system alert while an issue occurs. Traditional surveillance systems use a method where video pixel changes are analyzed and situations are flagged where a certain number of pixels (that corresponds to a certain size) changes in a certain timeframe. Think of a dog jumping over a fence or a door being opened.
While it is pretty simple to trigger an event if a door is opened that pretty much fills the camera frame, the situation changes drastically if the camera is recording a yard. Wind will move bushes and trees, and this will trigger the recording mechanism, resulting in a lot of recorded material. If you want to find out if someone is trespassing in your yard during the day, you'll have to look at hours and hours of video of moving trees.
There have been systems available that use algorithms to try to discern the shape of a person, for example. One of these that I have used personally is from Sighthound, Inc. which continued to develop the product I used several years ago ("Vitamin D"). Sighthound claims to have an SDK available that permits the training of a neural network used for facial recognition that runs locally on an iPad, using only that iPad's hardware.
While the training and recognition of individual faces may even work in a simple AI on an iPad, the video analysis that Nest offers goes several levels higher on the complexity scale. Think of it this way: instead of recognizing someone's face when they stand in front of a camera, a complete AI solution should be able to detect a lot more information, such as
"John Doe, wearing blue shorts and a Metallica T-Shirt. John just entered the dining room carrying a tablet computer and put that tablet computer on the table, leaving the room in the direction of the kitchen. John looks tired".
If that doesn't sound like big brother watching, I don't know what does!
Since the new Next camera not only records video in ultra high resolution but also sound in excellent quality, the data is sufficient for an AI to discern a lot of activities. Want to know how your aging mother is doing in her apartment three blocks down? In a scenario where the Nest service is connected to your Alexa account (which is planned), you could just say: "Alexa, what is mom doing right now?" and Alexa would answer "Your mother is sitting at the table doing a crossword puzzle." And while your mom might not appreciate being watched 24/7 by an online camera system, both of you will likely appreciate an urgent message being automatically sent out that your mom has been lying on the floor without motion for two minutes.
I would have an immediate use for a system like that: one that tells me when our cat is sitting in front of the terrace sliding door, waiting to be let in. I don't think I'll be putting these devices into my kids' bedrooms anytime soon.
These sensor boards (CMU calls them "Supersensors") use a plethora of different environmental sensors, such as:
- Radio interference
- Electromagnetic Noise (probably the same sensor as above)
- Motion X/Y/Z
- Light color
- Air pressure
- non-contact temperature
- Ambient temperature
The sensor data is fed into an AI that is trained to recognize events by their sensor signature, such as turning on a faucet, operating a microwave oven or even counting the number of paper towels used from a dispenser (Facility Managers listen up!).
While the project claims that the AI runs locally, my prediction is that - with the exception of large FM companies - most of these supersensors will likely feed event data into a cloud-based AI that is pre-trained in thousands of event types and continually learns from new signatures it receives.
While smart home automation is a great field for sensors like these, I see big advantages for healthcare as well. Attach one of these over each intense care bed and doctors as well as nurses - and most of all the patient - will benefit from registering key events such as shivering, shifting in bed, etc. Care-at-home patients will benefit just as much.
As with any data going into the cloud, I hope the Carnegie Mellon team is taking care to make sure there is no data being sent out that can be directly attributed to a household or an individual. Hack into one of these sensors and you'll figure our very quickly if someone is at home or not!
Where can you get your hands on one?
Well, the concept was just presented as a paper at CHI2017, so we're not talking ready-for-market devices.
Have a look at the project homepage, there are more details on the work done here. You can download the paper from this website.