I was most impressed with one story I heard at the Smart Cities Connect confab in Austin. It brought to mind Walt Disney’s Fantasia and Mickey Mouse as the Sorcerer’s Apprentice. For those of you who remember the cute little rodent who worked for the Sorcerer. The boss left him in charge, and who could resist putting on that pointy hat and picking up the magic wand.
Mickey was supposed to carry water but he found that with a few ‘waves’ of the wand, he could get the broom to carry the water without him having to break into a sweat. Of course all hell breaks loose as the brooms bring more and more water and begin to flood the castle. Mickey can’t stop the onslaught, and is swept up in the tsunami. The Sorcerer arrives in the nick of time and saves the day and a chastened mouse goes back to what he does best.
Oh yes, the story. It seems that an electricity supplier for a major city in the Northeast had a data collection operation going on. When it began a couple of decades ago, they were collecting a gigabyte of data a year. Time passed and they began to install sensors throughout their network. Today they collect a gigabyte every thirty minutes.
Is it possible to be able to actually use that nearly 18 terabytes of raw data? Who is the sorcerer and what kind of magic wand is going to be used? And keep in mind that the data equaling 18 terabytes was from the electric company alone. What happens when you add in streets, trash collection, lights, traffic signals, police, fire, CCTV cameras, and the rest.
It seems to me that the collection of the data is the easy part. The big job is the processing of the data into something usable. To do that you first have to know what your goal is, and how the data will help you reach it. Then you can perhaps process the data in such a way that it makes sense.
The parking folks know about what they need for parking, the water and power folks know what they need, as to the street, police, fire, and trash departments. Who knows enough to pull all that data together and make sense from it?
I”m not saying it can’t be done. I just think that this is where the focus needs to be.
JVH
One Response
Nice point, John! As a data scientist, I’d say yes and no to your assessment. On the one hand, you are absolutely right that processing this amount of data expeditiously and insightfully is close to impossible because the data is constantly updating. Google server farms is the first that comes to mind but developing productive algorithms would require first to understand the value of all data in its entirety – and that’s very complex for a human being. So, even most complicated algorithms would still be limited by what humans can digest and aggregate (machine learning offers some paths to scale but that’s more dangerous than beneficial given some recent examples).
On the other hand, most of what we currently do to answer social science questions with available evidence is look at administrative data – that is, data that was not collected for the purpose of answering that specific question (of course, there are other data types but admin data is most interesting from this perspective). Surveys are very expensive and if we can utilize administrative data and link it to other datasets – we significantly enrich our understanding of social, economic and mobility phenomena.
So, yes and no: we might never be able to meaningfully analyze 18 terabytes of vaguely structured data (and growing) but we might well find interesting data points in all this jungle that would answer our most pressing questions. As such, the data is more of an enabler rather than a direct answer to those questions.