You must be using a very big data warehouse to store all the data. How do you collect and analyze it so quickly? (Please take us inside the head of an analyst who's using all these difficult algorithms:)
"Sense Networks has a lot of data and we have done a fair amount of engineering to quickly calculate various statistics over it. Part of this involved building the right data structures so that you don't have to search a giant database from A to Z each time you want to solve a query. For instance, we use efficient tree structures to avoid brute force search. Another important aspect of our algorithms is that they work with machine learning, the combination of statistics and computer science. Since the statistical methods are mostly about finding patterns and predictive trends in the data, they don't have to look at the whole database to lock onto a hypothesis once there is enough evidence for it. Similarly, we use computational algorithmic thinking to solve problems efficiently. One key tool we use fairly often is dimensionality reduction. This reduces the high dimensional measurements (consistent of hundreds of thousands of numbers) we obtain about movement in a city into a handful of dimensions while avoiding any real loss of information. The important relationships in the data are preserved but the raw data size itself has been compressed 100,000 fold. For instance, we can summarize a street corner in the city with a handful of numbers but the raw initial data we collected about it involved hundreds of thousands of GPS latitude/longitude/time pings."
Can you use Citysense on all future phones having a browser and GPS?
"We do hope that CitySense will be on many future phones with browsers and GPS capability. Following the BlackBerry version, we are releasing the iPhone version now and are in talks with other handset manufacturers to get it onto other platforms. Also, instead of building everything ourselves and porting CitySense to each platform, we are releasing parts of our API so that other companies (and even individuals) can work with us on their own development projects that build off of our data and our algorithms."
1 | 2 | 3
all interviews
