How Accuracy Has Changed
Posted by 
Share this Post
AM radio used to be good enough, but like accuracy, our expectations and needs have evolved
There was a time when AM radio was all we had - and that was fine. There also used to be only a handful of television channels, which was fine, too. I don't remember longing for a wider variety of music or more channels. We had what we had, and it was all fine - it was all "good enough."
There was also a time when the level of accuracy that our intelligence and law-enforcement systems offered was "fine." We connected the dots well enough to eliminate the greatest threats.
Not anymore.
Today, there is an intense push for accuracy in our data and, particularly, in our ability to accurately "connect the dots." But what's changed?
What is accuracy?
I'm a mathematician. When I think of accuracy I think of numbers and percentages, of false-negatives and false-positives. But for law enforcement or intelligence officials, accuracy means tracking down and mitigating a potential risk before it happens. Both perspectives are critical.
Mathematically, accuracy is a pair of numbers. Accuracy compares the number of times you "miss" (present a false negative) and the number of times you incorrectly "hit" (present a false positive). Accuracy measures how well your process makes a decision - how well it can find a "true-positive" result amid the false negatives and false positives.
When you hear a phrase like, "our system is 95 percent accurate," it usually refers to the false-negative rate - or the connections it missed. To gauge the true accuracy of that system, you also need to know the false-positive rate.
If the system floods you with false positives, and touts a 95 percent accuracy rate (focusing on the things it missed), that's not going to get you very far. You're going to be spending all your time chasing false threats.
From an intelligence perspective, accuracy is just as much about keeping data apart as it is putting it together. If there were a security threat in a particular airport at a particular time, a less-than-accurate system might flag every person who was in the airport at that time.
A highly accurate system would be able to parse through the vast numbers of individuals in the airport at that time, "connect the dots" between those people and other data points within other records, and present a highly targeted list of suspects.
Accuracy is being able to make the best use of all the information you have - putting data together where necessary, and keeping it apart where necessary, to create a highly targeted list of "true-positive" results.
Why now?
The description of accuracy hasn't changed since the time of AM-only radio, but the need has changed - because our circumstances have changed. Two primary factors are driving the push for greater accuracy:
More information. Today's law enforcement officials have to deal with millions of terabytes more data than ever before. Not only are there more records about more people - a simple function of our digital times - there is significantly more travel, more people traveling on visas, more types of communication and a wider variety of threats.
More fragmentation. As the amount of information grows, so do the different locations and different types of information. From local police records to state databases to federal watch lists - and all the different types of entities (people, phone numbers, weapons) - intelligence and law enforcement officials are faced with a daunting task of connecting dots among all this information.
Plus, the risks are greater for missing important connections. Being marginally accurate is no longer good enough.
Finding the right technology
Finding the right solution is all about understanding what accuracy is and what you should expect from a highly accurate system. The most effective technology will:
- Let you look at the right 10,000 things, not just the top 10,000 things.
- Help you reduce the noise, particularly the signal-to-noise ratio
- Account for errors
- Overcome purposeful deception, common among persons of interest trying to avoid detection.
- Recognize and account for unique factors in different languages
- Allow you to introduce new information and new agents
- Securely exchange information between reliable sources
- Be flexible, so you can readjust your priorities according to changing threats, threat levels and resources
Accuracy involves many factors. And, it's a moving target. Because of the evolving nature of threats, we must keep working at this - we must keep pushing the envelope.
"Good enough" just doesn't work anymore. As soon as we can do more, we have to do more. And we have to keep pushing for greater accuracy and a greater ability to connect the dots. Our national security is at stake.
This post is part of a longer article, Dot Connecting in the New Data World, that originally appeared on ebizQ.com on June 23, 2010. For more on the topic, view our recent webinar replay: Connecting the Dots Isn't What It Used to Be.
3 Responses »
Trackbacks
Leave a Response
Entries(RSS)
I would add that there is more pressure on our data to perform than ever before.
Let me expand on that with a personal example...
A telecoms company I consulted with had a collection of legacy systems that had previously performed extremely well citing no major data quality issues.
The client wished to improve their service quality through innovations such as one touch note such automation, ERP integration, and much faster engineer routing to various equipment sites on their network.
The problem we found was that whilst the data was perfectly accurate of the needs of the old system it simply didn't support the new business processes and advanced customer services they wished to implement on the target system.
I think we're seeing organisations finally wake up to the value that their data assets contain and in many cases they are finding that the original accuracy simply isn't sufficient for this brave new world.
Scott,
Excellent post.
I would also like to add an additional driver for improving data accurracy - efficiency. As most organisations are now looking to do more things using more complex processing and with ever fewer staff, the accurracy of data becomes more critical. In the AM-radio days, there were enough staff to check outputs were correct and to use insight and knowledge to decide business outcomes without using complex analysis. Now organisations have less staff, they need to codify this knowledge (which can be challenging) and become more reliant on their data. Whereas in the past, at any point in the process if data was not accurrate or appeared wrong, staff intuition would flag this up for more checking, something which complex analysis algorithms find hard to achieve.
Julian