While most of the attention has been on infrastructure like virtualization / hypervisors, Hadoop, and NoSQL data storage systems, we think those are really the enablers of the killer app for Cloud- which is making sense of data to solve information overload. Without next generation analytics and supporting technology, it is essentially impossible to:

Analyze a flow of data from multiple sensors deployed in a factory

Process mobile traffic at a telephone company

Make sense of unstructured and structured information flowing through an email system

Identify key entities and their importance in a stream of financial news and transaction data.

These are the real world problems that have engaged me for many years. I founded Digital Reasoning to automatically make sense of data because I believed that someday all software would learn and that would unleash the next great revolution in the Information Age. The demand for this revolution is inevitable because while data has increased exponentially, human attention has been essentially static in comparison. Technology to create better return on attention would go from “nice to have” to utterly essential. And now, that moment is here.

Digging a little deeper, Digital Reasoning has created a way to take human communication and use algorithms to make sense of it without having to depend on a human design, an ontology, or some other structure. Our system looks at patterns and the way a word is used in its context and bootstraps the understanding much like a human child does – creating associations and building into more complex relationships.

In 2009, we migrated onto Hadoop and began taking on the problem of managing very large scale unstructured data and move the industry beyond counting things that are well structured and toward being able to figure out exactly what the data means that you are measuring.

Digital Reasoning asks the question: “How do you take loose, noisy information that is disconnected and unstructured and then make sense of it so that you can then apply analytics to it in a way that is valuable to business?”

We identify actors, actions, patterns, and facts and then put it into the context of space and time in an efficient and scalable way. In the government scenario, that can mean to finding and stopping bad guys. In the legal environment they want to answer the questions of “who”, “what”, “where”, and “when”.

Digital Reasoning initially set our focus on the complex task of making sense out of massive volumes of unstructured text within the US Government Intelligence Community after the events of 9/11. But we also believe that our Synthesys software can be utilized in the commercial sector to create great value from the mountains of unstructured data that sit in the Enterprise and streaming in from the Web.

Companies with large-scale data will see value in investing in our technology because they cannot hire 100,000 people to go through and read all of the available material. This matters if you are a bank and trying to make financial trades. This matters for companies doing electronic discovery. This matters for health sectors that need help organizing medical records and guarding against fraud.

We are an emerging firm, growing rapidly and looking to have the best and the brightest join our quest to empower users and customers to make sense of their data through revolutionary software. With the recent investment from In-Q-Tel and partners of Silver Lake, I believe that Digital Reasoning has a great future ahead. We are on the bleeding edge of what is going on with Hadoop and Big Data in the engineering area and how to make sense of data through some of the most advanced learning algorithms in the world. Most of all we care that people are empowered with technology so that they can recover value and time in the race to overcome information overload.

I recently sat down with James Gardner to discuss Big Data and the promise of Automated Understanding. James is a senior research scientist, and a doctoral candidate at Emory University, who works principally in the area of natural language processing – from tokenization to named entity extraction, entity resolution, fact extraction and relationship extraction.

Since his undergraduate studies, James has been interested in Artificial Intelligence. That interest extended to natural language processing and machine learning, which he saw could be applied to increasing privacy in medical records.

The following is our interview:

-What is Big Data and what the inherent problems?

Data is being generated with more velocity and variability now due to social networking, blogging, webpages, etc.

This velocity and, even more so, the variability (or heterogeneity) defines big data. Think terabytes and petabytes (eventually exabytes and zettabytes) of unstructured or semi-structured information.

Big data increases storage cost, computation cost, and variety cost. Where the variety or variability cost is the most expensive due to the large amount of human effort required to make the connections in the data or development time necessary to integrate the systems.

I think the benefits of Big Data significantly outweigh the cost. It may be difficult for engineers to make response times fast for complex analytics over very large datasets, but those of us on the machine learning side really welcome Big Data, because more data in many cases allows for better predictions, especially in unsupervised or semi-supervised machine learning methods.

-What is Automated Understanding and how can it resolve the inherent problems within Big Data?

The velocity and variability of this data is unreadable even by all humans. I think that most people read the text that they write, but they definitely aren’t reading all of there social networking, or Internet traffic.

That last statement is alluding more toward the need to handle textual and “graphical” relationship content, but there is also a variety of video, audio, and temporal data associated with these events, in either meta-data or even in the content itself.  Computer aided summarization of these complex relationships and discoveries of entities is crucial for business and intelligence analysts to be able to make decisions quickly without having to read every document or data signal necessary to make a decision.

Human-aided summarization of data, where the human is helping the computer, is the future of big data.

Automated Understanding’s goal is to minimize the amount of human effort necessary to summarize these entity relationships.

-Is Entity Oriented Analytics the best approach to understanding unstructured data?

Humans think in terms of entities and therefore computers need to present this information especially when dealing with big data.

When there are millions of mentions of a single individual in a dataset you definitely don’t want to have to read every one of the documents, and you might not even care how many documents the entity occurs in if you can see relationships and other facts associated with the given entity.

This brings us to the point of how we even define entities. Entities, in the most general philosophical sense, are sets of subsets that are invariant over space and time. Notice the recursive definition. For most practical purposes we can define entities as things that exist in the real world and have certain properties that make them unique.

-What are some of the toughest problems we have to resolve to better make sense of unstructured data?

Named Entity Recognition is considered by many to be a nearly solved problem. This is the case in many situations, but is not necessarily true for all languages or for messy data. Synthesys has made great strides in accomplishing this goal by using the latest greatest techniques from the academic literature.

Even in those cases where the automated processes are not able to make a decision we give the users the ability to train Synthesys.

This training allows Synthesys to continually improve and adapt to the users needs. This training may be time-consuming but it can get the job done much more quickly. We can teach Synthesys a new language very quickly. It takes years to teach a human a new language.

Entity resolution (including disambiguation and clustering), association generation, and relationship generation are the hardest problems with the most promise for really affecting how users interact with and make use of big data in the future.

-What does the recent release of Siri mean to the AI community?

Siri is a pretty cool feature recently introduced for iOS devices. Speech recognition technology has finally evolved to the point that handheld devices can process audio and determine which application or task the user wishes to accomplish. This technology is actually not that new and we use very similar technologies and algorithms for dealing with various aspects of our text analytics, but Apple has really put together a neat combination of existing technologies (as they have always been very good at) that really allows users to more quickly, or at least with less effort, interact with their handheld devices.

I think Siri is like the Chatterbot systems from the 90′s with the added ability to interact with other applications on the device. This is likely going to turn out to be a great success for Apple, but I’m not really sure as to how much this will really affect the AI community. I think it’s more likely to influence developers that it’s probably worth while to accept natural language commands to perform task rather than having users have to follow a strict API (or language) to communicate with their software. This,I think, is extermely important for human computer interaction.

I think that developers should work to not only make it easy for humans to interact with the systems, but also create frameworks for systems to learn from one another through probabilistic interactions. Maybe call this computer-computer interaction.

We have been viewing computers in a binary (right, wrong, 0-1) light since the 50′s. This paradigm leaves us stuck with having to make a decision, and if the computer makes the wrong decision we automatically jump to the conclusion that computers are unable to reason as humans do.  But think of the mistakes that humans make all the way through life. The whole “touch a hot stove and get burned” example. Gyorgy Buzaski in the Rhythms of the Brain explains that this is the way learning happens. I interact with the environment and I get some sort of response. Sometimes those random interactions (or predictions) are accepted by the environment and are considered correct or valid interactions, while others aren’t and you end up with a first-degree burn. These unsuccessful predictions or mistakes are necessary for the learning process.

Alan Turing, way back in 1947, made the point that computers can never be both infallible and intelligent. Machine Learning research is now using this idea in full force. Many learning and inference algorithms are based on the idea that if the model makes a mistake then update the model, otherwise the model is already making the correct prediction so why change it. Very much like Occam’s razor combined with work only as long or as much as you need to.

I recently read an article written by the Nobel Prize winning psychologist, Daniel Kahneman, where he was describing his discovery of the “illusion of validity.” Eight soldiers try to get a log and themselves over a wall without ever touching it.

They watched the soldiers and tried to infer who was the best candidate to be a leader. It turns out that the teachers in the leadership school told them that they were failing to make good predictions.

Kahneman explained that in spite of knowing that their predictions were wrong they continued to use the same metrics to make the predictions. This allowed him to conclude that WYSIATI, “What you see is all there is.”

I don’t believe that the test was flawed, this was more likely an issue with understanding the goal (as in the great business novel by Goldratt).

It’s very likely that if they had measured and weighted the appropriate observable components of the exercise relative to how well the soldiers performed in officer school rather than how well they performed the log-task, the psychologist would have made much better predictions.

This is one example where machine learning could be used to learn how to make accurate leadership predictions, considering the presence of both the prior data (log-task features) and the ground truth from the teachers in officer school. I think it’s important that we always consider the goal of computing.  We developed computers to help us accomplish tasks more quickly.

1. It Takes a Network

2. It Takes Integrity

3. Don’t Confuse Activity with Productivity

4. You can’t improve anything unless you measure it

5. Small Efforts Repeated Add Up to Great Results

1. It Takes a Network:

Retired US General Stanley McChrystal wrote an article entitled “It Takes a Network: The new front line of modern warfare” for the March/April 2011 edition of Foreign Policy. In it he wrote the following:

“In bitter, bloody fights in both Afghanistan and Iraq, it became clear to me and to many others that to defeat a networked enemy we had to become a network ourselves. We had to figure out a way to retain our traditional capabilities of professionalism, technology, and, when needed, overwhelming force, while achieving levels of knowledge, speed, precision, and unity of effort that only a network could provide.

Further into the article, he writes:

“It was clear, though, that in this fusion process we had created only a partial network: Each agency or operation had a representative in the tent, but that was not enough. The network needed to expand to include everyone relevant who was operating within the battlespace. Incomplete or unconnected networks can give the illusion of effectiveness, but are like finely crafted gears whose movement drives no other gears.

This insight allowed us to move closer to building a true network by connecting everyone who had a role — no matter how small, geographically dispersed, or organizationally diverse they might have been — in a successful counterterrorism operation. ”

When I returned to Digital Reasoning a few weeks ago, I saw striking parallels with what we were trying to accomplish with Big Data. In this particular instance, it truly takes a robust, connected network that is often highly distributed to make sense of the ever-growing volume, velocity and complexity of data. However, something more fundamental struck me – one of approach. In order to produce the best technology and become a learning organization requires 360 degrees of situational inputs and awareness. Bottomline, our various organizations must include everyone. Is marketing talking to the engineers? Is research and development talking to the end user? Is the president talking to the developers? Put another way, you have to remove the protected fiefdoms and silos for fusion to truly exist. At the end of the day in Afghanistan, the infantry Soldiers our forward were able to succeed, because they had a dedicated network of support – the same is true here at Digital Reasoning.

2. It Takes Integrity:

In a previous deployment to Iraq, I had the distinct honor of serving as Gen. Martin Dempsey’s Deputy for Public Affairs. Bar none, he has been my best mentor within the military. While visiting Bagram earlier this year, he and I had a few moments to speak. During our conversation he shared that while he could requisition anything for war, he could not requisition integrity.

I am especially proud of my team at Digital Reasoning and have always believed that integrity outweighs ability. You can teach anyone almost anything except how to choose the hard right over the easy wrong – that has to be inherent and natural.

One of the resounding criticisms of Army Public Affairs, especially in Afghanistan, is that we are often too slow. In a 2007 study entitled “The Taliban’s Information Warfare”, the author notes:

“The Taliban are often on international media with their messages within 60 minutes of a major event, considerably faster than ISAF can counter the Taliban’s messages, due to the requirement to investigate, confirm and gain approval through the chain of command before it can release a press statement to rebut or counter Taliban’s claims.” Ultimately, this is a matter of integrity. The Taliban simply are not held to nor do they hold themselves to any ethical standards of truthfulness. While, admittedly, Coalition Forces are slower with the news – we are first with the truth.

In a 2001 article for Fast Company, author Jim Collins wrote about exceptional companies and commented that “leaders of companies that go from good to great start not with “where” but with “who.” They start by getting the right people on the bus, the wrong people off the bus, and the right people in the right seats. And they stick with that discipline—first the people, then the direction—no matter how dire the circumstances.”

Ultimately, whether in business or war, you are asking yourself “Do I trust the person to my left and right?”. Trust simply cannot exist without integrity.

3. Don’t Confuse Activity with Productivity

While at Bagram Airfield in Afghanistan, I was involved in an experimental cell, which aimed to rebut, refute or confirm civilian casualties, which has become a leading concern for warfighters throughout Afghanistan. As it true of all organizations, people or businesses early in their life cycle, more attention is often given to the volume of work you are doing rather than the effect your work is actually having. Ultimately, it can become a chore of accounting, which is decoupled from optimization. For instance, you can pay more attention to how many press releases you are sending out to your target audience when you should be asking “Does my audience care about what I am saying and are they doing anything once they know what I have told them?”. Many organizations have adopted social media and will gladly tell you how many Twitter followers they have, how many likes they received on Facebook, and how many unique visitors they have had over the month. The next question ought to be “So what?”

In a 2009 Stanford University published its research findings regarding multitasks. After extensive research they concluded that “[p]eople who are regularly bombarded with several streams of electronic information do not pay attention”. According to Professor Clifford Nass, one of the researchers, these “high-tech jugglers” were “suckers for irrelevancy” and easily distracted.

In order to yield the best possible results from our efforts, we must engage in focused activity, which is tied to a measurable, specific outcome.

4. You can’t Improve Anything Unless You Measure It

This lesson is closely associated with Lesson #3, because it also requires focus. The problem is not one, per se, of measurement. As an Army at war, we measure a lot, which is the underlying issue. We provide leadership with so many metrics that we dilute the impact of the most important metrics. In the words of the American cognitive scientist Herbert Simon “[w]hat information consumes is rather obvious: it consumes the attention of its recipients. Hence a wealth of information creates a poverty of attention, and a need to allocate that attention efficiently among the overabundance of information sources that might consume it.” So then the question is no longer about if you should measure it is what should be measured. What are my key indicators for any stated goal or desired outcome?

So, at the end of the day, it is critical to first choose what should measure, and then measure it consistently. Peter Morville in his book Ambient Findability argues that what we find shapes what we become. By extension, what we look for to measure shapes the results.

While on combat patrols with an infantry company in Afghanistan, I spoke to the company commander and asked him how many insurgents he had killed. It seemed like a good question and, at first glance, appeared to be a useful metric. His answered surprised me – he didn’t keep count. To him, it was more useful to count other things such as the number of times local villagers informed Coalition Forces where improvised explosives were or the number of indirect fire attacks or the number of local elders that were willing to sit down and speak to him and his men. At the end of our conversation he remarked that we can’t simply kill our way out of this problem, we have to build trust with the Afghan people and demonstrate that what we can provide is far better than the insurgents.

5. Small Efforts Repeated Add Up to Great Results

Consistency is the hallmark of maturity. Regardless of circumstances, we must do the right thing every time, every day. I joined a mounted convoy to Mehtar Lam with an infantry platoon just a few days before many of them were to return to the United States. The platoon leader dismounted elements of his platoon to clear several danger zones of improvised explosive devices. Though no IEDs were found, he admitted it must be done every time. He said, “[y]ou cannot let the enemy dictate your next move. I will dismount every time, because the one time you don’t is the one time you may end up losing a Soldier due to complacency.”

“Success”, wrote Robert Collier,”is the sum of small efforts, repeated day in and day out.” But, again, they must be the right efforts – efforts guided by good sense and good character.

On the passing of Steve Jobs, I find myself surprisingly moved for someone I did not know personally. The extent of my relationship to Steve Jobs is that I was one of those hundreds or thousands that sent an email to Steve at sjobs@apple.com in 2004 and he wrote back (unexpectedly) to start a brief chat. At the time he was battling pancreatic cancer, but he still found time to send back a note to a random user of his technology. I’d hoped that after a few years of growth at Digital Reasoning, someday we’d meet Steve and his team at Apple and find a way to work with them. If and when that day comes, he will no longer be there and only his vision will carry forward. Even so, parts of that vision live on in many of us beyond Apple as a responsibility to make technology simple by making it smarter and making it for everyone.

A friend and mentor of mine who had worked with Jobs in the early days of Apple had recently told me that Steve was a great “assembler of technologies” versus an inventor of them. The more I think about it, the more I see the wisdom in that description. Did he invent the MP3 player? No- but he put “1,000 songs in your pocket” before anyone else did in a form you actually felt cool to have and take with you. Did he invent the graphical user interface? No, but his team at Apple made it usable with major innovations such as allowing windows to overlap each other (something the Xerox Star OS at PARC couldn’t do)- giving depth and utility to the desktop metaphor that we now take for granted in all computing devices. Did he invent the tablet PC? No- but he brought a device to market that made touch a second nature interface combined with elegant software delivery to make iPads something that many families have for reading to games to light office work just two years later. He had a singular instinct to know when and how to democratize technology by making it beautiful and culturally relevant that only improved with age.

In a time when the role of America in the world is sometimes uncertain and looked on negatively, Apple’s products are a symbol that innovation and elegance that makes the world better still begin and succeed here. The irony of Apple outstripping Exxon as the most valuable company in the world over the last few weeks is an amazing metaphor into our civilization. One massive company selling something we have to have for civilization to continue being surpassed by a company that sells hundreds of millions of devices all over the world that no one needs but most of us choose is a testimony to the unique and compelling vision of Steve Jobs.

Democratizing technology is a calling of great moral significance. Whether it is Twitter allowing dissidents to escape the limitations of speech in oppressive regimes, or getting clean water technology into small form factors for impoverished regions, or protecting the privacy and lives of our users through better software that people can own – we have only begun to empower and improve the lives of humanity through innovation.

There is much left to do. Much more than we have already done. The world has computers and they are connected to each out. The ecology is in place because of Steve’s generation, but the Cambrian explosion of linking and understanding data is just beginning. And that should make us excited and driven – even as we are saddened by the loss of a trailblazer that came before us.

As I write this, I’m listening to “Homeward Bound” by Simon & Garfunkel in my iTunes. If ever there was a “poet and a one-man band” of the technology industry, it was the two Steves of Apple. Now we say goodbye to the poet even as I type this email on a laptop his company designed and with music playing from a store he invented through a program that brought more music to the world than anything in history.

Life is short. Live brave and be fearless. You might just change the world.

There is no story of us without the story of 9/11. Even ten years after, words should be sparing. Our memories still speak to us more vividly than any writing.

The day was singular because all of us shared something that day where it did not matter our differences for a while after that morning. We were all on notice that the simple fact that we were Americans made us targets that day. We were one.

We shared heroes. And we still do. We have the great privilege to work with those heroes. Together we strive because we remember that any day can move from surreal calm to calamity in an instant. We work and we hope and we pray that our efforts hold back those days so long as we strive to our fullest. For ten years, we have had no days of calamity here at home – and this is not by chance or luck. It is the gift of heroes.

And on this morning- quiet for now like ten years ago, we can hear the echoes of those heroes that fell – from 40 men and women over the skies of Pennsylvania to 30 Navy Seals hunting the still vicious disciples of the killers of 9/11 in the heart of darkness in Afghanistan. I hear them say in the silence and peace they have made…

“It is a far, far better thing that I do, than I have ever done; it is a far, far better rest that I go to than I have ever known.”

The way many companies did business changed in an instant on Sept. 11, 2001. The terrorist attacks ushered in an era of fear and uncertainty for government and businesses. Contingency planning for emergencies and disasters took center stage. Spending on security skyrocketed, and for some Nashville-area companies, that meant opportunities….

Digital Reasoning was a year-old technology startup with two employees before the attacks. Soon after, it became a key part of the United States’ global counter-terrorism initiative. The company’s information-sifting software landed a five-year, $7 million contract from the U.S. Army National Ground Intelligence Center and helped the Franklin-based company grow to more than 30 employees.

“It was just a product or idea; what 9/11 did was give it a reason to exist,” said Tim Estes, founder and chief executive officer of Digital Reasoning. At the time, the weakness of U.S. intelligence agencies wasn’t a lack of data, the 9/11 commission later noted. Rather it was the inability to “connect the dots” of cross-departmental intelligence data. That’s where Digital Reasoning comes in. Its software is able to analyze massive amounts of data and find links that previously could only be discovered if someone were to actually read through it all. While demand in the intelligence sector shows no signs of slowing down, the company has begun going after new markets.

“Eventually every hedge fund in the world will want analytic capabilities that are the most advanced to make decisions for obvious reasons,” Estes said. “In the next year and a half, we expect there will be a lot of people investing in capabilities to understand data on the Web, data inside their enterprises for gain or for risk mitigation.”…

Tim Estes, CEO and founder of Digital Reasoning, was a featured author in a recent IQT Quarterly publication. In this article, Tim helps the reader understand the meaning of “Entity Oriented Analytics” including how the mission has evolved over the past decade to bring us to this place and why entity-orientation is so necessary in today’s big data analytics solution architecture.

In this article, Tim references a decade of lessons learned in the intelligence community addressing topics including:

• How to deal with challenges when you don’t know what you need in advance
• Why static ontology can be limiting
• Challenges Relational DBs face with entity-centric missions
• Why analytics tools can be overwhelmed by these new data sets
• That there is no “uber algorithm” solution to all needs

In summary, Tim presents the case for an architecture that is both revolutionary and enduring.

View the full article by clicking here

“Our work with Digital Reasoning bolsters Cloudera’s commitment to fostering Hadoop adoption in the U.S. government market,” said Mike Olson, CEO of Cloudera. “With CDH3 integration, Digital Reasoning’s government customers can extract valuable and actionable insight from the enormous amounts of data they generate.”

We are very happy to have such a solid working relationship with Cloudera. Both companies are committed to bringing the best solutions for highly scalable Entity Oriented Analytics to leading enterprise and government customers.