Nashville, TN – July 20, 2011 – Digital Reasoning™, the leader in unstructured data analytics at scale, today announced a recently published Impact Report by independent analyst firm The 451 Group titled, “Digital Reasoning Positions Military-tested Text Analysis Tools for Commercial Market,” is now available for download on the company’s website.

In the report, Nick Patience Research Director, Information Management at The 451 Group states, “Digital Reasoning’s patented text analysis technology, deep level of experience and apparent success in the proving grounds of US government intelligence show that Digital Reasoning has a product to be reckoned with.”

“We are pleased to be recognized for our expertise in big data analytics,” said Tim Estes, CEO at Digital Reasoning. “Our proven and patented software Synthesys, uncovers actionable intelligence from web content, email, social media, reports, and other unstructured data.”

Digital Reasoning recently announced Chinese language support for big data analytics in its flagship product Synthesys®. Synthesys is an entity oriented cloud-scale analytic solution that enables enterprises and government agencies to automatically make sense of complex data. Built to address the most complicated data analytics challenges, Synthesys excels at extracting, resolving and linking entities and concepts to provide context to the newly discovered information.

Arlington, VA and Nashville, TN – June 7, 2011 –Digital Reasoning^™, the leader in unstructured data analytics at scale, today announced Chinese language support for its flagship product Synthesys®. Synthesys can now analyze the unstructured data from a variety of sources in both English and Chinese to uncover potential threats, fraud, and political unrest. By automating this process, intelligence analysts can gain actionable intelligence in context quickly and without translation.

While English is still the most widely used language on the web, a recent report from The Next Web suggests that “[i]t could be less than five years before Chinese becomes the dominant language on the Internet.”

“This is a significant milestone for our company”, said Rob Metcalf, President and COO of Digital Reasoning,” said Rob Metcalf, President and COO of Digital Reasoning, “Whether for the public sector, financial services, health care or other enterprise applications, the next generation of Big Data solutions for unstructured data will need to natively support the world’s most widely spoken languages.”

I recently sat down with James Gardner to discuss Big Data and the promise of Automated Understanding. James is a senior research scientist, and a doctoral candidate at Emory University, who works principally in the area of natural language processing – from tokenization to named entity extraction, entity resolution, fact extraction and relationship extraction.

Since his undergraduate studies, James has been interested in Artificial Intelligence. That interest extended to natural language processing and machine learning, which he saw could be applied to increasing privacy in medical records.

The following is our interview:

-What is Big Data and what the inherent problems?

Data is being generated with more velocity and variability now due to social networking, blogging, webpages, etc.

This velocity and, even more so, the variability (or heterogeneity) defines big data. Think terabytes and petabytes (eventually exabytes and zettabytes) of unstructured or semi-structured information.

Big data increases storage cost, computation cost, and variety cost. Where the variety or variability cost is the most expensive due to the large amount of human effort required to make the connections in the data or development time necessary to integrate the systems.

I think the benefits of Big Data significantly outweigh the cost. It may be difficult for engineers to make response times fast for complex analytics over very large datasets, but those of us on the machine learning side really welcome Big Data, because more data in many cases allows for better predictions, especially in unsupervised or semi-supervised machine learning methods.

-What is Automated Understanding and how can it resolve the inherent problems within Big Data?

The velocity and variability of this data is unreadable even by all humans. I think that most people read the text that they write, but they definitely aren’t reading all of there social networking, or Internet traffic.

That last statement is alluding more toward the need to handle textual and “graphical” relationship content, but there is also a variety of video, audio, and temporal data associated with these events, in either meta-data or even in the content itself.  Computer aided summarization of these complex relationships and discoveries of entities is crucial for business and intelligence analysts to be able to make decisions quickly without having to read every document or data signal necessary to make a decision.

Human-aided summarization of data, where the human is helping the computer, is the future of big data.

Automated Understanding’s goal is to minimize the amount of human effort necessary to summarize these entity relationships.

-Is Entity Oriented Analytics the best approach to understanding unstructured data?

Humans think in terms of entities and therefore computers need to present this information especially when dealing with big data.

When there are millions of mentions of a single individual in a dataset you definitely don’t want to have to read every one of the documents, and you might not even care how many documents the entity occurs in if you can see relationships and other facts associated with the given entity.

This brings us to the point of how we even define entities. Entities, in the most general philosophical sense, are sets of subsets that are invariant over space and time. Notice the recursive definition. For most practical purposes we can define entities as things that exist in the real world and have certain properties that make them unique.

-What are some of the toughest problems we have to resolve to better make sense of unstructured data?

Named Entity Recognition is considered by many to be a nearly solved problem. This is the case in many situations, but is not necessarily true for all languages or for messy data. Synthesys has made great strides in accomplishing this goal by using the latest greatest techniques from the academic literature.

Even in those cases where the automated processes are not able to make a decision we give the users the ability to train Synthesys.

This training allows Synthesys to continually improve and adapt to the users needs. This training may be time-consuming but it can get the job done much more quickly. We can teach Synthesys a new language very quickly. It takes years to teach a human a new language.

Entity resolution (including disambiguation and clustering), association generation, and relationship generation are the hardest problems with the most promise for really affecting how users interact with and make use of big data in the future.

-What does the recent release of Siri mean to the AI community?

Siri is a pretty cool feature recently introduced for iOS devices. Speech recognition technology has finally evolved to the point that handheld devices can process audio and determine which application or task the user wishes to accomplish. This technology is actually not that new and we use very similar technologies and algorithms for dealing with various aspects of our text analytics, but Apple has really put together a neat combination of existing technologies (as they have always been very good at) that really allows users to more quickly, or at least with less effort, interact with their handheld devices.

I think Siri is like the Chatterbot systems from the 90′s with the added ability to interact with other applications on the device. This is likely going to turn out to be a great success for Apple, but I’m not really sure as to how much this will really affect the AI community. I think it’s more likely to influence developers that it’s probably worth while to accept natural language commands to perform task rather than having users have to follow a strict API (or language) to communicate with their software. This,I think, is extermely important for human computer interaction.

I think that developers should work to not only make it easy for humans to interact with the systems, but also create frameworks for systems to learn from one another through probabilistic interactions. Maybe call this computer-computer interaction.

We have been viewing computers in a binary (right, wrong, 0-1) light since the 50′s. This paradigm leaves us stuck with having to make a decision, and if the computer makes the wrong decision we automatically jump to the conclusion that computers are unable to reason as humans do.  But think of the mistakes that humans make all the way through life. The whole “touch a hot stove and get burned” example. Gyorgy Buzaski in the Rhythms of the Brain explains that this is the way learning happens. I interact with the environment and I get some sort of response. Sometimes those random interactions (or predictions) are accepted by the environment and are considered correct or valid interactions, while others aren’t and you end up with a first-degree burn. These unsuccessful predictions or mistakes are necessary for the learning process.

Alan Turing, way back in 1947, made the point that computers can never be both infallible and intelligent. Machine Learning research is now using this idea in full force. Many learning and inference algorithms are based on the idea that if the model makes a mistake then update the model, otherwise the model is already making the correct prediction so why change it. Very much like Occam’s razor combined with work only as long or as much as you need to.

I recently read an article written by the Nobel Prize winning psychologist, Daniel Kahneman, where he was describing his discovery of the “illusion of validity.” Eight soldiers try to get a log and themselves over a wall without ever touching it.

They watched the soldiers and tried to infer who was the best candidate to be a leader. It turns out that the teachers in the leadership school told them that they were failing to make good predictions.

Kahneman explained that in spite of knowing that their predictions were wrong they continued to use the same metrics to make the predictions. This allowed him to conclude that WYSIATI, “What you see is all there is.”

I don’t believe that the test was flawed, this was more likely an issue with understanding the goal (as in the great business novel by Goldratt).

It’s very likely that if they had measured and weighted the appropriate observable components of the exercise relative to how well the soldiers performed in officer school rather than how well they performed the log-task, the psychologist would have made much better predictions.

This is one example where machine learning could be used to learn how to make accurate leadership predictions, considering the presence of both the prior data (log-task features) and the ground truth from the teachers in officer school. I think it’s important that we always consider the goal of computing.  We developed computers to help us accomplish tasks more quickly.

Tim Estes, CEO and founder of Digital Reasoning, was a featured author in a recent IQT Quarterly publication. In this article, Tim helps the reader understand the meaning of “Entity Oriented Analytics” including how the mission has evolved over the past decade to bring us to this place and why entity-orientation is so necessary in today’s big data analytics solution architecture.

In this article, Tim references a decade of lessons learned in the intelligence community addressing topics including:

• How to deal with challenges when you don’t know what you need in advance
• Why static ontology can be limiting
• Challenges Relational DBs face with entity-centric missions
• Why analytics tools can be overwhelmed by these new data sets
• That there is no “uber algorithm” solution to all needs

In summary, Tim presents the case for an architecture that is both revolutionary and enduring.

View the full article by clicking here

While technology has provided several good paradigms for dealing with structured data (i.e. data that is structured in such a way to be easily decomposed into pre-defined fields), it has not kept pace with unstructured data, such as emails, blogs, web site content, etc. Thus, critical information is often kept “secret” through the obscurity of the sheer volume of data one must process, often manually, to reveal this information.

In response to this challenge, Digital Reasoning Systems, Inc has developed a comprehensive set of analytical tools packaged into product called Synthesys^® that essentially decomposes unstructured text into meaningful information easily understood and manipulated by a user.

This technology is based on the premise that there is order inherent in all languages that can be discovered and mathematically modeled. This has led to the development of our advanced data analytics and knowledge abstraction for unstructured data, based on a distinctive, patented mathematical approach to natural language processing.

For a better understanding of Synthesys^® and its capabilities, a down-loadable white paper (Synthesys – Technology Overview) providing a high-level overview can be found here.