Arlington, VA and Nashville, TN – June 7, 2011 –Digital Reasoning^™, the leader in unstructured data analytics at scale, today announced Chinese language support for its flagship product Synthesys®. Synthesys can now analyze the unstructured data from a variety of sources in both English and Chinese to uncover potential threats, fraud, and political unrest. By automating this process, intelligence analysts can gain actionable intelligence in context quickly and without translation.

While English is still the most widely used language on the web, a recent report from The Next Web suggests that “[i]t could be less than five years before Chinese becomes the dominant language on the Internet.”

“This is a significant milestone for our company”, said Rob Metcalf, President and COO of Digital Reasoning,” said Rob Metcalf, President and COO of Digital Reasoning, “Whether for the public sector, financial services, health care or other enterprise applications, the next generation of Big Data solutions for unstructured data will need to natively support the world’s most widely spoken languages.”

Synthesys is a “big data” analytics solution that automatically “reads” text, transforming it into a “network” or underlying facts, connections and associations.

You can also view our Synthesys^® Demo here

Digital Reasoning, Santa Clara, CA – February 2, 2011 – Digital Reasoning^™, the leader in complex, large scale unstructured data analytics, today announced Synthesys^® Platform, an innovative, open, and scalable version of our market- leading software.

In order to promote innovative uses of Synthesys^® and demonstrate its capabilities, we are pleased to offer hacked program access to this forthcoming version of our Synthesys product. This platform version of our software will provide beta users immediate API-level access to our analytics software and access to tools that will be added through the beta program.

“We are excited to introduce Synthesys^® Platform to the market,” said Matthew Russell, vice president engineering at Digital Reasoning. “By allowing users to upload their data into the cloud for analysis, many more users will get the opportunity to experience next generation data analytics while exploring their own data.”

During our beta period we are inviting interested parties to register for acceptance into the program. Our beta program participants will contribute to making Synthesys® Platform a powerful new solution for cloud-based data analytics.

“The O’Reilly Strata Conference is the perfect venue for introducing Synthesis Platform and our beta program,” added Rob Metcalf, President and COO of Digital Reasoning.  “We see the introduction of Synthesys® Platform as a fundamental step in advancing our market and discovering new markets and opportunities for “big data” analytics.”

Synthesys® Platform is available now. Stop by our booth at Strata Conference (#305) or select “Platform” our Website for more information.

About Digital Reasoning^®

Digital Reasoning Systems (www.digitalreasoning.com) solves the problem of information overload by providing the tools people need to understand relationships between entities in vast amounts of unstructured and structured data.

Digital Reasoning builds data analytic solutions based on a distinctive mathematical approach to understanding natural language. The value of Digital Reasoning is not only the ability to leverage an organization’s existing knowledge base, but also to reveal critical hidden information and relationships that may not have been apparent during manual or other automated analytic efforts. Synthesys^® is a registered trademark of Digital Reasoning Systems, Inc.

Tim Estes, CEO to speak “Generating Dynamic Social Networks from Unstructured Data”

Matthew Russell, VP Engineering to speak “Unleashing Twitter Data for Fun and Insight”

Visit us at Booth #305

While most of the attention has been on infrastructure like virtualization / hypervisors, Hadoop, and NoSQL data storage systems, we think those are really the enablers of the killer app for Cloud- which is making sense of data to solve information overload. Without next generation analytics and supporting technology, it is essentially impossible to:

Analyze a flow of data from multiple sensors deployed in a factory

Process mobile traffic at a telephone company

Make sense of unstructured and structured information flowing through an email system

Identify key entities and their importance in a stream of financial news and transaction data.

These are the real world problems that have engaged me for many years. I founded Digital Reasoning to automatically make sense of data because I believed that someday all software would learn and that would unleash the next great revolution in the Information Age. The demand for this revolution is inevitable because while data has increased exponentially, human attention has been essentially static in comparison. Technology to create better return on attention would go from “nice to have” to utterly essential. And now, that moment is here.

Digging a little deeper, Digital Reasoning has created a way to take human communication and use algorithms to make sense of it without having to depend on a human design, an ontology, or some other structure. Our system looks at patterns and the way a word is used in its context and bootstraps the understanding much like a human child does – creating associations and building into more complex relationships.

In 2009, we migrated onto Hadoop and began taking on the problem of managing very large scale unstructured data and move the industry beyond counting things that are well structured and toward being able to figure out exactly what the data means that you are measuring.

Digital Reasoning asks the question: “How do you take loose, noisy information that is disconnected and unstructured and then make sense of it so that you can then apply analytics to it in a way that is valuable to business?”

We identify actors, actions, patterns, and facts and then put it into the context of space and time in an efficient and scalable way. In the government scenario, that can mean to finding and stopping bad guys. In the legal environment they want to answer the questions of “who”, “what”, “where”, and “when”.

Digital Reasoning initially set our focus on the complex task of making sense out of massive volumes of unstructured text within the US Government Intelligence Community after the events of 9/11. But we also believe that our Synthesys software can be utilized in the commercial sector to create great value from the mountains of unstructured data that sit in the Enterprise and streaming in from the Web.

Companies with large-scale data will see value in investing in our technology because they cannot hire 100,000 people to go through and read all of the available material. This matters if you are a bank and trying to make financial trades. This matters for companies doing electronic discovery. This matters for health sectors that need help organizing medical records and guarding against fraud.

We are an emerging firm, growing rapidly and looking to have the best and the brightest join our quest to empower users and customers to make sense of their data through revolutionary software. With the recent investment from In-Q-Tel and partners of Silver Lake, I believe that Digital Reasoning has a great future ahead. We are on the bleeding edge of what is going on with Hadoop and Big Data in the engineering area and how to make sense of data through some of the most advanced learning algorithms in the world. Most of all we care that people are empowered with technology so that they can recover value and time in the race to overcome information overload.

At Digital Reasoning Systems, we say that the model of the data comes from the data itself without imposing a pre-determined model. There is a lot that is being said there and it would help to talk about the many assumptions found in that statement and the reason for this choice.

If you are not familiar with the concept of Ontology here it is in a nutshell. The study of Ontology examines metaphysics, the existence of things. It also suggests that all objects can be grouped, related to each other in a hierarchy and concepts classified by their similarity and differences to other concepts.

Classical ontologies are hierarchical and form trees. At the top the standing assumption is the world can be divided between concrete objects and abstract objects. Well at least typically. You could root the tree with a more complex asymptotic relationship. For example it is considered by the Buddhists that reality is a very personal perception. Thus one can suggest the root ontology consists things I think exist, things I think don’t exist, things I imagine and things I can’t imagine. Some things have a dual nature. For example, money is an abstract concept while cash is a concrete object that refers to the abstract concept of money. If you don’t understand the difference consider the case of hyper-inflation where cash loses nearly all connection to value. Where before you could expect to buy a loaf of bread for a few dollars in a normal economy; one beset by hyper-inflation may require literally millions of dollars.

A tree like hierarchy of things assumes they fall into neat categorization. It doesn’t take much digging before you find conflicting cases. How to create your ontological tree depends upon opinion more than science. You have two solutions possible with conflicts. One is to create new hybrid categories. The other is to shoehorn the things that don’t clearly fit into a category into what in your opinion is the nearest category.

Creating new categories every time there is an apparently ill-fitting object defeats the purpose of having a neat, traversable tree. Is a tomato a fruit or a vegetable? You could just simply state it is a fruit since it flowers. Botanically you are correct. You may applaud yourself for sticking with science. Hmm, but, there is that nagging bit; tomatoes aren’t sweet like fruit. They are savory. When you look at the genetics of the tomato you notice that they are in the same family as nightshade plants (Solanaceae.) Some of these plants are quite toxic. The family also includes the potato, mandrake, paprika, eggplant and many more. Well those don’t sound like fruit. Tobacco is in the same family but I am certain your doctor would not consider smoking to be a method for getting your daily requirement of fruit! The use of the tomato in the culinary arts is similar to other vegetables. So, based upon that, one could claim it is ok to refer to the tomato as a vegetable.

Ontologies can be slippery things when we create a model for them. We can get even further confused when we discuss the concept of identity. Let’s assume we have an accepted ontology we have put into use of common things. Let’s say we are interested in ships. When is a ship, a ship? Is it a ship when it is on the drawing boards? Perhaps it is an abstract ship at that point. How about during assembly? How far along into assembly does it go from an abstract concept? With an imposed model there is no gradient between Abstract.Ship and Concrete.Ship. Let’s say the ship is complete but still up on the dry dock. Is it a ship? It isn’t functioning as a ship, though it could if it were in water. What makes it different than a very complex sculpture of a ship? If it is in the water and sailing we can assume that it is a Concrete.Ship as it is functioning like a ship and has all the attributes of a ship. Let’s say the anchor falls off. Is it still a ship? I would say yes. What if the engine has trouble or the rudder becomes stuck? These are various bits of state change but I would say it is still a ship. If more and more things start to fall off the ship, is it still a ship? It may reach a point where it is less complete than it was when it was under construction when we said it wasn’t yet a ship. At what point before it sinks is it no longer a ship? Finally, consider the Queen Elizabeth. She caught fire in Victoria Harbor, Hong Kong in 1972. She eventually capsized. If she was no longer a ship was it due to the initial fire or the later capsizing? If she could have been righted and repaired would she be a ship again? If she had never had the fire and had become a floating university as intended would she still be a ship?

The point of all this is that while these are fairly simple and common state changes they point to some interesting issues with enforcing a manmade ontology onto objects – it’s not always clear what identity an object has and over time it that identity may change.

Association as Ontology

Digital Reasoning’s technology employs the concept of the association network. This network examines the properties and use of an object in human language, it derives the most meaningful associations and it proceeds to create a network map. A meaningful association can be something as simple as being a subject or object in relationship to another concept. There could be a meaningful chain of associations.  For example “K278” (submarine) may be associated with ”190MW” through its association to “reactor”. K278 and reactor may share an association to “fire,” which was the cause of the submarine’s sinking.

Unlike a hierarchical tree, an association network does not imply complimentary association.

It does not imply a top down order either. Every node represents a concept. Every concept has a list of associations. Think of this as saying to yourself, “when I think of this concept, I also think about these other concepts.” These can range from attributes of the concept to other similar concepts. This can be extremely helpful. When we conduct a search we know what we are looking for. This is great in general. The power of Google™ is obvious to anyone who uses the internet today. Simply type in what you are looking for and you will get access to hundreds of mostly relevant documents. Sometimes we are looking for just one specific document and we don’t remember exactly what is written in it. Or perhaps we never knew but we still suspect it exists. We know the topic so we start there with our search. Associations can help us out here in a powerful way. Consider the following example:

You are searching for a document about Quantum Cryptology. Let’s say a search using “quantum cryptology” doesn’t produce the document because that exact phrase isn’t in the document. Associations can help. The associations of Quantum Cryptology would include other types of cryptography such as secret key, public key, hash functions as well as lots of terms associated with cryptology in general such as cipher feedback, Rivest cipher, NIST and many other terms. However the real power comes from having access to concepts tightly connected to the topic concept that might help find the document you need. Terms like quantum communication and quantum processing, quantum channel, QKD and other concepts can be used to augment the search. Associations can help you provide these terms.

In an association network the inter-relationships can be represented by a hypergraph. You can get a full visualization of conceptual relationships. Each relationship has a strength and thus a distance which can push two concepts apart or pull them closer together. The complication is that in many cases Concept A is near and related to Concept B. Concept A is also near and related to Concept C but Concept B and Concept C have no direct relationship. This is where associations start to take on the look of Venn diagrams.

Let’s look at an example where we can see how this plays out. We will look at the example from the point of view of determining the ontology of a “Golf Cart.” You know what a golf cart is. You’ve seen them, maybe ridden in them and have a general understanding of them. The typical golf cart has 4 wheels, a roof, a steering wheel, storage space, a motor and so forth. Many have electric motors but some are fueled by propane or gasoline. So lets quickly come up with a chain of categories that a golf cart would be in:

Concrete_object.machine.vehicle.cart.golf_cart

That is nice but ANSI standard Z130.1 says carts are not self-propelled and that golf cart is a misnomer and it should be called a car. Ok, an easy fix.

Concrete_object.machine.vehicle.car.golf_cart

This is lovely but it has a problem. Cars bring up a certain image and set of properties. For a wide variety of things we call cars they all share properties that the golf cart does not have. Cars can run on roads, golf carts can’t. Golf carts can run on fairways and cars cannot. The local quick oil change place will work on your car but likely won’t on your cart. OK so a golf cart isn’t in the family of cars so we create a concept called “golf equipment” and move the cart there. Now we are done.

But wait! You aren’t done because “golf carts” are found at other sporting events like football games where they sometimes are used as ambulances for injured players, at airports as inter-terminal transport, and in parks. So some are “sports equipment” and some are clearly some sort of vehicle. There are several possible designations we can now apply and they are all correct in context. The new Semantic Wikimedia will deal with this problem in exactly this way by examining in context what exactly is this instance of golf cart.

With an association network the uses and meanings of golf cart, golf buggy and golf car as concepts will build relationships both strong and weak and while not related to a car the closeness to car will be apparent by the shared associations and position.  A golf cart is not a car but it is closer to a car than a toy car. It is closer to a car than a bulldozer. This will be obvious by the relationships the network has for all of these concepts. The question never comes up with an association network of whether something is or isn’t a particular class of item. Instead of forcing a concept to be something it sees it as a series of networked relationships. It allows the concept to be many things, closer or more distant to other concepts.

In the real world this can be incredibly useful. If I am researching message traffic and I am interested in finding “purchase/acquire transactions” for man portable air defense systems (shoulder fired anti-aircraft missiles) then I’d want the associations for the generic acronym “MANPAD” and from that I will get various attributes that might help me find what I am looking for. One thing that will associate strongly will be names of specific systems such as Stinger, Olga, Redeye, Strela, but also other names will come up such as Javelin, RPG-7 and Saxhorn. The last 3 are designed for anti-tank work. Why would they come up? The fact is that if you laid all 7 of these systems out on a table they all, at a gross level, look the same. The basic concept in design is the same for them all. The differences are all in the details. Here, the ontology of MANPAD would normally exclude these systems. Why you would be interested in including these results are varied but some rationalizations are that if the document in question is an eyewitness account then perhaps the witness was incorrect about what they saw. Another is that if you can smuggle a Javelin then they most certainly can smuggle a Strela. Further – let’s say you never heard of the QianWei MANPAD you most certainly would want it to be suggested as part of your search augmentation. Other associations that would come up would be attributes of MANPADs and be ontologically dissimilar but still highly important. For example sub-parts such as rocket motor designations, warhead design, related technology you were previously unaware of and so forth. The document you are looking for might not contain the term MANPAD or the name of a system but instead be about someone very interested in spare parts to maintain their cache or to build their own.

The association network found in Digital reasoning’s Synthesys™ goes beyond a strict, imposed ontology. It offers a refinement by building the model from the data. It augments our ability to find what we need using search by giving us freedom from relying solely upon keywords we have guessed at.

We’re excited about Monday’s announcement from Steve Arnold that he has launched a new site called Inteltrax (http://inteltrax.com/).  Steve Arnold is the well-known search and technology analyst who has been conveying his opinions, advice and reviews on Beyond Search for some time.  As Steve explains the reason for the name “Inteltrax” is: “The “intel” in our name refers to the systems, often nurtured in the traditional data mining and text analysis needs of government agencies, justice officials, and enforcement professionals. The “trax” means blazing a trail.

We are pleased that Inteltrax is including information on Digital Reasoning together with other key technologies and vendors addressing the challenges of “Big Data”.  Steve and other market leaders understand that new solutions are required in order to make actionable use out of the flood of information – largely unstructured – that is common in today’s market.  Our flagship product Synthesys^® uniquely addresses this challenge by extracting information and underlying linkages out of unstructured and structured data on a scale of hundreds of millions of documents or more – thus fundamentally changing the way analysts can approach “cloud scale” data problems.

About the ongoing content in Inteltrax, Steve further writes: “What we want to do is document the shift from key word information retrieval to more sophisticated content processing, analytics, and output of actionable information”.

We applaud Steve Arnold’s launching of Inteltrax – a site focused on developing a better understanding of the players and technologies that will surely transform the state of making enterprise data useful to the business and intelligence communities.  Here are some of the references to Digital Reasoning on Inteltrax. We at Digital Reasoning will be working to be a significant part of this coming evolution.

Here at Digital Reasoning, we have been dealing with the challenges of automated understanding of massive amounts of unstructured data for years. Knowing that Tim Estes, our CEO, might have a different view on this issue, I decided to interview him. Tim has worked within the realms of unstructured data analytics, artificial intelligence, and machine learning for the past decade.

The following is our interview:

Jason Beck – In the article, one researcher suggests that developers shouldn’t build analytics robots, but rather “exoskeletal systems”. Do you agree?

Tim Estes - I think that it’s a matter of degree. The range of judgments that a machine can make as a proxy for the human is constantly and necessarily expanding. Even R2D2 was most famous for taking orders from Luke Skywalker trying to accomplish tasks from fixing the X-wing in flight to cracking into computer networks.

Just to be a little more accurate – Iron Man wouldn’t work without an AI that is close to R2D2. Jarvis (the AI program that runs’ the Stark house and the Iron Man suit) is always chatting up Tony Stark about what’s going on with the suit and the risks that are present around him. The Iron Man analogy means we seed the full situational awareness (the sensory and data input space) to the machine with the human making key decisions on the filtered and prioritized information. I think that’s about right.

R2D2 is distinct in having a measure of its own intentionality (i.e. it is autonomous in more dramatic ways than Jarvis/Iron Man suit) but they are much more close than you might think. Should humans get out of the loop in making analytic judgements? No more than we should have pilots out of the loop in flying commercial airlines at this time. But show me a pilot that can fly a 747 without computer assistance and guidance? We are already in the hybrid space. And the complexity of our technology and the explosion of the information created by machines and man assisted by machines means we will need ever increasing automation in understanding.

JB – Doesn’t the exponential growth of data and decreasing levels of available talent necessitate automated systems?

TE - Exactly. The notion that “augmented intelligence” can solve the full data problem is wishful thinking. Something has to read everything and that can no longer be a human as a matter of scale. We have to make strides to catch up intelligent systems with the complexity and scale of the data we are being inundated with.

JB – Is this an Either-Or situation? Just because someone may prefer automated systems, does this assume that there won’t be any human in the loop?

TE - I think that’s the real issue – where is the dividing line right now and where is it going to be in 5 years? Right now – machines have to read and organize everything. The race is to see who can do it accurately, at scale, and focused on the entity-level vs. the document level. In five years, the information overload will be so substantial that autonomous proxies or agents will likely be the baseline for all of these systems. In both situations, humans are in the loop. Now – they have much greater heavy lifting because nearly all of our enterprise information systems don’t really understand their data that well so the burden is on the reader. That has to change. Even when it does, we will just be enabling the humans to make better decisions in less time and less interruption of their daily lives.

JB – Does the delineation between these two approaches represent a common split in the overall text analytics community?

TE - I think so. We can either be satisfied with augmenting the status quo or we can get to the root of the issue – that software doesn’t understand natural signals that make up unstructured data. We are in a place of diminishing returns with simple classifiers and ETL (Extract, Transform, Load) architecture. The more exciting alternative, however, is to go at the semantic and scale problems with the appropriate technologies and transform the enterprise to be entity-oriented.

JB – Can you think of any example where someone tried to completely automate text mining?

TE - Not off the top of my head. I’m sure there have been. But a lot of text mining is feeding either fancy search engines (such as faceted navigation and data enriched topic clustering) or Business Intelligence frameworks.

JB – What does the future look like regarding automation?

TE - Its going to go from being reactive (search, research, and investigation) to being proactive (push, warnings, summaries). Its going to go from two major silos inside the enterprise – the human curated/ structured data and the content management/unstructured data – to being one, unified entity-oriented data store. Once this is done, programs will constantly monitor this unified data store for areas of interest to users and start to screen most everything and prioritize it. Eventually, we’ll get some real next generation automation out of this because there will be a class of actions that will be autonomously executed without requiring human intervention (such as determining the defense policy in a detected cyber-attack).

JB – What other thoughts do you have about this?

TE - I think that as we weigh the risks or errors in additional automation, we need to be wary of irrational risk aversion. The poverty of attention that most people suffer from has very real consequences even if we don’t fully understand that right now. Solutions which give small, incremental gains are unlikely to get ahead of this increasingly detrimental phenomenon. Without something reading everything and getting smarter, we are simply rolling the dice on what we don’t have time to read or consider. That’s the other side of the coin of the incremental approach.

Johnson’s point is that the small digital footprint of text and nearly infinite computing power of the PC (and now the iPad), means that it’s now only the copyholders that prevent instant access to everything every written, and thus the end of an era.

I disagree.  I think we’re just getting started.

It’s true that computers and networks have dramatically amplified human capacity to generate, store and share text.  It’s also true hardware and software have converged to integrate vast stores of digital information in our every day lives.

However, we’re still remarkably distant from computers being able to understand what we mean when we write.  Sure, gmail can post adds for Coca-Cola when I’m writing to my friends about steps in the steel refinery process, but that remains far from true understanding.

As our devices get more sophisticated, what must happen next is an era of understanding.  While “understanding” requires interpreting myriad inputs, the cornerstone of understanding humans is the ability to comprehend the written word.

This isn’t a new problem, and intelligent people have been working on a solution for quite some time.  The building blocks are clear and solutions are beginning to emerge in the market.

First, you won’t be able to rely on dictionaries to sort out meaning, for the simple reason that words change based on who speaks and in what context.  “Park” is a noun symbolizing where I have a picnic, a verb for what I do with my car when I’m at the store, or, with slightly less frequency, a proper last name describing an individual from a certain family.

Second, you’d better bring a big computer (and have some very good shortcuts) because speed matters.  A human of average intelligence uses about 10,000 words and adjusts the meaning of those words based on tone, location, speaker, non-verbal cues, etc.  While the field of human psychology is rife with examples of our cognitive shortcuts and their corresponding failings, the human brain does a remarkable job with a very computationally intense process.

For a moment, consider a world where the computer will understand text with the same speed and depth of humans.  A new era will be upon us.   You’ll CC your digital assistant on an email and it will schedule a meeting with right people at the right time, book flights for all attendees and make sure you’re eating at a restaurant that can accommodate your co-workers’ special dietary needs.   If you are a lawyer, your computer will suggest arguments with the greatest chance of success for a specific judge, based on the judge’s published opinions, all while you are writing the initial brief.   For doctors and nurses, the computer will suggest and rule out possible diagnosis as you dictate a patient’s symptoms.  Or, even more commonly, as you and I are writing typing our thoughts on a topic of interest, your computer will find people with similar interests and cite relevant passages of everything ever written.

No, not the end of an era.  Far from it.

I’d say the more important era is just beginning.

Special thanks to Intelligent Decisions, a solid partner and VAR reseller for us.

Net effect: It’s easier than ever for government agencies to take advantage of Digital Reasoning solutions. Please contact us if you have any questions or would like to learn more about procuring our solutions via the GSA schedule.