FRANKLIN, Tenn.– March 15, 2011 –Digital Reasoning^™, the leader in complex, large scale unstructured data analytics, today announced it has been issued United States patent #7,882,055 for its distributed system of intelligent software agents for discovering the meaning in text.

The pioneering invention entails intelligent software agents that extract meaning from text as humans do – by analyzing concepts and entities in context. The software learns as it runs, continually comparing new text to existing knowledge.  Associated entities and synonym relationships are automatically discovered and relevant documents are identified from across extremely large corpora.

The patent specifically covers the mechanism of measurement and the applications of algorithms to develop machine-understandable structures from patterns of symbol usage. In addition, it covers the semantic alignment of those learned structures from unstructured data with pre-existing structured data – a necessary step in creating enterprise-class entity-oriented systems. The technology as implemented in Synthesys^® provides a unique and now protected means of bringing automated understanding to end users in the enterprise and beyond.

“Digital Reasoning has been awarded two significant patents that form the basis of our flagship product Synthesys,” said Tim Estes, CEO and founder of Digital Reasoning.  “Our patents and recent strategic agreement with In-Q-Tel (IQT) will help us to accelerate our vision of helping decision makers discover, visualize and act on important information that may not be readily apparent or is buried in different and disparate data sources.”

The Company recently announced a strategic investment, licensing, and development agreement from IQT, the investment firm that identifies innovative technology solutions to support the missions of the U.S. Intelligence Community. The transaction will make Digital Reasoning’s technology available for use within the U.S. Intelligence Community.

Patent #7,882,055 along with Patent #7,249,117, which was awarded in 2007, are the foundation for the company’s flagship product, Synthesys, an entity oriented cloud-scale analytic solution that enables enterprises and government agencies to automatically make sense of complex data. Built to address the most complicated data analytics challenges Synthesys excels at extracting, resolving and linking entities and concepts from unstructured and structured data.  By uncovering hidden connections, the system empowers analysts to make smart decisions faster.

About Digital Reasoning

Digital Reasoning (www.digitalreasoning.com) solves the problem of information overload by providing the tools people need to understand relationships between entities in vast amounts of unstructured and structured data.

Digital Reasoning builds data analytic solutions based on a distinctive mathematical approach to understanding natural language. The value of Digital Reasoning is not only the ability to leverage an organization’s existing knowledge base, but also to reveal critical hidden information and relationships that may not have been apparent during manual or other automated analytic efforts. Synthesys is a registered trademark of Digital Reasoning Systems, Inc.

# # # #

Tim Estes, CEO to speak “Generating Dynamic Social Networks from Unstructured Data”

Matthew Russell, VP Engineering to speak “Unleashing Twitter Data for Fun and Insight”

Visit us at Booth #305

FRANKLIN, Tenn.–(BUSINESS WIRE)–Digital Reasoning™ Systems Inc., the leader in complex, large scale unstructured data analytics announced Synthesys v3.0, the latest release of their flagship solution for cloud-scale analytics in government and commercial markets. Synthesys v3.0 builds upon Digital Reasoning’s tradition of innovative software advances in achieving automated understanding of structured and unstructured data.

The explosive growth of data and the increasing demand for timely, actionable intelligence requires innovative solutions. Synthesys meets this need by automating the analysis of structured and unstructured data, and empowering users to identify facts and associations with far less reading. Synthesys converts massive amounts of unstructured data into the underlying people, places, organizations and their connections.

“Synthesys 3.0 is the culmination of a decade of solving large scale data analytics problems for the intelligence community,” said Tim Estes, CEO and founder of Digital Reasoning Systems. “With this release we are bringing to the market unprecedented understanding of unstructured data in cloud-scale architectures with our integration of Hadoop and Cassandra.”

Synthesys v3.0 achieves unprecedented scalability with our integration with Cassandra and Hadoop. Synthesys is one of the first products to fully leverage the latest version of these technologies – Hadoop CDH3 and Cassandra v0.7. These “cloud-scale” enablers are essential to meeting typical Synthesys requirements of hundreds of millions of documents and other text files.

Synthesys v3.0 incorporates entity extraction/resolution, link analysis, concept resolution and summarization to automate much of what used to be human reading. What makes Synthesys particularly unique is that this is accomplished without the assistance of a taxonomy or ontology. This “unbiased” approach to data analytics is the only way to uncover the non-obvious connections hidden in the data and to understand the alternate meanings of words – prevalent in “coded” or “disguised” messages.

“This is an important milestone,” said Rob Metcalf, President and COO of Digital Reasoning. “Synthesys 3.0 will allow us to address a number of new markets demanding our unique text analytic capability.”

Synthesys v3.0 is available now. Demonstrations, information and pricing is available by contacting Digital Reasoning by clicking here.

About Digital Reasoning™

Digital Reasoning Systems solves the problem of information overload by providing the tools people need to understand relationships between entities in vast amounts of unstructured and structured data.

Digital Reasoning builds data analytic solutions based on a distinctive mathematical approach to understanding natural language. The value of Digital Reasoning is not only the ability to leverage an organization’s existing knowledge base, but also to reveal critical hidden information and relationships that may not have been apparent during manual or other automated analytic efforts. Synthesys^® is a registered trademark of Digital Reasoning Systems, Inc.

The explosion of unstructured data represents over 75 percent of the world’s information. The increasing demand for actionable intelligence from this information requires innovative solutions to make timely, accurate judgments on large quantities of information. It saves lives in the intelligence community, and it saves time and money in the enterprise. The ability to distribute analytical capabilities is changing how information is used. Riptano and Digital Reasoning Systems are teaming up to promote and simplify the evolution to these cloud-scale analytic intelligence solutions.

Riptano is the authority on Apache™ Cassandra and delivers software, services and training to help businesses and organizations build their applications on Cassandra. Digital Reasoning’s flagship product Synthesys^® is deployed over the largest Cassandra instance in the government intelligence infrastructure. Riptano and Digital Reasoning are working to promote both commercial and government scalable Cassandra solutions and will be involved in the analytic and data storage layers of a 400-node Cassandra architecture. The database is designed to allow for the analysis of hundreds of millions of intelligence documents.

“We are excited to be working with Riptano,” said Tim Estes, CEO of Digital Reasoning Systems. “Riptano is clearly the leading expert on Apache Cassandra, a technology we depend upon to achieve our extraordinary scalability advantages.”

One of the first results of this partnership will be the deployment and support of an open source Cassandra management solution – called “PyStratus” – used by Digital Reasoning to dramatically simplify creating, configuring, and managing a Cassandra, Hadoop, or hybrid-Cassandra/Hadoop clusters in Amazon EC2. This package is available on http://www.github.com/digitalreasoning/PyStratus. This partnership will deliver similar scripting solutions for Rackspace and other cloud providers in the near future.

“Apache Cassandra is the database for big data,” said Matt Pfeil, CEO and co-founder and of Riptano. “Digital Reasoning is doing interesting work within the intelligence community that is resulting in a deployment of Cassandra across hundreds of nodes for millions of documents. We’re excited Digital Reasoning is giving back to the community by open sourcing their software to deploy Cassandra on Amazon EC2. This will save time and make it easier to run Cassandra in the cloud.”

Digital Reasoning Systems and Riptano are leaders among a growing group of companies that recognize the benefits that Cassandra brings to the challenges of extracting hidden value locked within “big data.”

About Riptano®

Riptano is the company for Apache Cassandra, the leading scalable and high performance open source database. Riptano offers software, support, and training for Cassandra. Capable of both online transactions and analytical workloads, Cassandra offers distribution of data across multiple data centers and incremental scalability with no single points of failure. Cassandra is the logical choice when you need reliability without compromising performance, and Riptano brings Cassandra to the enterprise. For information on Riptano visit www.riptano.com

About Digital Reasoning^™

Digital Reasoning solves the problem of information overload by providing the tools people need to understand relationships between entities in vast amounts of unstructured and structured data.

Digital Reasoning builds data analytic solutions based on a distinctive mathematical approach to understanding natural language. The value of Digital Reasoning is not only the ability to leverage an organization’s existing knowledge base, but also to reveal critical hidden information and relationships that may not have been apparent during manual or other automated analytic efforts. For more information on Digital Reasoning visit www.digitalreasoning.com

Contacts

Digital Reasoning Systems

Dave Danielson, 603-429-8866

dave.danielson@digitalreasoning.com

At Digital Reasoning Systems, we say that the model of the data comes from the data itself without imposing a pre-determined model. There is a lot that is being said there and it would help to talk about the many assumptions found in that statement and the reason for this choice.

If you are not familiar with the concept of Ontology here it is in a nutshell. The study of Ontology examines metaphysics, the existence of things. It also suggests that all objects can be grouped, related to each other in a hierarchy and concepts classified by their similarity and differences to other concepts.

Classical ontologies are hierarchical and form trees. At the top the standing assumption is the world can be divided between concrete objects and abstract objects. Well at least typically. You could root the tree with a more complex asymptotic relationship. For example it is considered by the Buddhists that reality is a very personal perception. Thus one can suggest the root ontology consists things I think exist, things I think don’t exist, things I imagine and things I can’t imagine. Some things have a dual nature. For example, money is an abstract concept while cash is a concrete object that refers to the abstract concept of money. If you don’t understand the difference consider the case of hyper-inflation where cash loses nearly all connection to value. Where before you could expect to buy a loaf of bread for a few dollars in a normal economy; one beset by hyper-inflation may require literally millions of dollars.

A tree like hierarchy of things assumes they fall into neat categorization. It doesn’t take much digging before you find conflicting cases. How to create your ontological tree depends upon opinion more than science. You have two solutions possible with conflicts. One is to create new hybrid categories. The other is to shoehorn the things that don’t clearly fit into a category into what in your opinion is the nearest category.

Creating new categories every time there is an apparently ill-fitting object defeats the purpose of having a neat, traversable tree. Is a tomato a fruit or a vegetable? You could just simply state it is a fruit since it flowers. Botanically you are correct. You may applaud yourself for sticking with science. Hmm, but, there is that nagging bit; tomatoes aren’t sweet like fruit. They are savory. When you look at the genetics of the tomato you notice that they are in the same family as nightshade plants (Solanaceae.) Some of these plants are quite toxic. The family also includes the potato, mandrake, paprika, eggplant and many more. Well those don’t sound like fruit. Tobacco is in the same family but I am certain your doctor would not consider smoking to be a method for getting your daily requirement of fruit! The use of the tomato in the culinary arts is similar to other vegetables. So, based upon that, one could claim it is ok to refer to the tomato as a vegetable.

Ontologies can be slippery things when we create a model for them. We can get even further confused when we discuss the concept of identity. Let’s assume we have an accepted ontology we have put into use of common things. Let’s say we are interested in ships. When is a ship, a ship? Is it a ship when it is on the drawing boards? Perhaps it is an abstract ship at that point. How about during assembly? How far along into assembly does it go from an abstract concept? With an imposed model there is no gradient between Abstract.Ship and Concrete.Ship. Let’s say the ship is complete but still up on the dry dock. Is it a ship? It isn’t functioning as a ship, though it could if it were in water. What makes it different than a very complex sculpture of a ship? If it is in the water and sailing we can assume that it is a Concrete.Ship as it is functioning like a ship and has all the attributes of a ship. Let’s say the anchor falls off. Is it still a ship? I would say yes. What if the engine has trouble or the rudder becomes stuck? These are various bits of state change but I would say it is still a ship. If more and more things start to fall off the ship, is it still a ship? It may reach a point where it is less complete than it was when it was under construction when we said it wasn’t yet a ship. At what point before it sinks is it no longer a ship? Finally, consider the Queen Elizabeth. She caught fire in Victoria Harbor, Hong Kong in 1972. She eventually capsized. If she was no longer a ship was it due to the initial fire or the later capsizing? If she could have been righted and repaired would she be a ship again? If she had never had the fire and had become a floating university as intended would she still be a ship?

The point of all this is that while these are fairly simple and common state changes they point to some interesting issues with enforcing a manmade ontology onto objects – it’s not always clear what identity an object has and over time it that identity may change.

Association as Ontology

Digital Reasoning’s technology employs the concept of the association network. This network examines the properties and use of an object in human language, it derives the most meaningful associations and it proceeds to create a network map. A meaningful association can be something as simple as being a subject or object in relationship to another concept. There could be a meaningful chain of associations.  For example “K278” (submarine) may be associated with ”190MW” through its association to “reactor”. K278 and reactor may share an association to “fire,” which was the cause of the submarine’s sinking.

Unlike a hierarchical tree, an association network does not imply complimentary association.

It does not imply a top down order either. Every node represents a concept. Every concept has a list of associations. Think of this as saying to yourself, “when I think of this concept, I also think about these other concepts.” These can range from attributes of the concept to other similar concepts. This can be extremely helpful. When we conduct a search we know what we are looking for. This is great in general. The power of Google™ is obvious to anyone who uses the internet today. Simply type in what you are looking for and you will get access to hundreds of mostly relevant documents. Sometimes we are looking for just one specific document and we don’t remember exactly what is written in it. Or perhaps we never knew but we still suspect it exists. We know the topic so we start there with our search. Associations can help us out here in a powerful way. Consider the following example:

You are searching for a document about Quantum Cryptology. Let’s say a search using “quantum cryptology” doesn’t produce the document because that exact phrase isn’t in the document. Associations can help. The associations of Quantum Cryptology would include other types of cryptography such as secret key, public key, hash functions as well as lots of terms associated with cryptology in general such as cipher feedback, Rivest cipher, NIST and many other terms. However the real power comes from having access to concepts tightly connected to the topic concept that might help find the document you need. Terms like quantum communication and quantum processing, quantum channel, QKD and other concepts can be used to augment the search. Associations can help you provide these terms.

In an association network the inter-relationships can be represented by a hypergraph. You can get a full visualization of conceptual relationships. Each relationship has a strength and thus a distance which can push two concepts apart or pull them closer together. The complication is that in many cases Concept A is near and related to Concept B. Concept A is also near and related to Concept C but Concept B and Concept C have no direct relationship. This is where associations start to take on the look of Venn diagrams.

Let’s look at an example where we can see how this plays out. We will look at the example from the point of view of determining the ontology of a “Golf Cart.” You know what a golf cart is. You’ve seen them, maybe ridden in them and have a general understanding of them. The typical golf cart has 4 wheels, a roof, a steering wheel, storage space, a motor and so forth. Many have electric motors but some are fueled by propane or gasoline. So lets quickly come up with a chain of categories that a golf cart would be in:

Concrete_object.machine.vehicle.cart.golf_cart

That is nice but ANSI standard Z130.1 says carts are not self-propelled and that golf cart is a misnomer and it should be called a car. Ok, an easy fix.

Concrete_object.machine.vehicle.car.golf_cart

This is lovely but it has a problem. Cars bring up a certain image and set of properties. For a wide variety of things we call cars they all share properties that the golf cart does not have. Cars can run on roads, golf carts can’t. Golf carts can run on fairways and cars cannot. The local quick oil change place will work on your car but likely won’t on your cart. OK so a golf cart isn’t in the family of cars so we create a concept called “golf equipment” and move the cart there. Now we are done.

But wait! You aren’t done because “golf carts” are found at other sporting events like football games where they sometimes are used as ambulances for injured players, at airports as inter-terminal transport, and in parks. So some are “sports equipment” and some are clearly some sort of vehicle. There are several possible designations we can now apply and they are all correct in context. The new Semantic Wikimedia will deal with this problem in exactly this way by examining in context what exactly is this instance of golf cart.

With an association network the uses and meanings of golf cart, golf buggy and golf car as concepts will build relationships both strong and weak and while not related to a car the closeness to car will be apparent by the shared associations and position.  A golf cart is not a car but it is closer to a car than a toy car. It is closer to a car than a bulldozer. This will be obvious by the relationships the network has for all of these concepts. The question never comes up with an association network of whether something is or isn’t a particular class of item. Instead of forcing a concept to be something it sees it as a series of networked relationships. It allows the concept to be many things, closer or more distant to other concepts.

In the real world this can be incredibly useful. If I am researching message traffic and I am interested in finding “purchase/acquire transactions” for man portable air defense systems (shoulder fired anti-aircraft missiles) then I’d want the associations for the generic acronym “MANPAD” and from that I will get various attributes that might help me find what I am looking for. One thing that will associate strongly will be names of specific systems such as Stinger, Olga, Redeye, Strela, but also other names will come up such as Javelin, RPG-7 and Saxhorn. The last 3 are designed for anti-tank work. Why would they come up? The fact is that if you laid all 7 of these systems out on a table they all, at a gross level, look the same. The basic concept in design is the same for them all. The differences are all in the details. Here, the ontology of MANPAD would normally exclude these systems. Why you would be interested in including these results are varied but some rationalizations are that if the document in question is an eyewitness account then perhaps the witness was incorrect about what they saw. Another is that if you can smuggle a Javelin then they most certainly can smuggle a Strela. Further – let’s say you never heard of the QianWei MANPAD you most certainly would want it to be suggested as part of your search augmentation. Other associations that would come up would be attributes of MANPADs and be ontologically dissimilar but still highly important. For example sub-parts such as rocket motor designations, warhead design, related technology you were previously unaware of and so forth. The document you are looking for might not contain the term MANPAD or the name of a system but instead be about someone very interested in spare parts to maintain their cache or to build their own.

The association network found in Digital reasoning’s Synthesys™ goes beyond a strict, imposed ontology. It offers a refinement by building the model from the data. It augments our ability to find what we need using search by giving us freedom from relying solely upon keywords we have guessed at.

On the surface, requirements gathering would seem to be a rather simple process of just sitting down with the customer and asking them what they want. It’s been my experience, however, that requirements are a lot like minerals found in nature. They are rarely found in their pure state and require a great deal of refinement. You don’t want the customer to tell you that you gave them what they asked for, but not what they really needed.

I’m sure that many of you have seen the cartoon to the right that provides a humorous view on the requirements gathering process.  This cartoon is equally applicable for projects and products and if you have been involved in the process, your laughter may come along with some formerly repressed bad memories. The main point, of course, is how far a project can drift from the “real” customer requirements. But this cartoon also points out that customers often have difficulty describing their own requirements.

While very funny, this cartoon unfortunately tends to describe the norm, rather than the exception. This lack of understanding of the customer’s “real” requirements explains in part why many delivered software systems never achieve their desired goals.

While requirements gathering and analysis is hard work when done correctly, with few short cuts, it is not rocket science either. All it takes is just a little common sense.  While every project is different, potentially requiring unique approaches to requirements gathering, I have found two simple methods that have worked well for me in the past.

#1: Take Time to Observe

The first is just to spend significant time with the end user, talking with them in detail about their job functions, and observing them performing their duties. Years ago, while working as a systems engineer at a refinery, I was tasked with writing a process control program for one of the automated plants in the refinery. This proved to be a daunting task for two reasoning. First, a previous effort to provide this exact same function, implemented by a seasoned systems engineer, had failed. The second reason was my lack of previous experience with process control systems, and how this particular plant operated. Therefore, I spent several weeks just working with the plant operators, learning how they performed their jobs, and even assisting them with some of their tasks. I learned as much of their daily routine as I could. When I finally started to design and implement the process control program, I did so from the perspective of a plant operator, not an outside engineer. The system proved to be a success, in large part because I was intimately familiar with primary function that was to be automated by this program, including all the little nuisances related to that function.

#2: Refine Requirements with help from Customer Peer Groups

A group of key customer personnel that are representative of the people who will be using and managing the new system being created is an excellent refining source for the requirements. The primary purpose for this group is to meet periodically and review the system requirements and design, using prototypes to demonstrate and validate key requirements of the system. I used this process very successfully on a project involving the redesign of a grant management system for a large government program. Our user’s group consisted of approximately 25 people, representing all levels of users that would be directly and indirectly impacted by this new system, from the end-users all the up to the managing director responsible for that program. Throughout the development of the system, we met 2-3 days every quarter to review, validate, and refine the requirements. We made extensive use of prototyping to demonstrate those portions of the system reviewed during previous meetings. We would then incorporate this feedback into the design and make sure it was reflected in the prototypes used for the next meeting to validate what we heard.

As a result, the customer was able to visually see the system at all of its various stages of development, and provide critical mid-course corrections to the design. One positive side-effect of using this process was that the customer assumed ownership of the system as it was being developed. It became “their system”, exactly fitting their needs, not something just dropped on them at the conclusion of the project. As one might guess, they enthusiastically embraced the final product.

In conclusion, gathering and refining requirements is an iterative, collaborative process with the customer. It is not a passive activity, but requires hard work to get past the obvious needs to determine the often hidden secondary and tertiary requirements that play such a crucial role in the overall acceptance of the end product. You know you’ve been successful when the end-user remarks that whoever designed the system must have done their job before, because they addressed all of the little nuances that would only be known by someone intimately familiar with the entire process workflow making up that job.

Communications is a critical component of any successful project. It governs not only how well the project team works together, but also impacts the public perception of how well the project is going. Communications is a multi-faceted function that serves a variety of different purposes. I would like to focus on just two aspects of project communications; managing the customer’s perception of the project, and internal project team communications.

Managing a customer’s expectations of the project outcome is one of the most important jobs of a project manager. How often have you heard of a company having a real good earnings report, only to have their stock go down because they didn’t “meet analyst’s expectations”? For a project, not meeting expectations often results from poorly defined project requirements and deliverables, combined with inadequate communication of project status, causing the customer to assume capabilities that the project was never designed to deliver. Therefore, it is very important to explicitly communicate exactly what the project will deliver, as well as what it will specifically NOT deliver.

Keeping the customer apprised of detailed project status involves more than just holding periodic project meetings. These can be a big time waster if not properly structured and combined with other communications methods.

One ancillary communications vehicle that I have found useful in the past is to create/maintain a project notebook for each customer stakeholder.  While each project will have different communications requirements, the following list provides some examples of the content that I have found useful in the past:

• a detailed description of the project requirements
• important project milestones and deliverables
• all meeting minutes
• significant communications with key vendors contributing to the project
• internal project memos
• change request logs
• analysis/tracking of major project risks and issues/problems
• tracking of key project dependencies – especially with outside entities

Prior to each Prior to each Prior to each Prior to each Prior to each Prior to each project meeting with stakeholders, I send out revisions/updates to the project notebook reflecting the latest project status. This allows us to focus the meetings on important project issues and not waste time on the more routine status items.

You might reasonably argue that some of this information could be considered project minutia that wouldn’t be very meaningful to a project stakeholder. However, providing this level of detail tends to build an element of trust indicating that you’re not holding anything back from the stakeholders. In other words, you are making the stakeholders part of the project team, dissolving the typical “them vs. us” mentality that often exists between the project team and the end customer. Providing this level of information to the customer goes a long way in preventing unrealistic expectations from arising.

The other area of communications that is often undervalued is internal project communications. There is a school of thought that communications with team members should be confined to just those areas that they are working on, the theory being that it helps people focus on their individual tasks at hand, rather than being inundated with information extraneous to their function. While there may be some truth to this, my experience has been that some of this “extraneous” information actually increases people’s effectiveness because they are more aware of how their efforts fit into the global scheme of things. Furthermore, potential integration issues between major project components tend to be identified earlier due to this increased awareness.

A secondary benefit is that a well-informed project team functions more like a team, has higher morale, and presents a more unified project “face” to the customer (i.e. no matter which project member the customer talks to, they get the same information).

In summary, open and honest communications with the customer and internal team members increases the chances of success.  There is an old adage that if you don’t provide adequate information to people, they’ll make it up. Most projects have enough real technical challenges to deal with, without having to address issues caused solely by poor communications that could easily have been avoided.

Here at Digital Reasoning, we have been dealing with the challenges of automated understanding of massive amounts of unstructured data for years. Knowing that Tim Estes, our CEO, might have a different view on this issue, I decided to interview him. Tim has worked within the realms of unstructured data analytics, artificial intelligence, and machine learning for the past decade.

The following is our interview:

Jason Beck – In the article, one researcher suggests that developers shouldn’t build analytics robots, but rather “exoskeletal systems”. Do you agree?

Tim Estes - I think that it’s a matter of degree. The range of judgments that a machine can make as a proxy for the human is constantly and necessarily expanding. Even R2D2 was most famous for taking orders from Luke Skywalker trying to accomplish tasks from fixing the X-wing in flight to cracking into computer networks.

Just to be a little more accurate – Iron Man wouldn’t work without an AI that is close to R2D2. Jarvis (the AI program that runs’ the Stark house and the Iron Man suit) is always chatting up Tony Stark about what’s going on with the suit and the risks that are present around him. The Iron Man analogy means we seed the full situational awareness (the sensory and data input space) to the machine with the human making key decisions on the filtered and prioritized information. I think that’s about right.

R2D2 is distinct in having a measure of its own intentionality (i.e. it is autonomous in more dramatic ways than Jarvis/Iron Man suit) but they are much more close than you might think. Should humans get out of the loop in making analytic judgements? No more than we should have pilots out of the loop in flying commercial airlines at this time. But show me a pilot that can fly a 747 without computer assistance and guidance? We are already in the hybrid space. And the complexity of our technology and the explosion of the information created by machines and man assisted by machines means we will need ever increasing automation in understanding.

JB – Doesn’t the exponential growth of data and decreasing levels of available talent necessitate automated systems?

TE - Exactly. The notion that “augmented intelligence” can solve the full data problem is wishful thinking. Something has to read everything and that can no longer be a human as a matter of scale. We have to make strides to catch up intelligent systems with the complexity and scale of the data we are being inundated with.

JB – Is this an Either-Or situation? Just because someone may prefer automated systems, does this assume that there won’t be any human in the loop?

TE - I think that’s the real issue – where is the dividing line right now and where is it going to be in 5 years? Right now – machines have to read and organize everything. The race is to see who can do it accurately, at scale, and focused on the entity-level vs. the document level. In five years, the information overload will be so substantial that autonomous proxies or agents will likely be the baseline for all of these systems. In both situations, humans are in the loop. Now – they have much greater heavy lifting because nearly all of our enterprise information systems don’t really understand their data that well so the burden is on the reader. That has to change. Even when it does, we will just be enabling the humans to make better decisions in less time and less interruption of their daily lives.

JB – Does the delineation between these two approaches represent a common split in the overall text analytics community?

TE - I think so. We can either be satisfied with augmenting the status quo or we can get to the root of the issue – that software doesn’t understand natural signals that make up unstructured data. We are in a place of diminishing returns with simple classifiers and ETL (Extract, Transform, Load) architecture. The more exciting alternative, however, is to go at the semantic and scale problems with the appropriate technologies and transform the enterprise to be entity-oriented.

JB – Can you think of any example where someone tried to completely automate text mining?

TE - Not off the top of my head. I’m sure there have been. But a lot of text mining is feeding either fancy search engines (such as faceted navigation and data enriched topic clustering) or Business Intelligence frameworks.

JB – What does the future look like regarding automation?

TE - Its going to go from being reactive (search, research, and investigation) to being proactive (push, warnings, summaries). Its going to go from two major silos inside the enterprise – the human curated/ structured data and the content management/unstructured data – to being one, unified entity-oriented data store. Once this is done, programs will constantly monitor this unified data store for areas of interest to users and start to screen most everything and prioritize it. Eventually, we’ll get some real next generation automation out of this because there will be a class of actions that will be autonomously executed without requiring human intervention (such as determining the defense policy in a detected cyber-attack).

JB – What other thoughts do you have about this?

TE - I think that as we weigh the risks or errors in additional automation, we need to be wary of irrational risk aversion. The poverty of attention that most people suffer from has very real consequences even if we don’t fully understand that right now. Solutions which give small, incremental gains are unlikely to get ahead of this increasingly detrimental phenomenon. Without something reading everything and getting smarter, we are simply rolling the dice on what we don’t have time to read or consider. That’s the other side of the coin of the incremental approach.

While technology has provided several good paradigms for dealing with structured data (i.e. data that is structured in such a way to be easily decomposed into pre-defined fields), it has not kept pace with unstructured data, such as emails, blogs, web site content, etc. Thus, critical information is often kept “secret” through the obscurity of the sheer volume of data one must process, often manually, to reveal this information.

In response to this challenge, Digital Reasoning Systems, Inc has developed a comprehensive set of analytical tools packaged into product called Synthesys^® that essentially decomposes unstructured text into meaningful information easily understood and manipulated by a user.

This technology is based on the premise that there is order inherent in all languages that can be discovered and mathematically modeled. This has led to the development of our advanced data analytics and knowledge abstraction for unstructured data, based on a distinctive, patented mathematical approach to natural language processing.

For a better understanding of Synthesys^® and its capabilities, a down-loadable white paper (Synthesys – Technology Overview) providing a high-level overview can be found here.

(Strong project leadership and enlightened “people” management.)

I almost consider the term “project manager” a misnomer. Successful projects are “led”, not “managed”. Projects are too dynamic to lend themselves to just being managed. What are the hallmarks of a good project leader? I believe that a strong leader possesses a strong sense of purpose that instills confidence to the project team. A strong leader thoroughly understands the goals of a project, and the requirements to achieve those goals. They are also very decisive when faced with difficult decisions. A project team needs clear direction and quick conflict resolution. Lack of timely decisions or unresolved conflicts often lead to project objectives not being met on schedule, and can also lead to morale issues on the team.

An IT project should not be run as a democracy, but rather as a benevolent dictatorship. While it is good to obtain consensus when making major decisions, it should not be a requirement for the project leader to come to a timely decision. A project awash in indecision is ripe for failure. Also, once a decision is made, it should stay “made”, unless new compelling information comes to light that was not factored into the original decision. I have seen projects where the de facto project motto seemed to be that any good decision was worth making several times. This leads to confusion and wasted effort.

Part of being a strong leader is conveying a very concise message regarding the project goals and how those goals are going to be met. Just as important is a good understanding of the non-goals of the project (i.e. specific goals that the project will NOT address).  If you don’t paint a clear target, don’t be surprised if no one hits it. A strong leader creates this clear project focus by imparting an unambiguous understanding of the following to the project team:

1. The project goals – the team needs a crystal clear understanding of the project goals. These goals need to be simply stated and non-ambiguous. If you don’t know where you are going, how do you ever expect to get there. The team should also clearly understand what the non-goals are (i.e. the functions/ features/capabilities that are specifically NOT being provided by design). The level to which these goals are understood will directly impact the effectiveness and “correctness” of every major decision made in the project.
2. Each team member’s role in meeting the project goals – just as important to understanding the overall project goals is every team member’s understanding of their own individual roles in meeting these goals. Knowing where your tasks fit into the overall scheme of things provides the context necessary to disambiguate and prioritize the minor issues that always arise in the execution of individual tasks. In other words, it helps everyone in the boat to row in the same direction and not work against each other.
3. A conflict resolution process – the project team needs to know how conflicts will be resolved. A conflict can be a difference of opinion on some aspect of the project design, or any obstacle preventing someone from accomplishing their tasks. A strong leader resolves conflicts as quickly as possible to minimize their impact on the project (i.e. conflicts DO NOT improve with age).

Another hallmark of an effective leader is a good understanding of human nature. A project leader sets the tone of the project primarily through how they treat the project team. Team members need to be treated as individual people, each with their own personal aspirations, and NOT as project resource units. Everyone needs some level of affirmation for a sense of accomplishment. The project leader needs to know his or her team members well enough to provide the right level of affirmation to each person. Treating them with respect creates the type of atmosphere that successfully sustains the project team through the difficult times that often occur during the course of a project. Teams that are led through fear and intimidation often fail because this type of leadership divests team members from project ownership and discourages them from going that “extra mile” often required to get through the difficult times.

Finally, a strong leader NEVER takes credit for accomplishments rightfully belonging to individual team members or the project team as a whole.  A project leader’s success (or lack thereof) should just be a reflection of the project team’s success, and not measured by their own individual efforts.