What is Federated Search and why is it important?

Analysts have tough jobs and are often exposed to considerable time pressure. The results of their analyses must be professionally researched to be convincing and reliable. Sometimes their queries result in answers of enormous complexity, yet they must be reported in a comprehensible and concise form. In addition, OSINT analysts often have to deal with hidden data sources the discovery of which being a time-consuming job requiring a considerable degree of insider knowledge.

 

We accepted the challenge and collected feedback indicating that analysts are tired of spending hours going through search engines manually just to find irrelevant and poor-quality results. Many analysts are frustrated and stressed by the sheer number of confusing data and browser tabs they have to handle.

 

What if there was a more efficient solution to their problem by using Federated Search?

 

In this article we’d like to familiarize the analyst with the approach of using Federated Search to solve his/her search problems, corroborating its advantages by presenting figures from R&D. Eventually, we will reveal Traversals methodology to increase even more the value already added by using a conventional Federated Search.

What is Federated Search?

Federated Search is the technology used to simultaneously search multiple data sources, finding the most relevant and pertinent information in the shortest possible period of time. Instead of manually sifting a score of data sources e.g. Google, Facebook, Yahoo, or others, manually, Federated Search provides one single unified search interface

 

Federated search is the opposite of having dozens of unorganized browser tabs.

 

Efficient OSINT analysis requires Federated Search including the following data sources: 

 

• Search engines, such as Google, Bing, Yahoo, and others.
• Deep web search engines, such as DuckDuckGo or Ahmia.
• Social media, for example, Twitter, Weibo, Instagram, Facebook, or LinkedIn.
• RSS feeds from websites of interest.
• New Data-as-a-Service providers, e.g. CapitalIQ.

 

Especially at the beginning of an analysis task one is happy about the broad coverage. Maybe you are lucky and find the jackpot in a data source where you would not have searched or which you did not know before.

 

Once you enter the keywords in the federated search, it will query all connected data sources in parallel. Your one search is converted in the background into multiple searches against the different data sources.

 

The advantages of a federated search are already obvious.

Why is it important?

Especially from executives we often get the question where the added value is compared to Google. 

 

“Why is it better than using Google alone and why should we pay for it if Google is for free?”

 

The following numbers show how important a federated search is for your costs and quality.

 

 

Susan Feldman explains in her IDC White Paper the cost of not finding information:

 

• 76% of company executives consider information to be “mission-critical” and their company’s most important asset.
• 60% feel that time constraints and lack of understanding of how to find information were preventing their employees from finding the information they needed.
• Typical analysts spend about 2.5 hours per day, or roughly 30% of the workday, searching for information. 

 

CMSWire’s research signifies the importance of a federated search. According to their research:

 

• Over 90% of enterprise data is unstructured and hidden inside thousands of documents.
• Over 60% of searches require at least 4 different data sources to be searched.

 

Even a classic federated search that only unifies data sources makes your analysis work more efficiently and increases the quality of your reports. 

What makes Traversals’ Federated Search unique?

The entire experience of our developers and advisory board is used to create an advantage for you. Traversals’ Federated Search is heavily supported by AI algorithms.

It Speaks More Than 100+ Languages

Starting the research process, the analyst very frequently runs into language problems, since the information searched or retrieved or both very often are in an unfamiliar foreign language. To overcome that language barrier – as demonstrated in our Yemen Covid-19 analysis (….) –  it is crucial for the search system to be able to handle as many foreign languages as possible.

 

Traversals’ Federated Search has integrated a powerful machine translation that allows you to search and understand multilingual content. This feature made it possible for our analysts to look for Arabic Covid-19 news in Yemen without speaking the language.

It Understands Your Context

Getting back hundreds of results from dozens of data sources does not make your analysis job easier. Each data source has its own way of ranking the results, e.g. Twitter sorts by time and Google by generic relevancy. An inconsistent result ranking, however, blurs the analysis process and considerably reduces search and retrieval efficiency.

 

A lot of AI-based Natural Language Processing modules are running in the background to enrich search results with information that is not provided by the data sources, such as text sentiment or text entities. This extra information is used to estimate a ranking for your context. The analyst can thus focus on the top places of the result list and will not suffer a disruption of the analysis flow.

It Saves Time and Money

Especially when it comes to foreign language information, analyses become very time-consuming and thus expensive. In order to not miss any important information, results must be translated into the language you understand. This is a manual process done with Google Translate.

 

It takes approximately 10 seconds to process a single foreign language result. For 100 hits, it, therefore, takes at least 15 minutes. With more data sources and larger result lists, concentration decreases whereas analysis overhead (e.g. managing multiple browser tabs) increases. The average processing time can thus quickly double or even triple.

 

Our Federated Search includes a machine translation and other powerful text processing and optimizations automating most of the manual work. This results in time savings of up to 90%. You will be up to 10 times faster by using our Federated Search and improve the reliability and quality of your results.

It is Growing

Especially in the OSINT area new data sources, such as new search engines or data services, are secretly published weekly. It will cost you a lot of free time to read all OSINT blogs just to keep an overview and to think about the integration into your analysis workflow.

 

The modular structure of our Federated Search allows efficient and transparent integration of new data sources. Our engineers and advisors are constantly looking for new data sources to be integrated in our system, while the analyst can fully devote his/her attention to his genuine job.

It is a Core Element of the R&D Roadmap

Our Federated Search is part of various research projects. Together with smart researchers, we discuss new concepts and new support measures to make the search even more efficient and powerful.

Key Takeaways

As you can see, the Federated Search is a crucial element, and more than important for enterprise businesses, large corporations, and governmental organizations:

 

• It saves time and money.
• It gives you more accurate and relevant results.
• It searches dozens of different data sources at the same time.
• It is able to find qualified information when you need it the most.
• It can find one piece of information inside thousands of unstructured documents and data elements.

Our Federated Search is one of the driving modules in our solutions like Strategic Intelligence, Medical Intelligence or Competitor Intelligence. Start using our Federated Search to benefit from all the advantages.