The upcoming release 2.4 of MongoDB will include a first, experimental support for full text search (FTS). This feature was requested early in the history of MongoDB as you can see from this JIRA ticket: SERVER-380 . FTS is first available with the developer release 2.3.2 .
Full Text Search 101
Before looking at how MongoDB implemented its initial full text search, we need to learn a little bit about the basics. There are (at least) two important concepts in order to unterstand full text search:
Stop Words
Stop words are used to filter words that are irrelevant for searching. Examples are is, at, the etc. Let’s have a look at the following sentence …
I am your father, Luke
… and these stop words: am, I, your. After applying the stop words, that’s what’s left of our sentence:
father Luke
The remains are processed in the next step. Please note that stop words are langugage dependent and may also vary from domain to domain.
Stemming
Stemming is the process of reducing words to their root, base or .. well .. stem. Remember things like declension and conjugation ? These typically change the stem of a word. Example
waiting, waited, waits
have all the same stem wait. This processing is also language dependent. Implementations for stemming are called stemmers.
The following diagram sums up the whole process:
So let’s see how we can use MongoDB for full text search.
Enable Text Search
Up to now, text search is disabled by default. You have to enable it at server start with the follwing command line option:
1$ ./mongod --setParameter textSearchEnabled=true
Create a text index
First of all, you define a special kind of index on a field, similar to geospatial indexes:
1db.txt.ensureIndex( {txt: "text"} )
Language settings are important with FTS. MongoDB uses the open source stemmer Snowball and a custom set of stop words for every language supported by that stemmer. The default language is English.
If you have a look at the indexes, our special text index shows up:
1> db.txt.getIndices() 2[ 3 { 4 "v" : 1, 5 "key" : { 6 "_id" : 1 7 }, 8 "ns" : "txt.txt", 9 "name" : "_id_" 10 }, 11 { 12 "v" : 0, 13 "key" : { 14 "_fts" : "text", 15 "_ftsx" : 1 16 }, 17 "ns" : "txt.txt", 18 "name" : "txt_text", 19 "weights" : { 20 "txt" : 1 21 }, 22 "default_language" : "english", 23 "language_override" : "language" 24 } 25]
Insert documents
If you insert a document to the above collection, MongoDB applies filtering of stop words and stemming to the content of the indexed text field. Each stem is added to the index pointing to the current document.
1db.txt.insert( {txt: "I am your father, Luke"} )
You can easily see that the stop word filtering happened, because there are only 2 keys in the index txt.txt.$txt_text:
1> db.txt.validate() 2{ 3 "ns" : "txt.txt", 4 ... 5 "nIndexes" : 2, 6 "keysPerIndex" : { 7 "txt.txt.$_id_" : 1, 8 "txt.txt.$txt_text" : 2 9 }, 10 ... 11}
Search
If you want to perform a full text search, you run a command on the collection holding the text index:
1db.txt.runCommand( "text", { search : "father" } )
Again, the language (this time the language of the search phrase) defaults to English.
The result looks like this:
1> db.txt.runCommand("text", {search: "father"} ) 2{ 3 "queryDebugString" : "father||||||", 4 "language" : "english", 5 "results" : [ 6 { 7 "score" : 0.75, 8 "obj" : { 9 "_id" : ObjectId("50e820689068856d0ac6a801"), 10 "txt" : "I am your father, Luke" 11 } 12 } 13 ], 14 "stats" : { 15 "nscanned" : 1, 16 "nscannedObjects" : 0, 17 "n" : 1, 18 "timeMicros" : 114 19 }, 20 "ok" : 1 21}
We have one hit for “father” using the index. The ObjectId of the document is return alongside with the full text.
This doesn’t feel like rocket science? Ok, then try a more advanced example:
1> db.txt.insert({txt: "I'm still waiting"}) 2> db.txt.insert({txt: "I waited for hours"}) 3> db.txt.insert({txt: "He waits"}) 4> db.txt.runCommand("text", {search: "wait"}) 5{ 6 "queryDebugString" : "wait||||||", 7 "language" : "english", 8 "results" : [ 9 { 10 "score" : 1, 11 "obj" : { 12 "_id" : ObjectId("50e82dc9c95b73b63ec5f5aa"), 13 "txt" : "He waits" 14 } 15 }, 16 { 17 "score" : 0.75, 18 "obj" : { 19 "_id" : ObjectId("50e82db5c95b73b63ec5f5a9"), 20 "txt" : "I waited for hours" 21 } 22 }, 23 { 24 "score" : 0.6666666666666666, 25 "obj" : { 26 "_id" : ObjectId("50e82dabc95b73b63ec5f5a8"), 27 "txt" : "I'm still waiting" 28 } 29 } 30 ], 31 "stats" : { 32 "nscanned" : 3, 33 "nscannedObjects" : 0, 34 "n" : 3, 35 "timeMicros" : 148 36 }, 37 "ok" : 1 38}
That’s pretty cool, isn’t it? As you can see, the resulting documents are sorted in descending order according to the score. There is a metric applied that measures the distance between the search word and the indexed stems.
Examples
All examples can be found on github . Try them yourself.
Summary
Of course, this implementation of a full text search won’t enable MongoDB to compete with search engines like Apache Solr or Elastic Search , but it is a step in the right direction. I think there are many use cases where this kind of FTS is absolutely sufficient. And don’t forget: this is the first release. We probably will see other interesting features in the future.
If I had to write a wish list, I would write the following:
- Enable users to provide their own stop word lists (w/o compiling). This could be done via a command line option pointing to a file or a new system collection like
system.fts.stopwords - Use a stemmer implementation that supports more languages than these . What about all the Asian langugages?
- Introduce the concept of a dictionary in order to handle
- synonyms,
- irregular words and
- compound words that are common in various European languages, something like the German words Volltextsuche (full text search) or Erdbeermarmeladenglas (jar of strawberry jam).
What’s next
In my next blog article I will have a closer look at more advanced features and non-English languages .
In the meantime: try text search yourself, especially if you have huge product data sets. Report any errors or suggestions to the Mongo JIRA .
Was this post helpful?
More articles
fromTobias Trelle
17.12.2018 Your job at codecentric?
Jobs
Agile Developer und Consultant (w/d/m)
Alle Standorte
More articles in this subject area
Discover exciting further topics and let the codecentric world inspire you.
Gemeinsam bessere Projekte umsetzen.
Wir helfen deinem Unternehmen.
Du stehst vor einer großen IT-Herausforderung? Wir sorgen für eine maßgeschneiderte Unterstützung. Informiere dich jetzt.
Hilf uns, noch besser zu werden.
Wir sind immer auf der Suche nach neuen Talenten. Auch für dich ist die passende Stelle dabei.
Blog author
Tobias Trelle
Software Architect
Do you still have questions? Just send me a message.
Do you still have questions? Just send me a message.