Thanks to internet search engines like Google, Yahoo, and Bing, everyone has become accustomed to searching their data effortlessly. Consequently, as developers of web-based applications, we increasingly find that our customers' requirements include being able to search their application's data in a "Google-like" way.
For a very small subset of applications that only need to do full-text search on a few database columns, there a some very simplistic solutions, ranging from using a SQL 'like' clause on every column to database-vendor supplied full-text search functionality for designated columns.
Frequently, however, these simplistic approaches are not enough, either because the data to be searched is large and complex, because the approach's search functionality is limited, or because database neutrality is desired. For these situations, a more advanced solution is needed. Fortunately, there are now a number of solutions available for all of the most popular platforms, languages, and frameworks, and several hosted solutions that are development stack-neutral.
Lucene and Compass
One of the most popular full-text search projects is Apache Lucene, a Java-based, open-source indexing and searching solution. Lucene itself targets indexing and searching of 'documents', but several other projects, most notably Compass, have adapted Lucene for indexing and searching relational data via integration with Object-Relational Mapping libraries such as Java's JPA, Hibernate, and others.
The combination of Lucene and Compass provides Java developers with an easy way to implement Google-like searching of their relational data, including fuzzy searching (spell-check / 'Did you mean...'), stemming ('fishing', 'fished', 'fish', and 'fisher' all match 'fish'), wildcards, and boolean logic.
For users of the Java JPA ORM, usage of Compass is fairly straightforward. For users of the Grails framework, the Grails Searchable Plugin makes implementation of full-text searching almost effortless. I recently added this plugin to a Grails application with a moderately complex data model, and had my full-text search functionality up and running in a few hours, with fine-tuning taking only a few more hours. As the application's data model has grown, very little time has had to be spent on keeping the search functionality up to date.
There are many other programming languages and platforms in use besides Java, so not surprisingly, Lucene has been ported to many of them, including C, C++, .NET, Objective-C, Python, Perl, and Ruby. The most popular of these, due to the popularity of the programming languages, are Lucene.net, which is a C# port targeted at the .NET runtime, and Ferret, which is a Ruby port targeted at the Ruby on Rails framework.
While Lucene and its variants currently dominate the open-source full-text search landscape, there are other well-known projects, including Xapian.
Hosted Solutions
There may be situations where it makes more sense move your full-text search functionality 'to the cloud'. Several hosted solutions are available, some based on Lucene or other open-source technologies, as well as other, proprietary offerings.
Websolr is a hosted version of Apache's Solr which is itself built on Lucene. Other offerings include Flying Sphinx for Rails (which uses the Sphinx search engine), and IndexTank, which is a proprietary offering.
Both Flying Sphinx and IndexTank are available as add-ons for the popular EngineYard and Heroku Ruby-based hosting platforms.
I recently tried out IndexTank on Heroku. IndexTank itself targets indexing and searching documents, but the Tanker gem does a great job of integrating IndexTank with Rails' ActiveRecord ORM. IndexTank's documentation doesn't mention it, and while not available "out of the box", IndexTank does support both fuzzy searching and stemming. All you need to do is send an email to their customer support address, and ask to have it turned on.
If your application can benefit from full-text search, there are plenty of interesting choices, regardless of your platform or programming language.
Tuesday, August 9, 2011
Wednesday, August 3, 2011
Groovy Script to Tame Gmail Attachments
I like Gmail but it has a big short coming: you can't sort mail by attachment size. This means that if you are running out of space in your Gmail account, you can't easily find large attachments like that 25MB video that your friend sent you of his kid's graduation or other large attachment that you really would like to get rid of. There are sites like findbigmail.com that will help you find large attachments, but the downside is that you have to trust them and allow them to read all of your email. I'm a little paranoid so that wasn't something I was willing to do.
I also like Groovy and so I wrote a little Groovy script that finds large attachments in Gmail and gives them a custom label so you can easily find them. The script uses JavaMail which depends on the JavaBeans Activation Framework (activation.jar). Both jars need to be in the classpath along with the Groovy jar.
The first part of the code makes a connection to Gmail.
I also like Groovy and so I wrote a little Groovy script that finds large attachments in Gmail and gives them a custom label so you can easily find them. The script uses JavaMail which depends on the JavaBeans Activation Framework (activation.jar). Both jars need to be in the classpath along with the Groovy jar.
The first part of the code makes a connection to Gmail.
import javax.mail.*
import java.util.Properties
def emailAddress = "you@gmail.com"
def password = "password"
def server = "imap.gmail.com"
def port = 993
def OneMB = 1024000
def FiveMB = 5120000
def props = new Properties()
props.setProperty("mail.store.protocol", "imaps")
props.setProperty("mail.imaps.host", server)
props.setProperty("mail.imaps.port", port.toString())
def session = Session.getDefaultInstance(props,null)
def store = session.getStore("imaps")
store.connect(emailAddress, password)
println "connected to Gmail"Then we define the labels that we want to add. Labels in Gmail are called folders in JavaMail.def bigAttachmentFolder = store.getFolder("1MB+ attachment")
if (!bigAttachmentFolder.exists()) {
bigAttachmentFolder.create(Folder.HOLDS_MESSAGES)
}
def hugeAttachmentFolder = store.getFolder("5MB+ attachment")
if (!hugeAttachmentFolder.exists()) {
hugeAttachmentFolder.create(Folder.HOLDS_MESSAGES)
}
Next the script iterates through all of the emails looking for attachments. If a large attachment is found, it is copied into a folder which in Gmail adds the appropriate label to it.def folder = store.getFolder("[Gmail]/All Mail")
folder.open(Folder.READ_ONLY)
folder.messages.each { msg ->
try {
def content = msg.content
if (content instanceof Multipart) {
for (i in 0..(content.count - 1)) {
BodyPart bodyPart = content.getBodyPart(i)
if (Part.ATTACHMENT.equalsIgnoreCase(bodyPart.disposition)
&& bodyPart.size > OneMB) {
println "big attachment: ${bodyPart.fileName} = ${bodyPart.size}"
folder.copyMessages([msg] as Message[], bigAttachmentFolder);
if (bodyPart.size > FiveMB) {
folder.copyMessages([msg] as Message[], hugeAttachmentFolder);
}
}
}
}
} catch (Exception ignore) {
println "error processing message, no big deal, moving on"
}
}
println "-----------done------------"That's it! If you modify the script with your email address and password, and then run it, your Gmail account will get 2 new labels: "1MB+ attachments" and "5MB+ attachments" that will allow you find and delete big attachments quickly. The code is pretty concise, and the JavaMail API looks better in Groovy in my opinion.
Subscribe to:
Posts (Atom)