DEVONagent for Custom Searching Sponsor
DEVONtech is sponsoring Macdrifter this week with DEVONagent.
Last week I outlined the basics of DEVONagent. This week I want to dive a little deeper and build a user defined search engine for some of my favorite sites.
Plugins
DEVONagent comes with a large number of pre-made search plugins. A plugin defines one or more sites to use as a source for searching. Without any additional effort, DEVONagent allows you to search Lexis Web for legal documents or the USPTO for patents. You just choose the plugin that works for you.
But I search some more obscure sites like LeanCrew. There are no pre-existing plugin for that site. So I made one.
Open the Plugins and Scanners window, Ctrl+CMD+P, and create a new empty plugin. Enter the information for this new plugin, including the Web site URL.
In this case, I’m going to create a Web site plugin and rely on DEVONagent to doing the crawling. After creating the plugin, you can specific additional details about the plugin as well.
That’s it. Now you can leverage the searching capabilities of DEVONagent, such as the NEAR modifier which only returns results with matches within a few words of each other.
So, if you are looking for Dr. Drang’s post about Multimarkdown tables, it’s easy:
:::text
multimarkdown NEAR/10 table
These are the simplest plugins. DEVONagent crawls the source and returns search results. If you want to build on top of a site’s existing search engine, then that is pretty darn easy too. Just get the query string from the site and do a little experimentation to understand how it works. DEVONagent will execute the query and parse the results. I’ll mention how to do this at the end.
Search Sets
With a collection of custom search plugins, I connect them together as a Search Set. Search sets will execute the same query against a collection of defined plugins, returning results from all sites ranked in order of relevance.
Open the Search Sets window, Opt+CMD+S, and create a new set. Next, add all of the plugins to include in the single query.
My most used set searches LeanCrew.com, BrettTerpstra.com, Macdrifter.com, Apple StackExchange, and Mac OS X Hints in one single query. DEVONagent then ranks all results for relevance. That’s awesome. With some care and tailoring, this can be a very powerful tool and huge time-saver.
DEVONagent also allows you to specify the type of documents to search. For most of my needs, I only care about HTML.
Next is where the care and tailoring helps deliver better results. DEVONagent is a web site crawler. That means it follows links on a site looking for anything that matches the query. It doesn’t care if the result is an article, an archive page or a tag page. However, we can tell it how to care.
In the Secondary Query field, we tell DEVONagent what types of URLs are OK to follow. In this case, I’ve instructed DEVONagent to ignore aggregation pages that are common on all of the plugins for this set.
The Secondary Query option will speed up searching but will also reduce the number of results. It’s worth taking the time to understand the consequences. For me, I have a nice balance that gets the results I want without having to do multiple searches on different sites.
But I Like Google
That’s cool. DEVONagent likes Google too. Let’s go back and create a LeanCrew plugin that uses Google to do the initial search and DEVONagent to do the smart filtering.
Visit the Google Advanced Search page and create a site specific search for LeanCrew with a simple query string.
When this search is run, you will get a long URL in your browser. Copy it and head over to DEVONagent. Create a new plugin but make it a Search instead of web site plugin.
In the URL text box paste the long URL from the Google search. Replace the query string you used with agentQuery. DEVONagent will pass any new queries into this token.
Now you can use the Google search engine to return initial results but use the superior DEVONagent query language to find better results faster. You can also tailor the search on the fly and DEVONagent will remember your settings. Here are the settings I prefer.
If you turn on Follow Links you will get more accurate result ranking but searching will be much slower as DEVONagent will follow the Google results and links found on those pages looking for the ideal set of matching pages.
DEVONagent is a deep Internet search tool. The query language and result ranking is far superior to Google or DuckDuckGo. If you spend a lot of time searching the web and scouring results, you should probably be spending a fraction of that time in DEVONagent.