Announcing NerdQuery
Did you ever start something just to scratch your own itch.? Then it started to be better than you expected so you kept building and polishing until it became something you wanted to exist in the real-world? Yeah? I just did that.
Backstory
I steal a lot. Everyone steals. When I write a script or macro, I’m not ignorant of those that came before me. Before I embark, I do a few searches. I find myself searching the same few sites. I search BrettTerpstra.com then I go to leancrew.com and maybe visit SimplicityisBliss.com. It’s the same process over and over and I was sick of it.1
So I made a thing…
NerdQuery
NerdQuery.com is a site specific search engine limited to the few nerds I identify with and create great stuff. I’m making it publicly available to anyone that likes the same stuff I do.
I’d like to thank Erik Hess (@themindfulbit) for all of the design work It looks pretty because he’s good at his job.2 I can’t recommend his work enough and guess what? You can work with him at high90.
Content
At this moment, there are 8 sites, 7343 links and 77,369 keywords in the index in two primary categories. Sites may come and go but these are some of my most frequented.
Development
Workflow
Man at Work
I make no guarantees. It’s a work in progress and things will change and occasionally break. In fact, I bet this announcement breaks it in some way.
Re-indexing is running daily. Indexing WordPress sites takes a rather long time. Especially sites with as much content as these. It’s unlikely I will increase the rate of indexing and I may actually decrease it.
Functionality
The search is built on Sphider Plus which I recently mentioned. It responds to several flags to help narrow results. The NerdQuery help section describes most of the useful features.
I’ve also assigned each site to a category. This allows results to be narrowed to a specific site or group of sites. It’s still not my ideal and there are some bugs. But, as they say, perfect is the enemy of done.3
Priorities of results are determined by a few ranking parameters, like the number of occurrences and if the hit is found in the title or URL. I may tweak the weighting over time.
There are no ads or sponsored links. No one pays to be in the index or results. I’d rather have a clear conscience than a new business model. If you want to support the site, there’s a Stripe donation page. If you would rather donate your expertise, then that is always welcome too. Just email me.
Tips
Here’s a URL for Launchbar.
http://nerdquery.com/?media_only=0&query=*&search=1&category=-1&catid=&type=and&results=50&db=0&prefix=0
Just go into the Launchbar index and add a new utf-8 search source.
Here’s a Keyboard Maestro macro for doing the same thing:
When triggered, it presents a pop-over panel for entering search terms. Hitting enter opens a new Safari window with the results.
Here’s the search URL that is used in the macro:
Here’s how to configure Alfred to use NerdQuery:
Just use this Query URL: http://nerdquery.com/?media_only=0&query={query}&search=1&category=-1&catid=&type=and&results=50&db=0&prefix=0
And here’s how to configure iCab for iOS. Go into the iCab in-app settings and add a new search engine. Use this string for the URL:
Oligarchy
The current list of indexed sites contain my preferred sources. I request permission from each site owner before crawling.4 I will probably never index highly-commercial sites. I’d like to keep the number of indexed sites small. It makes the results more useful. Honestly, no one looks past the third page of results on most search engines. Each person that I index gets some say in what else gets indexed. They also get the admin credentials for the indexer. That seems pretty fair and allows them to decide what gets indexed from their own site. That also means I only index sites of people I trust.
I do not simply crawl each site and index them. I write site specific exclusion rules to reduce result pollution by generic summary pages. Configuring the exclusion takes time and effort. As a consequence, new sites will be added slowly.
My primary motivation for making this is so that I can use it. I’m making it public in case anyone else finds it useful too.
Enjoy.
-
Remember when Yahoo was good? You could go to a category and find good results from a limited pool of sites? Good times. ↩︎
-
Seriously, the Sphider front-end is fugly. Without Erik’s help I would not have made this public. It was too embarrassingly ugly. ↩︎
-
It’s actually, “Perfect is the enemy of good” but this is good too. ↩︎
-
I don’t want to be an asshole. Sphider Plus will actually let me ignore robots.txt, although I do not. ↩︎