GunCrawler: creation, death, rebirth

The story of how I built and then abandoned my firearm shopping assistant, GunCrawler, and how I plan to reboot it

Way back in 2013, I was doing some online gun shopping and I was getting annoyed by how laborious the process was.

Even if I knew exactly what product I wanted, the process for finding the best price on a gun or accessory was ridiculous - google the name of the product (if I’m lucky, I have a UPC or MPN), click through the first result, note the price, click back, click the second result, note the price, click back…

Even in 2013 this was antiquated and was more like shopping in 2003. In 2013 we were all happily shopping on Amazon - the unified interface made it oh so easy to quickly find the best product for the task and do comparison shopping. It helped that the reviews were trustworthy and the products weren’t counterfeit Chinese garbage!

Scratching my own itch

As a programmer, I knew I had to build a tool to make this process easier. Even if nobody else would use it, I certainly would! So I got to work, chipping away at the problem with the precious hours I could invest after my day job and family life, and eventually, over a year later, I had a functional prototype I called GunCrawler. Here’s what the landing page looked like:

And here’s an example product listing page:

(This example doesn’t have multiple vendors listed for this product, but if they were, each would be listed in the “Vendor details” table, and the green price button at the top would show the lowest price and link to that vendor’s page)

Pretty basic, and it could definitely use a designer’s touch with the basic Bootstrap theming and the cluttered product images, wasted whitespace and so on, but the core functionality was there and it was good enough for an MVP.

I really should have launched it at this point, however at the time I knew nothing about startups, so I thought more work needed to be done because there was already SlickGuns (since renamed Gun.deals) already doing a similar thing. I felt I needed something extra to differentiate myself from them.

What I decided I needed was a powerful search feature, which then led me to feel the need for collaborative product data editing (possibly wiki-style).

Search

To implement search I first added Elasticsearch, which enabled text-based search. Elasticsearch is a clever tool - it handles misspellings, word stemming (searching for “reloading” will return results that contain “reload,” “reloader,” etc.), “did you mean” suggestions, things like that.

One important feature it supports that bogged me down though is facets. If you’ve ever browsed almost any online shopping site ever, you’ve probably seen filters dynamically appear that are relevant to what you’re searching for:

Picture courtesy Searchkick documentation

To enable this feature you have to tag your products. However, due to its simplicity, tagging is a very blunt tool. Let’s say I have a tag 7.62x39mm. If someone searches for that or clicks on that tag, what should I show? There is 7.62x39mm ammunition, but also firearms in that caliber (AK-47, SKS, vz. 58 to name a few). It would be confusing to have cartridge filters like steel-case (vs. brass-case) alongside filters for firearms (say fixed-stock vs folding-stock).

Further, if someone were to search on the AK-47 tag, how do I differentiate AK-47 complete firearms from parts? Do I need to introduce a firearm label and a part label? But if I’m labeling a part, like a barrel, it would be silly to require both barrel and part labels together - part should be implied from the fact that a barrel is-a part.

Collaboration

As I began to get bogged down in this data modeling problem, I realized it was going to be a whole lot of work for me to do this by myself. Every new product added to the site would need to be properly tagged. That’s a whole lot of work considering many firearm retail websites have from thousands to tens of thousands of products.

That’s not even considering the fact that while I enjoy firearms myself, I’m far from an expert, so my solo tagging attempt would undoubtedly be sub-optimal. Additionally, it’s guaranteed that I’d make mistakes that would somehow need to be found and corrected.

I also haven’t even gotten into the data ingestion process (how product data from across the web is gathered), which was fed by a bunch of scrapers that I hand-wrote for various websites, which, due to the nature of screen-scraping were eventually guaranteed to introduce dirty data that would need correction.

The obvious solution I latched on for all of the above is crowdsourcing the problem using a collaborative editing solution, in the style of a wiki - i.e. letting any user add, modify, or delete tags.

Unfortunately, that’s a lot of work to implement - on top of needing a well-built UI for editing, you need version control, conflict resolution, possibly a reputation system to prevent abuse (both from the ill-intentioned as well as well-intentioned).

Double unfortunately, both in 2013 as well as today, there aren’t any out-of-the-box solutions for adding this to a web app using a library or framework. I found libraries that added Wikipedia-style collaboration around rich text, but what I needed was for individual data fields. The closest thing today would be to spin up a Wikidata clone using Wikibase, but that would require me to build a website around their database model, which for various reasons would be difficult and restrictive.

Screenshot of Wikidata’s page on the venerable AK-47 (Q37116)

Burnout and fade away

Sadly the combination of having to build a complex collaborative editor (I hate UI-heavy work) and also solve the data modeling problems sapped away my energy to work on the project; I spent less and less time working on it, so it slowly withered away, never to see the light of day. 😩

A very unfortunate outcome for the many hours that I poured into it. Looking back, the most important lesson I learned is to launch your product no matter what. That means before you think it’s ready. Even if it sucks and nobody uses it, at least you took your shot - it’s better than never knowing. Plus I think if I got even a handful of users that could have reignited my passion to finish it properly.

That lesson is why I’m restarting this process with a newsletter from the very beginning - I need your help dear reader to help guide me as we build the tools that will improve gun shopping for all of us!

Reboot and expanded vision

Although I let GunCrawler fall into dormancy for seven long years, it never truly left my mind. Occasionally this puzzle would rise to the fore of my thoughts, and I’d dust it off and turn it around in my mind, but not make any progress solving it.

It’s not until recently that I’ve stumbled on some elegant new programming tools (actually “new old stock” would be more accurate) that I’m confident are going to finally cut the Gordian knot of this puzzle.

This new approach is affirming that my instincts around a collaborative, rich data model with powerful querying (search) was correct.

It’s going to take a bit of time to get a working prototype built, as I’m currently winding down my work on another startup and still in research/learning mode for this project (since these new tools require new knowledge), but I’m really excited to share progress updates with you as I proceed.

In exchange, you’ll be the first to test out these new tools! In particular, besides a basic product search, I anticipate the new data model is going to enable a couple powerful new features that I think you’ll find useful -

Parts picker for gun builds - think PCPartPicker for AR-15s and beyond! You’ll be able group products together arbitrarily into “builds” and share; that might be for an AR-15 you’re building, or for an IFAK you’re putting together. You’ll be able to compare to what other people are building, get and share feedback, as well as get modification suggestions from the website itself.

Optimized cross-vendor, “big picture” shopping (especially good for ammo shopping!). Imagine you have a list of parts you’re looking for; some parts are cheapest at Vendor A, some at Vendor B, some at Vendor C, some at Vendor D. You may not actually be better off buying the “lowest price” at each separate vendor after you take tax and shipping fees into consideration. This is especially obvious when you’re shopping for ammunition as the shipping fees can massively skew the final price! Figuring this out manually would require trying hundreds or even thousands of different combinations of products between vendors. Instead, we’ll provide an automated solution.

I have other ideas of course, but that’s what I thought I’d share for now. 🤫

Let’s get to work!

There’s a lot of work to be done, but I’m excited to get after it. And it couldn’t come at a better time. As industries have continued to consolidate around singular big players (Amazon for retail shopping, YouTube for videos, etc.), the treatment of guns as taboo on all these platforms has only gotten worse.

YouTube of course is well-known for demonetizing gun videos and even deleting videos as well as whole channels of popular creators.

Amazon meanwhile tolerates some gun accessories, but that noose continues to tighten - in fact I wrote an article for The Truth About Guns six months ago about how Amazon suddenly purged hundreds of gun accessories from their website that dared mention the unutterable words “AR-15” or “AK-47”. I expect the list of gun-related products will continue to shrink on Amazon rather than expand.

While it’s frustrating that gun owners are continually excluded from the popular platforms and forced to build our own solutions, the flip-side of that is that the big platforms are also excluding themselves from our market segment. That gives us a big opportunity to innovate without interference from the big players sucking all the oxygen out of the room. Put another way, while it’s inconvenient we can’t buy most gun accessories on Amazon, the upshot is dangerous Chinese counterfeits and AmazonBasics clones won’t be able to dominate the market.

---

I think we have the chance to build something really special together here, and I’m looking forward to sharing updates with you and also hearing back from you! Please feel free to comment on my posts here or contact me privately at _@abevoelker.com.

Lastly, if you have any friends that would be interested in a better solution for firearm-related shopping, please share this with them! 😀