Talk to any SEO who has been evaluating sites for a while and eventually they’ll mention Xenu for its ability to quickly crawl a site and provide the foundation for a site audit. I, too, am a fan, but I’m an even bigger fan of Link Examiner. The feature list of both tools is too long to describe, so let’s focus on the important differences from an SEO’s point of view.


URL Pattern Matching
There’s a special version of Xenu you can download that lets you exclude URLs, but it only supports basic wildcards (*). Link Examiner allows you to exclude AND require certain patterns which you can specify as *(anything), ? (characters), and # (numbers). Link Examiner makes it easier to ignore links that will clutter the report while also making it possible to zero in on a specific section of a site.

Links Into Page (Internal)
Both tools show internal links into a page, but Xenu only shows the count in a pop-up window. That means, in order to find interesting link counts, you need to right click on each URL of interest in the report – a really slow way of doing things. Link Examiner, on the other hand, includes a column with internal inbound link counts, making it easy to spot what pages are receiving the most link juice. Combined with the wildcard setting, you can look at how pages within individual topic areas link between each other.

Link Type
For quick filtering, link type identification can be quite handy. Are most links image-based or nice, clean text-based links? Inquiring minds want to know.

Obey Robots
Link Examiner lets you simulate the same robots exclusions that would apply to a search engine’s crawler. Aside from verifying complex robots.txt files, this also can be used to give you a better page count to compare against index counts reported by the search engines.

Obey Nofollow
Similar to obeying the robots.txt, obeying the nofollow attribute can help you better assess your use of nofollow internal links, as well as what external links truly get an endorsement from you. Even with Google’s recent change in how nofollow affects the distribution of PageRank, this data is another piece in the puzzle of understanding your site’s information architecture.

Skip if Recently Modified
For sites that generate pages on the fly – via calendars or other tools – this option prevents the crawl from running forever. Assuming a last modified date is returned, Link Harvester will stop crawling URLs that have been recently modified (e.g. within a few seconds of the current time). I’ve often left Xenu running overnight, only to have it still be running when I came back in the morning because it got hung up on these sorts of dynamic URLs.

Identify Duplicate Pages
While the definition of duplicate content varies depending on who you ask, Link Examiner’s metric can be handy when doing a spot check of a site (competitor or prospect) to gauge the level of any previous SEO effort.

Skip Images
Verifying that image paths are valid is certainly a good thing to do, but it’s not something I look at when auditing a site; having my report cluttered with image paths just results in more cleanup work. Link Examiner lets me exclude these before the crawl starts.

I think you’ll agree that Link Examiner does almost everything that Xenu does, while also providing some additional useful features. However, Link Examiner does have one annoying flaw – it doesn’t allow you to save your project and resume it at another time. For really big sites, being able to pause a project and save it means being able to shut down my laptop, go home, and continue the crawl the following day. And one area where both tools fall short is that they don’t use a disk-based, open database format for storing data, meaning you can’t easily query and manipulate data from large datasets without it taking a long time and requiring a lot of memory. But…both tools are free so I’m not really complaining.

Thanks to Ann Smarty of Search Engine Journal for being the first to introduce me to Link Examiner!