Fast for the right reason
The speed comes from the language and from fetching pages side by side. Not from skipping checks.
SEO crawler
CrawlStat
A site crawler that does not keep you waiting.
CrawlStat is an SEO crawler I wrote in Rust. It does the job of Screaming Frog: it goes through every page of a site and tells you what is wrong. It just does it faster, so you can run it often and get on with your day. It also does something the others do not: UX Testing, which checks whether a page does the one job it is there for.
CrawlStat Visit crawlstat.com (opens in a new tab)
- Role
- Design and build
- Type
- SEO and UX tool
- Built with
- Rust
01 The problem
Checking a siteshould not take all day.
If you look after a website, you need to know what is broken on it. Dead links. Pages that redirect, then redirect again. Pages with no title, or the same title as ten others.
The tools that find these things work. But on a big site they can be slow, and a slow check is a check people stop running. I wanted one I would run every week. So I built it.
02 What I did
What I did
- 01
Wrote it in Rust
Rust is a language made for speed. It lets the crawler fetch lots of pages at once and stay steady while it does.
- 02
Checked the basics, on every page
Broken links, redirects, missing titles, missing descriptions. The things search engines trip over, checked page by page.
- 03
Put the problems first
A crawl can turn up thousands of rows. What you need from it is a short list of what to fix. That is what I built it around.
- 04
Taught it to test pages, not just links
Then I added UX Testing, which checks a page against the one thing it is there for. Both are the first things I run on a new client site, so I see the whole picture before I change anything.
03 Details
Details that matter
The checks you already know
It looks for the same kinds of problems the well known crawlers look for. Nothing new to learn about what counts as broken.
Sees the site the way Google does
It follows links from page to page, the way a search engine finds its way around. If people cannot reach a page, neither can Google.
Quick enough to run often
When a check is quick, you run it after every change, not once a quarter. That is how small problems stay small.
UX Testing: does the page do its job?
Give it one page. It works out what kind of page it is and what it asks people to do, like book, buy or get in touch. Then it runs around 60 checks picked for that goal and ranks what to fix first.
Honest about what it could not test
Every check ends as pass, fail, needs review, not tested or blocked. A check that needs a real browser says so instead of passing. Run it again after the fix and you can show what now passes.
- 62
- UX checks to pick from
- 9
- Page types it tells apart
- 45
- SEO and accessibility audits
04 For your site
What you can take from it.
Look at the jobs you keep putting off. Most of the time the job is fine and the tool is the problem: it is slow, or the answer is buried. Make the task quicker and it gets done. The same goes for your customers. The step they skip on your site is usually the one that asks too much of them.
Want something like this?
Need your site checked properly, a page tested for whether it does its job, or a tool built around the way you work? Tell me about it.
Start a project
Tell me what you want to build.
Pick what you need and see the price right here, or just send a few lines. I read every message and reply myself.
What happens next
See all prices and what each includes