A page crawler told me a website with 18 services had almost none. The tool didn't misread: what it read was an anti-robot verification screen, and it marked it as "success". This is the third repo in the series and the one that taught me the most about distrusting the word success.
Crawl4AI is an open project that visits a page and hands it back as clean text, ready for a model to read. The repo is here: github.com/unclecode/crawl4ai. What follows are my measurements and my stumbles, not its manual.
What it does and why I added it
When you ask an agent to research a business, the first thing it needs is to read the site. If you give it raw HTML, it receives a load of tags, menus and scripts it can't use. If you give it a screenshot, it has to read an image. Crawl4AI opens the page in a real browser, waits for whatever is painted with JavaScript to load and hands you the content already clean in markdown, which is the format a model reads best.
I first saw it in my records on August 17 and put it to work on September 16. My case: researching SVDIGINET prospects before writing to them. I want to know what services a local business offers, whether it has a contact form and which emails it publishes, and I want every fact to come with the page it came from. A service of mine, Scout, does that, and Scout asks Crawl4AI for the pages.
Source: Crawl4AI repository
How I set it up, with numbers
I run it on a small private virtual server: two processors, 7.8 GB of memory and no swap. That last figure decides everything. Without swap, if a program eats the memory, the system kills the first thing it finds, and I have another important service on the same machine.
| Decision | Value | Why |
|---|---|---|
| Image | Version 0.9.3, pinned by its digest | So an update doesn't change behavior without my deciding it |
| Address | 127.0.0.1, port 11235 | Only the machine itself sees it. No port open to the internet |
| Resource cap | 3 GB of memory, 1.5 processors, 512 processes | If it runs away, it protects the other service |
| Idle use | about 561 MiB and under 1% of a processor | Measured on deploy day |
| Access | A separate internal network; containers I connect and whoever holds the token can enter | Without the token, nobody can ask it for pages |
| Password | A token on every request | Without the token, 401 |
And this is what I checked that day, before letting it work:
| Test | Result |
|---|---|
| Request /health | 200, no token needed |
| Request /crawl without a token | 401 |
| Request /crawl with a token | 200 |
| Crawl a public test page | 490 characters of real markdown |
| See on the server which ports listen | Only 127.0.0.1:11235, not the public address |
A detail that took me a while to understand: in this version, if you don't set a token, the published port stops working without a warning. It doesn't end up open without a password, it ends up mute. Setting the token does two things at once: it turns on authentication and makes the port reachable. So "no authentication" is not an option left open by accident, and that removed a worry. It also taught me not to trust that a service fails loudly.
The day it said success and read nothing
On September 21 Scout was researching a local business's site. The result said it had little service content. I knew they offered 18. I started looking at what it had actually read.
- The requestScout asks Crawl4AI for the page.
- The responseThe site returns an anti-robot verification challenge: about 7 KB of code and barely 61 characters of text.
- The labelCrawl4AI marks the response as success, because the site responded, though not with the page.
- The errorScout takes that success for a readable page and concludes the business has few services.
Two defects together. First, my broken-page detector required very little text AND very little HTML; the challenge's code was long enough to fail the second condition. Second, the module that builds the findings never asked whether the page had been readable. The system went from "it responded" to "I read it" without anyone checking.
Worst of all, the test was intermittent. In later runs, the same site let Crawl4AI through. A live retry proves nothing, for or against. What did work was feeding the new code the exact bytes I had captured that day.
The fix and its limit
I added a content judge that decides, page by page, whether what was received can be read, and classifies what can't: challenge, refusal, a page that never painted, or empty. Whatever doesn't pass that judge never enters extraction or produces a finding. If no page is readable, the result is "review by hand", not an invented conclusion.
- Every page goes through the judge before it is used for anything.
- The one that fails is kept in a separate list, with the reason.
- If none is readable, the prospect is flagged for manual review.
- Only then, and only for those reasons, a second crawler is tried (below).
The real limit: the fix was proven on one captured case (7 KB of code, 61 characters of text) and I never saw the block again live, because it was intermittent. I wrote 23 automatic tests for that version, with both crawlers simulated so they run in seconds and without touching the internet. A limitation I found while building them: Crawl4AI blocks internal server names for security, so I couldn't stand up a test site of my own on the same network and point it there. That is why the tests simulate the response instead of using a real server.
The rule I took from this works for any tool that hands you a status: "success" says the connection worked, not that what came back is useful. I now measure the content, not the response code. If the useful text is minimal compared with the weight of the page, something is wrong even if everything says 200, and I would rather have the system say it doesn't know than invent a conclusion with great confidence.
That version (1.3) was frozen on September 21: no new features and no architecture changes to what already existed. A week later I added a separate contacts endpoint without touching its logic. A defect that costs trust gets fixed and then left alone.
And Firecrawl, what for?
When the judge says "challenge" or "refusal", Firecrawl comes in, which is a service from another company, with credits. It is a last-resort tool: capped at 3 calls per request, and its response goes through the same judge. If it fails or there is no key, the result goes back to "review by hand". It never drops to an invented finding.
| Crawl4AI | Firecrawl | |
|---|---|---|
| Where it runs | On my server | In another company's cloud |
| When I use it | Always, first | Only if the judge rejects the page |
| What worried me | That it said success on a challenge | The spend: on October 12 it was at 50% of credits |
One primary tool, and another that only wakes up when the first fails in a way I recognize. It is the most reusable part of the whole setup.
What I don't know
- How many sites return a challenge in a normal month. I log the rate of each request, but I haven't done the analysis.
- How it behaves on a new version. My tests are for 0.9.3 and I haven't upgraded.
- The license. I didn't write it down, so check it in the repo yourself before using it with clients.
- Whether another backup crawler suits me. I left one as a candidate and I'll decide with the numbers, not before.
- How well it handles a very heavy site. My tests were on small local-business pages.
Check your ports
If you have something running on a server, this is the check that gave me the most value in this whole setup. Paste this into Claude Code, inside the server. It only reads:
Check, without changing anything, which Docker services I have running on this server and which address each one listens on. For every published port, tell me whether it listens only on 127.0.0.1 or on all interfaces (0.0.0.0), and whether it has any kind of password or token. In a short list, flag the ones that are open to the outside without authentication.
If you want to see how I use agents so I don't repeat this kind of mistake, read the Wick post, the second repo in the series.
