Drupal 11.4 blocks search query URLs in robots.txt: what to check
Drupal 11.4 landed on July 1, 2026, and the headlines went to performance. Fair enough — the numbers are spectacular. But hiding between them is a two-line change to a file most teams never look at, and it is the one item every Drupal site owner should verify this month. Fittingly, both stories are about queries: the ones crawlers send, and the ones your database answers.
The change: search result pages with query parameters are now blocked
Drupal core’s default robots.txt now ships with these additional rules:
Disallow: /search?
Disallow: /index.php/search?Any core search URL carrying a query string — /search?keys=drupal+expert, plus every pager, sort and filter
combination hanging off it — is no longer crawlable by well-behaved bots.
Why core did it
Search result pages are a crawler trap: each keyword, pager and parameter combination mints a new URL, so bots happily burn through an infinite URL space of thin, near-duplicate pages. That wastes crawl budget that should go to your real content, adds server load for zero SEO value — and the problem has grown teeth in the AI era, where aggressive crawlers hammer exactly this kind of parameterized URL. Blocking internal search results has been an SEO best practice for years (it is one of the first things I fix in technical audits); with 11.4, core finally makes it the default.
What it does not do
Three nuances that matter before you rely on it:
- Blocking crawling is not deindexing. URLs already in Google’s index can linger as “indexed, though blocked
by robots.txt”. If search URLs are polluting your index today, you need a
noindexmeta tag on those pages first (let Google recrawl and drop them), and the robots.txt block as the steady state. - It only covers
/search. A custom search built with Views at/find, a translated path like/recherche, or a Search API page at/search/sitewith its own parameters is not covered — replicate the pattern for your own routes. - Faceted search deserves a decision, not a default. If your facet URLs are part of your SEO strategy (category + filter landing pages that earn traffic), do not blanket-block them — choose deliberately which parameter combinations should live. That is exactly the kind of trade-off I tuned on DOGA’s Solr search.
Who has to act manually
This is the trap in the release: the new rules do not reach every site automatically.
| Your setup | What happens on update |
|---|---|
Stock robots.txt, composer scaffold enabled |
The file is re-scaffolded — you get the new rules for free |
Customized robots.txt (excluded from scaffold) |
Nothing changes — add the two rules yourself |
| RobotsTxt module (file served from the database) | Nothing changes — add the rules in its settings |
| Customized file, scaffold not excluded | Danger in reverse: the update may overwrite your customizations — diff it after upgrading |
The two-minute verification, right now:
curl -s https://your-site.example/robots.txt | grep 'search?'No output on a Drupal 11.4 site = you are in one of the manual cases. While you are there, check Search Console’s
crawl stats for how much of your budget currently goes to /search? URLs — that number is your before/after proof.
The other queries: the 11.4 performance diet
The same release cuts the queries on the inside:
- Half the database queries vs 11.3 across a wide range of requests, thanks to optimizations in entity field loading — and just over one third of the lookups compared to Drupal 11.0/10.6 on a cold cache.
- Fewer joins in entity listings, which particularly benefits JSON:API — relevant if you run a decoupled front end.
- Brotli compression for aggregated CSS/JS (15–25% smaller than gzip; requires the PHP Brotli extension) — a direct Core Web Vitals win.
- Cacheable 404 responses — the quiet cousin of the robots.txt change: bot storms on dead URLs no longer punch through to PHP on every hit.
- Translation update checks run 87% faster, and 11.4.x is security-supported until June 2027.
The takeaway
Upgrade to 11.4 for the performance, then spend ten minutes on the robots.txt check above — it is the rare SEO improvement that ships in core but still needs your eyes if your file is customized. And if you are reading this from a Drupal 10 site: 11.4 is what you are missing, and the clock runs out on December 9, 2026.
Not sure whether your search URLs, facets and crawl budget are configured deliberately or by accident? That is a one-day audit.