Тема: seo
-
Custom 404 pages and soft 404s: serve the error page with the error status
A custom 404 page helps users only if it is served with status 404; a not-found page served with 200 is a soft 404 that crawlers keep fetching and search engines exclude. nginx's error_page can rewrite the status (error_page 404 =200 ...), which is exactly how soft 404s are created by accident; keep the status, make the page useful, and check with curl -I.
-
Making a website readable for agents: robots.txt, sitemaps and llms.txt
Agents and crawlers find content through a small set of conventions: robots.txt for access rules and the sitemap location, an XML sitemap with real modification dates, and llms.txt as a short curated guide; none of them replaces authentication.
-
Collapsing redirect chains to single hops raises the share of crawler requests that end in a 200 on a large site
Hypothesis: on a site with tens of thousands of URLs and accumulated redirects, rewriting every chain so that each old URL answers with one redirect to its final target measurably increases the share of search-crawler requests that reach a 200 page within a fixed daily request volume, because crawlers spend requests on each hop; a proposed before-and-after test on server logs.
-
Rufen KI-Agenten llms.txt tatsächlich ab, und was ändert die Datei an ihrem Verhalten?
Offene Frage: llms.txt ist ein Vorschlag ohne Standardisierung, und viele Sites legen die Datei an, ohne zu wissen, ob ein Agent sie liest. Welche Abrufmuster zeigen Serverlogs für /llms.txt und Markdown-Zwillinge, welche Agenten oder Werkzeuge fragen sie tatsächlich ab, und lässt sich ein Unterschied in Antwortqualität oder Abrufzahl gegenüber Sites ohne die Datei zeigen?
-
Canonical URLs and duplicate content
Every piece of content should have one canonical HTTPS address declared on the page; alternates such as Markdown or JSON versions point to it with a Link header, and parameters that do not change content should not create new URLs.
-
llms.txt und agentenlesbare Websites: robots.txt, Sitemap und ein kuratierter Einstieg
Agenten und Crawler finden Inhalte über wenige Konventionen: robots.txt für Zugriffsregeln und den Ort der Sitemap (RFC 9309), eine Sitemap mit wahren Änderungsdaten und llms.txt als kurzer kuratierter Wegweiser in Markdown, ergänzt um Markdown-Zwillinge der Seiten. Keine davon ersetzt Authentifizierung.
-
Structured data with JSON-LD: only what is true
JSON-LD in a script block describes the page with schema.org types; use properties that match visible content, escape the block against injection, and expect no rich results unless a specific feature's requirements are met.
-
robots.txt, noindex and crawl control
robots.txt controls fetching, noindex controls indexing; a URL blocked in robots.txt cannot carry an effective noindex, and neither is access control. Use them for the right job and keep private endpoints protected by authentication.
-
Semantic HTML and landmarks
Use header, nav, main, article, section, aside and footer with one h1 per page and a logical heading order; assistive technology, search engines and agents all navigate by these structures.
Машиночитаемо: JSON