Laravel robots.txt, Sitemap, and Canonical URLs: My Production SEO Checklist

In ICanUp, robots.txt, sitemap.xml, and canonical URLs were implemented at different stages of the SEO work.

Each part looked simple in isolation. In production, however, I learned that verifying them independently was not enough.

All of these mechanisms need to describe the same reality: which URLs are public, which can be crawled, which should be discoverable through the sitemap, and which URL is the primary identity of a page.

The biggest change in my approach was to stop treating these mechanisms as interchangeable.

If a sitemap contains a URL, robots.txt accidentally blocks it, and the page declares another canonical address, the crawler receives conflicting signals.

I did not want a static file where the production domain or sitemap URL could drift away from the actual application configuration.

/robots.txt is therefore generated by the application and returned as a public text/plain response.

The exact list of restricted paths changes with the application, but the principle remains the same: public content should never be caught by an overly broad Disallow.

My sitemap started with Posts, but more public content types were added as ICanUp grew.

The important criterion is no longer the model type itself but whether the resource is actually available to a normal visitor and a crawler.

• Public homepage URLs.
• Active public Categories.
• Published Posts.
• Published Pages.
• Active public PostSeries.
• Localized URLs only for available language versions.

Drafts, future publications before published_at, deleted records, and other non-public resources should not appear in the sitemap.

I treat the sitemap as a projection of public eligibility, not simply a list of database rows.

The sitemap tells a crawler that a URL exists and matters for discovery. Canonical has a different role.

On the page itself, I explicitly declare the URL the application considers primary for the current content and locale.

For a multilingual Post, the EN canonical should not point to the Ukrainian page simply because Ukrainian is the default locale.

Each available language version has its own self-canonical URL.

I covered this contract in more detail in my article about canonical and hreflang for multilingual Laravel pages.

In the current architecture, Posts and Categories have one shared slug across locales.

Ukrainian is the default locale and has no prefix. English uses /en.

The URLs should therefore be generated through the same locale-aware routing infrastructure instead of being manually concatenated by individual SEO services.

The same rule is reused for sitemap URLs, hreflang, Open Graph URLs, and other SEO signals.

One of the most dangerous designs would be to let every SEO subsystem independently decide whether content is public.

A sitemap could then include a scheduled Post while the public controller returns 404 and IndexNow has already submitted the URL.

The SEO layer should not invent its own definition of published content.

• A Draft is not included in the sitemap.
• A scheduled Post is not included before its publication time.
• Soft-deleted content is excluded.
• An inactive Category or Series is not advertised as an indexable URL.
• A missing translation does not create a fake localized URL.
• A public URL should actually resolve under the same application contract.

Another value I no longer verify superficially is lastmod.

When a Post uses Content Blocks or localized content, the public page can change without an obvious update to the main posts.updated_at.

The lastmod value therefore needs to represent the latest relevant change to that public page version.

I documented a real locale-specific Content Block case in my article about Laravel sitemap lastmod and why the model updated_at value was not always the real page modification time.

Inertia SSR changed how I verify production SEO.

What matters is not only what DevTools shows after hydration, but also what a crawler receives in the initial server response.

This is especially useful for canonical, hreflang, robots metadata, Open Graph, and JSON-LD.

• /robots.txt returns HTTP 200.
• /robots.txt has the correct text/plain Content-Type.
• /sitemap.xml returns HTTP 200.
• The sitemap returns an XML Content-Type.
• Both endpoints are available without authentication.
• robots.txt references the current sitemap URL.
• The sitemap contains no admin or other private URLs.

A sitemap can look fine in a browser while an escaping or XML structure error makes it invalid for a crawler.

Another simple production check is to take several URLs directly from the sitemap and request them.

If the sitemap advertises a URL that returns 404, the problem is no longer the XML. Two application contracts have drifted apart.

After adding IndexNow, I had another indexing signal available.

Its role is different: the sitemap provides a stable inventory of public URLs, while IndexNow can notify search engines about individual changes faster.

I do not remove the sitemap simply because the application can now submit URLs through IndexNow.

I described that pipeline separately in my article about automatically submitting new and updated Laravel URLs through IndexNow.

After several SEO changes, I stopped checking only the feature I had just modified.

Even a small routing or publication change can affect several SEO signals at once.

• Open /robots.txt without authentication.
• Verify the robots.txt Content-Type.
• Verify the current Sitemap: URL.
• Confirm that public routes are not blocked by an accidental Disallow.
• Open /sitemap.xml.
• Verify HTTP 200 and the XML Content-Type.
• Validate the XML.
• Find several real UK and EN URLs in the sitemap.
• Confirm that Draft and future content are absent.
• Verify lastmod after a real content change.
• Open several sitemap URLs and verify HTTP 200.
• Inspect canonical in raw HTML.
• Verify self-canonical separately for UK and EN.
• Verify hreflang and x-default.
• Confirm that URLs come from application routing.
• Verify that SSR exposes SEO metadata before JavaScript runs.
• After a significant SEO change, check Google Search Console and Bing Webmaster Tools.

The production checklist does not replace tests, and tests do not replace a production smoke check.

They protect different layers for me.

• robots.txt returns the expected status and Content-Type.
• robots.txt contains the sitemap URL.
• The sitemap returns a valid XML response.
• Published content appears in the sitemap.
• Draft content does not appear.
• Future-published content is excluded before its publication time.
• Inactive content is not advertised as indexable.
• Localized URLs match actual routes.
• Canonical matches the current locale.
• SEO metadata exists in the SSR HTML.
• Content Block changes affect lastmod where appropriate.

• Treating Disallow in robots.txt as a complete substitute for noindex.
• Adding a URL to the sitemap simply because a database record exists.
• Adding a scheduled Post before its real publication time.
• Building locale URLs manually instead of using routing infrastructure.
• Pointing an EN canonical at the UK URL.
• Assuming one updated_at always represents a complex Content Block page.
• Checking canonical only after client-side hydration.
• Not checking the HTTP status of URLs already published in the sitemap.
• Treating IndexNow as a replacement for the sitemap.
• Changing routing without rechecking dependent SEO signals.

• robots.txt, sitemap, and canonical URLs have different responsibilities.
• Public eligibility should be shared across SEO infrastructure.
• A sitemap should not advertise a URL that the application itself returns as 404.
• Canonical URLs should use the same locale-aware routing contract.
• UK and EN should have their own self-canonical URLs.
• lastmod should reflect real public-page changes.
• SEO metadata should be verified in raw SSR HTML.
• SEO endpoints should be checked as normal HTTP responses.
• IndexNow complements the sitemap rather than replacing it.
• Both automated tests and a short production check are useful after deployment.

Implementing robots.txt, sitemap, or canonical URLs individually is not especially difficult. The harder part is keeping them consistent after many later changes to routing, localization, and publication logic.

In ICanUp, I eventually arrived at a simple rule: one public URL contract comes first, while robots.txt, sitemap, canonical, hreflang, and IndexNow describe different aspects of it.

When all of these signals come from the same application reality, production SEO becomes much easier to verify and much harder to break accidentally.