Website Launch Checklist: robots.txt and Social Preview Cards

A launch check should confirm two separate things: whether crawlers can access the pages you intend to expose, and whether a shared link shows the right title, description, and image. These checks use different metadata and serve different purposes.

Review robots.txt rules

Open the site’s /robots.txt file and confirm that its rules match the pages you want search crawlers to fetch. The Robots.txt Generator can help draft directives; review the output before publishing it.

Google explains that robots.txt controls crawling, not reliable removal from search results. A blocked URL may still be known to Google from other links. Do not use a disallow rule as a substitute for authentication or a noindex directive when you need to restrict access or indexing.

Preview social metadata

Open Graph tags such as og:title, og:description, og:image, and og:url help social services form a shared-link preview. Use the Open Graph Checker to inspect the tags returned for a URL and the X Card Validator to review the card-specific metadata your page exposes.

Test the final public URL

1. Test the canonical production URL rather than a staging or preview address.
2. Check that the page can be fetched without a login or unexpected redirect.
3. Make sure the preview image is publicly accessible and represents the page.
4. After changing tags, allow the platform’s cache to refresh or use its official debugger if available.

These checks help catch configuration mistakes; they do not guarantee indexing, rankings, or a particular preview on every platform.

Sources

• Google Search Central: robots.txt introduction
• Open Graph Protocol