robots-txt
Use when applies to any public website. Use when auditing technical SEO foundations or diagnosing crawl coverage gaps.
DeepseekModel
キュレーション済みスキル
品質 優秀 · 90
v1.0.0
取得
https://deepseekmodel.com/api/download.php?id=thedaviddias-front-end-checklist-skills-robots-txt-skill-md&format=skill
ダウンロード .skill
標準形式。system_prompt と model_config を収録し、任意の Agent で利用可能
.skill ファイルの system_prompt フィールドの実際の内容。
name robots-txt description Use when applies to any public website. Use when auditing technical SEO foundations or diagnosing crawl coverage gaps. metadata {"category":"seo","priority":"high","difficulty":"beginner","estimatedTime":"10","source":"frontendchecklist.io","url":"https://frontendchecklist.io/en/rules/seo/robots-txt"} Publish a robots.txt file robots.txt is the first file crawlers fetch; misconfigured directives can silently block search engines from crawling your entire site, killing organic visibility. Quick Reference Serve a valid robots.txt at /robots.txt on the production domain, returning HTTP 200 Include a Sitemap: directive pointing to your XML sitemap Never disallow crawling of CSS/JS assets that render your pages Avoid blocking all crawlers with Disallow: / on a live site Check Fetch /robots.txt on the live domain and verify it returns HTTP 200, uses correct User-agent / Disallow / Allow syntax, and includes a Sitemap: directive pointing to the XML sitemap. Check for accidental Disallow: / directives. Fix Create or update robots.txt at the web root with valid directives. Add a live Sitemap: line for the production sitemap URL. Remove any Disallow: / rules that block the whole site or resources needed for rendering. Explain Explain how robots.txt controls crawler access, why an accidental Disallow: / can delist a site, and why CSS/JS must remain accessible for rendering-based indexing. Code Review Review metadata generation, rendered HTML, structured data, and response headers related to Publish a robots.txt file. Flag exact routes or templates where search-facing output violates the rule, and describe how to verify the final page output. For full implementation details, code examples, and framework-specific guidance, see references/rule.md . Rule page: https://frontendchecklist.io/en/rules/seo/robots-txt
このスキルを起動するキーワード。クリックでコピーできます。
このスキルにはトリガーワードがありません。
ダウンロードした .skill に含まれるフィールド。
| フィールド | 説明 |
|---|---|
| format | フォーマット識別子（skill/v1） |
| skill_id | スキル固有 ID |
| name | スキル名 |
| version | バージョン |
| description | 説明 |
| category | カテゴリ（配列） |
| trigger_words | トリガーワード |
| tags | タグ |
| source | ソース |
| source_url | ソース URL（本ページ） |
| exported_at | エクスポート日時（ダウンロード毎） |
| system_prompt | システムプロンプト本文 |
| model_config | モデル設定：provider / model / temperature / max_tokens / top_p |
| examples | サンプル |
| install_guide | 各プラットフォームの導入説明（Coze / Dify / Claude / カスタム） |
