scrapy-mcp
A headless web-scraping MCP server built on Scrapy, providing tools for polite fetching, CSS/XPath extraction, link/table extraction, sitemap and robots.txt reading, and bounded asynchronous crawls.
Browser Automation
0 platforms
0 features
Install
uvx scrapy-mcp
Paste the configuration above into your MCP client config (claude_desktop_config.json for Claude Desktop) and restart the client.
{
"mcpServers": {
"scrapy-mcp-scraper": {
"command": "uvx",
"args": [
"scrapy-mcp"
],
"env": {
"SCRAPY_MCP_JOB_DIR": "<SCRAPY_MCP_JOB_DIR>",
"SCRAPY_MCP_LOG_LEVEL": "<SCRAPY_MCP_LOG_LEVEL>",
"SCRAPY_MCP_MAX_BYTES": "<SCRAPY_MCP_MAX_BYTES>",
"SCRAPY_MCP_USER_AGENT": "<SCRAPY_MCP_USER_AGENT>",
"SCRAPY_MCP_OBEY_ROBOTS": "<SCRAPY_MCP_OBEY_ROBOTS>",
"SCRAPY_MCP_RETRY_TIMES": "<SCRAPY_MCP_RETRY_TIMES>",
"SCRAPY_MCP_AUTOTHROTTLE": "<SCRAPY_MCP_AUTOTHROTTLE>",
"SCRAPY_MCP_JOB_TTL_DAYS": "<SCRAPY_MCP_JOB_TTL_DAYS>"
}
}
}
}Paste the configuration above into your MCP client config (claude_desktop_config.json for Claude Desktop) and restart the client.
Sources: public MCP Server directories. This is an independent third-party directory with no affiliation to or endorsement from the maintainers of the listed servers.
