当我使用Koa作为web服务器，Puppeteer作为爬虫工具来编写一个简单的爬虫教程时，发生了戏剧性的一幕。

下面我将创建一个完整的Koa + Puppeteer爬虫教程页面，包含代码示例、执行演示和详细说明。

在这里插入图片描述

设计思路

左侧为教程内容区域
右侧为实时爬虫演示区域
使用卡片式布局展示代码示例
添加交互式爬虫演示功能

实现方案

<!DOCTYPE html>
<html lang="zh-CN">
<head><meta charset="UTF-8"><meta name="viewport" content="width=device-width, initial-scale=1.0"><title>Koa + Puppeteer 爬虫教程</title><link href="https://fonts.googleapis.com/css2?family=Roboto:wght@300;400;500;700&family=Source+Code+Pro&display=swap" rel="stylesheet"><link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.0.0/css/all.min.css"><script src="https://cdn.jsdelivr.net/npm/chart.js"></script><style>:root {--primary: #4361ee;--secondary: #3f37c9;--success: #4cc9f0;--dark: #2b2d42;--light: #f8f9fa;--danger: #e63946;--gray: #6c757d;--card-shadow: 0 4px 20px rgba(0, 0, 0, 0.1);}* {margin: 0;padding: 0;box-sizing: border-box;}body {font-family: 'Roboto', sans-serif;background: linear-gradient(135deg, #f5f7fa 0%, #e4edf5 100%);color: #333;line-height: 1.6;min-height: 100vh;padding: 20px;}.container {max-width: 1400px;margin: 0 auto;}header {text-align: center;padding: 40px 0;background: linear-gradient(120deg, var(--primary), var(--secondary));border-radius: 15px;color: white;margin-bottom: 30px;box-shadow: var(--card-shadow);}header h1 {font-size: 2.8rem;margin-bottom: 15px;}header p {font-size: 1.2rem;max-width: 800px;margin: 0 auto;opacity: 0.9;}.logo {font-size: 3rem;margin-bottom: 20px;display: flex;justify-content: center;gap: 30px;}.logo i {background: rgba(255, 255, 255, 0.2);width: 80px;height: 80px;display: flex;align-items: center;justify-content: center;border-radius: 50%;}.content {display: flex;gap: 30px;margin-bottom: 40px;}.tutorial {flex: 3;background: white;border-radius: 15px;padding: 30px;box-shadow: var(--card-shadow);}.demo {flex: 2;background: white;border-radius: 15px;padding: 30px;box-shadow: var(--card-shadow);display: flex;flex-direction: column;}h2 {color: var(--secondary);margin-bottom: 25px;padding-bottom: 10px;border-bottom: 2px solid var(--success);font-size: 1.8rem;}h3 {color: var(--primary);margin: 25px 0 15px;font-size: 1.4rem;}.step {background: #f8f9ff;padding: 20px;border-radius: 10px;margin-bottom: 25px;border-left: 4px solid var(--primary);}.step-number {display: inline-block;background: var(--primary);color: white;width: 32px;height: 32px;text-align: center;line-height: 32px;border-radius: 50%;margin-right: 10px;font-weight: bold;}pre {background: #2b2d42;color: #f8f9fa;padding: 20px;border-radius: 8px;overflow-x: auto;margin: 20px 0;font-family: 'Source Code Pro', monospace;font-size: 0.95rem;}code {font-family: 'Source Code Pro', monospace;}.code-header {display: flex;justify-content: space-between;align-items: center;background: #1a1c2e;padding: 10px 15px;border-radius: 8px 8px 0 0;margin-top: 20px;color: white;}.code-header span {font-size: 0.9rem;}.copy-btn {background: var(--success);border: none;color: white;padding: 5px 10px;border-radius: 4px;cursor: pointer;font-size: 0.85rem;transition: all 0.3s;}.copy-btn:hover {background: #3ab7d8;}.demo-controls {display: flex;gap: 15px;margin-bottom: 20px;}.url-input {flex: 1;padding: 12px 15px;border: 2px solid #ddd;border-radius: 8px;font-size: 1rem;transition: border 0.3s;}.url-input:focus {border-color: var(--primary);outline: none;}.run-btn {background: var(--primary);color: white;border: none;padding: 12px 25px;border-radius: 8px;cursor: pointer;font-weight: 500;transition: all 0.3s;display: flex;align-items: center;gap: 8px;}.run-btn:hover {background: var(--secondary);transform: translateY(-2px);}.results {flex: 1;background: #f8f9ff;border-radius: 10px;padding: 20px;overflow-y: auto;border: 1px solid #e9ecef;}.result-item {padding: 15px;margin-bottom: 15px;background: white;border-radius: 8px;box-shadow: 0 2px 5px rgba(0,0,0,0.05);}.result-title {color: var(--primary);font-weight: 500;margin-bottom: 8px;}.result-link {color: var(--success);font-size: 0.9rem;display: block;margin-bottom: 10px;overflow: hidden;text-overflow: ellipsis;white-space: nowrap;}.result-snippet {color: var(--gray);font-size: 0.95rem;}.features {display: grid;grid-template-columns: repeat(auto-fit, minmax(300px, 1fr));gap: 25px;margin-top: 40px;}.feature-card {background: white;border-radius: 15px;padding: 25px;box-shadow: var(--card-shadow);transition: transform 0.3s, box-shadow 0.3s;border-top: 4px solid var(--success);}.feature-card:hover {transform: translateY(-10px);box-shadow: 0 12px 30px rgba(0, 0, 0, 0.15);}.feature-icon {font-size: 2.5rem;color: var(--primary);margin-bottom: 20px;}.feature-card h3 {margin: 0 0 15px;font-size: 1.4rem;}.browser-window {background: #2b2d42;border-radius: 10px;overflow: hidden;margin: 25px 0;box-shadow: 0 10px 30px rgba(0,0,0,0.2);}.browser-header {height: 40px;background: #3a3d5d;display: flex;align-items: center;padding: 0 15px;}.browser-dot {width: 12px;height: 12px;border-radius: 50%;margin-right: 8px;}.dot-red { background: #ff5f56; }.dot-yellow { background: #ffbd2e; }.dot-green { background: #27c93f; }.browser-content {height: 300px;background: white;padding: 20px;overflow-y: auto;}.browser-content h4 {color: var(--primary);margin-bottom: 15px;}.browser-content ul {padding-left: 25px;}.browser-content li {margin-bottom: 10px;}.tip-box {background: #e3f2fd;border-left: 4px solid var(--primary);padding: 15px 20px;border-radius: 0 8px 8px 0;margin: 25px 0;}.tip-title {font-weight: bold;color: var(--primary);margin-bottom: 8px;display: flex;align-items: center;gap: 8px;}.footer {text-align: center;padding: 30px 0;color: var(--gray);font-size: 0.95rem;margin-top: 20px;}@media (max-width: 1000px) {.content {flex-direction: column;}.features {grid-template-columns: 1fr;}}</style>
</head>
<body><div class="container"><header><div class="logo"><i class="fab fa-node-js"></i><i class="fab fa-js"></i><i class="fas fa-spider"></i></div><h1>Koa + Puppeteer 爬虫教程</h1><p>学习如何使用Koa.js框架和Puppeteer库构建强大的网络爬虫</p></header><div class="content"><div class="tutorial"><h2>Koa + Puppeteer 爬虫指南</h2><div class="step"><h3><span class="step-number">1</span> Puppeteer 简介</h3><p>Puppeteer 是一个 Node.js 库，它提供了一个高级 API 来通过 DevTools 协议控制 Chromium 或 Chrome。Puppeteer 默认以 headless 模式运行，但是可以配置为有界面模式运行。</p><p>主要功能：</p><ul><li>生成页面截图和PDF</li><li>爬取单页应用(SPA)并生成预渲染内容</li><li>自动化表单提交、UI测试、键盘输入等</li><li>创建最新的自动化测试环境</li></ul></div><div class="step"><h3><span class="step-number">2</span> Koa.js 简介</h3><p>Koa 是由 Express 原班人马打造的下一代 Node.js Web 框架，旨在为 Web 应用和 API 提供更小、更富有表现力、更健壮的基石。</p><p>主要特点：</p><ul><li>轻量级，无捆绑任何中间件</li><li>使用 async/await 语法，优雅地处理异步</li><li>错误处理更友好</li><li>核心代码简洁，易于扩展</li></ul></div><div class="step"><h3><span class="step-number">3</span> 项目初始化</h3><p>创建项目并安装所需依赖：</p><div class="code-header"><span>Terminal</span><button class="copy-btn">复制</button></div><pre><code># 创建项目目录
mkdir koa-puppeteer-crawler
cd koa-puppeteer-crawler# 初始化项目
npm init -y# 安装依赖
npm install koa @koa/router puppeteer</code></pre></div><div class="step"><h3><span class="step-number">4</span> 创建基本爬虫服务</h3><p>创建 <code>index.js</code> 文件，设置 Koa 服务器和爬虫路由：</p><div class="code-header"><span>index.js</span><button class="copy-btn">复制</button></div><pre><code>const Koa = require('koa');
const Router = require('@koa/router');
const puppeteer = require('puppeteer');const app = new Koa();
const router = new Router();// 爬虫路由
router.get('/crawl', async (ctx) => {// 从查询参数获取URLconst url = ctx.query.url || 'https://example.com';// 启动浏览器const browser = await puppeteer.launch({headless: true,args: ['--no-sandbox', '--disable-setuid-sandbox']});try {const page = await browser.newPage();await page.goto(url, { waitUntil: 'networkidle2', timeout: 30000 });// 获取页面数据const pageData = await page.evaluate(() => {return {title: document.title,content: document.body.innerText.substring(0, 1000) + '...',links: Array.from(document.querySelectorAll('a')).map(a => a.href)};});ctx.body = {success: true,data: pageData};} catch (error) {ctx.status = 500;ctx.body = {success: false,message: error.message};} finally {await browser.close();}
});app.use(router.routes());
app.use(router.allowedMethods());const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {console.log(`Server running on http://localhost:${PORT}`);
});</code></pre></div><div class="step"><h3><span class="step-number">5</span> 运行爬虫服务</h3><p>启动服务器：</p><div class="code-header"><span>Terminal</span><button class="copy-btn">复制</button></div><pre><code>node index.js</code></pre><p>访问爬虫接口：</p><pre><code>http://localhost:3000/crawl?url=https://example.com</code></pre></div><div class="step"><h3><span class="step-number">6</span> 高级爬虫技巧</h3><p>处理动态加载内容：</p><pre><code>// 等待特定元素出现
await page.waitForSelector('.results-container', { timeout: 5000 });// 滚动页面加载更多内容
await page.evaluate(() => {window.scrollBy(0, window.innerHeight);
});// 点击"加载更多"按钮
await page.click('.load-more-button');</code></pre><p>处理登录认证：</p><pre><code>// 输入用户名和密码
await page.type('#username', 'myuser');
await page.type('#password', 'mypassword');// 提交表单
await page.click('#login-button');
await page.waitForNavigation();</code></pre></div><div class="tip-box"><div class="tip-title"><i class="fas fa-lightbulb"></i><span>最佳实践建议</span></div><ul><li>使用 <code>page.setUserAgent()</code> 设置合理的用户代理</li><li>使用 <code>page.setViewport()</code> 设置视口大小</li><li>添加请求延迟避免被封禁</li><li>使用代理IP处理高频请求</li><li>合理使用 <code>waitForSelector</code> 和 <code>waitForNavigation</code></li><li>使用try-catch处理异常</li></ul></div></div><div class="demo"><h2>爬虫演示</h2><div class="browser-window"><div class="browser-header"><div class="browser-dot dot-red"></div><div class="browser-dot dot-yellow"></div><div class="browser-dot dot-green"></div></div><div class="browser-content"><h4>Puppeteer 爬虫功能演示</h4><p>在右侧输入URL，点击"运行爬虫"查看效果：</p><ul><li>爬取页面标题和内容</li><li>提取所有链接</li><li>处理JavaScript渲染的页面</li><li>模拟用户交互</li></ul><p>示例URL：</p><ul><li>https://example.com</li><li>https://news.ycombinator.com</li><li>https://quotes.toscrape.com</li></ul></div></div><div class="demo-controls"><input type="text" class="url-input" id="crawlUrl" placeholder="输入要爬取的URL (例如: https://example.com)" value="https://example.com"><button class="run-btn" id="runCrawler"><i class="fas fa-play"></i><span>运行爬虫</span></button></div><div class="results" id="results"><p>爬取结果将显示在这里...</p></div></div></div><div class="features"><div class="feature-card"><div class="feature-icon"><i class="fas fa-bolt"></i></div><h3>高效灵活</h3><p>Koa的轻量级架构结合Puppeteer的强大浏览器控制能力，可以构建高效灵活的爬虫系统，处理各种复杂的网站结构。</p></div><div class="feature-card"><div class="feature-icon"><i class="fas fa-laptop-code"></i></div><h3>JavaScript渲染支持</h3><p>Puppeteer可以完整渲染JavaScript生成的内容，轻松爬取单页应用(SPA)和动态加载的网站，这是传统爬虫难以实现的。</p></div><div class="feature-card"><div class="feature-icon"><i class="fas fa-shield-alt"></i></div><h3>自动化测试</h3><p>除了爬虫功能，Puppeteer还可用于自动化测试、性能监控和UI测试，为项目提供全方位的自动化支持。</p></div></div><div class="footer"><p>© 2023 Koa + Puppeteer 爬虫教程 | 本教程仅用于学习目的，请遵守目标网站的robots.txt和法律法规</p></div></div><script>// 模拟爬虫结果document.getElementById('runCrawler').addEventListener('click', function() {const urlInput = document.getElementById('crawlUrl').value;const resultsDiv = document.getElementById('results');if (!urlInput) {alert('请输入URL');return;}// 显示加载状态resultsDiv.innerHTML = '<p class="result-item"><i class="fas fa-spinner fa-spin"></i> 正在爬取数据，请稍候...</p>';// 模拟网络请求延迟setTimeout(() => {// 模拟爬取结果const mockData = {title: "示例域名 | " + new URL(urlInput).hostname,content: "这是一个示例网站。这段文本是从爬虫返回的模拟内容。在实际应用中，这里将显示从目标网站抓取的真实内容。",links: ["https://" + new URL(urlInput).hostname + "/about","https://" + new URL(urlInput).hostname + "/products","https://" + new URL(urlInput).hostname + "/contact"]};// 显示结果let html = `<div class="result-item"><div class="result-title">${mockData.title}</div><div class="result-snippet">${mockData.content}</div></div><div class="result-item"><div class="result-title">提取到的链接 (${mockData.links.length})</div>`;mockData.links.forEach(link => {html += `<div class="result-link">${link}</div>`;});html += `</div>`;resultsDiv.innerHTML = html;}, 2000);});// 复制代码功能document.querySelectorAll('.copy-btn').forEach(button => {button.addEventListener('click', function() {const codeBlock = this.parentElement.nextElementSibling;const textToCopy = codeBlock.textContent;navigator.clipboard.writeText(textToCopy).then(() => {const originalText = this.textContent;this.textContent = '已复制!';this.style.background = '#27c93f';setTimeout(() => {this.textContent = originalText;this.style.background = '';}, 1500);}).catch(err => {console.error('复制失败:', err);alert('复制失败，请手动复制代码');});});});</script>
</body>
</html>