Web Scraper

Project Structure

Organize your code into the following structure:

web-scraper/
├── scraper.py           # Main entry point
├── fetcher.py           # HTTP request handling
├── parser.py            # BeautifulSoup parsing logic
├── exporter.py          # CSV export functionality
├── config.py            # Configuration settings
├── utils.py             # Helper functions
├── data/
│   ├── books_YYYY-MM-DD.csv    # Scraped output (auto-generated)
│   └── categories.csv          # Category summary
├── tests/
│   ├── test_fetcher.py
│   ├── test_parser.py
│   └── test_exporter.py
├── requirements.txt     # Dependencies
└── README.md            # Project documentation

Fetcher Module (HTTP Requests)

Handle all HTTP operations:

• get_page(url): Fetch a single URL and return response
• get_pages(urls): Fetch multiple URLs with rate limiting
• Headers: Set User-Agent and Accept headers
• Timeouts: Implement request timeouts (10 seconds)
• Retries: Retry failed requests up to 3 times
• Rate limiting: Wait 1 second between requests

import requests
import time
from typing import Optional

class Fetcher:
    BASE_URL = "https://books.toscrape.com"
    
    def __init__(self, delay: float = 1.0):
        self.delay = delay
        self.session = requests.Session()
        self.session.headers.update({
            'User-Agent': 'BookScraper/1.0 (Student Project)',
            'Accept': 'text/html,application/xhtml+xml'
        })
    
    def get_page(self, url: str) -> Optional[str]:
        """Fetch a single page and return HTML content."""
        try:
            time.sleep(self.delay)  # Rate limiting
            response = self.session.get(url, timeout=10)
            response.raise_for_status()
            return response.text
        except requests.RequestException as e:
            print(f"Error fetching {url}: {e}")
            return None

Parser Module (BeautifulSoup)

Parse HTML and extract data:

• parse_book_list(html): Extract all books from a listing page
• parse_book_detail(html): Extract full details from a book page
• parse_categories(html): Extract all category links
• parse_pagination(html): Find next page link if exists
• clean_price(text): Convert "£51.77" to float 51.77
• clean_rating(class_name): Convert "star-rating Three" to 3

from bs4 import BeautifulSoup
from dataclasses import dataclass
from typing import List, Optional

@dataclass
class Book:
    title: str
    price: float
    rating: int
    availability: str
    category: str = ""
    upc: str = ""
    description: str = ""
    url: str = ""

class Parser:
    RATING_MAP = {
        'One': 1, 'Two': 2, 'Three': 3, 
        'Four': 4, 'Five': 5
    }
    
    def parse_book_list(self, html: str) -> List[Book]:
        """Extract all books from a listing page."""
        soup = BeautifulSoup(html, 'html.parser')
        books = []
        for article in soup.select('article.product_pod'):
            title = article.h3.a['title']
            price = self._clean_price(article.select_one('.price_color').text)
            rating = self._get_rating(article.select_one('.star-rating'))
            availability = article.select_one('.availability').text.strip()
            url = article.h3.a['href']
            books.append(Book(title, price, rating, availability, url=url))
        return books

Exporter Module (CSV)

Export data to CSV files:

• export_books(books, filepath): Export book list to CSV
• export_categories(categories, filepath): Export category summary
• Encoding: Use UTF-8 encoding for Unicode support
• Timestamps: Include date in filename
• Headers: Include column headers in first row

import csv
from datetime import datetime
from typing import List
from parser import Book

class Exporter:
    def export_books(self, books: List[Book], filepath: str = None) -> str:
        """Export books to CSV file."""
        if filepath is None:
            date_str = datetime.now().strftime('%Y-%m-%d')
            filepath = f"data/books_{date_str}.csv"
        
        with open(filepath, 'w', newline='', encoding='utf-8') as f:
            writer = csv.writer(f)
            writer.writerow(['title', 'price', 'rating', 'availability', 
                           'category', 'upc', 'description', 'url'])
            for book in books:
                writer.writerow([
                    book.title, book.price, book.rating, book.availability,
                    book.category, book.upc, book.description, book.url
                ])
        return filepath

Error Handling

Implement robust error handling:

• Network errors: Handle connection timeouts and failures
• HTTP errors: Handle 404, 500, and other status codes
• Parsing errors: Handle missing elements gracefully
• Logging: Log all errors with timestamps
• Recovery: Continue scraping even if some pages fail

CLI Interface

Command-line arguments:

• python scraper.py - Scrape all books
• python scraper.py --category Travel - Scrape specific category
• python scraper.py --pages 5 - Limit to first N pages
• python scraper.py --output data/my_books.csv - Custom output file
• python scraper.py --verbose - Show detailed progress