Web Scraping Ethics Guide

1. Respect robots.txt and Terms of Service
   - Check robots.txt before scraping any site
   - Review the site's ToS for explicit prohibitions
   - When in doubt, contact the site operator

2. Minimize server impact
   - Use rate limiting (1-2 requests per second maximum)
   - Scrape during off-peak hours when possible
   - Cache responses to avoid redundant requests
   - Use conditional requests (If-Modified-Since headers)

3. Collect only what you need
   - Define your data requirements before scraping
   - Do not scrape personal data without ethical justification
   - Anonymize or pseudonymize personal information

4. Attribution and transparency
   - Set a descriptive User-Agent header with contact info
   - Be prepared to identify yourself if contacted
   - Credit data sources in publications

5. Institutional compliance
   - Check if your IRB/ethics board requires approval for web data
   - Follow your institution's acceptable use policy
   - Consider data protection regulations (GDPR, CCPA)

import urllib.request
import urllib.robotparser


def check_robots_txt(base_url: str, target_path: str,
                     user_agent: str = "*") -> dict:
    """
    Check if a URL is allowed by robots.txt.

    Args:
        base_url: The website's base URL (e.g., 'https://example.com')
        target_path: The path you want to scrape (e.g., '/data/papers')
        user_agent: Your bot's user agent string
    """
    robots_url = f"{base_url}/robots.txt"

    rp = urllib.robotparser.RobotFileParser()
    rp.set_url(robots_url)

    try:
        rp.read()
    except Exception as e:
        return {
            "robots_txt_found": False,
            "error": str(e),
            "recommendation": "Proceed with caution; use conservative rate limiting"
        }

    full_url = f"{base_url}{target_path}"
    allowed = rp.can_fetch(user_agent, full_url)
    crawl_delay = rp.crawl_delay(user_agent)

    return {
        "robots_txt_found": True,
        "url_checked": full_url,
        "allowed": allowed,
        "crawl_delay": crawl_delay or "Not specified (use 1-2 seconds)",
        "recommendation": (
            "Proceed with specified crawl delay"
            if allowed
            else "Do NOT scrape this path -- it is disallowed"
        )
    }

import time
import urllib.request


def scrape_with_rate_limit(urls: list[str],
                            delay: float = 1.0,
                            user_agent: str = None) -> list[dict]:
    """
    Scrape a list of URLs with rate limiting and proper headers.

    Args:
        urls: List of URLs to fetch
        delay: Seconds to wait between requests
        user_agent: Custom user agent string
    """
    if user_agent is None:
        user_agent = (
            "ResearchBot/1.0 (Academic research; "
            "contact: researcher@university.edu)"
        )

    results = []

    for i, url in enumerate(urls):
        try:
            req = urllib.request.Request(url, headers={
                "User-Agent": user_agent,
                "Accept": "text/html",
            })

            response = urllib.request.urlopen(req, timeout=30)
            content = response.read().decode("utf-8", errors="replace")

            results.append({
                "url": url,
                "status": response.status,
                "content_length": len(content),
                "success": True
            })

        except Exception as e:
            results.append({
                "url": url,
                "error": str(e),
                "success": False
            })

        # Rate limiting
        if i < len(urls) - 1:
            time.sleep(delay)

    return results

United States:
  - CFAA (Computer Fraud and Abuse Act): Unauthorized access is illegal
  - hiQ v. LinkedIn (2022): Scraping public data is generally permissible
  - Key question: Is the data publicly accessible without authentication?

European Union:
  - GDPR: Personal data requires legal basis for processing
  - Database Directive: Protects substantial investment in databases
  - Text and Data Mining exception (DSM Directive, Art. 3-4):
    Research organizations can mine lawfully accessible content

General guidance:
  - Public data is more defensible than data behind login walls
  - Scraping that circumvents technical measures is riskier
  - Academic fair use / research exceptions vary by jurisdiction
  - When in doubt, consult your institution's legal counsel

def assess_irb_requirements(data_type: str,
                             contains_pii: bool) -> dict:
    """
    Assess whether web scraping requires IRB review.

    Args:
        data_type: Type of data being collected
        contains_pii: Whether data includes personally identifiable information
    """
    if contains_pii:
        return {
            "irb_required": "Likely yes",
            "rationale": (
                "Data that identifies or can re-identify individuals "
                "generally requires ethics review, even if publicly posted."
            ),
            "steps": [
                "Submit IRB protocol describing data collection",
                "Justify why PII is necessary for the research",
                "Describe de-identification procedures",
                "Explain data storage and security measures",
                "Plan for data destruction after the study"
            ]
        }

    return {
        "irb_required": "Possibly exempt, but check with your IRB",
        "rationale": (
            "Non-human-subjects data (e.g., product prices, publication "
            "metadata) typically does not require IRB review, but policies "
            "vary by institution."
        ),
        "recommendation": "Submit an exemption request to be safe"
    }

Before scraping:
  [ ] Check robots.txt
  [ ] Review Terms of Service
  [ ] Consider whether an API exists (prefer API over scraping)
  [ ] Assess IRB requirements
  [ ] Define minimal data needed

During scraping:
  [ ] Set descriptive User-Agent with contact email
  [ ] Implement rate limiting (min 1 second between requests)
  [ ] Handle errors gracefully (do not retry aggressively)
  [ ] Log all requests for reproducibility
  [ ] Cache responses to avoid re-fetching

After scraping:
  [ ] Anonymize personal data if present
  [ ] Store data securely
  [ ] Document the scraping methodology for your paper
  [ ] Credit the data source
  [ ] Consider whether the scraped data can be shared
      (check copyright and ToS)