Web Scraping Olostep

Olostep - Web Scraping & Crawling API

Setup

Read your credentials from ~/.gooseworks/credentials.json:

export GOOSEWORKS_API_KEY=$(python3 -c "import json;print(json.load(open('$HOME/.gooseworks/credentials.json'))['api_key'])")
export GOOSEWORKS_API_BASE=$(python3 -c "import json;print(json.load(open('$HOME/.gooseworks/credentials.json')).get('api_base','https://api.gooseworks.ai'))")

If ~/.gooseworks/credentials.json does not exist, tell the user to run: npx gooseworks login

All endpoints use Bearer auth: -H "Authorization: Bearer $GOOSEWORKS_API_KEY"

Powerful web scraping, crawling, and AI-powered content extraction.

Capabilities

• Create Scrape: Initiate a web page scrape
• Create Answer: The AI will perform actions like searching and browsing web pages to find the answer to the provided task
• Maps: This endpoint allows users to get all the urls on a certain website
• Start Crawl: Starts a new crawl
• Start Batch: Starts a new batch
• Batch Items: Retrieves the list of items processed for a batch
• Crawl Info: Fetches information about a specific crawl
• Crawl Pages: Fetches the list of pages for a specific crawl
• Get Answer: This endpoint retrieves a previously completed answer by its ID
• Get Scrape: Can be used to retrieve response for a scrape
• Batch Info: Retrieves the status and progress information about a batch
• Retrieve Content: Retrieve page content of processed batches and crawls urls

Usage

Create Scrape

Initiate a web page scrape

Parameters:

• url_to_scrape* (string) - The URL to start scraping from.
• wait_before_scraping (integer) - Time to wait in milliseconds before starting the scraping.
• formats (string[]) - Formats in which you want the content.
• remove_css_selectors (string) - Option to remove certain CSS selectors from the content. Optionally, you can also pass a JSON stringified array of specific selectors you want to remove. The CSS selectors removed when this option is set to default are ['nav','footer','script','style','noscript','svg',[role=alert],[role=banner],[role=dialog],[role=alertdialog],[role=region][aria-label*=skip i],[aria-modal=true]] Available options: default, none, array
• actions (object[]) - Actions to perform on the page before getting the content.
• country (string) - Residential country to load the request from. Supported values are: * US (United States) * CA (Canada) * IT (Italy) * IN (India) * GB (England) * JP (Japan) * MX (Mexico) * AU (Australia) * ID (Indonesia) * UA (UAE) * RU (Russia) * RANDOM Some operations, like scraping Google Search and Google News, support all countries.
• transformer (string) - Specify the HTML transformer to use, if any. Postlight's Mercury Parser library is used to remove ads and other unwanted content from the scraped content. Available options: postlight, none
• remove_images (boolean) - Option to remove images from the scraped content. Defaults to false.
• remove_class_names (string[]) - List of class names to remove from the content.
• parser (object) - When defining json as a format, you can use this parameter to specify the parser to use. Parsers are useful to extract structured content from web pages. Olostep has a few parsers built in for most common web pages, and you can also create your own parsers.
• llm_extract (object)
• links_on_page (object) - With this option, you can get all the links present on the page you scrape.
• screen_size (object) - Configuration for screen size. Preset dimensions are available through screen_type: desktop (1920x1080), mobile (414x896), or default (768x1024).
• metadata (object) - User-defined metadata. Not supported yet

curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \
  -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"api":"olostep","path":"/v1/scrapes","body":{"url_to_scrape":"https://example.com/page"}}'

Create Answer

The AI will perform actions like searching and browsing web pages to find the answer to the provided task. Execution time is 3-30s depending upon complexity. For longer tasks, use the agent endpoint instead.

Parameters:

• task* (string) - The task to be performed.
• json_format (object) - The desired output JSON object with empty values as a schema, or simply describe the data you want as a string.

curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \
  -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"api":"olostep","path":"/v1/answers","body":{"task":"What are the latest AI developments?"}}'

Maps

This endpoint allows users to get all the urls on a certain website. It can take up to 120 seconds for complex websites. For large websites, results are paginated using cursor-based pagination

Parameters:

• url* (string) - The URL of the website for which you want the links
• search_query (string) - An optional search query to sort the links by search relevance.
• top_n (number) - An optional number to limit to only top n links for a search query.
• include_subdomain (boolean) - Include subdomains of the given URL. true by default.
• include_urls (string[]) - URL path patterns to include using glob syntax. For example: /blog/** to only include blog URLs. Only URLs matching these patterns will be returned.
• exclude_urls (string[]) - URL path patterns to exclude using glob syntax. For example: /careers/**. Excluded URLs will supersede included URLs.
• cursor (string) - OPTIONAL: Pagination cursor from a previous response. When provided, returns the next set of URLs from where the previous request left off due to response size limit.

curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \
  -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"api":"olostep","path":"/v1/maps","body":{"url":"https://example.com"}}'