CLI tool to fetch a webpage's contents after it has finished loading

Overview

webpage-dl

Fetch a webpage's contents via CLI after it has finished loading.

Scraping HTML from webpages can't always be done via curl as it is often useful to let the page's JavaScript play out first. webpage-dl launches a WKWebView and only reads the source of the page once loading has completed.

webpage-dl even allows you to provide your own JavaScript expression, so scraping specific HTML nodes has never been easier.

Installation

  1. Clone this repo
  2. Run make install (requires Xcode 13+)

Usage

USAGE: webpage-dl [--width <width>] [--height <height>] [--timeout <timeout>] [--delay <delay>] [--expression <expression>] <url>

ARGUMENTS:
  <url>                   The URL of the webpage to download.

OPTIONS:
  -w, --width <width>     The width of the simulated web page. (default: 500)
  -h, --height <height>   The height of the simulated web page. (default: 500)
  --timeout <timeout>     The maximum number of seconds the program should run
                          for before exiting (in the event of slow loading).
                          (default: 20.0)
  -d, --delay <delay>     An additional time to wait before executing
                          'expression'.
  --expression <expression>
                          The JavaScript expression to run. The output from
                          this expression with be printed to stdout before the
                          program exits. (default:
                          document.documentElement.outerHTML.toString())
  -h, --help              Show help information.
You might also like...
Headline News Widget for Better Touch Tool. You can display the articles fetched by rss.

BTTPluginHeadLineNews This is a headline news widget plugin for BTT(Better Touch Tool) You can display the articles fetched by rss. (Pock version is h

iAppStroe is an Apple Store tool app written in SwiftUI.
iAppStroe is an Apple Store tool app written in SwiftUI.

iAppStore 简介 iAppStroe 是一款使用 SwiftUI 打造的苹果商店工具类 App。 1、提供苹果实时榜单查询,包含 iOS 和 iPad 的热门免费榜、热门付费榜、畅销榜,还有新上架榜、新上架免费榜、新上架付费榜等。 2、提供查询 app 详细页面内容、搜索 app、订阅 ap

Watchbuild - Get a notification once your iTunes Connect build is finished processing
Watchbuild - Get a notification once your iTunes Connect build is finished processing

fastlane WatchBuild Get a notification once your App Store Connect build is finished processing When you upload a new binary from Xcode to App Store C

This tool leverages the research linked below to understand the contents of TCC.db.
This tool leverages the research linked below to understand the contents of TCC.db.

SwiftParseTCC Description This tool leverages the research linked below to understand the contents of TCC.db. Uses "Full Disk Access" permissions to r

Erik is an headless browser based on WebKit. An headless browser allow to run functional tests, to access and manipulate webpages using javascript.
Erik is an headless browser based on WebKit. An headless browser allow to run functional tests, to access and manipulate webpages using javascript.

Erik Erik is a headless browser based on WebKit and HTML parser Kanna. An headless browser allow to run functional tests, to access and manipulate web

An NSURL extension for showing preview info of webpages
An NSURL extension for showing preview info of webpages

URLPreview An NSURL extension for showing preview info of webpages. You may want to use it if you want to mimick Facebook app's behavior when you post

Erik is an headless browser based on WebKit. An headless browser allow to run functional tests, to access and manipulate webpages using javascript.
Erik is an headless browser based on WebKit. An headless browser allow to run functional tests, to access and manipulate webpages using javascript.

Erik Erik is a headless browser based on WebKit and HTML parser Kanna. An headless browser allow to run functional tests, to access and manipulate web

SwiftyOpenGraph - A swift library that retrieves structured open graph data from webpages.

SwiftyOpenGraph Usage Initialization Base Properties Types Installation License Usage Initialization You use SwiftyOpenGraph by initializing OpenGraph

☠️ An elegant way to show users that something is happening and also prepare them to which contents they are awaiting
☠️ An elegant way to show users that something is happening and also prepare them to which contents they are awaiting

Features • Guides • Installation • Usage • Miscellaneous • Contributing 🌎 README is available in other languages: 🇪🇸 . 🇨🇳 . 🇧🇷 . 🇰🇷 . 🇫🇷 To

Infinite paging controller, scrolling through contents and title bar scrolls with a delay
Infinite paging controller, scrolling through contents and title bar scrolls with a delay

PageController PageController is infinite paging controller, scrolling through contents and title bar scrolls with a delay. Then it provide user inter

☠️ An elegant way to show users that something is happening and also prepare them to which contents they are awaiting
☠️ An elegant way to show users that something is happening and also prepare them to which contents they are awaiting

Features • Guides • Installation • Usage • Miscellaneous • Contributing 🌎 README is available in other languages: 🇪🇸 . 🇨🇳 . 🇧🇷 . 🇰🇷 . 🇫🇷 To

Infinite paging controller, scrolling through contents and title bar scrolls with a delay
Infinite paging controller, scrolling through contents and title bar scrolls with a delay

PageController PageController is infinite paging controller, scrolling through contents and title bar scrolls with a delay. Then it provide user inter

☠️ An elegant way to show users that something is happening and also prepare them to which contents they are awaiting
☠️ An elegant way to show users that something is happening and also prepare them to which contents they are awaiting

Features • Guides • Installation • Usage • Miscellaneous • Contributing 🌎 README is available in other languages: 🇪🇸 . 🇨🇳 . 🇧🇷 . 🇰🇷 . 🇫🇷 To

DGDynamicHeightTableView - UITableView that supports auto calculated height by contents size
DGDynamicHeightTableView - UITableView that supports auto calculated height by contents size

DGDynamicHeightTableView UITableView that supports auto calculated height by con

A Simple iOS QR code scanner that allows users to enable their camera and local Photo Library accesses to scan the contents of the input QR codes.

iOS QR Code Scanner A Simple iOS QR code scanner using Swift UI. Jump to: ContentView.swift screenshot 1.1.5 NOTE: be aware of the new horizontal line

CLI tool for macOS that transcribes speech from the microphone using Apple’s speech recognition API, SFSpeechRecognizer. (help.)
CLI tool for macOS that transcribes speech from the microphone using Apple’s speech recognition API, SFSpeechRecognizer. (help.)

CLI tool for macOS that uses SFSpeechRecognizer to transcribe speech from the microphone. The recognition result will be written to the standard output as JSON string.

SuccessorCLI A CLI tool to restore iOS Devices on versions they're already on, based on the original Succession

A tool to restore iOS Devices on versions they're already on, based on the original Succession

A CLI tool for localization resource management on Xcode. Built with Google Translator.
A CLI tool for localization resource management on Xcode. Built with Google Translator.

Supported by Jetbrains Open Source License Program Automatically translate and synchronize '.strings' files from the defined base language The basic c

An extremely simple CLI tool that was created to diagnose and further understand an issue in DriverKit causing kIOHIDOptionsTypeSeizeDevice to behave incorrectly when used in DriverKit system extensions.

IOKitHIDKeyboardTester This tool is NOT useful to, or intended for general users. IOKitHIDKeyboardTester is an extremely simple (one-file!) CLI tool t

Owner
Max Chuquimia
Principal iOS Engineer
Max Chuquimia
A companion app for after-cancer treatment patients

Amber - your smart companion We have developed the iOS app Amber - a companion for after-cancer treatment patients. It connects patients with their do

Alex Studnička 1 Oct 24, 2021
A SwiftUI app that fetch images from The Dog API and The Cat API

Cat-Or-Dog-App A SwiftUI app that fetch images from The Dog API and The Cat API, then classify the species of dogs or cats with confidence level. The

null 0 Oct 19, 2021
Weather - Use Open weather APIs to fetch live weather data

Weather ??️ Use Open weather APIs to fetch live weather data Use Core Location p

Vicky Lee 2 Jun 3, 2022
Aplikasi iOS Simulasi CRUD Stok Barang dengan SwiftUI, Firebase CLI & Firestore

iStockery Aplikasi iStockery adalah aplikasi stok inventory berbasis iOS yang dibuat menggunakan Firebase (Firestore) secara Local dengan fitur CRUD d

DK 7 Aug 1, 2022
Implemented MVVM-C (Coordinator) architecture pattern for the project. Which is satisfying SOLID principles altogether. Protocol oriented development has been followed.

BreakingBad BreakingBad API doc Implemented MVVM-C (Coordinator) architecture pattern for the project. Which is satisfying SOLID principples altogethe

Dhruvik Rao 2 Mar 10, 2022
The iOS application has the Pokémon list. In the app

The iOS application has the Pokémon list. In the app, you can search for Pokémon by name, edit the Pokémon list (mark favorites or delete unwanted ones ?? ), and re-update the Pokémon list from the network.

Volodya 0 Nov 16, 2021
Want to know the current weather around the globe? Clima has your back!

Clima (a weather app) Dreaming about going on vacation somewhere? Use Clima to find real time weather from around the world or use your GPS to get loc

null 0 Dec 27, 2021
This app has always been open source! It began with the Big Brain Hackathon and now during Hacktoberfest

Brain Marks This app has always been open source! It began with the Big Brain Hackathon and now during Hacktoberfest. It is available for free on the

Mikaela Caron 79 Jan 4, 2023
SafeTimer es una app para iPhone y Apple Watch que te ayuda a controlar el tiempo que has usado una mascarilla.

SafeTimer es una app para iPhone y Apple Watch que te ayuda a controlar el tiempo que has usado una mascarilla.

Chubby Apps 13 Sep 20, 2022
Pegase is a beautifully easy tool to keep track of your financial life on all your macOS

Pegase ?? Features ?? Documentation Personal account software Pegase is a beautifully easy tool to keep track of your financial life on all your macOS

null 2 Oct 12, 2021