Scrape a page into rows
Takes every item on a page as a row (fields, numbers, dates, pagination), from a tab you never look at or from a plain fetch.
Scrape selector into into
Details #
selector is the repeating item (.result, role=listitem, or selection for whatever you have selected). Leave it empty and Switchboard picks the page's main list: the biggest group of siblings with the same shape, the way reader mode picks the article. Each row is then {text, href?, src?}. A table's rows are read by its header names.
fields makes the columns: {name: {selector, what, attribute, regex, as, default, all}}, each relative to the item. what is text (the default), html, attribute, value, exists or count. regex keeps its first capture group. all gives a list for a field that repeats inside the item. as turns the text into number (currency signs, thousands separators, 1.2k, a minus), date (ISO, Oct 25, 2 hours ago; a date with no time is YYYY-MM-DD), url (absolute), boolean or list. A value that won't turn into one is null, and the step's note says which row and why.
paginate keeps going: {next: selector} clicks the pager's Next (or follows its link), {scroll: true} scrolls an infinite list. It stops at no next page, no new rows, max pages (20 by default, 200 at most), or when until matches ({field, op, value} on a row). key names the field that identifies a row, so rows seen twice are dropped.
Where the page comes from. With no url the step reads a tab (the front one, or tab): signed in, scripts run, what you see. With url (one address or a list, one visit each) it fetches the page the way a browser would, with via: fetch, needing no browser at all; via: tab opens it in a background tab instead; via: auto tries the fetch first and, if the page needs its scripts to show rows, tries a tab and says so. html runs the recipe over a string, such as {{ res.body }} from an http step.
The result is {rows, count, pages, source: {url, title, at}}. {{ res.rows }} is a list of dictionaries for repeat, Save rows to a file or the clipboard and return. A step that ends in return {{ res.rows }} is a tool an agent can call.
When a page that gave rows last time gives none, the site changed: the step's note says "fields stopped matching" and the trace keeps the page's HTML (1 MB, passwords and tokens removed).
Parameters #
| Parameter | Type | Required | Values | What it does |
|---|---|---|---|---|
browser | one of | safari chrome front | ||
fields | record | The columns. | ||
html | text | |||
impersonate | one of | auto chrome safari none | ||
into | text | The variable that receives the result. | ||
key | text | , limit: Identify rows; at most 10,000 rows. | ||
limit | number | |||
pace | number | |||
paginate | record | max next pace scroll until | next, scroll, max, until, pace. | |
sample | any value | |||
selector | text | The repeating item. Empty: the page's main list. | ||
signIn | text | , impersonate, pace: For a fetch: a saved sign-in, the browser it looks like, seconds between requests to one site. | ||
tab | any value | , browser, timeout. | ||
timeout | number | |||
url | any value | , html, via: Where the page comes from. | ||
via | one of | tab fetch auto |
Every step also accepts label, onFailure, timeout and dryRunSafe.
Examples #
- Press ⌃⌥R
- Scrape “.result” into res
- Save rows to “results.csv”
Gotchas #
Scraping never drives a hidden automation browser. A page that needs scripts is read in your own browser, in a background tab that shows in the tab strip while it is open.
A fetch looks like a browser on the wire and can be signed in. It solves no challenge: a site that answers with one fails the step with
challenge, andvia: tabis the answer.paceis at least a quarter second per site, shared by every scrape, visit and page watch.A dry run doesn't read any page.
See also #
Open a page in a background tab, Snapshot the page for an agent, A web page changes.