FoxScrape
|
Docs

JavaScript Scenarios

Interact with the webpage you want to scrape.

💡 Important:

This page explains how to use a specific feature of our main web scraping API! If you are not yet familiar with FoxScrape web scraping API, you can read the documentation here.

Basic usage

If you want to interact with pages you want to scrape before we return you the HTML you can add JavaScript scenario to your API call.

For example, if you wish to click on a button, you will need to use this scenario.

JSON
1
{
2
    "instructions": [
3
       {"click": "#buttonId"}
4
    ]
5
}

You can use both CSS and XPath selectors in all instructions. All selectors beginning with / will be treated as XPath selectors. All other selectors will be treated as CSS selectors.

And so our scraper will scrape the webpage, click on the button #buttonId and then return you the HTML of the page.

Important: JavaScript scenarios are JSON formatted, and in order to pass them to a GET request, you need to stringify them.

Node.js

JAVASCRIPT
1
// Using Axios
2
const axios = require('axios');
3


4
axios.get('https://www.foxscrape.com/api/v1', {
5
    params: {
6
        'api_key': 'YOUR_API_KEY',
7
        'url': 'https://www.foxscrape.com/blog',
8
        'js_scenario': '{"instructions": [{ "click": "#buttonId" }]}',
9
    }
10
}).then(function (response) {
11
    // handle success
12
    console.log(response);
13
})

You can add multiple instructions to the scenario, they will get executed one by one on our end.

Important: We strongly advise you to use JS scenario with json_response set to true (learn more), as it will return you a detailed report of the scenario execution under the js_scenario_report key. Very useful for debugging for example.

Below is a quick overview of all the different instruction you can use.

JSON
1
{"evaluate": "console.log('foo')"} # Run custom JavaScript
2
{"click": "#button_id"} # Click on an element
3
{"wait": 1000} # Wait for a fixed duration in ms
4
{"wait_for": "#slow_div"} # Wait for an element to appear
5
{"wait_for_and_click": "#slow_div"} # Wait for an element to appear and then click on it
6
{"scroll_x": 1000} # Scroll the screen in the horizontal axis, in px
7
{"scroll_y": 1000} # Scroll the screen in the vertical axis, in px
8
{"fill": ["#input_1", "value_1"]} # Fill some input
9
{"evaluate": "console.log('toto')"} # Run custom JavaScript code
10
{"infinite_scroll": # Scroll the page until the end
11
    {
12
        "max_count": 0, # Maximum number of scroll, 0 for infinite
13
        "delay": 1000, # Delay between each scroll, in ms
14
        "end_click": {"selector": "#button_id"} # (optional) Click on a button when the end of the page is reached, usually a "load more" button
15
    }
16
}

Of course, you can choose to use them in the order you want, and you can use the same one multiple time in one scenario.

Here is an example of a scenario that wait for a button to appear, click on it and then scroll, wait a bit, and scroll again.

JSON
1
{
2
    "instructions": [
3
        {"wait_for_and_click": "#slow_button"},
4
        {"scroll_x": 1000},
5
        {"wait": 1000},
6
        {"scroll_x": 1000},
7
        {"wait": 1000}
8
    ]
9
}