Headers

forward_headersbooleanFalse

You might need to forward specific headers to the website that you want to scrape.

In order to forward headers, you must set forward_headers to true and then pass your custom headers.

You must then prefix the headers to forward to the website with Spb- (for FoxScrape).

This prefix will be trimmed by FoxScrape and headers will be forwarded to the target web page.

Example:

If you want to send the header Accept-Language: En-US, add the header: Spb-Accept-Language: En-US and the parameter forward_headers=true to the request sent to the FoxScrape API.

Note:

If you are using the FoxScrape Python or Node library, no need to prefix headers with Spb- or to use forward_headers=True.

Please note that in order to make your request look like real ones, FoxScrape adds several headers to all API requests. Use forward_headers_pure=True to avoid this behavior.

1
curl "https://www.foxscrape.com/api/v1?api_key=YOUR_API_KEY&url=http%3A%2F%2Fhttpbin.org%2Fheaders%3Fjson&forward_headers=true" \
2
  -H "Spb-Accept-Language:En-US"

Here we are scraping httpbin.org/headers?json, a page that demonstrates and displays the headers it received.

The following is the response from the above code. Note the Accept-Language header in the response.

1
{
2
  "headers": {
3
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
4
    "Accept-Encoding": "gzip, deflate",
5
    "Accept-Language": "fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7",
6
    "Host": "httpbin.org",
7
    "Accept-Language": "En-US", // <-- Your header
8
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36"
9
  }
10
}

forward_headers_purebooleanFalse

If you want to forward specific headers to the website that you want to scrape, and don't need FoxScrape to add any headers to your request, you should use forward_headers_pure=True.

You must then prefix the headers to forward to the website with Spb- (for FoxScrape).

This prefix will be trimmed by FoxScrape and headers will be forwarded to the target web page.

Example:

If you want to only send the header Accept-Language: En-US, add the header: Spb-Accept-Language: En-US and the parameter forward_headers_pure=true to the request sent to the FoxScrape API.

Note:

If you are using the FoxScrape Python or Node library, no need to prefix headers with Spb- or to use forward_headers_pure=True.

1
curl "https://www.foxscrape.com/api/v1?api_key=YOUR_API_KEY&url=http%3A%2F%2Fhttpbin.org%2Fheaders%3Fjson&forward_headers_pure=true" \
2
  -H "Spb-Accept-Language:En-US"

Here we are scraping httpbin.org/headers?json, a page that demonstrates and displays the headers it received.

The following is the response from the above code. Note the Accept-Language header in the response.

1
{
2
  "headers": {
3
    "Accept-Encoding": "gzip, deflate", // Technical header sent with all requests
4
    "Host": "httpbin.org", // Technical header sent with all requests
5
    "Accept-Language": "En-US" // <-- Your headers, and only your headers
6
  }
7
}

This parameter is only useful when using render_js=False.