Common HTTP Headers (requests and responses)

Modifying and sending the correct HTTP headers are required in almost every web scraping project.

For example, you may need to route through different User Agents, or change the Content-Language header to get a different response.

The following list is to help you understand what the HTTP headers you are observing mean. How you use them within your scraper is up to you. Try and be creative, for example - often changing the Content-Type, or Accept headers will result in a different response - sometimes even changing HTML to json and making your life a lot easier!

!!! More info coming soon, the below list is by no means exhaustive!

HTTP Request Headers

Name Example Description
Accept Accept: json The content types accepted by the client - for example text/html
Accept-Charset Accept-Charset: utf-8 The character sets the client accepts
Accept-Encoding Accept-Encoding: gzip, deflate, br The encoding the client accepts
Accept-Language Accept-Language: en-AU A list of accepted languages
Accept-Datetime Accept-Datetime: Sat, 01 Jan 2019 21:21:00 GMT Accepted time format

HTTP Response Headers

Checkout our Web Scraping Service