Modifying and sending the correct HTTP headers are required in almost every web scraping project.
For example, you may need to route through different User Agents, or change the
Content-Language header to get a different response.
The following list is to help you understand what the HTTP headers you are observing mean. How you use them within your scraper is up to you. Try and be creative, for example - often changing the Content-Type, or Accept headers will result in a different response - sometimes even changing HTML to json and making your life a lot easier!
!!! More info coming soon, the below list is by no means exhaustive!
||The content types accepted by the client - for example
||The character sets the client accepts|
||The encoding the client accepts|
||A list of accepted languages|
||Accepted time format|