Tools
/
Crawler
/
Crawler: RequestOptions
Jan. 17, 2022
Crawler: RequestOptions
Type:
object
Parameter syntax
requestOptions: {
proxy: 'http://proxy'
timeout: 10000
retries: 2
headers: {
...
}
}
About this parameter# A
Modify all crawler’s requests behavior.
The Crawler only accepts headers from our list of allowed headers.
The Cookie header will be replaced by the cookie fetched in login.
Examples# A
Copy
1
2
3
4
5
6
7
8
9
10
11
{
requestOptions: {
proxy: 'http://algolia:password@1.1.1.1:8080',
timeout: 10000,
retries: 2,
headers: {
Authorization: 'Bearer Aerehdf==',
Cookie: 'session=1234;',
}
}
}
Parameters # A
proxy
# |
type: string (URL)
Optional
Set a proxy for your crawler’s requests. |
timeout
# |
type: number
default: 30000
Optional
Set a timeout (in milliseconds) to your crawler’s requests. |
retries
# |
type: number
default: 3
Optional
Set a maximum retry to your crawler’s requests. |
headers
# |
type: Object
Optional
Specify headers to be sent in the crawler’s requests. Supported headers:
|
Did you find this page helpful?
