Hey friends!
I’m working on an analytics tool in the music space, and need some help scraping data from Instagram for insights. I’m capable of scraping all the data, but after a short while the accounts I use usually get banned or blocked.
Need some help navigating this space in terms of the right proxies request intervals, and general best practices for automating these requests to Instagram!
Try using the recommended api limits from Jarvee’s Support:
For Enable API Limits:
Wait between 2500 and 12000 milliseconds when API excced 2 in last 5 seconds OR
Wait between 2500 and 7500 milliseconds when API exceed 1 in the last 2 seconds.
For Delay all tools when API errors exceed a given value:
Wait between 60 and 120 minutes when API errors exceed 1 per hour AND
wait between 350 and 550 minutes when API errors exceed 2 per day.
For Delay all tools when API calls exceed a given value:
Wait between 10 and 30 minutes when API calls exceed 50 per hour AND
wait between 350 and 550 minutes when API calls exceed 500 per day.
I have a good experience using Max 20 api calls an hour and max 140 api calls a day - using ~13 scrapers per main account. good luck.
wow id have to be running like 2000 scrapers for my setup to do that
This is with strict filtering? Or mains using specific users?
are you using any type of container orchestration? or central daemon service to manage horizontal scaling? shouldn’t be too difficult to deploy
It’s still better this way, to use more strict limits than to have to replace Scrapers all the time.
I have been using jarvee most of the time but have been looking for alternative measures