What is your guy’s mass scraping strategy at the moment? I am scraping 20k+ Accounts a day. Using 50 Accounts each of them 400req/day running on 4g rotating mobile proxies. I want to scale up to 100k+ accounts per day, but at the moment I have an extremely high scraper burning rate. I split the accounts 50 accounts on 5 proxies each of them working on 3 shifts: 3 mornings, 3 midday, 4 afternoons. Also, i have a request delay of 6-20 seconds. But I have the feeling, that i am doing something wrong, since after a week, half of my scrapers are getting banned from instagram. Are there some scraper pro’s here on the forum, would love to hear your experiences and ideas on this topic 
Ur accounts are burning because your scraping too much . You need more accounts to uphold that
So 400Req/day for one account is too much? Nice to know, because i thought that 400req are not that much for a scrapper. What would you advice 200req/day, 100req/day?
What are your target sources like? Followers of other accounts? Or hashtags? Or people who interacted with other profiles?
It depends on what type of scraping you are doing .
Just for refrence, if you are scraping followers each API request will give you about 100 users, so for scraping 20000 you need 200 API calls, and that`s really close to the maximum limit .
For profile scraping on the other hand, depending on the quality of the accounts you could theorethically go as high as 1.5 to 2.5k api calls in a single day (opposed to 10k not so long ago sigh) but that will instantly kill your account .
On the other hand, sticking to lower limits will make the account last longer but at the end of the day, if you are scraping daily even with 200req day your account will get burned , don`t expect it to last for a whole month .
Personally I just churn and burn, I feel that it doesn`t really have a significative benefict to take it slow but that would ofcourse depend on the type of scraping you are doing, if it;s just as scrape for main accounts then ofcourse you would want to go slow …
SO consider the exampple limits I have explained and start from there . 200 req / day would still be way too much if scraping followers !
1 Like
Yes this was also my evaluation. For fetching followers I had the feeling that around 150/req a day is the maximum (approximately 15k profiles). To get specific profile information I personally went with 400 Api Calls per scrapper account.
But it seems like I have to accept, that my scrappers will get banned anyway, so it doesn’t matter to lower the Api-Call-Rate. Thank you for the tip
After all, I wonder, if the accounts would last longer, if you let them perform random actions, like viewing stories, posting stories, liking timeline etc.
Well, I havent personally tested that but my suspicion is that the amount of work envolved isnt worth it, especially if you are mass scraping at a larger scale !
1 Like