Can't stop the scrapers from dying...Looking for advice

Hi guys, I’m looking for some recommendations on how to better my settings in Jv so my scrapers would last for at least 2-3 days. I’m not using Jv to automate my client’s accounts, all I’m doing is use it for scraping targeted users with filters for our manual growth service. So far I’ve been trying to use the limits & settings below and only scraping from one source at a time( we test sources and update them on a daily basis). The source has over 50k followers but for some reason the scraper only manages to spit out around 200 filtered users in a couple of hours which to us is very slow and painful. How can I make sure the scraper extracts at least 1k followers from the source? These are good sources so the option that Jv doesn’t extract more users because it can’t find enough users that match the filters is faluse. From 50k followers we should be able to get at least 1-2k filtered users but somehow only manage to get like 200. Any idea how to change this? Maybe use fewer filters…

Even though I got my scrapers from different sources that I tested and ended up buying aged accounts with og email just to make sure they survive… I thought this will ensure the scrapers will live a little longer but no luck. Then I moved them on a good quality 4G proxy but still, the same outcome-scraper dies in a couple of hours, max 1 day.

Currently facing two issues: 1. Jv doesn’t extract enough users from one source and 2. Scrapers are dying in a couple of hours even with safe settings and 4G proxy.

Any suggestion is welcomed and really appreciated. I’m also willing to pay for consulting on this if someone has a good method for my case or an alternative way, feel free to message me.

Do you have the option scrape with the eb where possible checked on your scrapers?

Can you check if your scraper accounts get API Scrape blocks (Social Profiles > actions on selected profiles > export api scrape blocks)?

I do think this setting is way too aggressive.

I’d suggest you play with the settings. Delaying the tools for only 5 minutes may not be enough. Try to increase the delay time, see if it helps.

Hey @ossi , I don’t think the scrapers are set to do actions trough eb but where can I check to make sure that’s the case? I exported the API blocks and it does look like it’s been blocked one time-see image attached

here you go, you can find the option here:

1 Like

Do you use this option on all your scrapers ? Isn’t using too much ressources?

Thanks @Luca , for some reason I never saw that setting:smile: It’s not checked on my scrapers and I see you also checked the full API emulation. Should I try with these two checked? Did you had good success with these scrape eb for scrapers?

You should check Enable Api full emulation on your scrapers. As for the option scrape with embedded browser where possible, I’d suggest you do a split test, have some scrapers with that option checked, and some other scrapers with that option unchecked, and see which group of account lives longer. Checking that options is recommended by Jarvee support to reduce the number of API calls but I’ve seen some users here avoid checking that option because it made their accounts die faster. I believe it has different effects on different accounts, so you should do a test to know which settings are better for your accounts.

1 Like

As Ossi has said, some accounts will work better with that option checked and some will not. It’s best to test both on your side, see what’s working better for you. There are really no concrete arguments to say that one option is better than the other.

1 Like

not all of them but yeah I do use that option a lot and no I did not notice any unusual ressources use, normal stuff.

you can use the scrape with embedded browser where possible on a few accounts like what Ossi suggest and see if the results are good, if yes then you can use both settings on your account, a test is what will give you the right answer

1 Like