The best way would be to scrape your page once a day, before you start doing actions, and send yourself a list of users who didn’t follow you back with their links. It will still be a lot of manual labor, but definitely less than a manual check…
You can find many scraping process around that would work, even for free, especially for that small kind of job
Theres no other way than scraping with tools or do it manually like you already do.
These are the limitations for manual growing so you might consider to just unfollow everyone after a certain period of time since this will save you lots of time.
I’ve also done the unfollow process manually a while ago by just going into my followings list and sorted it from old to new so I’d unfollow the oldest followings first.
I know you mentioned you are yet to try Jarvee but I’ll just add here that you can extract your followers via Jarvee’s Scrape Tools then you can also use the Check Follow Status section from the Scrape Tools to check if the users are following you or not.