I'm trying to use Tweepy to get the full list of followers from an account with like 500k followers, and I have a code that gives me the usernames for smaller accounts, like under 100, but if I get one that's even like 110 followers, it doesn't work. Any help figuring out how to make it work with larger numbers is greatly appreciated!
Here's the code I have right now:
import tweepy
import time
key1 = "..."
key2 = "..."
key3 = "..."
key4 = "..."
accountvar = raw_input("Account name: ")
auth = tweepy.OAuthHandler(key1, key2)
auth.set_access_token(key3, key4)
api = tweepy.API(auth)
ids = []
for page in tweepy.Cursor(api.followers_ids, screen_name=accountvar).pages():
ids.extend(page)
time.sleep(60)
users = api.lookup_users(user_ids=ids)
for u in users:
print u.screen_name
The error I keep getting is:
Traceback (most recent call last):
File "test.py", line 24, in <module>
users = api.lookup_users(user_ids=ids)
File "/Library/Python/2.7/site-packages/tweepy/api.py", line 321, in lookup_users
return self._lookup_users(post_data=post_data)
File "/Library/Python/2.7/site-packages/tweepy/binder.py", line 239, in _call
return method.execute()
File "/Library/Python/2.7/site-packages/tweepy/binder.py", line 223, in execute
raise TweepError(error_msg, resp)
tweepy.error.TweepError: [{u'message': u'Too many terms specified in query.', u'code': 18}]
I've looked at a bunch of other questions about this type of question, but none I could find had a solution that worked for me, but if someone has a link to a solution, please send it to me!
Alec :
To extend upon this:\n\nYou can harvest 3,000 users per 15 minutes by adding a count parameter:\n\nusers = tweepy.Cursor(api.followers, screen_name=accountvar, count=200).items()\n\n\nThis will call the Twitter API 15 times as per your version, but rather than the default count=20, each API call will return 200 (i.e. you get 3000 rather than 300).",
2016-12-28T22:08:24
Himanshu Punetha :
Twitter provides two ways to fetch the followers: -\n\n\nFetching full followers list (using followers/list in Twitter API\nor api.followers in tweepy) - Alec and mataxu have provided the\napproach to fetch using this way in their answers. The rate limit\nwith this is you can get at most 200 * 15 = 3000 followers in every\n15 minutes window.\nSecond approach involves two stages:-\n a) Fetching only the followers ids first (using followers/ids in\nTwitter API or api.followers_ids in tweepy).you can get 5000 *\n15 = 75K follower ids in each 15 minutes window.\n\nb) Looking up\ntheir usernames or other data (using users/lookup in twitter api or\napi.lookup_users in tweepy). This has rate limitation of about 100 * 180\n= 18K lookups each 15 minute window.\n\n\nConsidering the rate limits, Second approach gives followers data 6 times faster when compared to first approach.\nBelow is the code which could be used to do it using 2nd approach:-\n\n#First, Make sure you have set wait_on_rate_limit to True while connecting through Tweepy\napi = tweepy.API(auth, wait_on_rate_limit=True,wait_on_rate_limit_notify=True)\n\n#Below code will request for 5000 follower ids in one request and therefore will give 75K ids in every 15 minute window (as 15 requests could be made in each window).\nfollowerids =[]\nfor user in tweepy.Cursor(api.followers_ids, screen_name=accountvar,count=5000).items():\n followerids.append(user) \nprint (len(followerids))\n\n#Below function could be used to make lookup requests for ids 100 at a time leading to 18K lookups in each 15 minute window\ndef get_usernames(userids, api):\n fullusers = []\n u_count = len(userids)\n print(u_count)\n try:\n for i in range(int(u_count/100) + 1): \n end_loc = min((i + 1) * 100, u_count)\n fullusers.extend(\n api.lookup_users(user_ids=userids[i * 100:end_loc]) \n )\n return fullusers\n except:\n import traceback\n traceback.print_exc()\n print ('Something went wrong, quitting...')\n\n#Calling the function below with the list of followeids and tweepy api connection details\nfullusers = get_usernames(followerids,api)\n\n\nHope this helps.\nSimiliar approach could be followed for fetching friends details by using api.friends_ids inplace of api.followers_ids\n\nIf you need more resources for rate limit comparison and for 2nd approach, check below links:-\n\n\nhttps://github.com/tweepy/tweepy/issues/627\nhttps://labsblog.f-secure.com/2018/02/27/how-to-get-twitter-follower-data-using-python-and-tweepy/\n",
2019-10-04T10:10:41
mataxu :
I actually figured it out, so I'll post the solution here just for reference.\n\nimport tweepy\nimport time\n\nkey1 = \"...\"\nkey2 = \"...\"\nkey3 = \"...\"\nkey4 = \"...\"\n\naccountvar = raw_input(\"Account name: \")\n\nauth = tweepy.OAuthHandler(key1, key2)\nauth.set_access_token(key3, key4)\n\napi = tweepy.API(auth)\n\nusers = tweepy.Cursor(api.followers, screen_name=accountvar).items()\n\nwhile True:\n try:\n user = next(users)\n except tweepy.TweepError:\n time.sleep(60*15)\n user = next(users)\n except StopIteration:\n break\n print \"@\" + user.screen_name\n\n\nThis stops after every 300 names for 15 minutes, and then continues. This makes sure that it doesn't run into problems. This will obviously take ages for large accounts, but as Leb mentioned:\n\n\n The twitter API only allows 100 users to be searched for at a time...[so] what you'll need to do is iterate through each 100 users but staying within the rate limit.\n\n\nYou basically just have to leave the program running if you want the next set. I don't know why mine is giving 300 at a time instead of 100, but as I mentioned about my program earlier, it was giving me 100 earlier as well. \n\nHope this helps anyone else that had the same problem as me, and shoutout to Leb for reminding me to focus on the rate limit. ",
2015-06-23T14:42:53