For this recipe, we are going to use the Twitter API to pull JSON data about Twitter users. Each Twitter user, identified by either a screen name (such as SayHiToSean) or a unique integer, has a profile containing a rich set of information about someone.
The following steps guide you through retrieving a set of Twitter users' profiles:
- First, we create a function that will manage pulling twitter profiles:
def pull_users_profiles(ids): users = [] for i in range(0, len(ids), 100): batch = ids[i:i + 100] users += twitter.lookup_user(user_id=batch) print(twitter.get_lastfunction_header('x-rate-limit-remaining')) return (users)
- We put this function to use, pulling profiles of both friends and followers:
friends_profiles = pull_users_profiles(friends_ids) followers_profiles = pull_users_profiles(followers_ids)
- To check whether everything works, we use a list comprehension to extract all of the friends' screen names from the profiles:
friends_screen_names = [p['screen_name'] for p in friends_profiles] - Using the following command, you should be greeted by a list of your friends' screen names:
print(friends_screen_names)
The first step in this recipe was the creation of a function that manages the twitter.lookup_user method call. The Twitter users/lookup endpoint accepts 100 user IDs at a time. Thus, we need to loop over our list of friends' or followers' IDs and batch them into groups of 100 for the requests. Twitter returns a JSON object that Twython converts into a list of Python dictionaries, ready for use.
There is an alternative way of retrieving profiles. Instead of pulling friends' and followers' IDs and then using those to request user profiles, we could have queried the friends/list endpoint with simply the current user's screen name (in this case, mine, which is @SayHiToSean) or user ID. Twitter would then return up to 200 user profiles per request. If you work out the API limits, either path works out to the same number of user profiles pulled in the 15 minute default time window that Twitter uses for rate-limiting purposes.
The pull_users_profiles function that we created has an extra feature in the last line of the loop:
print(twitter.get_lastfunction_header('x-rate-limit-remaining'))
We retrieve the header of the response from the last API call and check the x-rate-limit-remaining value. This value tells us exactly how many API calls we have left in a given 15 minute window. Although we print this value out with each loop, we do absolutely nothing to prevent us from slamming up against Twitter's rate limit, which varies by the endpoint.
Further, the list comprehension that we used in step 3 can fail if, for some reason, one of the Twitter user profiles that were received did not have a screen_name key. Thus, it would be better to add a condition to the comprehension:
friends_screen_names = [p['screen_name'] for p in friends_profiles if 'screen_name' in p]
Or, as an alternative and potentially, a more Pythonic way, we could use the GET method of the dictionary:
friends_screen_names = [p.get('screen_name') for p in friends_profiles]
In this case, profiles that do not have a screen_name key are not skipped but are replaced with None, instead.
- The description of the Twitter user profile JSON at https://dev.twitter.com/docs/platform-objects/users
- The GET users/lookup documentation at https://dev.twitter.com/docs/api/1.1/get/users/lookup
- The GET friends/list documentation at https://dev.twitter.com/docs/api/1.1/get/friends/list