List of Listings

Chapter 1. Getting to know Redis

Listing 1.1. An example showing the SET, GET, and DEL commands in Redis

Listing 1.2. The RPUSH, LRANGE, LINDEX, and LPOP commands in Redis

Listing 1.3. The SADD, SMEMBERS, SISMEMBER, and SREM commands in Redis

Listing 1.4. The HSET, HGET, HGETALL, and HDEL commands in Redis

Listing 1.5. The ZADD, ZRANGE, ZRANGEBYSCORE, and ZREM commands in Redis

Listing 1.6. The article_vote() function

Listing 1.7. The post_article() function

Listing 1.8. The get_articles() function

Listing 1.9. The add_remove_groups() function

Listing 1.10. The get_group_articles() function

Chapter 2. Anatomy of a Redis web application

Listing 2.1. The check_token() function

Listing 2.2. The update_token() function

Listing 2.3. The clean_sessions() function

Listing 2.4. The add_to_cart() function

Listing 2.5. The clean_full_sessions() function

Listing 2.6. The cache_request() function

Listing 2.7. The schedule_row_cache() function

Listing 2.8. The cache_rows() daemon function

Listing 2.9. The updated update_token() function

Listing 2.10. The rescale_viewed() daemon function

Listing 2.11. The can_cache() function

Chapter 3. Commands in Redis

Listing 3.1. A sample interaction showing INCR and DECR operations in Redis

Listing 3.2. A sample interaction showing substring and bit operations in Redis

Listing 3.3. A sample interaction showing LIST push and pop commands in Redis

Listing 3.4. Blocking LIST pop and movement commands in Redis

Listing 3.5. A sample interaction showing some common SET commands in Redis

Listing 3.6. A sample interaction showing SET difference, intersection, and union in Redis

Listing 3.7. A sample interaction showing some common HASH commands in Redis

Listing 3.8. A sample interaction showing some more advanced features of Redis HASHes

Listing 3.9. A sample interaction showing some common ZSET commands in Redis

Listing 3.10. A sample interaction showing ZINTERSTORE and ZUNIONSTORE

Listing 3.11. Using PUBLISH and SUBSCRIBE in Redis

Listing 3.12. A sample interaction showing some uses of SORT

Listing 3.13. What can happen without transactions during parallel execution

Listing 3.14. What can happen with transactions during parallel execution

Listing 3.15. A sample interaction showing the use of expiration-related commands in Redis

Chapter 4. Keeping data safe and ensuring performance

Listing 4.1. Options for persistence configuration available in Redis

Listing 4.2. The process_logs() function that keeps progress information in Redis

Listing 4.3. The wait_for_sync() function

Listing 4.4. An example sequence of commands for replacing a failed master node

Listing 4.5. The list_item() function

Listing 4.6. The purchase_item() function

Listing 4.7. The update_token() function from section 2.5

Listing 4.8. The update_token_pipeline() function

Listing 4.9. The benchmark_update_token() function

Listing 4.10. Running redis-benchmark on an Intel Core-2 Duo 2.4 GHz desktop

Chapter 5. Using Redis for application support

Listing 5.1. The log_recent() function

Listing 5.2. The log_common() function

Listing 5.3. The update_counter() function

Listing 5.4. The get_counter() function

Listing 5.5. The clean_counters() function

Listing 5.6. The update_stats() function

Listing 5.7. The get_stats() function

Listing 5.8. The access_time() context manager

Listing 5.9. The ip_to_score() function

Listing 5.10. The import_ips_to_redis() function

Listing 5.11. The import_cities_to_redis() function

Listing 5.12. The find_city_by_ip() function

Listing 5.13. The is_under_maintenance() function

Listing 5.14. The set_config() function

Listing 5.15. The get_config() function

Listing 5.16. The redis_connection() function/decorator

Listing 5.17. The decorated log_recent() function

Chapter 6. Application components in Redis

Listing 6.1. The add_update_contact() function

Listing 6.2. The fetch_autocomplete_list() function

Listing 6.3. The find_prefix_range() function

Listing 6.4. The autocomplete_on_prefix() function

Listing 6.5. The join_guild() and leave_guild() functions

Listing 6.6. The list_item() function from section 4.4.2

Listing 6.7. The purchase_item() function from section 4.4.3

Listing 6.8. The acquire_lock() function

Listing 6.9. The purchase_item_with_lock() function

Listing 6.10. The release_lock() function

Listing 6.11. The acquire_lock_with_timeout() function

Listing 6.12. The acquire_semaphore() function

Listing 6.13. The release_semaphore() function

Listing 6.14. The acquire_fair_semaphore() function

Listing 6.15. The release_fair_semaphore() function

Listing 6.16. The refresh_fair_semaphore() function

Listing 6.17. The acquire_semaphore_with_lock() function

Listing 6.18. The send_sold_email_via_queue() function

Listing 6.19. The process_sold_email_queue() function

Listing 6.20. The worker_watch_queue() function

Listing 6.21. The worker_watch_queues() function

Listing 6.22. The execute_later() function

Listing 6.23. The poll_queue() function

Listing 6.24. The create_chat() function

Listing 6.25. The send_message() function

Listing 6.26. The fetch_pending_messages() function

Listing 6.27. The join_chat() function

Listing 6.28. The leave_chat() function

Listing 6.29. A locally aggregating callback for a daily country-level aggregate

Listing 6.30. The copy_logs_to_redis() function

Listing 6.31. The process_logs_from_redis() function

Listing 6.32. The readlines() function

Listing 6.33. The readblocks() generator

Listing 6.34. The readblocks_gz() generator

Chapter 7. Search-based applications

Listing 7.1. Functions to tokenize and index a document

Listing 7.2. SET intersection, union, and difference operation helpers

Listing 7.3. A function for parsing a search query

Listing 7.4. A function to parse a query and search documents

Listing 7.5. A function to parse and search, sorting the results

Listing 7.6. An updated function to search and sort based on votes and updated times

Listing 7.7. Some helper functions for performing ZSET intersections and unions

Listing 7.8. A function to turn a string into a numeric score

Listing 7.9. Helper functions for turning information about CPC and CPA ads into eCPM

Listing 7.10. A method for indexing an ad that’s targeted on location and ad content

Listing 7.11. Ad targeting by location and page content bonuses

Listing 7.12. A helper function for targeting ads based on location

Listing 7.13. Calculating the eCPM of ads including content match bonuses

Listing 7.14. A method for recording the result after we’ve targeted an ad

Listing 7.15. A method for recording clicks on an ad

Listing 7.16. A method for updating eCPMs and per-word eCPM bonuses for ads

Listing 7.17. A potential solution for finding jobs when a candidate meets all requirements

Listing 7.18. A function for indexing jobs based on the required skills

Listing 7.19. Find all jobs that a candidate is qualified for

Chapter 8. Building a simple social network

Listing 8.1. How to create a new user profile HASH

Listing 8.2. How to create a status message HASH

Listing 8.3. A function to fetch a page of recent status messages from a timeline

Listing 8.4. Update the following user’s home timeline

Listing 8.5. A function to stop following a user

Listing 8.6. Update a user’s profile timeline

Listing 8.7. Update a user’s followers’ home timelines

Listing 8.8. A function to delete a previously posted status message

Listing 8.9. Server and request handler for our streaming HTTP server

Listing 8.10. The code to actually start and run the streaming HTTP server

Listing 8.11. An example function to parse and store the client identifier

Listing 8.12. A function that will verify the request and stream data to the client

Listing 8.13. Updated create_status() from listing 8.2 to support streaming filters

Listing 8.14. Updated delete_status() from listing 8.8 to support streaming filters

Listing 8.15. A function to receive and process streamed messages

Listing 8.16. A factory function to dispatch to the actual filter creation

Listing 8.17. The function to handle firehose, gardenhose, and spritzer

Listing 8.18. A filter that matches groups of words that are posted in status messages

Listing 8.19. Messages posted by or mentioning any one of a list of users

Listing 8.20. Messages within boxes defined by ranges of latitudes and longitudes

Chapter 9. Reducing memory use

Listing 9.1. Configuration options for the ziplist representation of different structures

Listing 9.2. How to determine whether a structure is stored as a ziplist

Listing 9.3. Configuring the maximum size of the intset encoding for SETs

Listing 9.4. When an intset grows to be too large, it’s represented as a hash table.

Listing 9.5. Our code to benchmark varying sizes of ziplist-encoded LISTs

Listing 9.6. As ziplist-encoded LISTs grow, we can see performance drop

Listing 9.7. A function to calculate a shard key from a base key and a secondary entry key

Listing 9.8. Sharded HSET and HGET functions

Listing 9.9. Sharded IP lookup functions

Listing 9.10. A sharded SADD function we’ll use as part of a unique visitor counter

Listing 9.11. A function to keep track of the unique visitor count on a daily basis

Listing 9.12. Calculate today’s expected unique visitor count based on yesterday’s count

Listing 9.13. Base location tables we can expand as necessary

Listing 9.14. ISO3 country codes

Listing 9.15. A function for storing location data in sharded STRINGs

Listing 9.16. A function to aggregate location information for everyone

Listing 9.17. Convert location codes back to country/state information

Listing 9.18. A function to aggregate location information over provided user IDs

Chapter 10. Scaling Redis

Listing 10.1. A function to get a Redis connection based on a named configuration

Listing 10.2. Fetch a connection based on shard information

Listing 10.3. A shard-aware connection decorator

Listing 10.4. A machine and key-sharded count_visit() function

Listing 10.5. SORT-based search that fetches the values that were sorted

Listing 10.6. A function to perform queries against all shards

Listing 10.7. A function to merge sharded search results

Listing 10.8. ZSET-based search that returns scores for each result

Listing 10.9. Sharded search query over ZSETs that returns paginated results

Listing 10.10. An example of how we want our API for accessing shards to work

Listing 10.11. A class that implements sharded connection resolution based on key

Listing 10.12. Access follower/following ZSET shards

Listing 10.13. Sharded connection resolution based on ID pairs

Listing 10.14. A function that implements a sharded ZRANGEBYSCORE

Listing 10.15. Updated syndicate status function

Chapter 11. Scripting Redis with Lua

Listing 11.1. A function that loads scripts to be called later

Listing 11.2. Our function from listing 8.2 that creates a status message HASH

Listing 11.3. Creating a status message with Lua

Listing 11.4. Our final acquire_lock_with_timeout() function from section 6.2.5

Listing 11.5. A rewritten acquire_lock_with_timeout() that uses Lua

Listing 11.6. A rewritten release_lock() that uses Lua

Listing 11.7. The acquire_semaphore() function from section 6.3.2

Listing 11.8. The acquire_semaphore() function rewritten with Lua

Listing 11.9. A refresh_semaphore() function written with Lua

Listing 11.10. Our autocomplete code from section 6.1.2

Listing 11.11. Autocomplete on prefix using Redis scripting

Listing 11.12. The purchase item with lock function from section 6.2

Listing 11.13. The purchase item function rewritten with Lua

Listing 11.14. Functions for pushing items onto a sharded LIST

Listing 11.15. The Lua script for pushing items onto a sharded LIST

Listing 11.16. Our code to perform a blocking pop from a sharded LIST

Appendix A. Quick and dirty setup

Listing A.1. Installing build tools on Debian Linux

Listing A.2. Installing Redis on Linux

Listing A.3. Installing the Redis client libraries for Python on Linux

Listing A.4. Installing Redis on OS X

Listing A.5. Installing the Redis client library for Python on OS X

Listing A.6. Installing the Redis client library for Python on Windows

Listing A.7. Testing Redis from Python