Quickstart Guide¶
This section aims to get you started within a few couple of minutes, while still explaining what is going on, so that someone with only limited experience with Python can follow along.
Setup TL;DR
docker run --name redis_gears --rm -d -p 127.0.0.1:6379:6379 redislabs/redisgears:1.0.6
virtualenv -p python3.7 .venv
source .venv/bin/activate
pip install redgrease[all]
docker run --name redis_gears --rm -d -p 127.0.0.1:6379:6379 redislabs/redisgears:1.0.6
virtualenv -p python3.7 .venv
\venv\Scripts\activate.bat
pip install redgrease[all]
Note
This is not tested. If anyone is using this OS, please let me know if this works or not. :)
If this was obvious to you, you can jump straight to the first code examples.
Running Redis Gears¶
The easiest way to run a Redis Engine with Redis Gears is by running one of the official Docker images, so firstly make sure that you have Docker engine installed.
(Eh? I’ve been living under a rock. What the hedge is Docker?)
With Docker is installed, open a terminal or command prompt and enter:
docker run --name redis_gears --rm -d -p 127.0.0.1:6379:6379 redislabs/redisgears:1.0.6
This will run a single Redis engine, with the Redis Gears module loaded, inside a Docker container. The first time you issue the command it may take a little time to launch, as Docker needs to fetch the container image from Docker Hub.
Lets break the command down:
docker run
is the base command telling Docker that we want to run a new containerized process.--name redis_gears
gives the name “redis_gears” to the container for easier identification. You can change it for whatever you like or omit it to assign a randomized name.--rm
instructs Docker that we want the container to be removed, in case it stops. This is optional but makes starting and stopping easer, although the state (and stored data) will be lost between restarts.-d
instructs Docker to run the container in the background as a ‘daemon’. You can omit this too, but your terminal / command prompt will be hijacked by the output / logs from the container. Which could be interesting enough.-p 127.0.0.1:6379:6379
instructs Docker that we want your host computer to locally (127.0.0.1) expose port 6379 and route it to 6379 in the container. This is the default port for Redis communication and this argument is necessary for application on your computer to be able to talk to the Redis engine inside the Docker container.redislabs/redisgears:1.0.6
is the name and version of the Docker image that we want to run. This specific image is prepared by RedisLabs and has the Gears module pre-installed and configured.
Note
If its your first time trying Redis Gears, stick to the command above, but if you want to try running a cluster image instead, you can issue the following command:
docker run --name redis_gears_cluster --rm -d -p 127.0.0.1:30001:30001 -p 127.0.0.1:30002:30002 -p 127.0.0.1:30003:30003 redislabs/rgcluster:1.0.6
This will run a 3-shard cluster exposed locally on ports 30001-30003.
Refer to the official documentation for more information and details on how how to install Redis Gears.
Checking Logs:¶
You can confirm that the container is running by inspecting the logs / output of the Redis Gears container by issuing the command:
docker logs redis_gears
You can optionally add the argument
--follow
to continuously follow the log output.You can optionally add the argument
--tail 100
to start showing the logs from the 100 most recent entries only.
If you just started the single instance engine, the logs should hold 40 odd lines starting and ending something like this:
1:C 03 Apr 2021 07:41:37.250 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 03 Apr 2021 07:41:37.251 # Redis version=6.0.1, bits=64, commit=00000000, modified=0, pid=1, just started
...
...
...
1:M 03 Apr 2021 07:41:37.309 * Module 'rg' loaded from /var/opt/redislabs/lib/modules/redisgears.so
1:M 03 Apr 2021 07:41:37.309 * Ready to accept connections
Stopping¶
You can stop the container by issuing:
docker stop redis_gears
If successful, it should simply output the name of the stopped container: redis_gears
RedGrease Installation¶
For the client application environment, it is strongly recommended that you set up a virtual Python environment, with Python 3.7 specifically.
Note
The Redis Gears containers above use Python 3.7 for executing Gear functions, and using the same version the client application will enable more features.
Warning
The RedGrease client package works with any Python version from 3.6 and later, but execution of dynamically created GearFunction objects is only possible when the client Python version match the Python version on the Redis Gears server runtime.
If the versions mismatch, Gear function execution is limited to execution by string or execution of script files
With Python 3.7, and virtualenv installed on your system:
Create a virtual python3.7 environment
virtualenv -p python3.7 .venv
Python packages, including RedGrease that you install within this virtual environment will not interfere with the rest of your system.
Activate the environment
source .venv/bin/activate
.venv\Scripts\activate.bat
Install redgrease
pip install redgrease[all]
Note
The
[all]
portion is important, as it will include all the Redgrease extras, and include the dependencies for the RedisGears client module as well as the RedGrease Command Line Interface (CLI).See here for more details on the various extras options.
Basic Commands¶
In this section we’ll walk through some of the basic commands and interactions with the RedGrease Gears client, including executing some very basic Gear functions.
The next chapter “RedGrease Client”, goes into all commands in more details, but for now we’ll just look at the most important things.
You can take a sneak-peek at the full code that we will walk through in this section, by expanding the block below (click “▶ Show”).
If you find this rather self-explanatory, then you can probably jump directly to the next section where we do some RedGrease Gear Function Comparisons with “vanilla” RedisGears functions.
Otherwise just continue reading and we’ll, go through it step-by-step.
Full code of this section.
1from operator import add
2
3import redgrease
4
5# Create connection / client for single instance Redis
6r = redgrease.RedisGears()
7
8# # Normal Redis Commands
9# Clearing the database
10r.flushall()
11
12# String operations
13r.set("Foo-fighter", 2021)
14r.set("Bar-fighter", 63)
15r.set("Baz-fighter", -747)
16
17# Hash Opertaions
18r.hset("noodle", mapping={"spam": "eggs", "meaning": 8})
19r.hincrby("noodle", "meaning", 34)
20
21# Stream operations
22r.xadd("transactions:0", {"msg": "First", "from": 0, "to": 2, "amount": 1000})
23
24
25# # Redis Gears Commands
26# Get Statistics on the Redis Gears Python runtime
27gears_runtime_python_stats = r.gears.pystats()
28print(f"Gears Python runtime stats: {gears_runtime_python_stats}")
29
30# Get info on any registered Gear functions, if any.
31registered_gear_functions = r.gears.dumpregistrations()
32print(f"Registered Gear functions: {registered_gear_functions}")
33
34# Execute nothing as a Gear function
35empty_function_result = r.gears.pyexecute()
36print(f"Result of nothing: {empty_function_result}")
37
38# Execute a Gear function string that just iterates through and returns the key-space.
39all_records_gear = r.gears.pyexecute("GearsBuilder('KeysReader').run()")
40print("All-records gear results: [")
41for result in all_records_gear:
42 print(f" {result}")
43print("]")
44
45# Gear function string to count all the keys
46key_count_gearfun_str = "GearsBuilder('KeysReader').count().run()"
47key_count_result = r.gears.pyexecute(key_count_gearfun_str)
48print(f"Total number of keys: {int(key_count_result)}")
49
50
51# # GearFunctions
52# GearFunction object to count all keys
53key_count_gearfun = redgrease.KeysReader().count().run()
54key_count_result = r.gears.pyexecute(key_count_gearfun)
55print(f"Total number of keys: {key_count_result}")
56
57
58# Simple Aggregation
59add_gear = redgrease.KeysReader("*-fighter").values().map(int).aggregate(0, add)
60simple_sum = add_gear.run(on=r)
61print(f"Multiplication of '-fighter'-keys values: {simple_sum}")
Let’s look at some code examples of how to use RedGrease, warming up with the basics.
Instantiation¶
Naturally, the first thing is to import of the RedGrease package and instantiate Redis Gears client / connection object:
1from operator import add
2
3import redgrease
4
5# Create connection / client for single instance Redis
6r = redgrease.RedisGears()
7
This will attempt to connect to a Redis server using the default port (6379) on “localhost”, which, if you followed the instructions above should be exactly what you set up and have running. There are of course arguments to set other targets, but more on that later.
The imported add
function from the operator
module is not part of the RedgGrease package, but we will use it later in one of the examples.
Note
If you created a Redis cluster above then you have to specify the initial master nodes you want to connect to:
1import redgrease
2
3r = redgrease.RedisGears(port=30001)
Redis Commands¶
The instantiated client / connection, r
, accepts all the normal Redis commands, exactly as expected.
The subsequent lines populate the Redis instance it with some data.
9# Clearing the database
10r.flushall()
11
12# String operations
13r.set("Foo-fighter", 2021)
14r.set("Bar-fighter", 63)
15r.set("Baz-fighter", -747)
16
17# Hash Opertaions
18r.hset("noodle", mapping={"spam": "eggs", "meaning": 8})
19r.hincrby("noodle", "meaning", 34)
20
21# Stream operations
22r.xadd("transactions:0", {"msg": "First", "from": 0, "to": 2, "amount": 1000})
23
24
Gears Commands¶
The client / connection also has a gears
attribute that gives access to RedisGears Commands.
26# Get Statistics on the Redis Gears Python runtime
27gears_runtime_python_stats = r.gears.pystats()
28print(f"Gears Python runtime stats: {gears_runtime_python_stats}")
29
30# Get info on any registered Gear functions, if any.
31registered_gear_functions = r.gears.dumpregistrations()
32print(f"Registered Gear functions: {registered_gear_functions}")
33
34# Execute nothing as a Gear function
35empty_function_result = r.gears.pyexecute()
36print(f"Result of nothing: {empty_function_result}")
37
38# Execute a Gear function string that just iterates through and returns the key-space.
39all_records_gear = r.gears.pyexecute("GearsBuilder('KeysReader').run()")
40print("All-records gear results: [")
41for result in all_records_gear:
42 print(f" {result}")
43print("]")
44
45# Gear function string to count all the keys
46key_count_gearfun_str = "GearsBuilder('KeysReader').count().run()"
47key_count_result = r.gears.pyexecute(key_count_gearfun_str)
48print(f"Total number of keys: {int(key_count_result)}")
49
50
The highlighted lines show the commands Gears.pystats()
, Gears.dumpregistrations()
and Gears.pyexecute()
respectively and the output should look something like this:
Gears Python runtime stats: PyStats(TotalAllocated=41275404, PeakAllocated=11867779, CurrAllocated=11786368)
Registered Gear functions: []
Result of nothing: True
All-records gear results: [
b"{'event': None, 'key': 'Baz-fighter', 'type': 'string', 'value': '-747'}"
b"{'event': None, 'key': 'Bar-fighter', 'type': 'string', 'value': '63'}"
b"{'event': None, 'key': 'transactions:0', 'type': 'unknown', 'value': None}"
b"{'event': None, 'key': 'Foo-fighter', 'type': 'string', 'value': '2021'}"
b"{'event': None, 'key': 'noodle', 'type': 'hash', 'value': {'meaning': '42', 'spam': 'eggs'}}"
]
Total number of keys: 5
The command Gears.pystats()
gets some memory usage statistics about the Redis Gears Python runtime environment on the server.
The command Gears.dumpregistrations()
gets information about any registered Gears functions, in this cas none.
And finally, the command Gears.pyexecute()
is the most important command, which sends a Gears function to the server for execution or registration.
In the above example, we are invoking it three times:
Firstly (line 35) - We pass nothing, i.e. no function at all, which naturally doesn’t do anything, but is perfectly valid, and the call thus just returns
True
.Secondly (line 39) - We execute a Raw Function String, that reads through the Redis keys (indicated by the
'KeysReader'
) and just returns the result by running the function as a batch job (indicated by therun()
operation). The result is consequently a list of dicts, representing the keys and their respective values and types in the Redis keyspace, I.e. the keys we added just before.Thirdly (lines 46-47) - We pass a very similar function, but with an additional
count()
operation, which is a Gear operation that simply aggregates and counts the incoming records, in this case all key-space records on the server. The result is simply the number or keys in the database:5
.
There are other Gears commands too, and the next chapter, “Redgrease Client”, will run through all of them.
GearFunctions¶
Composing Gear functions by using strings is not at all very practical, so RedGrease provides a more convenient way of constructing Gear functions programmatically, using various GearFunction objects.
52# GearFunction object to count all keys
53key_count_gearfun = redgrease.KeysReader().count().run()
54key_count_result = r.gears.pyexecute(key_count_gearfun)
55print(f"Total number of keys: {key_count_result}")
56
57
This Gear function does the same thing as the last function of the previous example, but instead of being composed by a string, it is composed programmatically using RedGrease’s GearFunction objects, in this case using the KeysReader
class.
The output is, just as expected:
Total number of keys: 5
Warning
Note that execution of GearFunction objects only work if your local Python environment version matches the version on the Redis Gear server, i.e. Python 3.7.
If the versions mismatch, Gear function execution is limited to execution by string or execution of script files
The final basic example shows a GearFunction that has a couple of operations stringed together.
59add_gear = redgrease.KeysReader("*-fighter").values().map(int).aggregate(0, add)
60simple_sum = add_gear.run(on=r)
61print(f"Multiplication of '-fighter'-keys values: {simple_sum}")
This Gear function adds the values of all the simple keys, with names ending in “-fighter”, which were the first three keys created in the example.
And indeed, the result is:
Sum of '-fighter'-keys values: 1337
Here is a quick run down of how it works:
Firstly, the
KeysReader
is parameterized with a key pattern*-fighter
meaning it will only read the matching keys.Secondly, the
map()
operation uses a simple lambda function, to lift out thevalue
and ensure it is an integer, on each of the keys.Thirdly, the
aggregate()
operation is used to add the values together, using the importedadd
function, starting from the value 0.Lastly, the
run()
operation is used to specify that the function should run as a batch job. Theon
argument states that we want to run it immediately on our client / connection,r
.
The chapter “Readers” will go through the various types of readers, and the chapter operations will go through the various types of operations, and how to use them.
RedGrease Gear Function Comparisons¶
Now let’s move on to some more examples of smaller Gear functions, before we move on to some more elaborate examples.
The examples in this section are basically comparisons of how the examples in the official Gears documentation, could be simplified by using RedGrease.
Note
RedGrease is backwards compatible with the “vanilla” syntax and structure, and all versions below are still perfectly valid Gear functions when executing using RedGrease.
Word Count¶
Counting of words.
Assumptions¶
All keys store Redis String values. Each value is a sentence.
Vanilla Version¶
This is the the ‘Word Count’ example from the official RedisGears documentation.
gb = GearsBuilder()
gb.map(lambda x: x["value"]) # map records to "sentence" values
gb.flatmap(lambda x: x.split()) # split sentences to words
gb.countby() # count each word's occurances
gb.run()
RedGrease Version¶
This is an example of how the same Gear function could be rewritten using RedGrease.
from redgrease import KeysReader
KeysReader().values().flatmap(str.split).countby().run()
Delete by Key Prefix¶
Deletes all keys whose name begins with a specified prefix and return their count.
Assumptions¶
There may be keys in the database. Some of these may have names beginning with the “delete_me:” prefix.
Vanilla Version¶
This is the the ‘Delete by Key Prefix’ example from the official RedisGears documentation.
gb = GearsBuilder()
gb.map(lambda x: x["key"]) # map the records to key names
gb.foreach(lambda x: execute("DEL", x)) # delete each key
gb.count() # count the records
gb.run("delete_me:*")
RedGrease Version¶
This is an example of how the same Gear function could be rewritten using RedGrease.
from redgrease import KeysReader, cmd
delete_fun = KeysReader().keys().foreach(cmd.delete).count()
delete_fun.run("delete_me:*")
Basic Redis Stream Processing¶
Copy every new message from a Redis Stream to a Redis Hash key.
Assumptions¶
An input Redis Stream is stored under the “mystream” key.
Vanilla Version¶
This is the the ‘Basic Redis Stream Processing’ example from the official RedisGears documentation.
gb = GearsBuilder("StreamReader")
gb.foreach(
lambda x: execute("HMSET", x["id"], *sum([[k, v] for k, v in x.items()], []))
) # write to Redis Hash
gb.register("mystream")
RedGrease Version¶
This is an example of how the same Gear function could be rewritten using RedGrease.
from redgrease import StreamReader, cmd
StreamReader().foreach(lambda x: cmd.hmset(x["id"], x)).register( # write to Redis Hash
"mystream"
)
Automatic Expiry¶
Sets the time to live (TTL) for every updated key to one hour.
Assumptions¶
None.
Vanilla Version¶
This is the the ‘Automatic Expiry’ example from the official RedisGears documentation.
gb = GB()
gb.foreach(lambda x: execute("EXPIRE", x["key"], 3600))
gb.register("*", mode="sync", readValue=False)
RedGrease Version¶
This is an example of how the same Gear function could be rewritten using RedGrease.
from redgrease import KeysReader, cmd
expire = KeysReader().keys().foreach(lambda x: cmd.expire(x, 3600))
expire.register("*", mode="sync", readValue=False)
Keyspace Notification Processing¶
This example demonstrates a two-step process that:
Synchronously captures distributed keyspace events
Asynchronously processes the events’ stream
Assumptions¶
The example assumes there is a process
function defined, that does the actual processing of the deleted records. For the purpose of the example we can assume that it just outputs the name of the expired keys to the Redis logs, as follows:
def process(x):
"""
Processes a message from the local expiration stream
Note: in this example we simply print to the log, but feel free to replace
this logic with your own, e.g. an HTTP request to a REST API or a call to an
external data store.
"""
log(f"Key '{x['value']['key']}' expired at {x['id'].split('-')[0]}")
Vanilla Version¶
This is the the ‘Keyspace Notification Processing’ example from the official RedisGears documentation.
# Capture an expiration event and adds it to the shard's local 'expired' stream
cap = GB("KeysReader")
cap.foreach(lambda x: execute("XADD", f"expired:{hashtag()}", "*", "key", x["key"]))
cap.register(prefix="*", mode="sync", eventTypes=["expired"], readValue=False)
# Consume new messages from expiration streams and process them somehow
proc = GB("StreamReader")
proc.foreach(process)
proc.register(prefix="expired:*", batch=100, duration=1)
RedGrease Version¶
This is an example of how the same Gear function could be rewritten using RedGrease.
from redgrease import KeysReader, StreamReader, cmd, hashtag, log
# Capture an expiration event and adds it to the shard's local 'expired' stream
KeysReader().keys().foreach(
lambda key: cmd.xadd(f"expired:{hashtag()}", {"key": key})
).register(prefix="*", mode="sync", eventTypes=["expired"], readValue=False)
# Consume new messages from expiration streams and process them somehow
StreamReader().foreach(process).register(prefix="expired:*", batch=100, duration=1)
Reliable Keyspace Notification¶
Capture each keyspace event and store to a Stream.
Assumptions¶
…
Vanilla Version¶
This is the the ‘Reliable Keyspace Notification’ example from the official RedisGears documentation.
GearsBuilder().foreach(
lambda x: execute(
"XADD", "notifications-stream", "*", *sum([[k, v] for k, v in x.items()], [])
)
).register(prefix="person:*", eventTypes=["hset", "hmset"], mode="sync")
RedGrease Version¶
This is an example of how the same Gear function could be rewritten using RedGrease.
from redgrease import KeysReader, cmd
KeysReader().foreach(lambda x: cmd.xadd("notifications-stream", x)).register(
prefix="person:*", eventTypes=["hset", "hmset"], mode="sync"
)
Cache Get Command¶
As a final example of this quickstart tutorial, let’s look at how we can build caching into Redis as a new command, with the help of Redis Gears and RedGrease.
Full code.
It may look a bit intimidating at first, but theres actually not not that much to it. Most of it is just comments, logging or testing code.
import timeit
import redgrease
# Bind / register the function on some Redis instance.
r = redgrease.RedisGears()
# CommandReader Decorator
# The `command` decorator tunrs the function to a CommandReader,
# registerered on the Redis Gears sever if using the `on` argument
@redgrease.command(on=r, requirements=["requests"], replace=False)
def cache_get(url):
import requests
# Check if the url is already in the cache,
# And if so, simply return the cached result.
if redgrease.cmd.exists(url):
return bytes(redgrease.cmd.get(url))
# Otherwise fetch the url.
response = requests.get(url)
# Return nothing if request fails
if response.status_code != 200:
return bytes()
# If ok, set the cache data and return.
response_data = bytes(response.content)
redgrease.cmd.set(url, response_data)
return response_data
# Test caching on some images
some_image_urls = [
"http://images.cocodataset.org/train2017/000000483381.jpg",
"http://images.cocodataset.org/train2017/000000237137.jpg",
"http://images.cocodataset.org/train2017/000000017267.jpg",
"http://images.cocodataset.org/train2017/000000197756.jpg",
"http://images.cocodataset.org/train2017/000000193332.jpg",
"http://images.cocodataset.org/train2017/000000475564.jpg",
"http://images.cocodataset.org/train2017/000000247368.jpg",
"http://images.cocodataset.org/train2017/000000416337.jpg",
]
# Get all the images and write them to disk
def get_em_all():
for image_url in some_image_urls:
# This will invoke the cache_get function **on the Redis server**
image_data = cache_get(image_url)
# Quick and dirty way of getting the image file name.
image_name = image_url.split("/")[-1]
# Write to file
with open(image_name, "wb") as img_file:
img_file.write(image_data.value)
# Test it
# Time how long it takes to get images when the cache is empty.
t1 = timeit.timeit(get_em_all, number=1)
print(f"Cache-miss time: {t1:.3f} seconds")
# Time how long it takes to get the images when they are all in the cache.
t2 = timeit.timeit(get_em_all, number=1)
print(f"Cache-hit time: {t2:.3f} seconds")
print(f"That is {t1/t2:.1f} times faster!")
# Clean the database
def cleanup(r: redgrease.RedisGears):
# Unregister all registrations
for reg in r.gears.dumpregistrations():
r.gears.unregister(reg.id)
# Remove all executions
for exe in r.gears.dumpexecutions():
r.gears.dropexecution(str(exe.executionId))
# Clear all keys
r.flushall()
# Check that there are no keys
return len(r.keys()) == 0
# print(cleanup(r))
Let’s go through the code, step by step, and it will hopefully make some sense.
1import timeit
2
3import redgrease
4
5# Bind / register the function on some Redis instance.
6r = redgrease.RedisGears()
7
8
The instantiation of the client / connection is business as usual.
Cache-Get function¶
Lets now go for the core of the solution; The code that we want to run on Redis for each resource request.
14 import requests
15
16 # Check if the url is already in the cache,
17 # And if so, simply return the cached result.
18 if redgrease.cmd.exists(url):
19 return bytes(redgrease.cmd.get(url))
20
21 # Otherwise fetch the url.
22 response = requests.get(url)
23
24 # Return nothing if request fails
25 if response.status_code != 200:
26 return bytes()
27
28 # If ok, set the cache data and return.
29 response_data = bytes(response.content)
30 redgrease.cmd.set(url, response_data)
31
32 return response_data
33
34
Look at the highlighted lines, and notice:
The logic of handling requests with caching is simply put in a normal function, much like we would if the caching logic was handled by each client.
The argument of the function is what we could expect, the
url
to the resource to get.- The function return value is either:
The contents of the response to requests to the URL (line 32), or
A cached value (line 19)
Which is exactly what you would expect from a cached fetching function.
The really interesting part, however, is this little line, on top of the function.
10# The `command` decorator tunrs the function to a CommandReader,
11# registerered on the Redis Gears sever if using the `on` argument
12@redgrease.command(on=r, requirements=["requests"], replace=False)
13def cache_get(url):
14 import requests
15
16 # Check if the url is already in the cache,
17 # And if so, simply return the cached result.
18 if redgrease.cmd.exists(url):
19 return bytes(redgrease.cmd.get(url))
All the Redis Gears magic is hidden in this function decorator, and it does a couple of important things:
It embeds the function in a
CommandReader
Gear function.It ensures that the function is registered on our Redis server(s).
It captures the relevant requirements, for the function to work.
It ensures that we only register this function once.
It creates a new function, with the same name, which when called, triggers the corresponding registered Gear function, and returns the result from the server.
This means that you can now call the decorated function, just as if it was a local function:
result = cache_get("http://images.cocodataset.org/train2017/000000169188.jpg")
This may look like it is actually executing the function locally, but the cache_get
function is actually executed on the server.
This means that the registered cache_get
Gear function can not only be triggered by the client that defined the decorated function, but can be triggered by any client by invoking the Redis Gear RG.TRIGGER command with the the functions’ trigger name and arguments.
In our case, using redis-cli as an example:
> RG.TRIGGER cache_get http://images.cocodataset.org/train2017/000000169188.jpg
The arguments for the @command
decorator, are the same as to the OpenGearFunction.register()
method, inherited by the CommandReader
class.
Note
This simplistic cache function is only for demonstrating the command function decorator. The design choices of this particular caching implementation is far from ideal for all use-cases.
For example:
Only the response content data is returned, not response status or headers.
Cache is never expiring.
If multiple requests for the same resource is made in close successions, there may be duplicate external requests.
The entire response contents is copied into memory before writing to cache.
… etc …
Naturally, the solution could easily be modified to accommodate other behaviors.
Testing the Cache¶
To test the caching, we create a very simple function that iterates through some URLs and tries to get them from the cache and saving the contents to local files.
36some_image_urls = [
37 "http://images.cocodataset.org/train2017/000000483381.jpg",
38 "http://images.cocodataset.org/train2017/000000237137.jpg",
39 "http://images.cocodataset.org/train2017/000000017267.jpg",
40 "http://images.cocodataset.org/train2017/000000197756.jpg",
41 "http://images.cocodataset.org/train2017/000000193332.jpg",
42 "http://images.cocodataset.org/train2017/000000475564.jpg",
43 "http://images.cocodataset.org/train2017/000000247368.jpg",
44 "http://images.cocodataset.org/train2017/000000416337.jpg",
45]
46
47
48# Get all the images and write them to disk
49def get_em_all():
50
51 for image_url in some_image_urls:
52
53 # This will invoke the cache_get function **on the Redis server**
54 image_data = cache_get(image_url)
55
56 # Quick and dirty way of getting the image file name.
57 image_name = image_url.split("/")[-1]
58
59 # Write to file
60 with open(image_name, "wb") as img_file:
61 img_file.write(image_data.value)
62
63
64# Test it
65# Time how long it takes to get images when the cache is empty.
66t1 = timeit.timeit(get_em_all, number=1)
67print(f"Cache-miss time: {t1:.3f} seconds")
68
69# Time how long it takes to get the images when they are all in the cache.
70t2 = timeit.timeit(get_em_all, number=1)
71print(f"Cache-hit time: {t2:.3f} seconds")
72print(f"That is {t1/t2:.1f} times faster!")
73
74
Calling the this function twice reveals that the caching does indeed seem to work.
Cache-miss time: 10.954 seconds
Cache-hit time: 0.013 seconds
That is 818.6 times faster!
We can also inspect the logs of the Redis node to confirm that the cache function was indeed executed on the server.
docker logs --tail 100
And you should indeed see that the expected log messages appear:
1:M 06 Apr 2021 08:58:06.314 * <module> GEARS: Cache request #1 for resource 'http://images.cocodataset.org/train2017/000000416337.jpg'
1:M 06 Apr 2021 08:58:06.314 * <module> GEARS: Cache miss #1 - Downloading resource 'http://images.cocodataset.org/train2017/000000416337.jpg'.
1:M 06 Apr 2021 08:58:07.855 * <module> GEARS: Cache update #1 - Request status for resource 'http://images.cocodataset.org/train2017/000000416337.jpg': 200
...
1:M 06 Apr 2021 08:58:07.860 * <module> GEARS: Cache request #2 for resource 'http://images.cocodataset.org/train2017/000000416337.jpg'
The last piece of code is jut to clean up the database by un-registering the cache_get
Gear function, cancel and drop any ongoing Gear function executions and flush the key-space.
76def cleanup(r: redgrease.RedisGears):
77
78 # Unregister all registrations
79 for reg in r.gears.dumpregistrations():
80 r.gears.unregister(reg.id)
81
82 # Remove all executions
83 for exe in r.gears.dumpexecutions():
84 r.gears.dropexecution(str(exe.executionId))
85
86 # Clear all keys
87 r.flushall()
88
89 # Check that there are no keys
90 return len(r.keys()) == 0
91
92
93# print(cleanup(r))
That wraps up the Quickstart Guide! Good luck building Gears!