Executing Gear Functions

Gear functions can either be defined as a Raw Function String, in Script File Path, as or dynamically constructed GearFunction Object. There are some subtleties and variations the three types that we’ll go through in their respective section, but either type can be executed using the redgrease.Gears.pyexecute() method.

import redgrease

gear_fun = ...  # Either a function string, script file path or GearFunction object

connection = redgrease.RedisGears()

result = connection.gears.pyexecute(gear_fun)

print(result.value)
print(result.errors)

Raw Function String

The most basic way of creating and executing Gear Functions is by passing a raw function string to the redgrease.Gears.pyexecute() method:

import redgrease

raw_gear_fun = "GearsBuilder('KeysReader').map(lambda x: x['type']).countby().run()"

rg = redgrease.RedisGears()

result = rg.gears.pyexecute(raw_gear_fun)

Note

You would rarely construct Gear functions this way, but it is fundamentally what happens under the hood for all the other methods of execution, and corresponds directly to the underlying RedisGears protocol.

Script File Path

A more practical way of defining Gear Functions is by putting them in a separate script file, and executing it by passing the path to the redgrease.Gears.pyexecute() method:

import redgrease

gear_script_path = "./path/to/some/gear/script.py"

rg = redgrease.RedisGears()

result = rg.gears.pyexecute(gear_script_path)

These scripts may be plain vanilla RedisGears functions that only use the built-in runtime functions, and does not import redgrease or use any of its features. In this case the redgrease package does not need to be installed on the runtime.

If the function is importing and using any RedGrease construct from the redgrease package, then when calling redgrease.Gears.pyexecute() method, the enforce_redgrease must be set in order to ensure that the package is installed on the RedisGears runtime.

In most cases you would just set it to True to get the latest stable RedGrease runtime package, but you may specify a specific version or even repository.

A notable special case is when functions in the script are only importing RedGrease modules that do not require any 3rd party dependencies (see list in the Redgrease Extras Options section). If this is the case then you may want to set enforce_redgrease="redgrease" (without the extras “[runtime]”), when calling redgrease.Gears.pyexecute(), as this is a version of redgrease without any external dependencies.

Another case is when you are only using explicitly imported Builtin Runtime Functions (e.g. from redgrease.runtime import GB, logs, execute) , and nothing else, as you in this case do not need any version of RedGrease on your RedisGears server runtime. In this case you can actually set enforce_redgrease=False.

More details about the various runtime installation options, which modules and functions are impacted, as well as the respective 3rd party dependencies can be found in the Redgrease Extras Options section.

Note

By default all Gear functions run in a shared runtime environment, and as a consequence all requirements / dependencies from different Gear functions are all installed in the same Python environment.

GearFunction Object

You can dynamically create a GearFunction object, directly in the same application as your Gears client / connection, by using any of the constructs well talk about in the GearFunction section, such as for example GearsBuilder or the “Reader” classes such as KeysReader and StreamReader etc.

GearFunction objects can be executed in three different ways; Using pyexecute, the on-method or directly in run or execute.

Note

There are two drawbacks of executing GearFunction objects, compared to executing Gear functions using either raw string, or by script file:

  1. The Python version of local application must match the the Python version in the RedisGears runtime (Python 3.7, at the time of writing this).

    When executing Gear functions using either raw string, or by script file, it doesn’t matter which version of Python the application is using, as long as it is Python 3.6 or later and the the code in the raw string is compatible with the Python version in the RedisGears runtime,

  2. The redgrease[runtime] package must be installed on the RedisGears Runtime environment.

    If you pass a GearFunction to redgrease.Gears.pyexecute(), it will attempt to install the latest stable version of redgrease[runtime] on the server, unless already installed, or if explicitly told otherwise using the enforce_redgrease argument.

    When executing Gear functions using either raw string, or by script file, redgrease only have to be installed if the function is importing any redgrease modules, of course.

As an example let’s assume, that we have instantiated a Gear client and created a very simple “open” Gear function as follows:

# Function that collecs the keys per type
get_keys_by_type = redgrease.KeysReader().aggregateby(
    lambda record: record["type"],
    set(),
    lambda _, acc, rec: acc | set([rec["key"]]),
    lambda _, acc, lacc: acc | lacc,
)

In this example get_keys_by_type is our simple “open” Gear function, which groups all the keys by their type (“string”, “hash”, “stream” etc) , and within each group collects the keys in a set.

We call it “open” because it has not been “closed” by a run() or register() action. The output from the last operation, here countby(), can therefore be used as input for a subsequent operations, if we’d like. The chain of operations of the function is “open-ended”, if you will.

Once an “open” function is terminated with either a run() or register() action, it is considered “closed”, and it can be executed, but not further extended.

The GearFunction section, goes into more details of these concepts and the different classes of GearFunctions.

RedGrease allows for open Gear functions, such as key_counter to be used as a starting point for other Gear functions, so lets create two “closed” functions from it:

# Transform to a single dict from type to keys
get_keys_by_type_dict = (
    get_keys_by_type.map(lambda record: {record["key"]: record["value"]})
    .aggregate({}, lambda acc, rec: {**acc, **rec})
    .run()
)

# Find the most common key-type
get_commonest_type = (
    get_keys_by_type.map(lambda x: (x["key"], len(x["value"])))
    .aggregate((None, 0), lambda acc, rec: rec if rec[1] > acc[1] else acc)
    .run()
)

These two new functions, get_keys_by_type_dict and get_commonest_type, both extend the earlier get_keys_by_type function with more operations. The former function collates the results in a dictionary. The latter finds the key-type that is most common in the keyspace.

Note that both functions end with the run() action, which indicates that the functions will run as an on-demand batch-job, but also that it is ‘closed’ and cannot be extended further.

Let’s execute these functions in some different ways.

Execute with Gears.pyexecute() Client method

The most idiomatic way of executing GearFunction objects is just to pass it to Gears.pyexecute():

# Execute "closed" GearFunction object with a Gear clients' `pyexecute` method
r = redgrease.RedisGears()

result_1 = r.gears.pyexecute(get_keys_by_type_dict)

The result might look something like:

{
    'hash': {'hash:1', 'hash:0', ...},
    'string': {'string:0', 'string:1', 'string:2', ...}
    ...
}

If you pass an “open” gear function, like our initial get_keys_by_type, to Gears.pyexecute(), it will still try its best to execute it, by assuming that you meant to close it with an empty run() action in the end:

# Execute "open" GearFunction with a Gear clients' `pyexecute` method
result_3 = r.gears.pyexecute(get_keys_by_type)

The result from our function might look something like:

[
    {'key': 'hash', 'value': {'hash:1', 'hash:0', ...}},
    {'key': 'string', 'value': {'string:0', 'string:1', 'string:2', ...}},
    ...
]

Execute with ClosedGearFunction.on() GearFunction method

Another short-form way of running a closed GearFunction is to call its its on() method.

# Execute "closed" GearFunction object with its `on` method
result_2 = get_commonest_type.on(r)

This approach only works with “closed” functions, but works regardless if the function has been closed with the run() or register() action.

The result for our specific function might look something like:

("string", 3)

The API specification is as follows:

ClosedGearFunction.on(gears_server, unblocking: bool = False, requirements: Optional[Iterable[str]] = None, replace: Optional[bool] = None, **kwargs)[source]

Execute the function on a RedisGears. This is equivalent to passing the function to Gears.pyexecute

Parameters
  • gears_server ([type]) – Redis client / connection object.

  • unblocking (bool, optional) – Execute function unblocking, i.e. asynchronous. Defaults to False.

  • requirements (Iterable[str], optional) – Additional requirements / dependency Python packages. Defaults to None.

Returns

The result of the function, just as Gears.pyexecute

Return type

redgrease.data.ExecutionResult

Execute directly in run() or register()

An even more succinct way of executing GearFunction objects is to specify the target connection directly in the action that closes the function. I.e the run() or register() action.

RedGrease has extended these methods with a couple additional arguments, which are not in the standard RedisGears API:

  • requirements - Takes a list of requirements / external packages that the function needs installed.

  • on - Takes a Gears (or RedisGears) client and immediately executes the function on it.

# Execute GearFunction using `on` argument in "closing" method
result_4 = get_keys_by_type.run(on=r)

This approach only works with “closed” functions, but works regardless if the function has been closed with the run() or register() action.

The result for our specific function should be identical to when we ran the function using pyexecute:

[
    {'key': 'hash', 'value': {'hash:1', 'hash:0', ...}},
    {'key': 'string', 'value': {'string:0', 'string:1', 'string:2', ...}},
    ...
]

pyexecute API Reference

Gears.pyexecute(gear_function: Union[str, redgrease.runtime.GearsBuilder, redgrease.gears.GearFunction] = '', unblocking=False, requirements: Optional[Iterable[Union[str, packaging.requirements.Requirement]]] = None, enforce_redgrease: Optional[Union[bool, str, packaging.version.Version, packaging.requirements.Requirement]] = None) redgrease.data.ExecutionResult[source]

Execute a gear function.

Parameters
  • gear_function (Union[str, redgrease.gears.GearFunction], optional) –

    Function to execute. Either:

    Note

    • Python version must match the Gear runtime.

    • If the function is not “closed” with a run or register operation, an run() operation without additional arguments will be assumed, and automatically added to the function to close it.

    • The default for enforce_redgrease is True.

    Defaults to "", i.e. no function.

  • unblocking (bool, optional) –

    Execute function without waiting for it to finish before returning.

    Defaults to False. I.e. block until the function returns or fails.

  • requirements (Iterable[Union[None, str, redgrease.requirements.Requirement]], optional) –

    List of 3rd party package requirements needed to execute the function on the server.

    Defaults to None.

  • enforce_redgrease (redgrease.requirements.PackageOption, optional) –

    Indicates if redgrease runtime package requirement should be added or not, and potentially which version and/or extras or source.

    It can take several optional types:

    • None : No enforcement. Requirements are passed through, with or without ‘redgrease’ runtime package.

    • True : Enforces latest "redgrease[runtime]" package on PyPi,

    • False : Enforces that redgrease is not in the requirements list, any redgrease requirements will be removed from the function’s requirements. Note that it will not force redgrease to be uninstalled from the server runtime.

    • Otherwise, the argument’s string-representation is evaluated, and interpreted as either:

      1. A specific version. E.g. "1.2.3".

      2. A version qualifier. E.g. ">=1.0.0".

      3. Extras. E.g. "all" or "runtime". Will enforce the latest version on PyPi, with this/these extras.

      4. Full requirement qualifier or source. E.g: "redgrease[all]>=1.2.3" or "redgrease[runtime]@git+https://github.com/lyngon/redgrease.git@main"

    Defaults to None when gear_function is a script file path or a raw function string, but True when it is a GearFunction object.

Returns

The returned ExecutionResult has two properties: value and errors, containing the result value and any potential errors, respectively.

The value contains the result of the function, unless:

  • When used in ‘unblocking’ mode, the value is set to the execution ID

  • If the function has no output (i.e. it is closed by register() or for some other reason is not closed by a run() action), the value is True (boolean).

Any errors generated by the function are accumulated in the errors property, as a list.

Note

The ExecutionResult object itself behaves for the most part like its contained value, so for many simple operations, such as checking True-ness, result length, iterate results etc, it can be used directly. But the the safest was to get the actual result is by means of the value property.

Return type

redgrease.data.ExecutionResult

Raises

redis.exceptions.ResponseError – If the function cannot be parsed.

on API Reference

ClosedGearFunction.on(gears_server, unblocking: bool = False, requirements: Optional[Iterable[str]] = None, replace: Optional[bool] = None, **kwargs)[source]

Execute the function on a RedisGears. This is equivalent to passing the function to Gears.pyexecute

Parameters
  • gears_server ([type]) – Redis client / connection object.

  • unblocking (bool, optional) – Execute function unblocking, i.e. asynchronous. Defaults to False.

  • requirements (Iterable[str], optional) – Additional requirements / dependency Python packages. Defaults to None.

Returns

The result of the function, just as Gears.pyexecute

Return type

redgrease.data.ExecutionResult


Courtesy of : Lyngon Pte. Ltd.