RedGrease Client

The RedGrease client / connection object gives access to the various commands defined by the RedisGears module.

Instantiation

The RedGrease client can be instantiated in a few different ways.

As a Redis Client

This instantiation method is useful if you are working with Redis from scratch and are not working on a legacy application that already has a bunch of Redis objects instantiated.

import redgrease

r = redgrease.RedisGears(host="localhost", port=6379)

This instantiates a Redis client object that is a subclass of the Redis class of the popular redis python client, but with an additional property gears through which the RedisGears commands can be accessed.

The constructor takes all the same arguments as the normal Redis client, and naturally it exposes all the same Redis command methods as the original.

It also means that if you are running a Redis instance on the default Redis port (6379) locally, as above, then you don’t need any arguments to instantiate the client object: r = redgrease.RedisGears().

Note

RedGrease also supports cluster-mode. Instantiating RedisGears will automatically try to figure out if it is against a cluster, and in that case instead instantiate an object which is a subclass of RedisCluster from the redis-py-cluster package (which in turn also is a subclass of Redis from the redis package).

The constructor takes all the same arguments as the normal Redis client, and naturally it exposes all the same Redis command methods as well.

If you want to be explicit which class to instantiate, then you can use redgrease.Redis and redgrease.RedisCluster for single and cluster mode respectively.

As a Gears object

If you already have code with instantiated Redis client objects, and you don’t want to create more connections, then you can instantiate only the redgrease.Gears object directly, using your existing Redis connection.

import redis  # Alt. `rediscluster`
import redgrease

# Legacy code
r = redis.Redis()  # Alt. `rediscluster.RedisCluster()`
...

# New code
gears = redgrease.Gears(r)

This instantiates a Gears object, which only exposes the RedisGears commands can be accessed, and not the normal Redis commands. This object is the same object that the above RedisGears client exposes through its gears property.

RedisGears Commands

The commands introduced by the RedisGears Redis module can be invoked through the Gears object, instantiates as per above. This section gives a run-down of the various commands and what they do, in the order of usefulness to most people.

You can find all the methods and functions in the API Reference, and in this section, we specifically look at the redgrease.Gears class.

Command Descriptions:

Executing Gear Functions

redgrease.Gears.pyexecute()

This is the most important RedGrease command, as it is the command for executing Gear functions. There are other ways, as we will go through in the next section Executing Gear Functions, but under the hood they all call this method.

RedisGears Command: RG.PYEXECUTE

Gears.pyexecute(gear_function: Union[str, redgrease.runtime.GearsBuilder, redgrease.gears.GearFunction] = '', unblocking=False, requirements: Optional[Iterable[Union[str, packaging.requirements.Requirement]]] = None, enforce_redgrease: Optional[Union[bool, str, packaging.version.Version, packaging.requirements.Requirement]] = None) → redgrease.data.ExecutionResult[source]

Execute a gear function.

Parameters

  • gear_function (Union[str, redgrease.gears.GearFunction], optional) –

    Function to execute. Either:

    Note

    • Python version must match the Gear runtime.

    • If the function is not “closed” with a run or register operation, an run() operation without additional arguments will be assumed, and automatically added to the function to close it.

    • The default for enforce_redgrease is True.

    Defaults to "", i.e. no function.

  • unblocking (bool, optional) –

    Execute function without waiting for it to finish before returning.

    Defaults to False. I.e. block until the function returns or fails.

  • requirements (Iterable[Union[None, str, redgrease.requirements.Requirement]], optional) –

    List of 3rd party package requirements needed to execute the function on the server.

    Defaults to None.

  • enforce_redgrease (redgrease.requirements.PackageOption, optional) –

    Indicates if redgrease runtime package requirement should be added or not, and potentially which version and/or extras or source.

    It can take several optional types:

    • None : No enforcement. Requirements are passed through, with or without ‘redgrease’ runtime package.

    • True : Enforces latest "redgrease[runtime]" package on PyPi,

    • False : Enforces that redgrease is not in the requirements list, any redgrease requirements will be removed from the function’s requirements. Note that it will not force redgrease to be uninstalled from the server runtime.

    • Otherwise, the argument’s string-representation is evaluated, and interpreted as either:

      1. A specific version. E.g. "1.2.3".

      2. A version qualifier. E.g. ">=1.0.0".

      3. Extras. E.g. "all" or "runtime". Will enforce the latest version on PyPi, with this/these extras.

      4. Full requirement qualifier or source. E.g: "redgrease[all]>=1.2.3" or "redgrease[runtime]@git+https://github.com/lyngon/redgrease.git@main"

    Defaults to None when gear_function is a script file path or a raw function string, but True when it is a GearFunction object.

Returns

The returned ExecutionResult has two properties: value and errors, containing the result value and any potential errors, respectively.

The value contains the result of the function, unless:

  • When used in ‘unblocking’ mode, the value is set to the execution ID

  • If the function has no output (i.e. it is closed by register() or for some other reason is not closed by a run() action), the value is True (boolean).

Any errors generated by the function are accumulated in the errors property, as a list.

Note

The ExecutionResult object itself behaves for the most part like its contained value, so for many simple operations, such as checking True-ness, result length, iterate results etc, it can be used directly. But the the safest was to get the actual result is by means of the value property.

Return type

redgrease.data.ExecutionResult

Raises

redis.exceptions.ResponseError – If the function cannot be parsed.

Get List of Executions

redgrease.Gears.dumpexecutions()

RedisGears Command: RG.DUMPEXECUTIONS

Gears.dumpexecutions(status: Optional[Union[str, redgrease.data.ExecutionStatus]] = None, registered: Optional[bool] = None) → List[redgrease.data.ExecutionInfo][source]

Get list of function executions. The executions list’s length is capped by the ‘MaxExecutions’ configuration option.

Parameters

  • status (Union[str, redgrease.data.ExecutionStatus], optional) – Only return executions that match this status. Either: “created”, “running”, “done”, “aborted”, “pending_cluster”, “pending_run”, “pending_receive” or “pending_termination”. Defaults to None.

  • registered (bool, optional) –

    If True, only return registered executions. If False, only return non-registered executions.

    Defaults to None.

Returns

A list of ExecutionInfo, with an entry per execution.

Return type

List[redgrease.data.ExecutionInfo]

Get Result of Asynchronous Execution

redgrease.Gears.getresults()

RedisGears Command: RG.GETRESULTS

Gears.getresults(id: Union[bytes, str, redgrease.data.ExecID, redgrease.data.ExecutionInfo]) → redgrease.data.ExecutionResult[source]

Get the results of a function in the execution list.

Parameters

id (Union[redgrease.data.ExecutionInfo, redgrease.data.ExecID, bytes, str]) – Execution identifier for the function to fetch the output for.

Returns

Results and errors from the gears function, if, and only if, execution exists and is completed.

Return type

redgrease.data.ExecutionResult

Raises

redis.exceptions.ResponseError – If the the execution does not exist or is still running

Get List of Registered Event-Based Functions

redgrease.Gears.dumpregistrations()

RedisGears Command: RG.DUMPREGISTRATIONS

Gears.dumpregistrations(reader: Optional[str] = None, desc: Optional[str] = None, mode: Optional[str] = None, key: Optional[str] = None, stream: Optional[str] = None, trigger: Optional[str] = None) → List[redgrease.data.Registration][source]

Get list of function registrations.

Parameters

  • reader (str, optional) – Only return registrations of this reader type. E.g: “StreamReader” Defaults to None.

  • desc (str, optional) – Only return registrations, where the description match this pattern. E.g: “transaction*log*” Defaults to None.

  • mode (str, optional) – Only return registrations, in this mode. Either “async”, “async_local” or “sync”. Defaults to None.

  • key (str, optional) – Only return (KeysReader) registrations, where the key pattern match this key. Defaults to None.

  • stream (str, optional) – Only return (StreamReader) registrations, where the stream pattern match this key. Defaults to None.

  • trigger (str, optional) – Only return (CommandReader) registrations, where the trigger pattern match this key. Defaults to None.

Returns

A list of Registration, with one entry per registered function.

Return type

List[redgrease.data.Registration]

Un-register an Event-Based Function

redgrease.Gears.unregister()

RedisGears Command: RG.UNREGISTER

Gears.unregister(id: Union[bytes, str, redgrease.data.ExecID, redgrease.data.Registration]) → bool[source]

Removes the registration of a function

Parameters

id (Union[redgrease.data.Registration, redgrease.data.ExecID, bytes, str]) – Execution identifier for the function to unregister.

Returns

True if successful

Return type

bool

Raises

redis.exceptions.ResponseError – If the registration ID doesn’t exist or if the function’s reader doesn’t support the unregister operation.

Trigger a ‘Command-Event’

redgrease.Gears.trigger()

RedisGears Command: RG.TRIGGER

Gears.trigger(trigger_name: str, *args) → List[Any][source]

Trigger the execution of a registered ‘CommandReader’ function.

Parameters

  • trigger_name (str) – The registered ‘trigger’ name of the function

  • *args (Any) – Any additional arguments to the trigger

Returns:6

List: A list of the functions output records.

Abort a Running Execution

redgrease.Gears.abortexecution()

RedisGears Command: RG.ABORTEXECUTION

Gears.abortexecution(id: Union[bytes, str, redgrease.data.ExecID, redgrease.data.ExecutionInfo]) → bool[source]

Abort the execution of a function mid-flight

Parameters

id (Union[redgrease.data.ExecutionInfo, redgrease.data.ExecID, bytes, str]) – The execution id to abort

Returns

True or an error if the execution does not exist or had already finished.

Return type

bool

Remove an Execution

redgrease.Gears.dropexecution()

RedisGears Command: RG.DROPEXECUTION

Gears.dropexecution(id: Union[bytes, str, redgrease.data.ExecID, redgrease.data.ExecutionInfo]) → bool[source]

Remove the execution of a function from the executions list.

Parameters

id (Union[redgrease.data.ExecutionInfo, redgrease.data.ExecID, bytes, str]) – Execution ID to remove

Returns

True if successful, or an error if the execution does not exist or is still running.

Return type

bool

Get List of Registered Dependencies

redgrease.Gears.pydumpreqs()

RedisGears Command: RG.PYDUMPREQS

Gears.pydumpreqs(name: Optional[str] = None, is_downloaded: Optional[bool] = None, is_installed: Optional[bool] = None) → List[redgrease.data.PyRequirementInfo][source]

Gets all the python requirements available (with information about each requirement).

Parameters

  • name (str, optional) – Only return packages with this base name. I.e. it is not filtering on version number, extras etc. Defaults to None.

  • is_downloaded (bool, optional) – If True, only return requirements that have been downloaded. If False, only return requirements that have NOT been downloaded. Defaults to None.

  • is_installed (bool, optional) – If True, only return requirements that have been installed. If False, only return requirements that have NOT been installed. Defaults to None.

Returns

List of Python requirement information objects.

Return type

List[redgrease.data.PyRequirementInfo]

Get Python Runtime Statistics

redgrease.Gears.pystats()

RedisGears Command: RG.PYSTATS

Gears.pystats() → redgrease.data.PyStats[source]

Gets memory usage statisticy from the Python interpreter

Returns

Python interpretere memory statistics, including total, peak and current amount of allocated memory, in bytes.

Return type

redgrease.data.PyStats

Get Cluster Information

redgrease.Gears.infocluster()

RedisGears Command: RG.INFOCLUSTER

Gears.infocluster() → redgrease.data.ClusterInfo[source]

Gets information about the cluster and its shards.

Returns

Cluster information or None if not in cluster mode.

Return type

redgrease.data.ClusterInfo

Refresh Cluster Topology

redgrease.Gears.refreshcluster()

RedisGears Command: RG.REFRESHCLUSTER

Gears.refreshcluster() → bool[source]

Refreshes the local node’s view of the cluster topology.

Returns

True if successful.

Return type

bool

Raises

redis.exceptions.ResponseError – If not successful

Get a Detailed Execution Plan

redgrease.Gears.getexecution()

RedisGears Command: RG.GETEXECUTION

Gears.getexecution(id: Union[bytes, str, redgrease.data.ExecID, redgrease.data.ExecutionInfo], locality: Optional[redgrease.data.ExecLocality] = None) → Mapping[bytes, redgrease.data.ExecutionPlan][source]

Get the execution plan details for a function in the execution list.

Parameters

  • id (Union[redgrease.data.ExecutionInfo, redgrease.data.ExecID, bytes, str]) – Execution identifier for the function to fetch execution plan for.

  • locality (Optional[redgrease.data.ExecLocality], optional) – Set to ‘Shard’ to get only local execution plan and set to ‘Cluster’ to collect executions from all shards. Defaults to ‘Shard’ in stand-alone mode, but “Cluster” in cluster mode.

Returns

A dict, mapping cluster ID to ExecutionPlan

Return type

Mapping[bytes, redgrease.data.ExecutionPlan]

Get and Set Gears Configurations

The above mentioned Gears object contains a property config of type redgrease.config.Config, through which various pre-defined and custom configuration setting can be read an written.

This object contains readable, and for certain options also writable, properties for the predefined RedisGears configurations. It also contains getter and setter methods that allows access to both the pre-defined as well as user defined configurations, in bulk.

RedisGears Commands: RG.CONFIGGET and RG.CONFIGSET

class redgrease.config.Config(redis: redis.client.Redis)[source]

Redis Gears Config

Instantiate a Redis Gears Confg object

Parameters

redis (redis.Redis) – Redis client / connection for underlying communication.

ValueTypes: Dict[str, Union[Type[redgrease.typing.T], Callable[[Any], redgrease.typing.T]]] = {'CreateVenv': <class 'bool'>, 'DependenciesSha256': <function safe_str>, 'DependenciesUrl': <function safe_str>, 'DownloadDeps': <class 'bool'>, 'ExecutionMaxIdleTime': <class 'int'>, 'ExecutionThreads': <class 'int'>, 'MaxExecutions': <class 'int'>, 'MaxExecutionsPerRegistration': <class 'int'>, 'ProfileExecutions': <class 'bool'>, 'PythonAttemptTraceback': <class 'bool'>, 'PythonInstallReqMaxIdleTime': <class 'int'>, 'PythonInstallationDir': <function safe_str>, 'SendMsgRetries': <class 'int'>}

Mapping from config name to its corresponding value type, transformer or constructor.

redis
get(*config_option: Union[bytes, str]) → Dict[Union[bytes, str], Any][source]

Get the value of one or more built-in configuration or a user-defined options.

Parameters

*config_option (Union[bytes, str]) – One or more names/key of configurations to get.

Returns

Dict of the requested config options mapped to their corresponding values.

Return type

Dict[Union[bytes, str], Any]

set(config_dict=None, **config_setting) → bool[source]

Set a value of one ore more built-in configuration or a user-defined options.

This function offers two methods of providing the keys and values to set; either as a mapping/dict or b key-word arguments.

Parameters

  • config_dict (Mapping[str, Any]) – Mapping / dict of config values to set.

  • **config_setting (Any) – Key-word arguments to set as config values.

Returns

True if all was successful, False oterwise

Return type

bool

get_single(config_option: Union[bytes, str])[source]

Get a single config value.

Parameters

config_option (str) – Name of the config to get.

Returns

The value of the config option.

Return type

Any

property MaxExecutions: int

Get the current value for the MaxExecutions config option.

The MaxExecutions configuration option controls the maximum number of executions that will be saved in the executions list. Once this threshold value is reached, older executions will be deleted from the list by order of their creation (FIFO).

Only executions that had finished (e.g. the ‘done’ or ‘aborted’ status ) are deleted.

property MaxExecutionsPerRegistration: int

Get the current value for the MaxExecutionsPerRegistration config option.

The MaxExecutionsPerRegistration configuration option controls the maximum number of executions that are saved in the list per registration. Once this threshold value is reached, older executions for that registration will be deleted from the list by order of their creation (FIFO).

Only executions that had finished (e.g. the ‘done’ or ‘aborted’ status ) are deleted.

property ProfileExecutions: bool

Get the current value for the ProfileExecutions config option.

The ProfileExecutions configuration option controls whether executions are profiled.

Note: Profiling impacts performance Profiling requires reading the server’s clock, which is a costly operation in terms of performance. Execution profiling is recommended only for debugging purposes and should be disabled in production.

property PythonAttemptTraceback: bool

Get the current value for the PythonAttemptTraceback config option.

The PythonAttemptTraceback configuration option controls whether the engine tries producing stack traces for Python runtime errors.

property DownloadDeps: bool

Get the current value for the DownloadDeps config option.

The DownloadDeps configuration option controls whether or not RedisGears will attempt to download missing Python dependencies.

property DependenciesUrl: str

Get the current value for the DependenciesUrl config option.

The DependenciesUrl configuration option controls the location from which RedisGears tries to download its Python dependencies.

property DependenciesSha256: str

Get the current value for the DependenciesSha256 config option.

The DependenciesSha256 configuration option specifies the SHA265 hash value of the Python dependencies. This value is verified after the dependencies have been downloaded and will stop the server’s startup in case of a mismatch.

property PythonInstallationDir

Get the current value for the PythonInstallationDir config option.

The PythonInstallationDir configuration option specifies the path for RedisGears’ Python dependencies.

property CreateVenv

Get the current value for the CreateVenv config option.

The CreateVenv configuration option controls whether the engine will create a virtual Python environment.

property ExecutionThreads

Get the current value for the ExecutionThreads config option.

The ExecutionThreads configuration option controls the number of threads that will run executions.

property ExecutionMaxIdleTime

Get the current value for the ExecutionMaxIdleTime config option.

The ExecutionMaxIdleTime configuration option controls the maximal amount of idle time (in milliseconds) before execution is aborted. Idle time means no progress is made by the execution.

The main reason for idle time is an execution that’s blocked on waiting for records from another shard that had failed (i.e. crashed). In that case, the execution will be aborted after the specified time limit. The idle timer is reset once the execution starts progressing again.

property PythonInstallReqMaxIdleTime

Get the current value for the PythonInstallReqMaxIdleTime config option.

The PythonInstallReqMaxIdleTime configuration option controls the maximal amount of idle time (in milliseconds) before Python’s requirements installation is aborted. Idle time means that the installation makes no progress.

The main reason for idle time is the same as for ExecutionMaxIdleTime .

property SendMsgRetries

Get the current value for the SendMsgRetries config option. The SendMsgRetries configuration option controls the maximum number of retries for sending a message between RedisGears’ shards. When a message is sent and the shard disconnects before acknowledging it, or when it returns an error, the message will be resent until this threshold is met.

Setting the value to 0 means unlimited retries.

Courtesy of : Lyngon Pte. Ltd.