Dealing with multiple **kwargs in Python

Our Problem Statement

When we write in large code bases, you will very often come across the case that you will have a large number of keywords. Much of this is because you want to allow control throughout the entire function. However — a significant number of arguments becomes increasingly difficult to handle over time and difficult to read. An update to one parameter means that there may be an issue with another parameter. This problem is explained deeper in Jeremy Howard’s blog “Make Delegation Work In Python”.

This quick article provides a general solution to dealing with multiple kwargs where you may have different functions that require different sets of kwargs. Howard’s blog provides a great starting point for this function that I had further extended upon to allow for feeding in different bodies of kwargs in different functions.

Nice bonuses when using this method:

• The ability to see parameters in Jupyter Notebooks
• Minimal re-writing (no one likes copy-pasting default parameters everywhere — so we will aim to inherit these functions themselves)

Note: The original intention of this code was to provide a simpler refactoring in our backend as the sheer number of keyword arguments in every function started to balloon — making the logic harder to work with over time with large number of parameters we had to copy and paste!

The Developer Experience

We start with the solution so users are able to understand what it is that the solution looks like for the developer.

class PaintMe:
    def paint(self, color='red', number_of_letters=5):
        self.color = color
        self.print(
            f"color is {color}", 
            number_of_letters=number_of_letters)    
    
    def verbose_print(self, string, number_of_letters=5):
        print('print began')
        print('a' * number_of_letters)
        print(string)
        print("print finished")
class PaintMeRainbow(PaintMe):
    @delegates({"paint_kwargs": PaintMe.paint, "print_kwargs": PaintMe.verbose_print})
    def rainbow_flag(self, print_kwargs={}, paint_kwargs={}):
        self.print_verbose(**print_kwargs)
        self.paint(**print_kwargs, **paint_kwargs)
PaintMeRainbow().rainbow_flag(
    paint_kwargs={"color": "orange"}, 
    print_kwargs={"number_of_letters": 10})

For classes like these, we will want to modularise our kwargs so that you can use them across separate functions. However, we can also converge them all into 1 simple **kwargs — if you are willing to go through every function and add a **kwargs parameter or include it in the decorator function.

In Jupyter Notebook, the functions are inherited and look as such:

How this appears in a Jupyter Notebook — with the defaults already suggested

How the code is used

Often times you will have multiple different functions each with a substantive number of arguments. For this — you simply place each function name into the decorator and assign it the relevant argument.

In the above example, “paint_kwargs” is the argument name but the arguments are inherited from the function PaintMe.paint — meaning we do not have to make a dictionary of functions with their default values separately!

Finally The Delegates Code

import inspect
def delegates(kwargs={}, keep=False):
    "Decorator: replace `**kwargs` in signature with params from `to`"
    def _f(f):
        from_f = f
        sig = inspect.signature(from_f)
        sigd = dict(sig.parameters)
        for name, kw in kwargs.items():
            sigd[name] = inspect.Parameter(
                name, inspect.Parameter.POSITIONAL_OR_KEYWORD, 
                default={k:v.default for k,v in inspect.signature(kw).parameters.items() if k not in sigd and 
                         v.default != inspect.Parameter.empty})
        if keep: sigd['kwargs'] = k
        from_f.__signature__ = sig.replace(parameters=sigd.values())
        return f
    return _f

Credit: Jeremy Howard for a great starting point and an impressive blog to go with it. In the article, he walks through the required “signature” dunder methods that allow our parameters to be seen at the highest level and explains it quite well.