What’s the most confusing part about a property-like data descriptor? Wrapping your head around the fact that it is being used to control instance-distinct attributes from its class.
What’s the hardest decision you have to make? Whether to store on the descriptor or on the instance (and then how you plan to accomplish that).
With instance properties, these issues are delegated to a nano framework so that you can concentrate on the important parts of your descriptor, creating a property that works the way you’d expect. Let’s get a little history to understand what I’m talking about.
Properties in Other Languages
When you see properties in other languages, such as C#, those properties work a lot like methods in that they’re defined on the class, but you get to focus on the instance while you’re working. In fact, they’re defined very much like methods and probably have the same or a similar implementation in the back.
Python’s property descriptor allows you to do something very similar, albeit in a slightly more verbose and unintuitive way, but you can still do it.
Next, we’ll look at Kotlin, which allows you to define properties in much the same way as C#, but they also have a secondary system called delegated properties. This is where you provide the property definition with an object that has get() and set() methods. Does this sound familiar? Sounds a lot like descriptors, right? There’s one big difference, though: there’s one delegated property object per instance. This makes it so that the delegated property only has to worry about what it’s doing with each instance. It also means that, since a new property is created with each instance, it can take a starting value in its constructor and never implement a set() method if it wants to be read-only; it doesn’t need set() to give it its first value. This is so much nicer than Python’s descriptors in most cases.
Back to Python
Now, don’t get me wrong; Python’s descriptors are an amazing feature, and the fact that they reside at the class level opens up a whole new world of possibilities. But the problem is that, arguably, most use cases for descriptors don’t need that. In fact, I would venture that most of the time, people just want a reusable property.
So, what can we do about this? We can make our own delegated properties, of course!
Accomplishing this went through at least four different iterations for me, starting off with using a completely different kind of Python metaprogramming. You can see the first two attempts on my blog, “Programming Ideas with Jake,” under my articles about descriptors.
Attempt 1
The first thing I tried was a more direct manipulation of how Python classes work to look and act more like it does in Kotlin. When you first set the attribute on an instance that you wanted with a delegated property, you assigned it an instance of that delegated property object. Then you would tweak __getattribute__() and __setattr__() so that if the attribute held a delegated property, it would call the appropriate method on it instead. Reusing the tweaked version of the __getattribute__() and __setattr__() could be done fairly easily with inheritance or a class decorator that does monkey patching.
As well as this works, it doesn’t sit well with me because I hate messing with those attribute access hooks. It seems too magical to me.
Attempt 2
I believe I was lying in bed about to fall asleep, when this idea came to me, causing me to stay up a little longer while I wrote it down. The idea was half-baked at first, but the basics of it run the rest of the attempts. Then, as I started to write it in code, I started to see certain issues and came up with a situation that will probably make you think of some jokes about Java frameworks.
The basics of the idea is that, instead of tweaking the attribute access methods, we move those changes out into a descriptor. That descriptor is called with any and all uses of its attribute, where it delegates to a delegated property object. It’s a pretty simple base of an idea.
From there, the problem came down to one question: how do we instantiate the delegated property instances? You may or may not already have a good guess, and the part that made my life so difficult was the idea that I thought the framework had to work in a way such that everything about the property had to be defined in that initial line on the class, and the constructor pretty much just needed to provide the starting value.
Be created without a value initially. For example, a lazy attribute where the lazy initialization function is provided with the factory. Or a property that can’t be None but might not have a value initially.
Could skip implementing the setter method to be read-only.
Could potentially take in some metadata, such as its name as well as what instance it’s for.
To do this, the first time the attribute was accessed, the descriptor created a blank version of the delegated property object and passed it and the metadata in an “InstancePropertyInitializer,” which had an initialize() method that you had to call in your normal constructor. This initializer method delegated to the initialize() method on the delegated property, sending in the metadata and whatever else the developer wanted to send into the property. The existence and flexibility of that initializer is what allowed delegated properties to accomplish this list of possibilities. If you don’t want an initial value, then just don’t give one to the initializer. If you want to skip having the setter method for a read-only property (but the framework can’t provide the initial value in the constructor), the initializer acts like a special backdoor setter. It’s also the vehicle for supplying the metadata.
The idea seemed pretty elegant to me at the time, but it dawned on me how cumbersome it was. First, the delegated property needed to provide an initializer method, plus it needed to provide a factory method. Also, initializing the attribute was weird, looking like self.attr.initialize(value) instead of just self.attr = value.
Attempt 3
Then, while I was on a camping trip and starting to work on my edits for this new edition, a better idea came to me. It followed mostly the same idea, but it made it nicer for properties that were given a starting value in the constructor.
To do this, the factory was changed to take in the metadata as well as an initial value. Now the delegated property could take in all of those things in the constructor. So, the first time that the attribute is set, the descriptor creates the property with all of that. This allowed the constructor code to go back to the self.attr = value format.
But what about ones that don’t want an initial value? Those classes have to take an extra step. Their factories had to have a default() method that on took in the metadata. This would be called if the delegated property still hadn’t been created for an instance but the descriptor’s __get__() method was being called. From there, the descriptor could start delegating to the property.
The reason that we have a default factory that is different than the normal factory is because most properties that would use a default factory also still allow the value to be initialized first.
Attempt 4
Before I was even done with that camping trip, I realized how dumb I had been all along and started work on this fourth, and hopefully final, attempt. We don’t need factories. Instead, at the class level, all you do is create the base InstanceProperty descriptor (shown below). The descriptor is just there to activate that attribute to use delegated properties. It simply assumes that the first assignment to the attribute is assigning the property itself instead of just a value. The descriptor doesn’t need to know what kind of property it will be storing or how to create it.
Instead, you create the delegate property instance in the class’ constructor. This has the added benefit of making sure that, if the descriptor stores the delegated property on the instance, the property is assigned in the constructor, which is recommended in Python 3.3+ due to key-sharing dictionaries. Sure, it’s no longer self.attr = value. Now it’s self.attr = PropertyType(value), which is more cumbersome but doesn’t feel nearly as weird, and it allows the design of the delegated property types to be notably easier.
There is still one awkward thing that needs to be dealt with on the property class. It needs a method for providing the metadata. It’s either do that or cause the attribute initialization line to look like self.attr = PropertyType(value, self, "attr", type(self).attr), assuming the property wants all three pieces of metadata (the instance, the attribute name, and the descriptor the property is controlled by).
The real one that’s included in the descriptor-tools library in version 1.1 (still unreleased at the time of writing) has more to it, allowing for the name to be set in versions that don’t support __set_name__(). The real one also makes the properties not deletable by default (a Deletable wrapper allows it) and allows you to use a simple wrapper for the descriptor that makes it read-only so that you don’t have create a mutable and read-only version of the delegated properties.
Example
This example also shows a small inconvenience with the framework: if you want a property that does some kind of validation and wants to use any of the metadata in the error message, you need to wait until set_meta() to do the initial validation. From the user’s perspective, this is effectively at the exact same point in time, but it’s awkward from the perspective of the person who has to write the property.
But you know what else this example shows? It shows how simple and intuitive the rest of creating a delegated property can be.
Just a little bit of extra work for a clean and easy way to have special attributes.
Go Nuts
While there is a default option for an instance attribute descriptor coming to descriptor-tools, that was designed to be as general as I knew how to make it. If you don’t care at all about the metadata, you can create your own instance attribute descriptor and strip that whole bit out. You’re nearly done with this book; you’ve got this!