Chapter 4: Working with Metaclasses
Metaclasses, the focal point of this chapter, can manipulate the way a new class is created by decorating the arguments without impacting the actual class definition itself. Metaclasses are not very frequently used in Python application development unless there is a need for more advanced implementations of frameworks or APIs that need features such as manipulation of classes or dynamic class generation and so on.
In the previous chapter, we looked at the concept of decorators with some examples. Understanding decorators helps in following metaclasses with more ease since both decorators and metaclasses deal with metaprogramming on Python 3 program objects by manipulating them externally.
In this chapter, we will cover the following main topics:
- Overview of metaclasses
- The structure of a metaclass
- The application of metaclasses
- Switching metaclasses
- Inheritance in metaclasses
- Manipulating class variables
By the end of this chapter, you should be able to create your own metaclasses, implement inheritance on metaclasses, and reuse ones that are already created.
Technical requirements
The code examples shared in this chapter are available on GitHub under the code for this chapter here: https://github.com/PacktPublishing/Metaprogramming-with-Python/tree/main/Chapter04.
Overview of metaclasses
Metaclasses are classes that can be created separately with certain features that can alter the behavior of other classes or can help in dynamically manufacturing new classes. The base class of all metaclasses is the type class and the object or instance of a metaclass will be a class. Any custom metaclass that we create will be inherited from the type class. type is the class of all data types in Python as well and everything else in Python 3 is an object of the type class. We can test this statement by checking the type of different program objects in Python, as follows:
class TestForType:
pass
type(TestForType)
type
type(int)
type
type(str)
type
type(object)
type
type(float)
type
type(list)
type
In this chapter, we will look at some examples of how to use these metaclasses, how to implement them, and how to reuse them. We will continue with our ABC Megamart examples to proceed further with the understanding of metaclasses.
The structure of a metaclass
A metaclass is like any other class, but it has the ability to alter the behavior of other classes that take it as their metaclass. Understanding the structure of a metaclass helps us create our own customized metaclasses, which can be used further in manipulating new classes. The superclass of a metaclass is the type itself. When we create a class with type as its superclass and override the __new__ method to manipulate the metadata of a class it returns, then we have created a metaclass. Let’s take a closer look with the help of some simple examples.
The __new__ method takes cls as its first argument, which is the class itself. The members of the class that has cls as its first argument can be accessed by the class name and the rest of the arguments as other metadata of the class, as seen here:
class ExampleMetaClass1(type):
def __new__(classitself, *args):
print('class itself: ', classitself) print('Others: ', args) return type.__new__(classitself, *args)
In the preceding code, we have created the class ExampleMetaClass1, which inherits the class type and overrides the __new__ method to print the class instance and its other arguments.
Let’s now create the class ExampleClass1 and add the preceding metaclass to it:
class ExampleClass1(metaclass = ExampleMetaClass1):
int1 = 123
str1 = 'test'
def test():
print('test') Running the preceding code displays the following result:
class itself: <class '__main__.ExampleMetaClass1'>
Others: ('ExampleClass1', (), {'__module__': '__main__', '__qualname__': 'ExampleClass1', 'int1': 123, 'str1': 'test', 'test': <function ExampleClass1.test at 0x00000194A377E1F0>})The first part of this output is the class instance <class '__main__.ExampleMetaClass1'> and the remaining arguments are the class name and the arguments of the class. A simple representation of the metaclass definition is as follows:
Figure 4.1 – Example metaclass definition
Let’s dive into a little more detail with another example in our next subsection.
Analyzing the arguments
We now will dig deeper into the arguments of the __new__ method of a metaclass. Analyzing the arguments of a metaclass will provide clarity on what information of a class can be customized using a metaclass. The data that can be manipulated in the classes that adds a metaclass while defining is represented in the following figure:
Figure 4.2 – Example metaclass with more arguments
Let’s now follow these steps to see how the behavior of arguments affects classes:
- First, look at the following code where we have all arguments of a metaclass segregated—class instance; class name; all parent classes, superclasses, or base classes of the class; and all variables and methods created within the class:
class ExampleMetaClass2(type):
def __new__(classitself, classname, baseclasses,
attributes):
print('class itself: ', classitself)
print('class name: ', classname)
print('parent class list: ', baseclasses)
print('attribute list: ', attributes)
return type.__new__(classitself, classname,
baseclasses, attributes)
- Next, we will be creating two parent classes—ExampleParentClass1 and ExampleParentClass2:
class ExampleParentClass1():
def test1():
print('parent1 - test1')
class ExampleParentClass2():
def test2():
print('parent2 - test2')
- Now, we will create the class ExampleClass2 where we will be inheriting both of the preceding parent classes and adding the metaclass as ExampleMetaClass2:
class ExampleClass2(ExampleParentClass1,ExampleParentClass2, metaclass = ExampleMetaClass2):
int1 = 123
str1 = 'test'
def test3():
print('child1 - test3')
- Executing the preceding code results in the following output:
class itself: <class '__main__.ExampleMetaClass2'>
class name: ExampleClass2
parent class: (<class '__main__.ExampleParentClass1'>, <class '__main__.ExampleParentClass2'>)
attributes: {'__module__': '__main__', '__qualname__': 'ExampleClass2', 'int1': 123, 'str1': 'test', 'test3': <function ExampleClass2.test3 at 0x00000194A3994E50>}
This example shows us the highlighted arguments that are returned by the metaclass and gives an overview of which values can possibly be manipulated from a class using metaprogramming.
- Let us look at the type of each of the classes created in this example:
type(ExampleParentClass1)
type
type(ExampleParentClass2)
type
type(ExampleMetaClass2)
type
type(ExampleClass2)
__main__.ExampleMetaClass2
As we can see, the type of all other classes is the type itself whereas the type of ExampleClass2 is ExampleMetaClass2.
Now that you understand the structure of a metaclass, we can look further into applications of metaclasses on our ABC Megamart example.
The application of metaclasses
In this section, we will look at an example where we will create a metaclass that can automatically modify the user-defined method attributes of any branch class that is newly created. To test this, let us follow these steps:
- Create a metaclass with the name BranchMetaclass:
class BranchMetaclass(type):
- Create a __new__ method with class instance, class name, base classes, and attributes as its arguments. In the __new__ method, import the inspect library, which can help inspect the input attributes:
def __new__(classitself, classname, baseclasses,
attributes):
import inspect
- Create a new dictionary, newattributes:
newattributes = {}
Iterate over the class attributes, check that the attributes start with __, and don’t change the value.
- Continue iterating over the other attributes and check if the attributes are functions. If they are functions, prefix branch to the class method and convert the method name into title case:
for attribute, value in attributes.items():
if attribute.startswith("__"):
newattributes[attribute] = value
elif inspect.isfunction(value):
newattributes['branch' +
attribute.title()] = value for a
attribute, value in
attributes.items():
if attribute.startswith("__"):
newattributes[attribute] = value
elif inspect.isfunction(value):
newattributes['branch' +
attribute.title()] = value
- If the preceding conditions are not met, save the value of the attribute as it is:
else:
newattributes[attribute] = value
- Return the new method with new attributes:
return type.__new__(classitself,
classname, baseclasses,
newattributes)
- Within the metaclass, also create a regular user-defined method, buy_product, to calculate the sales price of a product:
def buy_product(product,unit_price,quantity,statetax_rate,promotiontype):
statetax_rate = statetax_rate
initialprice = unit_price*quantity
sales_price = initialprice +
initialprice*statetax_rate
return sales_price, product,promotiontype
- Next, we will create another new class, Brooklyn, and add this metaclass to the class. By adding the metaclass, we want the methods in the class Brooklyn to have a prefix branch and change the methods to title case while creating the methods of Brooklyn.
The Brooklyn class has four variables, product_id, product_name, product_category, and unit_price. We will also create a method to calculate the maintenance cost and this method should be converted from maintenance_cost to branchMaintenance_cost due to the metaclass that alters the behavior of the newly created class. Here’s the new class:
class Brooklyn(metaclass = BranchMetaclass):
product_id = 100902
product_name = 'Iphone X'
product_category = 'Electronics'
unit_price = 700
def maintenance_cost(self,product_type, quantity):
self.product_type = product_type
self.quantity = quantity
cold_storage_cost = 100
if (product_type == 'Electronics'):
maintenance_cost = self.quantity * 0.25 +
cold_storage_cost
return maintenance_cost
else:
return "We don't stock this product"
- We can list all the arguments of the class Brooklyn and check if the metaclass has altered its behavior:
dir(Brooklyn)
['__class__',
'__delattr__',
'__dict__',
‚__dir__',
‚__doc__',
‚__eq__',
‚__format__',
‚__ge__',
‚__getattribute__',
‚__gt__',
‚__hash__',
‚__init__',
‚__init_subclass__',
‚__le__',
‚__lt__',
‚__module__',
‚__ne__',
‚__new__',
‚__reduce__',
‚__reduce_ex__',
‚__repr__',
‚__setattr__',
‚__sizeof__',
‚__str__',
‚__subclasshook__',
‚__weakref__',
'branchMaintenance_cost',
'product_category',
'product_id',
'product_name',
'unit_price']
- Let us now create an object and look at its methods and variables, as follows:
brooklyn = Brooklyn()
brooklyn.branchMaintenance_Cost('Electronics',10)
102.5
brooklyn.product_id
100902
brooklyn.product_name
'Iphone X'
brooklyn.product_type
'Electronics'
A simple representation of this example is as follows:
Figure 4.3 – Application of metaclass on ABC Megamart – Branch example
So far, we’ve looked at an overview of a metaclass, understood its structure, performed an analysis of its arguments, and applied our understanding by creating a custom metaclass on our core example. We will look at a few more applications in the following section.
Inheriting the metaclass
In this section, we will walk through an example where we will inherit the metaclass to check whether it can be inherited as a regular parent class without altering the behavior of the new class that is being created. Take a look at the following code:
class Queens(BranchMetaclass):
def maintenance_cost(product_type, quantity):
product_type = product_type
quantity = quantity
if (product_type == ‹FMCG›):
maintenance_cost = quantity * 0.05
return maintenance_cost
else:
return "We don't stock this product"
Let's now create an object for the preceding class to check if an object can be created:
queens = Queens()
We get the following TypeError:
Figure 4.4 – Error while creating an object for the class inheriting a metaclass
This error occurred as __new__ is a static method that is called to create a new instance for the class and it expects three arguments of the class, which are not provided while creating the class object. However, there is another way of calling the newly created class, Queens. The class can be called directly, and its methods can be used without having to create an object:
Queens.maintenance_cost('FMCG',120)6.0
The maintenance_cost method did not get modified into branchMaintenance_cost since the metaclass is not used as a metaclass but as a parent class. Since the metaclass is inherited, Queens also inherits the user-defined methods of BranchMetaclass as follows:
Queens.buy_product('Iphone',1000,1,0.04,None)(1040.0, 'Iphone', None)
Inheriting as a parent and metaclass
Let’s now look at what happens when we inherit a class as a parent and also add it as a metaclass while creating a new class:
class Queens(BranchMetaclass, metaclass = BranchMetaclass):
def maintenance_cost(product_type, quantity):
product_type = product_type
quantity = quantity
if (product_type == ‹FMCG›):
maintenance_cost = quantity * 0.05
return maintenance_cost
else:
return "We don't stock this product"
In the preceding code, we have added BranchMetaclass as the parent class for the class Queens and we have also added it as a metaclass. This definition should make the class Queens inherit the custom methods from BranchMetaclass and also change the maintenance_cost method into branchMaintenance_cost. Let’s see if it does:
Queens.branchMaintenance_Cost('FMCG',2340)117.0
In the preceding code execution and output, the maintenance_cost method is converted into the branchMaintenance_cost method as expected. Now run the following command:
Queens.buy_product('Iphone',1500,1,0.043,None)(1564.5, 'Iphone', None)
The buy_product method, which is a custom method from BranchMetaclass, is also inherited since it is a parent class.
Here is a simple representation of this example:
Figure 4.5 – Application of metaclass and also inheriting it on ABC Megamart branch example
Let us look further into examples of switching metaclasses from one class to another.
Switching metaclasses
We can now look into the concept of switching metaclasses for a class. You may think, why do we need to switch metaclasses? Switching metaclasses reinforces the reusability concept of metaprogramming and in this case, it helps in understanding how a metaclass created for use on one class can also be used for a different class without impacting the class definition.
In the example for this section, we will be creating two meta classes – IncomeStatementMetaClass and BalanceSheetMetaClass. For the Malibu branch of ABC Megamart, we will create a class to capture the information required for its financial statements. The two financial statements relevant for this example are Income Statement attributes and Balance Sheet attributes for the Malibu branch. To differentiate where a particular attribute or method of a class should go, we will be creating two metaclasses that look at the names of the attributes and tag them under Income Statement or Balance Sheet accordingly.
The following is a simple representation of the attributes that will be manipulated by the aforementioned metaclasses:
Figure 4.6 – Finance attributes used in this metaclass example
Take a look at the following code snippet:
class IncomeStatementMetaClass(type):
def __new__(classitself, classname, baseclasses,
attributes):
newattributes = {} for attribute, value in attributes.items():
if attribute.startswith("__"): newattributes[attribute] = value
elif («revenue» in attribute) or
("expense" in attribute) or ("profit" in attribute) or ("loss" in attribute): newattributes['IncomeStatement_' +
attribute.title()] = value
else:
newattributes[attribute] = value
return type.__new__(classitself, classname,
baseclasses, newattributes)
Here, the new method is modified to check for attributes that have the key as one of the parameters that belong to an income statement such as revenue, expense, profit, or loss. If any of this terminology occurs in the method name or variable name, we will add a prefix of IncomeStatement to segregate those methods and variables.
To test this metaclass, we will be creating a new class, Malibu, with four variables and four methods, as follows:
class Malibu(metaclass = IncomeStatementMetaClass):
profit = 4354365
loss = 43000
assets = 15000
liabilities = 4000
def calc_revenue(quantity,unitsales_price):
totalrevenue = quantity * unitsales_price
return totalrevenue
def calc_expense(totalrevenue,netincome, netloss):
totalexpense = totalrevenue - (netincome + netloss)
return totalexpense
def calc_totalassets(cash,inventory,accountsreceivable):
totalassets = cash + inventory + accountsreceivable
return totalassets
def calc_totalliabilities(debt,accruedexpense,
accountspayable):
totalliabilities = debt + accruedexpense +
accountspayable
return totalliabilities
In the preceding code, we have added the metaclass IncomeStatementMetaClass and we see that the attributes of the class Malibu modify the behavior of variables and methods as follows:
Figure 4.7 – Malibu without metaclass (left) and Malibu with metaclass (right)
We will further add another metaclass, BalanceSheetMetaClass, to deal with the balance sheet-related attributes in the class Malibu. In the following metaclass, the new method is modified to check for attributes that have the key as one of the parameters that belong to a balance sheet such as assets, liabilities, goodwill, and cash. If any of these terms occur in the method name or variable name, we will add a prefix of BalanceSheet to segregate those methods and variables:
class BalanceSheetMetaClass(type):
def __new__(classitself, classname, baseclasses,
attributes):
newattributes = {} for attribute, value in attributes.items():
if attribute.startswith("__"): newattributes[attribute] = value
elif («assets» in attribute) or
("liabilities" in attribute) or ("goodwill" in attribute) or ("cash" in attribute): newattributes['BalanceSheet_' +
attribute.title()] = value
else:
newattributes[attribute] = value
return type.__new__(classitself, classname,
baseclasses, newattributes)
In the preceding code, we have added the metaclass BalanceSheetMetaClass and we see that the attributes of the class Malibu modify the behavior of variables and methods as follows:
Figure 4.8 – Malibu with IncomeStatementMetaClass (left) and Malibu with BalanceSheetMetaClass (right)
Now that you know why we need to switch metaclasses, let us look at the application of metaclasses in inheritance.
Inheritance in metaclasses
Inheritance, in a literal sense, means a child acquiring the properties of a parent and it means the same in the case of object-oriented programming too. A new class can inherit the attributes and methods of a parent class and it can also have its own properties and methods.
In this example, we will look at how inheritance works on metaclasses by creating two classes, California and Pasadena – California being the parent class and Pasadena the child class.
Let’s check these steps out to understand inheritance better:
- In the previous section, we already created two metaclasses that inherited type as their parent class – IncomeStatementMetaClass and BalanceSheetMetaClass. We will start by creating the class California with the IncomeStatement metaclass:
class California(metaclass = IncomeStatementMetaClass):
profit = 4354365
loss = 43000
def calc_revenue(quantity,unitsales_price):
totalrevenue = quantity * unitsaleprice
return totalrevenue
def calc_expense(totalrevenue,netincome, netloss):
totalexpense = totalrevenue - (netincome + netloss)
return totalexpense
Here, we have defined only those attributes that can be modified by the IncomeStatement metaclass.
- Next, we will create another class, Pasadena, with the BalanceSheet metaclass:
class Pasadena(California,metaclass = BalanceSheetMetaClass):
assets = 18000
liabilities = 5000
def calc_totalassets(cash,inventory,
accountsreceivable):
totalassets = cash + inventory +
accountsreceivable
return totalassets
def calc_totalliabilities(debt,accruedexpense,
accountspayable):
totalliabilities = debt + accruedexpense +
accountspayable
return totalliabilities
We have defined here only those attributes that can be modified by the BalanceSheet metaclass.
Figure 4.9 – Error while executing a child class that has a different metaclass
This error was thrown since Pasadena inherited the parent class California, which has a different metaclass, IncomeStatementMetaClass, which is inherited from type, and Pasadena’s metaclass BalanceSheetMetaClass is also inherited from type.
- To resolve this error, we can redefine the BalanceSheetMetaClass with the parent class as IncomeStatementMetaClass instead of the type class, as follows:
class BalanceSheetMetaClass(IncomeStatementMetaClass):
def __new__(classitself, classname, baseclasses,
attributes):
newattributes = {}
for attribute, value in attributes.items():
if attribute.startswith("__"):
newattributes[attribute] = value
elif («assets» in attribute) or
("liabilities" in attribute) or
("goodwill" in attribute) or
("cash" in attribute):
newattributes['BalanceSheet_' +
attribute.title()] = value
else:
newattributes[attribute] = value
return type.__new__(classitself, classname,
baseclasses, newattributes)
- Let’s now rerun the California parent class and also the Pasadena child class to check if the behavior modification of both the metaclasses is implemented in the Pasadena class:
class California(metaclass = IncomeStatementMetaClass):
profit = 4354365
loss = 43000
def calc_revenue(quantity,unitsales_price):
totalrevenue = quantity * unitsaleprice
return totalrevenue
def calc_expense(totalrevenue,netincome, netloss):
totalexpense = totalrevenue - (netincome +
netloss)
return totalexpense
class Pasadena(California,metaclass = BalanceSheetMetaClass):
assets = 18000
liabilities = 5000
def calc_totalassets(cash,inventory,
accountsreceivable):
totalassets = cash + inventory +
accountsreceivable
return totalassets
def calc_totalliabilities(debt,accruedexpense,
accountspayable):
totalliabilities = debt + accruedexpense +
accountspayable
return totalliabilities
- Here is the output from the Pasadena class, and as we can see, both the BalanceSheet and IncomeStatement attributes are modified as per their metaclasses:
Figure 4.10 – Pasadena class with inheritance
A simple representation of this application is as follows:
Figure 4.11 – Inheritance in metaclasses
In this case, we have redefined the parent class of BalanceSheetMetaClass to be IncomeStatementMetaClass since Python does not automatically resolve their parent classes while they were both inherited by type and instead throws a metaclass conflict. Redefining the parent class of BalanceSheetMetaClass not only resolves the error but will also not impact the overall functionality of the class since IncomeStatementMetaClass is in turn inherited from type.
Let us look at another example where we will be adding additional information to class attributes.
Manipulating class variables
In this section, we will take an example to look at manipulating class variables further using metaclasses. We will be creating a metaclass named SchemaMetaClass and will define the __new__ method to manipulate attributes of a class if they are variables of data types that belong to integer, float, string, or boolean. Let’s go through the steps real quick:
- We will now create the SchemaMetaClass with the parent class as type and have modified the new method to check the following conditions:
class SchemaMetaClass(type):
- Create the dictionary object newattributes. If the class attribute is a built-in class method that starts with __, then the attribute’s value is stored as such in newattributes:
def __new__(classitself, classname, baseclasses,
attributes):
newattributes = {}
for attribute, value in attributes.items():
if attribute.startswith("__"):
newattributes[attribute] = value
- If the class attribute is an integer or float variable, then the class returns a dictionary item with the attribute name as ColumnName, the value as Value, Type as NUMERIC, and Length as the length of the value:
elif type(value)==int or type(value)==float:
newattributes[attribute] = {}
newattributes[attribute]['ColumnName']
= attribute.title()
newattributes[attribute]['Value']
= value
newattributes[attribute]['Type']
= 'NUMERIC'
newattributes[attribute]['Length'] = len(str(value))
- If the class attribute is a string variable, then the class returns a similar dictionary item with Type as VARCHAR:
elif type(value)==str:
newattributes[attribute] = {}
newattributes[attribute]['ColumnName']
= attribute.title()
newattributes[attribute]['Value']
= value
newattributes[attribute]['Type']
= 'VARCHAR'
newattributes[attribute]['Length']
= len(value)
- Similarly, if the class attribute is a boolean object, a similar kind of dictionary item with Type as BOOLEAN is returned:
elif type(value)==bool:
newattributes[attribute] = {}
newattributes[attribute]['ColumnName']
= attribute.title()
newattributes[attribute]['Value']
= value
newattributes[attribute]['Type']
= 'BOOLEAN'
newattributes[attribute]['Length']
= None
- Any other variable or method is stored like so in newattributes:
else:
newattributes[attribute] = value
return type.__new__(classitself, classname,
baseclasses, newattributes)
- We will now create the class Arizona with the metaclass as SchemaMetaClass, define all the variables for a product, and define a method that creates a schema out of the metaprogrammed class attributes:
class Arizona(metaclass = SchemaMetaClass):
product_id = 200443
product_name = 'Iphone'
product_category = 'Electronics'
sales_quantity = 2
tax_rate = 0.05
sales_price = 1200
profit = 70
loss = 0
sales_margin = 0.1
promotion = '20%Off'
promotion_reason = 'New Year'
in_stock = True
def create_schema(self):
import pandas as pd
tableschema = pd.DataFrame([self.product_id,
self.product_name,
self.product_category,
self.sales_quantity,
self.tax_rate,
self.sales_price,
self.profit,
self.loss,
self.sales_margin,
self.promotion,
self.promotion_reason,
self.in_stock])
tableschema.drop(labels = ['Value'], axis = 1,
inplace = True)
return tableschema
We have added product details of an example product (in this case, an iPhone) and the variables are a combination of different data types – string, integer, float, and bool. We will define the method create_schema, which imports the pandas library to create a DataFrame that gives a table-like structure to the variables and returns the data frame as a table schema.
- Now, consider a scenario where the metaclass is not added to the preceding code. Calling the product_name variable would have resulted in the following:
objarizona = Arizona()
objarizona.product_name
'Iphone'
- Since we have added the metaclass in the preceding Arizona class definition, calling the product_name results in the following:
objarizona = Arizona()
objarizona.product_name
{'ColumnName': 'Product_name',
'Value': 'Iphone',
'Type': 'VARCHAR',
'Length': 6}
- Similarly, we can look at the results of a few other variables as follows:
objarizona.product_category
{'ColumnName': 'Product_category',
'Value': 'Electronics',
'Type': 'VARCHAR',
'Length': 11}
objarizona.sales_quantity
{'ColumnName': 'Sales_quantity', 'Value': 2, 'Type': 'NUMERIC', 'Length': 1}
objarizona.tax_rate
{'ColumnName': 'Tax_rate', 'Value': 0.05, 'Type': 'NUMERIC', 'Length': 4}
- Using the metaprogrammed class variables further, we have defined the method create_schema to return a table schema:
objarizona.create_schema()
We get the following table, which includes all of the variables defined in the class:
Figure 4.12 – Output of the method create_schema
These are some examples of how metaclasses can be used in developing applications. Metaclasses can further be used in more complex scenarios such as automated code generation and framework development.
Summary
In this chapter, we have learned how to create metaclasses and some applications of metaclasses.
We then saw how to switch metaclasses, reuse the functionalities, and how to implement inheritance on classes that use metaclasses. Finally, we also saw how to manipulate the variables of metaclasses further.
All of these concepts are part of Python metaprogramming and they are used to change the behavior of a class externally and without impacting the internal functionalities of the class itself.
In the next chapter, we will be looking at the concept of reflection with different examples.