Introduction

Cython is a programming language that makes writing C extensions for Python as easy as Python itself. It lets us write Python code that calls back and forth from and to C/C++ code natively, and compiles to efficient C code usable as a Python extension module.

At Intersec , we use Cython in lib-common , our open-source C library. One of its key components is IOPy, the Python binding for our IOP serialization framework (similar to Protocol Buffers). IOPy dynamically creates Python classes from IOP type definitions at DSO load time, and each class needs to hold a pointer to its C type descriptor for fast serialization.

This blog post explores the issues and techniques used to have class attributes on Cython extension types.

We use a simple “greetings” module example: a C array of greeting classes (name + sentence), and the goal is to create Python classes dynamically, each with fast access to its C sentence pointer. The code of the examples is available on Intersec’s Github .

Extension Classes in Cython

A Cython cdef class (also called an extension type) compiles down to a CPython type object backed by a C struct. Here is a minimal example:

# simple-cdef-class/greetings.pyx

cdef class Hello:
    cdef str name

    def __init__(self, name: str = 'World'):
        self.name = name

    def greet(self) -> str:
        return f'Hello {self.name}!'

# simple-cdef-class/example.py

import greetings

hello = greetings.Hello()
print(hello.greet())       # Hello World!

hello2 = greetings.Hello('you')
print(hello2.greet())      # Hello you!

Under the hood, Cython generates a C struct for each cdef class. For Hello, the generated C code looks like:

struct __pyx_obj_9greetings_Hello {
  PyObject_HEAD
  PyObject *name;
};

This is a standard CPython object: PyObject_HEAD provides the reference count and type pointer, and name is stored as a direct struct field (not in a __dict__ like regular Python classes). This is why cdef attributes are fast: they are direct C struct accesses, not dictionary lookups.

The Problem: Storing Class-Level C Data

Now let’s say we have greeting classes defined in C:

// common/greetings.h
typedef struct greeting_class {
    const char *name;
    const char *sentence;
} greeting_class;

extern const greeting_class greetings[];

// common/greetings.c
const greeting_class greetings[] = {
    { "Hello", "Hello" },
    { "GoodMorning", "Good morning"},
    { NULL, NULL },
};

We want to create Python classes dynamically from this array, where each class knows its sentence. The sentence is a property of the class, not of individual instances.

Static Module-Level Variables

The simplest approach is to store each sentence as a module-level cdef variable:

# static-attribute/greetings.pyx

cdef const char *_hello_sentence = greetings[0].sentence

cdef class Hello:
    cdef str name

    def __init__(self, name: str = 'World'):
        self.name = name

    def greet(self) -> str:
        return f'{_hello_sentence.decode("utf-8")} {self.name}!'

This works, but it requires writing one class per greeting at compile time. Classes cannot be created dynamically from the C array, each class is hard-coded with its own module-level variable.

Dictionary Map

We can create classes dynamically and use a dictionary to map each class to its sentence pointer:

# dictionary-map/greetings.pyx

cdef dict _greet_map = {}


cdef class GreetHolder:
    cdef const char *sentence


cdef class Greet:
    cdef str _name

    def __init__(self, name: str = 'World'):
        self._name = name

    def greet(self) -> str:
        cdef GreetHolder holder = _greet_map[type(self)]
        cdef const char *sentence = holder.sentence
        return f'{sentence.decode("utf-8")} {self._name}!'


cdef GreetHolder _greet_holder
cdef int i = 0
while greetings[i].name != NULL:
    _cls = type(greetings[i].name.decode('utf-8'), (Greet,),
                {'__module__': 'greetings'})
    _greet_holder = GreetHolder.__new__(GreetHolder)
    _greet_holder.sentence = greetings[i].sentence
    _greet_map[_cls] = _greet_holder
    globals()[greetings[i].name.decode('utf-8')] = _cls
    i += 1

This works and handles dynamic creation, but has downsides:

• Dictionary lookup on every call: _greet_map[type(self)] is a hash table lookup every time greet() is called. As it is a hot path, this overhead matters.
• External state: The mapping lives outside the class. It is another thing to maintain and can get out of sync.

What we really want is to store the const char *sentence on the type object itself, as a C struct field, with direct pointer access.

What is a Type?

In Python, everything is an object, including classes. When we write class Foo: pass, the Foo class itself is an object. And like all objects, it has a type. The type of a regular class is type:

>>> class Foo: pass
>>> type(Foo)
<class 'type'>

A metaclass is the type of a class. type is the default metaclass. Custom metaclasses can be created by subclassing type. Classes built with that metaclass then carry additional attributes or behavior.

In Cython, we can do the same with cdef class:

cdef class GreetType(type):
    cdef const char *sentence

This creates a metaclass that extends type with an additional C field. Classes whose type is GreetType will have a sentence field stored directly on the type object’s C struct.

The Metaclass Solution

Here is the full metaclass approach:

# simple-metaclass/greetings.pyx

cdef class GreetType(type):
    cdef const char *sentence


cdef class Greet:
    cdef str _name

    def __init__(self, name: str = 'World'):
        self._name = name

    def greet(self) -> str:
        cdef GreetType cls = type(self)
        return f'{cls.sentence.decode("utf-8")} {self._name}!'


cdef int i = 0
while greetings[i].name != NULL:
    _name = greetings[i].name.decode('utf-8')
    _cls = GreetType.__new__(GreetType, _name, (Greet,),
                            {'__module__': 'greetings'})
    (<GreetType>_cls).sentence = greetings[i].sentence
    globals()[_name] = _cls
    i += 1

The generated C struct for GreetType extends PyHeapTypeObject (the base type for all heap-allocated type objects) with our custom field:

struct __pyx_obj_9greetings_GreetType {
  PyHeapTypeObject __pyx_base;
  char const *sentence;
};

When greet() accesses cls.sentence, Cython generates:

/* "greetings.pyx":23
 *     def greet(self) -> str:
 *         cdef GreetType cls = type(self)
 *         return f'{cls.sentence.decode("utf-8")} {self._name}!'
 */
__pyx_t_1 = ((struct __pyx_obj_9greetings_GreetType *)
              Py_TYPE(__pyx_v_self))->sentence;

This is a direct pointer dereference: Py_TYPE(self)->sentence. No dictionary lookup, just one pointer dereference from the object to its type, then a struct field access. This is as fast as it gets in CPython.

In IOPy

This is exactly the pattern IOPy uses . The _InternalStructUnionType metaclass stores the IOP type descriptor (iop_struct_t *) for each dynamically created struct class:

All IOPy struct classes (A, B, …) are instances of _InternalStructUnionType. Each holds its own iop_struct_t *desc pointing to the C descriptor. When serializing an object, IOPy accesses type(obj).desc, a single pointer dereference, exactly like our GreetType.sentence example.

The Inheritance Problem

This metaclass approach has a subtle but dangerous flaw when Python users subclass our dynamically created classes.

Consider this user code:

# metaclass-crash/example.py

import greetings

class FriendlyHello(greetings.Hello):
    pass

friendly = FriendlyHello()
friendly.greet()  # Segmentation fault!

What happens here?

1. greetings.Hello was created with GreetType as its metaclass, and sentence was set to "Hello".
2. When Python creates FriendlyHello(greetings.Hello), CPython’s type machinery determines the metaclass: since Hello’s metaclass is GreetType, FriendlyHello’s metaclass is also GreetType.
3. But FriendlyHello is a new GreetType instance. Its sentence field is zero-initialized, it is NULL.
4. greet() calls type(self).sentence which is NULL. Decoding a NULL const char * crashes with a segfault.

$ ./example.py
Hello World!
type(Hello)         = <class 'greetings.GreetType'>
type(FriendlyHello) = <class 'greetings.GreetType'>
Calling FriendlyHello().greet()...
Segmentation fault (core dumped)

Both Hello and FriendlyHello share the metaclass GreetType, but only Hello had its sentence set during registration.

The MRO Workaround

One fix is to walk the MRO (Method Resolution Order) to find the nearest parent with a sentence set:

# mro-metaclass/greetings.pyx

cdef inline const char *_get_sentence(GreetType cls):
    """Get sentence for a GreetType class, walking MRO if needed."""
    # Fast path: sentence is set on this class directly
    if cls.sentence != NULL:
        return cls.sentence

    # Slow path: walk MRO to find first parent with sentence set
    for parent in (<object>cls).__mro__:
        if (isinstance(parent, GreetType) and
                (<GreetType>parent).sentence != NULL):
            return (<GreetType>parent).sentence

    raise TypeError("No greeting sentence found in MRO")

Now FriendlyHello().greet() works: _get_sentence sees that FriendlyHello.sentence is NULL, walks the MRO, finds Hello with sentence set, and returns it.

The fast path (sentence is set directly) handles the common case, classes created by our registration code. Only Python subclasses trigger the MRO walk.

This is the approach IOPy uses today. For most use cases it is sufficient, because, in practice, no Python subclasses of IOP types exist in our codebase.

The Meta-Metaclass (Type of Type)

There is a more complex solution: instead of one shared metaclass for all greeting classes, create a per-class metaclass using a meta-metaclass.

# meta-metaclass/greetings.pyx

cdef class GreetMetaType(type):
    cdef const char *sentence

GreetMetaType is now a meta-metaclass, a type whose instances are themselves types (metaclasses). Each greeting gets its own metaclass:

cdef int i = 0
while greetings[i].name != NULL:
    _name = greetings[i].name.decode('utf-8')

    # Create a per-greeting metaclass
    _metacls = GreetMetaType.__new__(
        GreetMetaType, _name + 'Type', (type,),
        {'__module__': 'greetings'},
    )
    (<GreetMetaType>_metacls).sentence = greetings[i].sentence

    # Create the greeting class using this metaclass
    _cls = _metacls.__new__(
        _metacls, _name, (Greet,), {'__module__': 'greetings'},
    )
    globals()[_name] = _cls
    i += 1

Now the type chain looks like:

type(Hello)         = HelloType        (a GreetMetaType instance, sentence="Hello")
type(type(Hello))   = GreetMetaType

When a user subclasses Hello:

class FriendlyHello(Hello):
    pass

CPython resolves FriendlyHello’s metaclass to HelloType (inherited from Hello). HelloType already has sentence = "Hello" set. So type(type(FriendlyHello)).sentence just works, no MRO walk needed.

Access in greet() is:

cdef GreetMetaType metacls = type(type(self))  # Py_TYPE(Py_TYPE(self))

Two pointer dereferences, still O(1). The generated C is:

Py_TYPE(Py_TYPE(self))->sentence

$ ./example.py
Hello World!
Hello you!
Good morning World!
Good morning you!

type(Hello)         = <class 'greetings.HelloType'>
type(type(Hello))   = <class 'greetings.GreetMetaType'>

type(FriendlyHello) = <class 'greetings.HelloType'>
Hello World!
Hello you!

This was the old IOPy approach, using _InternalStructUnionType as a meta-metaclass that created per-class metaclasses (A_meta, B_meta):

This approach was removed from IOPy in commit d7b249499b in favor of the simpler single-level metaclass (see above). The reason: the meta-metaclass approach creates three objects per IOP type (the class, its metaclass, and a proxy), whereas the single-level approach only needs one. This resulted in ~30% higher memory usage for IOPy, which was not justified given that the MRO workaround handles the rare case of Python subclasses adequately.

Conclusion

The key insight is that Cython’s cdef class Foo(type) lets us extend the C struct of type objects, giving us zero-overhead class-level storage. This is the same mechanism CPython itself uses for built-in types; we are just using it from Cython instead of writing raw C.

Disclaimer: Designed by hand, written jointly with an LLM, reviewed by humans.