Pitfalls and solutions when building Metal Shaders for Core Image Kernels
Introduction
Core Image is a framework provided by Apple to help developers apply graphic effects to static images or video streams. Along with multiple built-in filters provided by the framework, it also allows us to write shaders to manipulate the image pixels, which is the same as fragment shaders in other graphics frameworks, utilizing the GPU to process efficiently.
Core Image is a framework with a long history. In the past, you would write your custom shaders using the Core Image Kernel language, as described in this archived documentation.
The Core Image Kernel Language is a simple shading language that you use to add custom image processing routines to a Core Image pipeline.
Writing in Core Image Kernel Language can be challenging since you are writing it as plain text in code, and you may not easily find the errors until you compile it at runtime. So in recent years, Apple has introduced a new way to implement a custom shader for Core Image, and that is through Metal shaders.
Core Image with Metal shader
There are some advantages when you implement your shader in Metal (Metal Shading Language, known as MSL):
You can get tips and auto-generation when writing Metal shaders, as it’s actually a C++ based language.
Errors or warnings can be found as you write them. So no need to run your code to debug the grammar error when writing your MSL.
Metal shaders are in dedicated
.metal
files like Swift or other languages do, which allows you to manage them.The Metal shader will be compiled at compile time. This is different compared to the Core Image Kernel Language, which will be compiled at runtime and may introduce time overhead if you have lots of shaders to be compiled.
There are a few WWDC videos introducing this new update:
https://developer.apple.com/wwdc21/10159
https://developer.apple.com/wwdc20/10021
To summarize the two videos above, you can build your Metal-based Core Image kernels in two ways:
Write your shader in a
.ci.metal
file and utilize Xcode’s build rules to manually build the metal files.Utilize the Metal dynamic library and use the
[[stitchable]]
attribute to avoid providing your custom build rules in Xcode.
However, since the videos above are not up-to-date, and due to the lack of documentation in text, there are some changes and pitfalls you should be aware of.
To learn more about the Core Image Kernel Language, Metal Shader Language, and Metal Shader Language for Core Image, please refer to these documentations:
https://developer.apple.com/metal/Metal-Shading-Language-Specification.pdf
https://developer.apple.com/metal/CoreImageKernelLanguageReference11.pdf
https://developer.apple.com/metal/MetalCIKLReference6.pdf
Build Metal shaders with Dynamic Library
Building metal shaders with Dynamic Library is a recommended way. However, there are something you should notice:
This feature requires Metal 2.4. That’s not an issue if you always use the latest Xcode with latest build tools to build your project.
The minimum supported OS are iOS 15.0 and macOS 12.0.
The Metal Dynamic Library is supported on some specified GPU, which is often ignored by developers since it’s not matched with the minimum OS versions.
More specified, the Metal Dynamic Library feature supports Apple GPU family 6 and above:
You can find this table in this address:
https://developer.apple.com/metal/Metal-Feature-Set-Tables.pdf
The minimum GPU family support for Dynamic Library means that a specified device, even if it can be upgraded to the newest iOS (as of now, iOS 17.3), cannot use the Dynamic Library. Since the compilation and linking of Metal shaders are device-independent, you will not notice the compatibility issue until you run your app on a specific device.
For example, although the iPhone X and iPhone Xs are compatible with iOS 16.0, they use the A11 and A12 chips, which belong to the Apple 4 and Apple 5 GPU families, respectively. Running Metal shaders with Dynamic Library can result in unexpected behavior: in my case, no errors were thrown in the code or error messages in the console, but the rendered result of the image using the shaders was always black.
You can find the Apple GPU family in the Metal-Feature-Set-Tables.pdf above.
In conclusion, before considering building Metal shaders with Dynamic Library, you should investigate about the minimum devices you are supporting and don’t assume that devices with latest OS version can always benefit the new features.
Best practice to compile and link Metal Shaders for Core Image
If you have watched the video above, by not using Metal with Dynamic Library, you will have to write your custom build rules to build the shaders, which is no need currently in 2024.
I will run you through the process on writing and building Metal Shaders for Core Image step by step, assuming you already have the experience on how to use Core Image and Metal to render and display images on screen.
The following steps are performed in Xcode 15.2.
Write your metal shaders
First, you should put your metal shaders in your main project that’s about to build, instead of a Swift Package or any other forms. That’s because the metal files in Swift Package will be compiled and linked using the Swift Package’s system.
I am aware that there are some cases where you may want to put the metal files into a dedicated library, but currently, I still find no way to do that.
Create a file that ends with the .metal
extension, and write your code:
In this example, I simply sample the original color and return it.
Some keys to be noticed:
You should include the header file of Core Image.
Use the sampler and destination with namespace of
coreimage
in parameters.
The function should be start with
extern "C"
.
Set custom compiler and linker flags
Then, you should specify the -fcikernel flags to use for Metal Compiler and Metal Linker. Go to the Build Settings in the Xcode, and append the -fcikernel flag to the Other Metal Compiler Flags and Other Metal Linker Flags. No custom Build Rules are needed any more, which is described in the WWDC video and it’s outdated.
The similar descriptions about setting flags for compiler and linker can be found below.
The
init(functionName:fromMetalLibraryData:)
constructor of CIKernel class:
https://developer.apple.com/documentation/coreimage/cikernel/2880194-init
The Xcode Integration part in Metal Shading Language for CoreImage Kernels:
https://developer.apple.com/metal/MetalCIKLReference6.pdf
In the latest Xcode 15.2, both flags should be -fcikernel
and it will throw a warning as build if you use -cikernel
for linker flag.
Load CIKernel with metal lib
Xcode will compile and link your Metal shaders into a file named default.metallib
. Here is the code to load metal shader with a specified function name.
To create the CIKernel for the example above:
Then you can use the CIKernel
to process your CIImage.
Conclusion
I always like the old and archived documents from Apple because it provides much detailed information about a framework or a technology. However, in recent years, the Apple’s new documentations are lacking so many tech details and hard to find. Moreover, some details are hidden in the WWDC videos and it’s not easy to retrieve by text.
I hope this article will help you with your image processing app with Core Image.