Karim and Stevie J discovered each other in the wild west of the digital art platform Hic et Nunc back in 2021. Stevie was blown away by one of Karim’s pieces, Little Red Riding Hood, and a creative friendship was immediately formed.
It would only be a matter of time before Stevie and his musical partner Jules Maxwell collaborated with Karim to provide a soundtrack to his piece The Trial – a dark interactive interpretation of chapter one of Franz Kafka’s novel of the same name, in late 2021. By this stage, Karim had produced a number of these immersive worlds inspired by horror stories and dark tales, and he had the idea to tie them all together in some way – thus The Cursed Library was born.
Stories and ideas
In The Cursed Library, there are six worlds in total. Three of them come directly from Karim’s questionable imagination: The Ogre of the Hill, The Hidden Wolf and The Night Taking Over The Kingdom.
The others are Karim’s personal interpretations of existing books or stories: Little Red Riding Hood, Bluebeard (Charles Perreault) and The Trial (Franz Kafka).
Each world was modelled as a space with multiple viewpoints which needed to be thought out carefully. It was important to find cinematic views of each scene from different angles. The 3D aspect made the task quite complex as “cheating” was simply not an option – each viewpoint had to tell a slightly different story while looking spectacular in its own right.
“It was important to make every scene look cinematic from every angle."
The Soundtrack
Having already composed soundtracks for two of Karim’s pieces, Stevie and Jules set about writing and recording music for all of the remaining worlds, and finding a way to incorporate them into the piece in a way that enhanced the narrative and atmosphere.
“We recorded improvisations while watching screen capture recordings that Karim sent us, just like scoring a movie."
Stevie deconstructed the parts into much smaller elements – loops and one-shot sounds – that could be composed in different ways and shared between different scenes to keep the file size as small as possible and let us build all the scenes for all the worlds from configuration files. The biggest challenge on the audio side was keeping the file size down, since the 3D model files were already pretty hefty and, after the initial loading stage, we never wanted to show a “Loading” message or wait for audio to load.
To achieve this, Stevie built a custom audio engine for the piece, based on the open-source WebAudio library ToneJS. The concept was fairly simple – to set up groups of sounds that shared as many elements as possible, using a crossfade technique to smoothly change the ambient sound as the viewer moved around each world.
Occasionally the engine would also choose random options from “pools” of sounds at the various camera viewpoints so it wasn’t always the same.
For example, in the Kafka world, there are 6 gnarly tremolo guitar lines in a kind of Marc Ribot / Tom Waits style. If a scene has a guitar in it, the engine chooses from the pool of guitar sounds for that scene. Once the sound has played, it can chain on to the next sound, play another random one or just stop. Having various options in the configuration files, meant we had quite a bit of flexibility to try out different arrangements without ever having to create any new audio files.
The engine became more and more advanced as the project went on, eventually providing options for voiceovers, multiple music layers, background world sounds, gesture audio, effects and a white noise generator.
One of our favourite uses of the engine is the gory layering of sound effects to create rhythms in The House of Bluebeard. Stevie took sound effects like chains, axes, blood dripping and his own evil grunting sounds to create a morphing rhythmic loop that accompanies the viewer through the visuals of the house. Accompanied by Jules and Stevie’s drunken accordion song, it makes for a pretty unusual soundtrack.
Narration
Karim thought that the project would benefit from some narration, or at least some words on the screen, as the whole project was about books. Not being a native English speaker, he turned to the internet’s favourite poet - ChatGPT - to create a poem for each scene.
“ChatGPT is amazing at many things, but alas poetry is not one of them."
To a native English speaker, the poems written by ChatGPT were terrible, with no musical rhythm and some excruciatingly angular rhymes! Stevie insisted that the only way to do it was the old-fashioned way – sitting in his favourite cafés of south east London and writing them all by hand in his notebook. He decided to make them all four lines long, the bare minimum to set the scene so the graphics and music could do the rest.
Intro poem – Chat GPT
In the cursed library, beware!
Scary books are everywhere,
Spectres haunt the dusty shelves,
Whispers fill the spooky delves.
Each book holds a scary tale,
With curses and spells that never fail,
But if you're brave and do not fear,
You'll find adventure lurking near.
So come and see the cursed hall,
But watch your step, don't trip or fall,
For in this library, the scary books,
Will leave you with spooky looks.
Intro poem – Stevie J
In this library dark and cursed
lies more than meets the eye
Portals six are hidden here
To find them you must try
Stevie recorded the spoken poems in his deep Northern Irish voice, adding a touch of formant shifting to push the vocals into pure horror territory. The final decision was to have both text on the screen and the spoken word as audio since English spoken in a Northern Irish accent may as well be some ancient language from Lord of The Rings; the Belgian test listeners had no idea what he was saying. Stevie did also provide the option to remove the voiceovers for a “minimal experience” (ie. to shut himself up once and for all).
3D tricks
Karim: “The idea of a library that hosts the 3D worlds hidden in its shelves seemed obvious to me. I wanted an old place with a dense atmosphere, dust and scary shadows.
The library was initially modelled as a confined space, but I ended up orienting the shelves to the outside as it was much easier for the user to navigate around. Sometimes I had to make hard decisions and occasionally ended up trading the sense of claustrophobia for usability.”
The library is filled with hundreds of books, arranged in controlled chaos. To do so, Karim roughly placed the books on the shelves and used Blender's physics simulation to stack them naturally. That saved a few days of manually placing each book and avoiding unpleasant intersections.
Most of the books were simple cubes, but for others that needed to be opened and closed, Karim created a more sophisticated model driven by a few "bones". These books were animated in Blender and share the same geometry, but are textured dynamically with ThreeJS.
Lighting
For every single WebGL project, there is that same question that raises itself over and over regarding lighting:
“Should we go with dynamic lighting, calculated at runtime, which allows switching between different ambiences at will, or should we stick with baked (precalculated) lighting which guarantees a boost of performance and tightly controlled art direction?”
There's this thing we call "Belgian compromise" that Karim learned to embrace, albeit not without difficulty. For this project, the atmosphere in the library was key. Karim wanted living lights and shadows, as if they were secondary actors. However, getting a real-time shadow to render in WebGL was too demanding in terms of performance especially for some devices, so dynamic lighting wasn’t an option.
The solution he ended up using was baking lightmaps (precalculated rendering) of three different lighting setups.
The fact that this whole project was mainly black and white allowed these three lightmaps to be combined in one texture, simply by encoding each lightmap in a different RGB channel.
This is how it looks in Blender:
In ThreeJS, a custom shader translates the values for each RGB channel to grayscale and mixes them in a pseudo-random way to simulate the dynamic lighting.
Textures
One signature of Karim’s illustrations is a grainy hatched effect which comes from his early days illustrating children's books with scratchboards.
Initially inspired by the work of Jaume Sanchez and his cross-hatching experiments applied on screen space (https://spite.github.io/sketch/cross-hatch-i/index.html) Karim wrote a custom shader that mimics that scratchboard effect, but this time applied on object space which permitted each element to have a different texture, grain density and lighting reaction.
Portals
Making portals is probably one of the most trending subjects among creative developers nowadays. It’s a tricky effect that involves a good understanding of ThreeJS, its renderer, render targets and how cameras work.
The principle is quite easy to understand, though. Let’s say on one side we have a main scene that contains a portal made of a simple plane and a basic material, and on the other we have a virtual scene that we want to show through the portal.
In each frame, the trick is to:
- Sync the cameras of both scenes, make them share the same position and rotation;
- Make the virtual camera frame the corners of the portal (this part is the hardest to grasp, but fortunately ThreeJS is kind enough to offer a CameraUtils.frameCorners() function that calculates this for us);
- Render the virtual scene and draw the result into a render target;
- Use that render target as a texture for the portal;
- Render the main scene, now it looks like the virtual scene is inside the main scene;
- Repeat for each frame
This is a demo on codepen of a simpler piece that showcases the portals effect.
[codepen : demo of infinite portals]
Conclusion
The Cursed Library was a challenging project to make for a multitude of reasons, not least the fact that statistically speaking, passion projects have very low chances of ever seeing the light of day. Nevertheless, Karim and Stevie succeeded in finding a smooth way to work together remotely from different countries, coding with four hands and catching up about twice a month to keep things moving. The project ended up being a spectacular blend of immersion, visuals and music.
Frontend Frameworks and Libraries: ThreeJS, GSAP, ToneJS
Tools: Blender, Photoshop, Visual Studio Code, Github, paper and pencils
Company Info
EPIC is a digital agency co-founded by Karim. It offers strategy, development and branding services to its clients. It also traditionally creates a space besides its clients’ work to experiment and push further the latest technologies. The Cursed Library is one of its side projects.
Karim Maaloul is a Creative Director, illustrator and coder. Despite the darkness of his illustrations, he’s surprisingly an optimistic and cheerful guy… most of the time.
Stevie J Jones is a musician and creative coder based in London. He has recorded and toured with artists such as AIR, Brian Eno, Bryan Ferry and David Holmes. He’s happiest in the place where code meets music.
Jules Maxwell is a composer and musician from Northern Ireland, and has been a close friend and collaborator of Stevie’s for many years.