Puppet Augmented: Using 3D technology in puppetry practise

PUPPETRY PRACTISE

Since the beginning of the 21st century, technology has become increasingly accessible. Currently, we can observe a trend in the use of various digital technologies on stage – robotic arms, virtual reality, interactive projections etc. In the following article I will focus on examples from my own practice, in which I will present the potential use of 3D technologies (3D scanning, modelling and animation) in puppetry. I will focus on their use in directing and puppeteers’ practice.

NEW APPROACHES IN PUPPET CONTROL

In the winter term 2021 I led a seminar for students of the Department of Puppetry at the Academy of Performing Arts in Bratislava. Our goal was to experiment and discover new possibilities of puppet control with the use of interactive and 3D technologies. The group consisted mainly of puppetry students, the minority were students of puppetry technology and scenography. When working with a group in the mentioned composition, it is more efficient to work actively, the so-called “hands-on,” as opposed to conducting long conceptual conversations. Working with interactive technologies proved to be very motivating and inspiring for the participants, as it offered them an immediate response to their actions. We worked with the Makey-Makey development board, which is controlled by the ATMega32u4 processor. It is a device that serves as a programmable microcontroller that can control the flow of electricity in a circuit. The advantage of the selected microcontroller is that it already has all the necessary resistors and other electrical components pre-installed, so no soldering is required. It also has pre-programmed outputs that can control individual keyboard keys on a personal computer. Its biggest advantage is the accessibility, as it is ready for immediate use and does not require pre-programming, as is the case with models of other Arduino microcontrollers.
Together we created simple electrical circuits into which we also wired everyday objects which are good conductors of electricity, for example an apple and aluminium foil. The Makey-Makey microcontroller works with a maximum electrical voltage of 5V, and therefore poses no risk.

In addition to the interactive level, we also experimented with the integration of sound. Our goal was to start a sound or a recording from a computer when we touched a specific object. In Ableton Live, we assigned individual keyboard keys to specific sounds. In this way the real object connected to the electrical circuit virtually turned into a key on the computer keyboard. The students’ bodies also gradually became involved in this setting, as for the circuit to be closed and functional, they had to touch each other, bridging the physical and the digital spaces. This highlighted the interactive and immersive potential of digital technologies in stagecraft, going beyond pushing buttons and switching switches.
We started with setting up computer keys to match piano keys. Using the method described above, the group then tried to play the piano. Later, we exchanged the musical instrument for recorded excerpts from favourite texts, which the participants of the seminar had chosen in advance. It was possible to mix individual recordings in real time, thus creating a personalized DJ-set.
Thanks to the great enthusiasm of the group, we continued to work with this technology and later added visual layer to our experiments. Following a setup similar to that of the DJ set, the group controlled a brush in Microsoft Paint, a basic raster image editor. Each person was limited to controlling a single function of the brush – for example, moving it to the side, making it touch the canvas, and so on. The real-time feedback and the necessity of mutual coordination turned the exercise into something akin a real-time game. The group started to spontaneously set themselves goals, such as “painting a house.” In turn, agreed on what the house consisted of – a square and a rectangle for example. Afterwards they divided the task into concrete steps and formulated execution plans, such as how they would move with the brush on the canvas, at what time it should touch it, and so on. In doing so the group intuitively emulated the procedures that a programmer abides by, e.g., to divide the larger task into smaller ones and to be particular about each step in the sequence. During this emerged a natural harmony between the technical and the performative.
In another exercise we created 3D scans of our own heads, using the Kinect v1 an infrared sensor and Skanect a 3D scanning software. Using the technique of photogrammetry, we created 3D models of seminar participants. “Photogrammetry encompasses methods of image measurement and interpretation in order to derive the shape and location of an object from one or more photographs of that object. In principle, photogrammetric methods can be applied in any situation where the object to be measured can be photographically recorded. The primary purpose of a photogrammetric measurement is the three-dimensional reconstruction of an object in digital form (coordinates and derived geometric elements) or graphical form (images, drawings, maps).”[1] This method is highly applicable in puppetry practice because it can transfer physical objects into digital space, making it possible to further work with the data and create various animations or even animate them live, which is the direction we want to explorein our next project.
For 3D scans, I created a body model in Houdini SideFX, Maxon Cinema 4D and MakeHuman, which I applied to each scan. The first phase of experimentation focused on creating our own animation, based on one of the participants. To do so, we used the online animation database
Mixamo. In addition, this tool automatically creates an invisible skeleton model (so-called rigg), thanks to which individual parts of the body can be animated. The participants ́ first task was to find several animated movements from the Mixamo database. The main purpose of this exercise was for them to acquaintance themselves with the ways they might be able to work with the digital material. The post-dramatic potential of the created 3D characters followed. Are storytelling methods changing when working with virtual imagery? I prepared nine pictures for the group, capturing one 3D model in different poses. The task was to create a storyboard from the material in a short time, which captures a random story. Subsequently, the group had to record it as a soundtrack. The aim of this exercise was to find out how inspiring it is for the group to work with purely virtual material. It resulted in improvised several recordings. It was interesting to see that the participants at first prepared a basic outline of one story in advance.
When finished, I left the recording on and the group went from a partially prepared story to free improvisation, which took the form of short sketches. The expressiveness of the individual poses was so strong that the group went from pre-made to improvised.
Originally it was planned to continue working with real-time 3D character animation using the Kinect v2 sensor and TouchDesigner software. We had to postpone this part of the study, due to anti-pandemic which prevented us from meeting in person.

Our approach of working with digital technologies was very experimental, which gave us the freedom to explore its potential for use in puppetry. Both directions of our explorations, have proven to be highly applicable and are worth exploring further. Incidentally, the Schubert Theater in Vienna recently presented the Future Lab project, that also examines the use of technology on stage and addresses similar issue.

USABLE TECHNOLOGIES IN PUPPET PRACTICE

The Distance-Dissonance project explored the boundaries of interactive costume as well as the use of interactive programmable components as part of the costume. The work consists of an object that has ultrasonic sensors built into it. In the Distance – Dissonance project there were three important aspects that we had considered: haptic, auditive and performative. We worked with haptics on different levels. At the beginning, we applied a ball robot on a textile. The robot moved freely, marking its trajectories. This was a physical contact between the robot and the textile. We followed the trajectories and applied a wire that stretched the whole textile forming a fluid shape. This combination has created two different tactile surfaces. Thirdly, the human touch is needed to turn on the whole electronic part of the costume.
Sensors built into the costume are capturing distance values of objects in the space around and transforming them into sounds of different frequencies. It was very important to us to intertwine textile with auditive elements. Different soundscapes are constantly surrounding us. What would it mean if our clothing or costume was audibly sensible? How can built-in sensors enhance our senses?
Performative aspect was also very important to us. We were consciously working with the notion of shifting the role of a visitor. Suddenly, by wearing a costume, a visitor becomes a performer. He/she can walk freely and explore the surroundings. What accompanies the visitor/performer is the soundscape from the costume. In the project Distance – Dissonance we applied methods of performative practices, sound design, digital art (e-textiles & interactive electronics) and 3D-printing. These various fields enabled us to start exploring the potential of interactive textiles and, especially, it allowed us to explore the fragile relationship between textiles and electronics.

This project was created as a visual interactive installation, but its elements can be very accurately transferred to a theatre costume or puppet. The connection of textiles and electronics has great potential for use because different types of sensors expand the controlling possibilities of the puppeteer.

PUPPET AUGMENTED – WHERE DOES IT COME FROM?

Puppet Augmented is an inclusive term that includes puppets connecting the physical and virtual worlds. It is based on the concept of augmented reality. „Augmented Reality (AR) refers to a live view of physical real world environment whose elements are merged with augmented computer-generated images creating a mixed reality. The augmentation is typically done in real time and in semantic context with environmental elements. By using the latest AR techniques and technologies, the information about the surrounding real world becomes interactive and digitally usable.“[2] In augmented reality, three aspects are important. The first is a combination of the real and virtual worlds. The user is still aware of his/her/their physical presence in the real world, because to display augmented reality it is necessary to have a device (tablet, mobile phone, etc.). Thanks to this, the user perceives the real world around and its 3D extension in the used device. Other important aspects are real-time interaction and accurate perception of physical and 3D objects. To run augmented reality, the user first needs to scan a special sign – it can be a QR code, a real object, or the user must be in a well-defined location. I see several parallels in the use of 3D technology in puppetry.
The first is live interaction. When controlling the puppet, there is a direct relationship between the puppet action and the material ́s / puppet ́s reaction. The performed movement is immediately transferred to the body of the puppet. There is a similar interaction when controlling a virtual puppet. The puppet performs the movement, which is reflected in the animation of the virtual puppet. It also combines the physical world with the virtual. The virtual puppet has a defined space in which it is located. We can’t work with it everywhere. The area on which, for example, the virtual puppet is projected must be prepared in advance, because it is necessary to define in advance the dimensions of the 3D model, with which the puppet actors will later work on the stage. In our case, a 3D scan of the participants became a virtual puppet. The 3D character was only in digital form. Digital technologies expand our possibilities for puppet-controlling as well as thinking about puppetry. Just as in augmented reality we perceive our real and also the expanded world, even with augmented reality we are aware of both worlds, but with the difference that the world of puppet augmented offers us many more possibilities to explore and is for us to a large extent terra incognita.
So let to discover new worlds!