The department anointed Shibuya as a visiting professor following the laboratory launch. He plans on working on a myriad of projects related to androids and music alongside frequent collaborators Hiroshi Ishiguro, a researcher, and Shintaro Imai, an electronic musician. We explore the present state and potential of the world’s first music and android laboratory.

The birth of a laboratory shedding light on the future of art and technology

Expo ‘70 in Osaka was a happy marriage of art and technology, as symbolized by the Pepsi Pavilion made by the American avant-garde group E.A.T. (Experiments in Art and Technology), Space Theater at the Steel Pavilion directed by Toru Takemitsu and Kei Usami, and Mitsui Group Pavilion with contributors like Katsuhiro Yamaguchi and Toshi Ichiyanagi. It’s well-known that the vision and creations shown at the expo had a palpable influence on the creative scene. 

More than half a century later, Osaka will welcome its second World Expo three years from now. It is also home to a groundbreaking laboratory that sheds light on a new future of creative expression and advanced technology. 

The Android and Music Science Laboratory (AMSL), located in the Art Science Department of Osaka University of Arts, has been described as the world’s first music and android laboratory. It’s operated by Keiichiro Shibuya, the protagonist of this series, robot researcher Hiroshi Ishiguro, who developed the Alter series (the android performer of Shibuya’s android opera), and electronic musician Shintaro Imai, who has been responsible for Alter’s speaking and singing voice since Shibuya’s Super Angels (premiered worldwide in 2021). Shibuya and Imai joined Ishiguro as visiting professors at the department this April, at the same time as the laboratory’s launch. 

I attended the opening ceremony and Shibuya’s performance at the end of April to witness the laboratory and its potential. 

A room out of a sci-fi film: brand-new android Alter4 and sound equipment 

The Art Science Department at the Osaka University of Arts was established in 2017 to cultivate creators of the next generation through interdisciplinary research and education combining art and science. The dean is Norihiro Hagata, who’s been researching and implementing the merge of robots and art. The department building was designed by visiting professor and one of Japan’s leading architects, Kazuyo Sejima. The space has an open feel with beautiful, organic curves and glass windows on all sides, 360 degrees. 

The laboratory is on the basement floor of the same building. Once I walked through the entrance hall and stepped inside the laboratory, a minimal space designed by Sejima, from the interior to furniture, and the newest model of the Alter series, Alter4, awaited me. Positioned among a grand piano and synthesizers, Alter4 looked at me while moving with ease. 

The sight in front of me went beyond my everyday imagination; it was like I was in one scene of a sci-fi film. I decided to take another good, hard look at Alter4, the main character of the room. I’ve seen Shibuya’s Alter series many times from his past works, but Alter4’s expressions and movements were much more varied.
According to the information provided that day, its expressions are more diverse than the previous models because of its enhanced mimetic muscles and tongue movements. The number of joints went from 43 to 53, making it possible for Alter4 to move more dynamically. Alter4, with its increased sense of mobility, had a strong presence that made me look forward to the laboratory’s future endeavors. 

Annihilating the conventions and mediocrity of art and technology 

Shibuya, Ishiguro, and Imai will create performances and installations using Alter4 at the laboratory, and the process will be available for the Art Science students to observe or participate in, according to their skill level. It may not be easy to partake in this world-class project, but the students are bound to learn and discover valuable lessons from experiencing the radical creations paving the way for the next generation at the intersection of art and technology. 

During the panel discussion before the opening ceremony, Shibuya pointed out that although many projects blend art and science or art and technology, quite a few are boring. The art and technology experts involved in these projects usually stay within the confines of their respective fields of expertise, thus producing a conventional outcome.

Shibuya said one could destroy that monotony by getting rid of one’s technical fixations, going beyond one’s expertise, and “intruding” into each other’s fields. That’s what Shibuya does with his work, and both Ishiguro and Imai possess adaptability and intellectual bravado. There’s no way to tell the specifics of the projects that will arise from the laboratory, but they’ll have nothing to do with conventions and mediocrity.  

An impromptu jam session with a human and android

After the involved parties cut the ribbon, the laboratory was officially opened. Shibuya then performed live to celebrate the occasion with Alter4, a grand piano, and a Rev4 synth. Released in 2020, the Rev4 is the renewed version of the famous Prophet-5 analog synthesizer introduced in 1978. 

Shibuya started by playing a drone-y, abstract sound in a low tone with the Rev4. Then, Alter4 began swaying to the sound and reciting a poem in English. Checking Alter4’s reaction, he gently changed the ambiance of the sounds with harmonic bells and melancholic leads.

While Shibuya played synth pad harmonies, Alter4 began singing in a distinct voice; somewhere between machine and human. The surprising thing was how Alter4’s singing wasn’t pre-programmed; it was singing impromptu by listening to Shibuya’s performance. Towards the end, Shibuya moved to the piano and played various sounds, from contemporary tone clusters to tender, lyrical phrases and harmonies.

Even then, it continued to sing perfectly with the piano. The spectacle of Shibuya and Alter4 listening to one another and producing sounds was unmistakably a live session, and whoever watched the show could tell that the nervousness and fulfillment were unprogrammable. The celebratory event came to a close after it clearly demonstrated the possibilities of the future of androids and music. 

One of the laboratory’s destinations is Expo 2025 in Osaka, three years from now. The expo, with the slogan, “Designing Future Society for Our Lives,” has eight themes. Ishiguro is the producer of the “Amplification of Lives” theme and will be responsible for many different exhibitions and events, and one could expect to see projects made in the laboratory there as well. 

How will the horizon of art and technology, which was pioneered at Expo ‘70, be further expanded? I can’t wait to see the creations and imaginations of Shibuya, Ishiguro, and Imai three years later. 

■Android and Music Science Laboratory (AMSL)
Visiting professors: Keiichiro Shibuya, Hiroshi Ishiguro, Shintaro Imai, Kazuyo Sejima (lab design)

Furniture and interior design: Kazuyo Sejima & Associates (Kazuyo Sejima, Yoshitaka Tanase, Takayuki Furuya, Naoya Harada, Risa Matsunaga)
Acoustic consultant: Yasushi Shimizu
Construction: Taisei Corporation (Keisuke Yamaura, Shunya Matsuhisa)
Appliances: Kuhsohsha, hhstyle.com (Jun Watanabe)
Curtains: Création Baumann (Masashi Nakajima)
Lighting cooperation: MinebeaMitsumi

Android production: A-Lab
Sound system: Jiro Kubo (Acoustic Field Inc.)

Project management: Natsumi Matsumoto (ATAK)
Producer: Hisamoto Naito

In cooperation with Yamaha Music Japan, Yamaha Corporation, Native Instruments, ATAK

Keiichiro Shibuya
Keiichiro Shibuya is a musician who graduated from Tokyo University of the Arts with a B.A. in Music Composition. In 2002, he founded the music label ATAK. His diverse soundscape covers areas such as cutting-edge electronic music, piano solos, opera, soundtrack music, sound installation, and so forth. His notable works include a Vocaloid opera comprised of no people called The End (2012) and the android opera Scary Beauty (2018). In September 2020, he created the soundtrack for the film Midnight Swan and won the Music Award at both the Mainichi Film Awards and the Japan Movie Critics Award. In August 2021, his opera Super Angels had its world premiere at New National Theater Tokyo. In March 2022, he showed his new android opera, MIRROR, a collaboration between an android, Buddhist music, shomyo, and an orchestra from the UAE at Expo 2020 Dubai. He explores the boundary between humans and technology and life and death.
http://atak.jp
Photography Mari Katayama

The post The Possibilities of the World’s First Android and Music Laboratory in Osaka — “Massive Life Flow; Inside the Mind of Keiichiro Shibuya” Part 6 appeared first on TOKION - Cutting edge culture and fashion information.

Increasing Alter3’s information content, so it could sing alongside a professional opera singer

One fall day in 2020, Keiichiro Shibuya visited Kunitachi College of Music. The purpose? To create android Alter3’s new voice for his opera piece, Super Angels, showing this summer. He had been using a machine-made voice, but he felt like it wasn’t enough. I asked him why.

“In Super Angels, Daichi Fujiki, a world-renowned countertenor singer, plays the part of the protagonist. Because Alter3 is an android, using a computer-made voice was the obvious choice. I assumed it would be interesting to contrast it with a professional human singer. But my opinion changed altogether when we had rehearsals at the New National Theatre, Tokyo, on August 23rd, 2020. With [Kazushi] Ono-san [Artistic Director of the New National Theatre, Tokyo] as the conductor, I played the piano as Fujiki-san and Alter3 sang along. That was the first time Alter3 sang with a human being. I realized Alter3 didn’t have the information content that Fujiki-san possessed in his voice. I realized something had to be done to [Alter3’s voice].”

As a result of him confiding in his collaborator, associate professor at Kunitachi College of Music Shintaro Imai, he decided to try out a different method: synthesizing a human voice. To achieve Alter3’s new voice, Imai used a particular program created via Max, a programming language for audio control and synthesis invented by himself. The following is his explanation of its technical traits and aims.

“The most difficult thing about creating a voice from scratch with formant synthesis is expressing ‘instability.’ It’s pretty easy to make something with a solid pitch, but it’s tough to make something with a lot of ‘noise,’ in the broadest sense of the term. I feel like elements of noise play an integral part in Shibuya-san’s music. Many people praise his work for its clear sound, but that clarity exists because it has noise. That’s what I wanted to keep with Alter3’s voice.

“Vocaloid [a popular singing synthesizer software] creates a voice by putting together fragments of sounds. But the thing is, it gets rid of noise right at the beginning; it ultimately creates a voice that’s too clean. This time around, I recorded speaking voices and stretched that, instead of breaking it down into fragments. By changing the pitch, I could then create a melody. This is how I preserve noise and the natural instability of human voices.” 

For their first-ever test run, Shibuya recorded his voice. After about an hour, Imai took that data and converted it into a new one that could sing or perform a melody. This voice was just temporary, made so that they could check how it sounded. Nonetheless, the voice Shibuya played with his keyboard convinced me of Alter3’s evolution. It maintained a human-like texture — that “noise” — while having an artificial feel. In other words, it sounded like something right between a human and a machine: an original voice appropriate for an android. 

In pursuit of the best quality possible, Imai continues to adjust the program today. I can’t wait to see what sort of voice will come out for the first performance of Super Angels this summer. 

How did Alter3’s movements become more melodic?

Incidentally, Imai works not only on developing Alter3’s voice; he also works on its movements. He incorporated a different technological program from the previous Alter series. Here, Shibuya gives some background on how the Alter android became more melodic. 

“Alter2 wasn’t initially made for music. So, when I had the first performance of Scary Beauty at the National Museum of Emerging Science and Innovation in 2018, it was hard because the android wasn’t moving smoothly right until showtime. One day, Alter2 broke, perhaps because I had used it too much despite it not being a robot specifically designed for music. When I was thinking about what to do, mixi Inc. showed an interest. That made it possible for us to develop a new android tailored to music: Alter3. 

“Later on, the University of Tokyo and Kunitachi College of Music started a collaborative research project regarding androids and music. Imai-san was a part of the research team, and that’s how we started working together closely. He has a distinct approach, so I get very inspired by working with him. Right now, he’s putting so much effort into Alter3’s vocals. Things have dramatically evolved because of his involvement, including improvisations.” 

What technology did Imai apply to achieve Alter3’s melodic movements? According to Imai, he got the idea from how synthesizers make sounds. 

“At first, I couldn’t even imagine how I could improve the movements of Alter3. I went back and forth in my head, and after I shifted my perspective to ‘combine music and movement,’ I wondered if I could apply the way synthesizers make sound onto Alter3’s movements. Alter3 has around 40 joints, and so I tried to allot three LFOs (often used to create a rhythmic pulse in synthesizers, a Low Frequency Oscillator is an oscillator that generates low-frequency soundwaves) per joint. I synchronized the tempo of the music and set the pace of the oscillator. I adjusted the three LFOs in a way that would allow me to change how it mixes. By working on the music at the same time, I was able to create complex movements.”

The desire to subvert the typical western notion of opera

Thus, Alter3 has successfully unified music and movement. However, there’s more; the android can sing and dance impromptu. Imai emphasizes the innovativeness of this aspect and Super Angels itself:

“I presume operas with robots have existed before, but such robots would only move a certain way for a certain amount of time. In Super Angels, Alter3 can match the beat of the music and perform on the spot. In this manner, I think you can say this is an opera piece like no other.”

So, this is a Robot Opera. In A Brief History of New Music by Hans Ulrich Obrist (Jrp|Ringier Kunstverlag Ag, 2014), Iannis Xenakis talks about how he imagined an opera in which robots perform, but because of various reasons, he couldn’t actualize it. The Greek-French composer left his mark on contemporary music and electronic music and sadly passed away right when the 21st century started. I can’t help but picture an alternate timeline where this maestro could see Alter3 perform on the spot. 

In any case, there’s no doubt that Super Angels is going to raise questions against the definition of “opera.” What made Shibuya, a musician who’s been creating innovative electronic music and sound installations, want to produce an opera piece, something that’s traditionally western? Shibuya reflects on the initial catalyst that led him to opera.

“THE END [a Vocaloid opera, shown for the first time in 2012] was my first-ever opera piece. That came about after YCAM asked if I could make something. If you look at my career until that point, you’ll probably picture my installation work. But at the time, there was a sudden increase in installations with a strong marketing and entertainment feel, so I didn’t feel like doing that sort of thing (laughs). Then, I wondered, ‘If I were to make something that represents me fully, what would I make?’ I also thought about what advantages I had compared to other artists. And I concluded that perhaps the opera format might work. Of course, I wasn’t planning on making a normal one. I thought about how I could break something that’s typically regarded as western. It wasn’t about doing something completely different from western music because I’m Japanese. I want to stand on the same ground as them and then strike back. I also feel it’s important not to do the same thing as the preceding generation. 

“I’ve always felt it was a bit strange to see people sing Italian opera, like the style of Bel canto, in Japanese. A ‘Japanese music’ approach doesn’t feel right. That’s why I turned my attention to Vocaloid and androids. I think it’s efficient because it’s contemporary, and it shows that European opera doesn’t necessarily always have to have human beings at the center.

“I’m also trying to compose in a way that’s never been done before in orchestra or opera. For instance, I sliced the songs from the first half of the act into 0.5 seconds or one second. I then changed the order around and reconstructed them into new songs in the latter half of the act. This remix-ish method is very detailed. And once a more linear countertenor voice goes on top of that, it creates this incredible sonic experience. It’s as if it interweaves different passages of time. This style of composition is something that could only be made with computers.” 

A sacred experience where a piece of work that belongs to no one unifies everyone

The opera piece is being honed day by day towards its first show in summer. As such, Super Angels continues to evolve and expand. As Imai’s assistant, Kairi Nagashima works on building different systems and operating them. One day in August, when they were having rehearsals at the New National Theatre, Tokyo, he felt convinced of their progress: 

“During rehearsals at the New National Theatre, Tokyo, we got Alter3 up and running, and as it started singing with Fujiki-san, the countertenor singer, I felt like something unique was born right there. The ensemble got better and better, as Alter3 and Fujiki-san’s relationship formed. It surpassed the context of music. It was like this spectacular experiment about the relationship between AI/robots and people. The experience was so exciting.” 

The only thing that continues to evolve isn’t the performer, however. Shibuya explains that The New National Theatre, Tokyo is more than just a venue. The creative-minded staff there are keeping busy to ensure the quality of Super Angels. 

“The staff of the New National Theatre, Tokyo, is amazing. They’re excellent at what they do and have an inquisitive spirit. They’re doing everything they can to make Super Angels as good as it can be. When production is this big, so many people from different backgrounds come together, such as those working in technology, art, sound, and orchestra. This piece doesn’t belong to anyone anymore. I, too, am merely one part of it. The time and experience used to march towards one collective goal are sacred. It’s making me grow in a big way. This sort of experience seldom happens.”

Many professionals have come together to build and refine the unprecedented opera that is Super Angels. Nobody can foresee its final form: not even Shibuya himself.

Keiichiro Shibuya graduated from Tokyo University of the Arts with a B.A. in Music Composition. In 2002, he founded the music label, ATAK. His diverse soundscape covers areas such as cutting-edge electronic music, piano solos, opera, soundtrack music, sound installation, and so forth. He released a Vocaloid opera starring Hatsune Miku, which comprised of no people, called THE END, in 2012. The opera was shown at Théâtre du Châtelet and other places around the world. In 2018, he released Scary Beauty, an android opera conducted by an AI-operated humanoid android that sings along. This was shown in Japan, Europe, and UAE. In September of 2019, Keiichiro then presented “Heavy Requiem,” a marriage between Buddhist music and chants and electronic music, at Ars Electronica in Austria. He explores themes of humanity and technology and the border between life and death with his work. His new opera piece, Super Angels is scheduled to be released in August 2021 at New National Theatre, Tokyo.
http://atak.jp

Shintaro Imai is a “computer” musician. He creates music by trimming and correcting detailed movements in sounds and objects. After his studies at Kunitachi College of Music and IRCAM in Paris, he received a grant from the Japanese Agency for Cultural Affairs and worked as a researcher at ZKM in Germany from 2002 to 2003. In 2004, he was an artist-in-residence at DAAD Berlin and composed music at the Berlin Institute of Technology. From 2008, he worked as the director of music for the Bauhaus Stage Projects, Dessau. He released Figure in Movement in 2015. His awards include: the Residence Prize at International Electroacoustic Music Competition of Bourges, the First Prize and Special Prize for Young Composer at MUSICA NOVA International Electroacoustic Music Competition, Prize at the EARPLAY Composers Competition, and the First Prize at ZKM International Competition for Electroacoustic Music. His works have been performed at various international festivals. He is an associate professor at Kunitachi College of Music and a part-time lecturer at the University of Tokyo.
http://www.shintaroimai.com

AMEYA
Photographer and Director based in Tokyo, Japan.
Instagram: @itsameyab

Photography and Videography by AMEYA

Translation Lena Grace Suda 

Latest articles of TOKION MUSIC

• Androids, Music, and Film—Android Aria “Seeds of Prophecy,” a New Piece Shown at LEFFEST, and Beyond: Massive Life Flow; Inside the Mind of Keiichiro Shibuya” Part 13
  by 藤川貴弘
  2024-02-29
•  A Vision of the Future Shown by a Dialogue Between Science and Art—IDEA – Dialogue on Love, Death, and Existence by Two Androids, Keiichiro Shibuya + Takashi Ikegami: “MASSIVE LIFE FLOW—Keiichiro Shibuya’s Current Thoughts,” Part 12
  by 藤川貴弘
  2024-02-29
• Interview with Otomo Yoshihide, The Mastery of Guitar & Turntable Achieved in His Mid-60s -Part 2-
  by 細田 成嗣
  2024-02-28

The post The Innovativeness of a Singing Android and Super Angels : “Massive Life Flow; Inside the Mind of Keiichiro Shibuya” Part 3: appeared first on TOKION - Cutting edge culture and fashion information.