Skip to content

News Center | Georgia Institute of Technology Georgia Institute of Technology

Search

Search form

  • Facebook
  • Twitter
  • YouTube
  • LinkedIn
  • News Home
  • Campus Map
  • Directory
  • Offices

News Center

Menu
Close
  • Calendar
  • Categories
    • Business and Economic Development
    • Campus and Community
    • Earth and Environment
    • Health and Medicine
    • Science and Technology
    • Society and Culture
    • Feature Stories
  • Media Contacts
  • Experts
    • Find an Expert
    • Featured Expert
  • Daily Digest
  • The Whistle
    • Home
    • Classifieds
    • Archives
  • Social Media
  • Subscribe
  • You are here:
  • GT Home
  • Georgia Tech NewsCenter
  • Home
  • Shimon: Now a Singing, Songwriting Robot

Science and Technology

Shimon: Now a Singing, Songwriting Robot

Marimba-Playing Robot Composes Lyrics and Melodies With Human Collaborators

February 25, 2020 • Atlanta, GA

Marimba-playing robot Shimon has learned some new skills, thanks to a class of machine learning known as deep learning. Working with human collaborators, Shimon writes lyrics and melodies — and soon, he's going on tour.

He has moves like Jagger (almost). And he’s coming to a music venue near you.

But he’s not like any performer you’ve ever seen. He’s not even human.

Shimon, the marimba-playing robot, has learned some new skills: He sings, he dances a little, he writes lyrics, he can even compose some melodies. Now he’s taking them on the road in a concert tour to support a new album — just like any other musician.

The new album will have eight to 10 songs Shimon wrote with his creator, Georgia Tech Professor Gil Weinberg. It will drop on Spotify later this spring.

“Shimon has been reborn as a singer-songwriter,” Weinberg said. “Now we collaborate between humans and robots to make songs together.”

[Listen to Shimon's first single, "Into Your Mind]

Weinberg will start with a theme — say, space — and Shimon will write lyrics around the theme. Weinberg puts them together and composes melodies to fit them. Shimon can also generate some melodies for Weinberg to use as he puts together a song. Then, with a band of human musicians, Shimon will play the songs and sing.

“I always wanted to write songs, but I just can’t write lyrics. I'm a jazz player,” Weinberg said. “This is the first time that I actually wrote a song, because I had inspiration: I had Shimon writing lyrics for me.”

Weinberg and his students have trained Shimon on datasets of 50,000 lyrics from jazz, prog rock, and hip-hop. Then Shimon uses deep learning, a class of machine learning algorithms, to generate his own words.

“There are lots of systems that use deep learning, but lyrics are different,” said Richard Savery, a third-year Ph.D. student who has been working with Shimon over the past year on his songwriting. “The way semantic meaning moves through lyrics is different. Also, rhyme and rhythm are obviously super important for lyrics, but that isn't as present in other text generators. So, we use deep learning to generate lyrics, but it's also combined with semantic knowledge.”

Savery offered this example of how it might work: “You'll get a word like ‘storm,’ and then it'll generate a whole bunch of related words, like ‘rain.’ It creates a loop of generating lots of material, deciding what's good, and then generating more based on that.”

When Shimon sings these songs, he really does sing, with a unique voice created by collaborators at Pompeu Fabra University in Barcelona. They used machine learning to develop the voice and trained it on hundreds of songs.

Along with his new skills — all developed in Weinberg’s lab — Shimon has some new hardware, too, that changes how he plays and moves on stage. To be clear, he’s still mostly stationary, but he has a mouth, new eyebrows, and new head movements designed to help convey emotion and interact with his bandmates. He also has new “hands,” that have totally changed how he plays the marimba.

“Shimon plays much faster — about 25 to 30 hertz at the maximum — and also much more expressively, playing from a soft dynamic range to a strong dynamic range,” said Ph.D. student Ning Yang, who designed all-new motors and hardware for Shimon. “That also allows [Shimon] to do choreography during the music being played.”

For example, Shimon can count in at the beginning of songs to cue the band, and sometimes he’ll wave his mallets around in time to the music. New brushless DC motors mean he has a much greater range of motion and control of that motion. Yang accomplished that by bringing his engineering knowledge and musical background together to create human-inspired gestures.

“It's actually a very, very good example at Georgia Tech that we can actually combine tech and arts together to create something that's brand new,” Yang said.

He worked closely with fellow Ph.D. student Lisa Zahray, who created a new suite of gestures for the robot — including how he uses those new eyebrows.

“We have to think about his role at each time during the song and what he should be doing,” Zahray said. “We also want to make sure he's interacting with the other musicians around him to give that feel that he's performing with people.”

That partnership with people is key for Weinberg. Teaching Shimon new skills isn’t about replacing musicians, he said.

“We will need musicians, and there will be more musicians that will be able to do more and new music because robots will help them, will generate ideas, will help them broaden the way they think about music and play music,” Weinberg said.

Shimon, Weinberg, and the entire band are building a touring schedule now with the goal of taking their unique blend of robot- and human-created music to more people. Weinberg said he hopes those shows will prove to be more than a novelty act.

“I think we have reached a level where I expect the audience to just enjoy the music for music’s sake,” Weinberg said. “This is music that humans, by themselves, wouldn't have written. I want the audience to think, ‘There's something unique about this song, and I want to go back and listen to it, even if I don't look at the robot.’”

Shimon was originally developed with support from the National Science Foundation Cyber-Human Systems program, grants No. 0713269, and 1017169.

Additional Videos

Music video for "Into Your Mind," the first single from the upcoming album "Shimon Sings."

Additional Photos

  • Shimon Singing
     

    Shimon Singing

    Along with his new singing and songwriting abilities, Shimon has some new hardware. That includes a mouth, eyes, new eyebrows, and a series of new movements. He also had new hands, allowing him to play the marimba faster than any human can. (Photo Courtesy: Gil Weinberg)

    Download Image
  • Shimon and the Band
     

    Shimon and the Band

    Musical robot Shimon plays with human musicians on one of his new songs. Shimon has learned to write lyrics and sing alongside his human collaborators. (Photo Courtesy: Gil Weinberg)

    Download Image

Related Links

  • "Into Your Mind" - Shimon's first single
  • Gil Weinberg
  • Georgia Tech Center for Music Technology
  • Connect with Shimon
  • Freethink Raps with Shimon

Contact Information

Joshua Stewart

404.894.6016

Categories

Science and Technology

News Categories

  • Business and Economic Development
  • Campus and Community
  • Earth and Environment
  • Health and Medicine
  • Science and Technology
  • Society and Culture

Expert Voices

James Wray

James Wray (Photo Fitrah Hamid)

More Hints of Life on Ancient Mars
James Wray
School of Earth and Atmospheric Sciences
Mariel Borowitz

Photo by Rob Felt

Half of Earth’s satellites restrict use of climate data
Mariel Borowitz
Sam Nunn School of Int'l Affairs

Featured Videos

Seth Osekre, a custodian in Facilities Management, follows safety protocols when cleaning. (Video by Allison Carter)

2020 graduate TJ Weiler talks about his journey to earning a Georgia Tech degree.

Georgia Tech Resources

  • Offices & Departments
  • News Center
  • Campus Calendar
  • Special Events
  • GreenBuzz
  • Institute Communications
  • Visitor Resources
  • Campus Visits
  • Directions to Campus
  • Visitor Parking Information
  • GTvisitor Wireless Network Information
  • Georgia Tech Global Learning Center
  • Georgia Tech Hotel & Conference Center
  • Barnes & Noble at Georgia Tech
  • Ferst Center for the Arts
  • Robert C. Williams Paper Museum

Colleges, Instructional Sites & Research

  • Colleges
  • College of Computing
  • College of Design
  • College of Engineering
  • College of Sciences
  • Ivan Allen College of Liberal Arts
  • Scheller College of Business
  • Instructional Sites
  • Georgia Tech-Lorraine
  • Georgia Tech-Savannah
  • Georgia Tech-Shenzhen
  • Georgia Tech Online
  • Professional Education
  • The Language Institute
  • Global Footprint
  • Global Engagement
  • Research
  • Georgia Tech Research Institute
  • Research at Georgia Tech
  • Executive Vice President for Research

Student & Parent Resources

  • Student Resources
  • Apply
  • BuzzPort
  • Buzzcard
  • Career Center
  • Co-ops & Internships
  • Commencement
  • Library
  • Student Life
  • Student Entrepreneurship
  • Study Abroad
  • T-Square
  • Parent Resources
  • Parent and Family Programs
  • Dean of Students
  • Scholarships & Financial Aid

Employee, Alumni, & Other Resources

  • Employees
  • Administration and Finance
  • Advising & Teaching
  • Faculty Affairs
  • Faculty Hiring
  • Human Resources
  • Office of the Provost
  • TechWorks
  • Alumni
  • Alumni Association
  • Alumni Career Services
  • Giving Back to Tech
  • Outreach
  • Startup Companies
  • Economic Development
  • Industry Engagement
  • Government & Community Partners
  • Professional Education
Map of News Center | Georgia Institute of Technology

Georgia Institute of Technology
North Avenue, Atlanta, GA 30332
Phone: (404) 894-2000

  • Contact Us
  • Site Feedback
  • Tech Lingo
  • Emergency Information
  • Legal & Privacy Information
  • Human Trafficking Notice
  • Accessibility
  • Accountability
  • Accreditation
  • Employment
Georgia Tech

© Georgia Institute of Technology