October 20

Lipdub App: Instant AI translations for your videos where the lips match up!

Oh boy. I’ve been waiting for this one.

My favorite captioning tool, aptly named Captions, released Lipdub this week, and it’s just as awesome (and as rough) as I thought it would be. The first half of this video is the original American English version. Then I do it again… in fluent Japanese, Italian and Spanish. (And even in Gen Z-speak at the bottom of the post!)

You HAVE to See It to Understand (64 seconds)

The End of Subtitles on Foreign Films?

Here’s how Lipdub works (as of today):

  • You load up a very short video of you talking straight at the camera.
  • You choose a language.
  • You wait a couple-three minutes for Lipdub to
    • Translate your video
    • Analyze your video
    • Dub your cloned voice and match up your lips to the new language.

5 Things to Know about the Lipdub App

Right now it’s Apple only

Don’t shoot the messenger. Y’all know how I try to review things that will work on multiple platforms. But I have been on this waiting list for a while, and I had to try it out anyway.

It’s amazing, but still kinda bad

As you can see in the video, it’s not really great yet. But come on, it’s PRETTY DARN GOOD! And right now it’s 100% free. You can’t really beat that. Who knows how much it will cost when they work out the kinks.

You can do so much with this technology

Beyond killing the foreign language subtitles and awkward voice overs in many movies I want to watch, I can see lots of potential. Imagine companies creating training videos for workers who speak other languages. Or charities who can reach more people who need help without having to translate material. Or content creators who want a wider reach. Those possibilities are right around the corner.

The translations seem pretty accurate

I don’t speak Japanese, Italian or Spanish, so I turned to Google Translate to test the translations. Google thinks it did pretty well. I welcome any feedback to the contrary.

You can definitely see the ChatGPT-ness of it

The more translations I created, the more off they got. The videos were shorter. They duplicated words. They left stuff out. They just didn’t come out the way the first ones did.

I thought it might be making translations from the translations, so I closed out the app and went all the way back in to try the Spanish version again. This version had different verbiage than the first one, and that reminded me of how you can put the same question in ChatGPT on two different days and get two different versions of the answer.

Check Out My Gen Z Translation (and more)

From Tagalog to Hindi and Finnish to Korean, Lipdub covers 26 different languages. Then we have a translator for the languages of …. Texas, Baby and Pirate. Oh, and my favorite, Gen Z. Here’s the Gen Z translation of my original video:

And here are the others. Like I said, I welcome feedback on the translations, quality, effectiveness and more. This is new, new, new technology. We’re all learning here.


Tags

communication, translation, video


You may also like

  • Beth, Lipdub looks amazing, looking forward to using it! It’s also worth noting that some films (Godzilla movies from Japan, for example) that need horrifically bad English overdubs for their weirdness to be fully appreciated.

  • I can see how this would be really useful for companies who do business worldwide. Great way to communicate with staff and partners, as well as prepare training videos. Equipment manufacturers could use this to translate videos that demonstrate how to use their products.

  • {"email":"Email address invalid","url":"Website address invalid","required":"Required field missing"}
    >