A.I. Tools for Video Pros, Filmmakers, & Content Creators #indepth [Updated]
Exciting times are upon us as Artificial Intelligence permeates our daily lives. A.I. tools empower video creators and filmmakers in ways that may surprise you. Easily transcribe audio, generate subtitles, eliminate background noise, or automatically blur faces in your videos. Repurpose lengthy footage, enhance video quality, and edit seamlessly with just a few prompts. As A.I. capabilities continue to improve exponentially, the possibilities for innovation are endless. And, as the capabilities of A.I. gets exponentially better and better, who knows where we will be in a year?
Updated 17 October 2023
New from Adobe MAX: The big news in the past month is Adobe’s new AI features, announced at Adobe MAX in October 2023. Below is a summary of what’s new, with some links so that you can check it out.
Adobe MAX Sneaks 2023
One of the most fun segments of Adobe MAX is Adobe Sneaks. To be clear, these are not products that will all created, but I’d imagine a lot these features will be integrated into Adobe tools in the future.
In this article
What is AI? How it help video pros.
- Is AI Taking your job?
- DaVinci Resolve Neural Engine
- Premiere Pro & Adobe Sensei UPDATE
- Edius 11 NEW
- Magix Vegas Pro
- Upscale footage. RE:Vision Effects REZup, Vimager ScaleUp
- Colorgrading & Color Effects NEW – AI Color Match
- Autodesk Flame
- Generative AI – Prompt-based generation and more.
- Adobe Sensei AI-Gen in FireFly and Photoshop. UPDATE
- Blace Local Diffusion – Text-to-image generator based on Stable Diffusion.
- Boris FX Silhouette – Stable Diffusion inside Silhouette.
- Diffusae – Bring Stable Diffusion into After Effects.
- Background Removal, Auto-Masking, Face Detection, and more. After Effects Roto Brush NEW, CoreMelt Shot Anvil NEW, Mask Prompter, Goodbye Greenscreen, AI Face Detection and Blurring, Keyper.
- Depth Map Generation and Estimation – Vegas Pro, Depth Scanner, AI Depth Map
- Noise Removal NEW
- Scripting
- Workflow tools NEW – Neural Enhancement Suite
Fix Audio. Remove noise and create pristine audio in your NLE.
More A.I. Tools to Check Out UPDATE
Conclusion: The Future of A.I. in video production and Visual Effects
What exactly is A.I.?
Artificial intelligence (A.I.) is an innovative area in computer science that aims to create smart systems capable of executing tasks that typically require human intelligence and interaction. This entails crafting intelligent tools that can generate knowledge from experience, and make decisions to achieve specific goals.
How A.I. Can Help Editors and Video Professionals
A.I. tools can automate repetitive tasks and enhance workflow efficiency. For example, use A.I. to transcribe footage, generate subtitles, perform morph cuts, recolor footage, smart reframing for social media, organize footage, upscale and sharpen footage, and so much more. They can remove unwanted noise, equalize frequencies, balance audio levels, and improve sound quality.
A.I. as a Collaborative Tool
Instead of replacing professionals, A.I. empowers them to focus on higher-level decision-making and experimentation. A.I. lacks the creativity and unique perspective that humans bring to video editing and filmmaking. By embracing A.I. as a collaborative tool, professionals can use its capabilities to enhance their own creativity and have a more fulfilling creative experience.
Is AI Taking your job?
A lot of artists I’ve talked to fear that A.I. will take their jobs. Studies have shown that AI is NOT taking jobs, but creating job opportunities. Listen below to an article from Wired.
A.I. should take on the tedious tasks for you and allow you to be more spend more time on creative work (ie. the fun stuff). It should help you be more productive, help you come up with new ideas, and help you get more balance in your life.
A.I. is here and it’s only becoming more utilized in software you use every day. Embrace it or you may be left behind.
Video Editing
You’d be shocked at the number of products that have incorporated A.I. into their algorithms, making them more efficient and taking care of tedious tasks for you. Several video editing tools that you’ve probably been using already have built-in AI Features for all sorts of repetitive tasks like organizing footage, transcribing audio, morph cutting, and lots more. And, of course, text-based editing! Streamline your workflow with A.I. Below, the products are grouped by task or category.
DaVinci Resolve Studio
The Neural Engine is a super fast A.I. engine, only available in the Studio version of DaVinci Resolve. The Resolve Studio version 18.5 beta received a big update with exciting new features. It allows users to do a slew of time-saving tasks, such as:
- Text-based editing. See the video below.
- Auto-generate subtitles from Audio.
- Audio Classification for media pool clips in Studio. See the video below.
- SuperScale. Upscale your footage by creating new pixels when increasing the resolution of an image. 18.5 Beta introduces a new 2x enhanced mode.
- Resolve FX Relight. Reposition virtual light sources in the scene.
Other previously introduced Neural-engine based tools in Resolve Studio
- Voice Isolation Track FX. Remove unwanted background noises from the dialog.
- Lens Distortion. Correct spherical lens distortion and aberrations.
- Smart Reframe. Reframe your clips to vertical or square for social media. It automatically identifies the movement in footage and repositions the image inside a new frame.
- Person Mask. Auto-detect people in shots and human movement.
- Face Detection. Helps editors sort footage.
- Face refinement. Automatically detect and track faces for beauty work like skin smoothing, sharpening eyes, and color grading skin.
- Magic Mask. Track objects or isolate parts of the body, like the face, clothing, legs, etc.
- Track Moving Warped Surfaces. Find it under the Fusion Tab.
- Background removal and object removal.
- Patch Replacer. Remove unwanted objects in shots by sampling other areas.
- SpeedWarp. Retime and change the speed of your footage using Optical Flow.
- Depth Map. Generate a depth map, useful in color grading. Find it under the Color page.
A.I.-text editing in DaVinci Resolve 18.5
Live from NAB 2023, Simon Hall, a product specialist for Blackmagic, talks about everything new in AI in Resolve 18.5. This includes transcribing clips, generating subtitles, and relighting footage.
Revolutionizing Audio Editing with DaVinci Resolve 18.5’s AI-Powered Audio Classification
In the video below, Jason Yadlovski walks you through the AI-Powered Audio Classification feature in Resolve 18.5.
Premiere Pro & Adobe Sensei Updated
Adobe Sensei is the AI engine in Adobe software. Sensei-infused tools in Premiere Pro help editors work faster. You’ll see some overlap with the Resolve Studio Neural Engine-driven tools here.
- Text-Based Editing. Read the overview.
- Scene Edit Detection. If you have a long clip that is already edited, apply Scene Edit Detection. Premiere Pro will analyze your footage and edit the clip for you.
- Remix. Automatically shorten audio so that you can create shorter clips that sound natural.
- Transcribe and create captions from audio. It’s as easy as the click of a button.
- Morph Cut. Minimize jump cuts in a locked-down shot when editing by automatically morphing between shots.
- Auto tracking. Track faces or anything else, then apply effects as needed.
- Color match. Choose a key moment in the video and Sensei will automatically match other shots to it.
- Speech-to-text for transcribing and captioning videos. “Speech to Text leverages AI technology to generate transcripts and Adobe Sensei machine learning to position captions on the timeline so that they match the pacing and cadence of the spoken words.” from the Adobe FAQ.
- Auto-ducking. Automatically adjust audio levels
- Auto-tag Audio. Learn more
- Auto Reframe. Resize clips and edits for social media.
New in Premiere Pro 2024
Several features that were in beta are now available in the release. New Transcription features, speech enhancement, and more.
Adobe Premiere Pro: Advanced AI Editing Features You NEED To TRY!
Vince Opra goes over Remix, Scene Edit Detection, and Auto-Tracking in Premiere Pro.
How AI Helps to Speed Up Video Editing in Premiere Pro | Adobe Creative Cloud
Adobe Sensei also runs text-based editing in Premiere Pro. Premiere Pro’s AI can help editors with tasks like morph cuts, smart reframing, organizing footage, relighting, color grading, and color match footage.
Grass Valley Edius NEW
Version 11, released in October 2023, adds AI Noise elimation. Utilizing AI based voice recognition, EDIUS 11 now allows you to eliminate even extreme noise better than ever before.
Magix Vegas Pro
Vegas Pro uses the ONNX Runtime Engine, a more powerful A.I. engine. A lot of the tools are similar to the tools found in Resolve Studio and Premiere Pro, so I’ll keep it brief.
- Z-Depth OFX Plug-in and Compositor. NEW in Vegas Pro 21, Z-Depth analyzes footage and estimates the depth between the foreground and background. It then creates a complex depth map from the data. See the tutorial video below under Z-Depth tools.
- Smart Mask. NEW in Vegas Pro 21, AI-assisted automatic masking. It analyzes clips, detects objects, and masks them.
- Text to Speech. (365 subscribers only) Generate high-audio-quality AI-generated voices. You can choose the emotional tone. Auto-translation is available for dozens of available languages.
- Colorization.
- Speech to Text.
- Upscaling footage
Upscale & Sharpen Footage
If your NLE doesn’t include built-in options or you want to try other options, below are some plugins to upscale and sharpen footage.
RE:Vision Effects REZup
With REZup, you can both upscale and enhance your image quality. REZup upscale footage using A.I., an improvement from the standard resizing filters like Lanczos or cubic. Rezup was trained on 2X, 3X, and 4X, so if you want to scale footage up even larger, you may need some pre-processing (e.g. denoising).
Host Support:
- After Effects CS6 (and up)
- Premiere Pro CS6 (and up)
- Final Cut Pro 10.1.4 (and up)
- Motion 5 (and up)
- Multiple OFX hosts (Nuke, HitFilm, Silhouette, Autograph, Vegas, and more)
Re:Vision Effects REZup
A new way to increase image quality
With Re:Vision Effects REZup, enhance the quality (“IQ”) of your content, increasing the resolution of your video without loss of detail or introducing artifacts.
Vimager ScaleUp
Another tool that uses AI for upscaling is Vimager ScaleUp. ScaleUp utilizes AI technology to enhance and scale footage up to 10 times with minimal quality loss.
Host Support:
- After Effects 2020 (and up)
- Premiere Pro 2020 (and up)
Check GPU Requirements and try a demo before purchasing ScaleUp
Note: The performance of ScaleUp is highly dependent on GPU, and some GPU cards are not supported. Please see the System Requirements tab for GPU details. Also, PLEASE TRY a DEMO of ScaleUp before purchasing.
Colorgrading and Color Effects NEW
AI Color Match
BSKL AI Color Match brings professional color correction and grading to After Effects and Premiere Pro. In short, you can copy the look from one shot and paste it to other shots. It matches the exposure, contrast, white balance and more, automagically.
Visual Effects
A.I. can generate content for you but it can also help with tedious tasks like rotoscoping and keying by automatically isolating faces, skies, and more.
Autodesk Flame
Flame‘s Machine learning image segmentation allows it to easily isolate skies, human bodies, heads, facial features, or other objects using AI-powered keyers.
Image & Video Generation
Several tools can bring AI into your host application. We’re at the early stages of video generation however, I expect this to explode in the coming months.
For background, there are 4 main image-generating tools available right now: Stable Diffusion, Midjourney, DALLE-E, and Adobe Firefly. More on Firefly below.
According to Superhuman – Zain Kahn, these are the best tools for AI-driven video generation at the moment:
- D-iD for animating faces
- Runway v2 for converting text to video
- Synthesia for generating avatar videos
- ElevenLabs for voice cloning
I have not tried any of the video tools above, and the tech behind them is certainly not perfect, but they are capable of generating characters, animations, voices, and scripts. I have used some of the tools that we sell that have AI generation built-in.
Generate Images with Adobe Firefly & Photoshop Update
New in FireFly v2 (beta): Adobe’s new firefly model’s Text to Image is a fantastic update. In fact, the image header for this article was created with the Text to Image tool.
Read: Adobe Shares Expansive Generative AI Strategy and Products
Other new features include an editorial channel where you can change the composition (Macrophotography, shot from below, etc.) Users can also adjust Aperture, Shutter Speed and Field of View. You can also include negative prompts, meaning that you can tell Firefly to not include certain things in your image. They’ve also added Generative Fill right in Firefly.
And before you say, “Firefly for still images. I work in video”, check out the video below.
ADOBE FIREFLY: The Future of AI Video Editing!
Blace Local Diffusion
Blace Local Diffusion is a Stable Diffusion based image generator running locally and natively inside AE.
Prompt interpolation example, using Depth Scanner for a perspective shift/.
Prompt: A city made of clay
Check out a tutorial: Create an AI animation in After Effects using Local Diffusion and Depth Scanner
Boris FX Silhouette
Silhouette 2023 introduced Stability AI’s DreamStudio integration, which allows users to enter prompts and generate content from Stable Diffusion right inside Silhouette.
You do need to install the API key and buy credits for Stable Diffusion, but it’s inexpensive.
Silhouette 2023 also introduced the Denoiser ML node, machine learning-trained denoising.
BSKL Diffusae
Diffusae from BSKL had been around for a few months and I’ve been playing with the beta on Mac. It works with prompts and allows users to generate content with Stable Diffusion, either from scratch or by applying AI-generated effects video effects to footage.
Background Removal, Auto-Masking, Face Detection, etc.
If you google A.I. background removal, you’ll find hundreds of tools, but you can do this with built-in tools like Resolve Studio, Premiere Pro, and Content-Aware Fill in After Effects. Autodesk Flame also excels with these features.
After Effects Advanced Roto Brush NEW
The new Roto Brush uses AI to make rotoscoping faster and b
CoreMelt Shot Anvil NEW
CoreMelt ShotAnvil
Tracked paint, masks, and graphics all in one plugin.
Unlock the power of CoreMelt ShotAnvil, a game-changing plugin that effortlessly combines tracked paint, masks, and graphics inserts. With just one click, experience the magic of AI person keying, all within this convenient all-in-one tool.
Goodbye Greenscreen (After Effects)
Blace Plugins’ Goodbye Greenscreen is a plugin for background matting and keying.
Mask Prompter (After Effects)
Simply use points and boxes and the A.I.-based plugin will create a mask from your prompt, for any object in your footage.
AI Face Detection & Blurring (After Effects)
Blace Plugins’ AI-based and automated face detection and blurring system for compositing and editing.
Sheffield Softworks Keyper (FCP X)
Keyper uses Machine Learning to extract people from their backgrounds without requiring greenscreen or prepared backdrops.
Depth Map Generation and Estimation
There are a couple of tools out there that can create depth maps, extremely useful when color grading, adding fog, converting footage to Stereo 3D, and more. Autodesk Flame has it built in but here are a couple of the other tools you may want to check out.
Vegas Pro 21
AEJuice AI Depth Map
Build depth maps from images and video.
Note: This is not a plugin. It opens a popup window that lets you convert images on your computer.
Blace Depth Scanner
Create Depth Maps for After Effects 2022 & 2023. Windows only.
Tutorial: Create an AI animation in After Effects using Local Diffusion and Depth Scanner
Noise Removal NEW
One of the first things I read about AI in visual effects was AI-enhanced noise reduction. There are multiple AI tools to remove noise from renders, images, and footage. Below are a few options.
Adobe AI Denoise
Part of Adobe Camera Raw and included with Lightroom, Photoshop, and more, AI Denoise was released in April 2023. More: Denoise demystified
NVIDIA OptiX AI Denoiser
Many products use NVIDIA OptiXTM Denoiser, and the list below is not even close to exhaustive.
- Adobe Substance Painter
- Autodesk Arnold
- Cebas Final Render
- Chaos V-Ray
- Chaos Coroona
- e-on Vue
- Maxon Redshift
- Pixar Flow
- SideFX Houdini
- Unity
Intel Open Source Image Denoiser
In July 2023, Intel released Open Image Denoise 2, an opensource AI denoiser, integrated into tools such as Cinema 4D, Blender, and Houdini. What’s different about the Intel denoiser is that it’s hardware agnostic and runs on GPU and CPU. Below are a few tools which use the Intel AI Image Denoiser. Again, not an exhaustive list.
- Autodesk Arnold
- Autodesk Maya
- Blender
- Chaos Corona
- Chaos Vantage
- Chaos V-Ray
- Lightwave 3D
- Maxon Cinema 4D
- Maxon Redshift
- Maxon Zbrush
- Renderman
- Houdini
- Modo
- Octane
- Unity
Scripting
Klutz GPT
If you need help writing expressions in After Effects, check out Klutz GPT, a free tool from Hyper Brew. It brings ChatGPT directly into After Effects.
Workflow tools
Neural Enhancement Suite
Neural Enhancement Suite is an A.I. powered toolset for AI-based video enhancement to achieve results not previously possible in After Effects. It features denoising, hint-based colorization, superresolution, low-light enhancement, and much more. Windows only.
Repair Audio problems in your NLE
A.I. tools trained on millions of audio clips can fix wind noise, remove bird sounds, or remove that low rumble of an air conditioner with ease. Premiere Pro and Resolve Studio each have their own tools but below are some plugins that can drastically improve your audio quality.
Boris FX CrumplePop
CrumplePop, recently acquired by Boris FX, is an Audio repair and restoration tool that works in your NLE. Host support includes Adobe Premiere Pro and After Effects, Apple Final Cut Pro and Motion, and a slew of OFX hosts (Nuke, Silhouette, HitFilm, Vegas, and many more).
iZotope RX
iZotope RX is an award-winning audio repair suite. It’s easy to use and is incredibly effective at noise removal and sound repair. RX comes in three versions, Elements, Standard, and Advanced versions. Each of these will run in Premiere Pro, and the Standard and Advanced also include a standalone version.
Other A.I. Tools for Video Pros, Filmmakers, and Creators
- NEW ssemble – an online AI video tool that looks great for filmmakers and social media managers. It includes Text to video, Text to speech voice generator in multiple languages, a background noise remover, a Chat GPT script writer, and much more. Check out all the features here. There’s a free version and a more robust Pro Version available.
- Jeremy Presner shared a List of A.I. tools for Filmmakers that you need to check out! This is a MASSIVE list of tools
- Nupur Mittal walks you through 12 Mind-Blowing AI Video Tools to Enhance Creation & Editing. She has a more curated list of tools, focusing on AI video generation, AI editing enhancements, and AI video repurposing tools.
Several NLEs have A.I. features built in, including Adobe Premiere Pro, DaVinci Resolve, Final Cut Pro X, and Magix Vegas. More on that below.
Create Better Videos INSTANTLY using THESE A.I. Tools!
Ignace Aleya has a list of his favorite tools for using A.I. as a video professional, which includes some on the list above.
Conclusion: The Future of A.I. in video production and Visual Effects
Expect to see A.I. dominate the video production and visual effects fields over the next few years. Expect to see a lot more time-saving features built into the software. We’ll be able to do a lot more in less time.
Since we’re talking about A.I., I decided to ask ChatGPT where it thought we would be in 5 years with artificial intelligence in the Film & Video fields.
Certainly! Here are some specific new technologies that we may see in the next five years in the film and video fields, focusing on AI tools and improved visual effects:
- AI-Generated Actors: Advancements in AI and deep learning could lead to the creation of completely virtual, AI-generated actors. These virtual actors could be designed to mimic human appearances and behaviors convincingly, opening up new possibilities for storytelling and character creation. Filmmakers could customize virtual actors to fit their narratives precisely, reducing reliance on human actors. (Joan is Awful?!)
- Real-time Style Transfer: AI algorithms can already apply artistic styles to images and videos. In the future, we may witness real-time style transfer during the filming process, where filmmakers can visualize the final look of a scene with a specific artistic style applied in real-time. This technology could streamline the production process by allowing filmmakers to make immediate creative decisions.
- AI-Enhanced Previsualization: Previsualization (previs) is commonly used to plan and visualize scenes before filming. AI-powered previs tools could go beyond basic wireframes and provide more realistic representations of scenes using machine learning algorithms. This would help filmmakers better understand how the final shots will look, improving decision-making during production.
- AI-Driven Emotion Analysis: AI algorithms can already analyze facial expressions and emotions. Future advancements may include AI-driven emotion analysis tools that can assess the emotional impact of specific scenes or sequences. Filmmakers could utilize this technology to optimize storytelling by identifying moments that elicit the desired emotional response from the audience.
- Automated Post-production Workflows: AI could revolutionize post-production workflows by automating tasks such as color grading, sound mixing, and visual effects compositing. Advanced AI algorithms would be capable of understanding the intended style and tone of a project and automatically applying the necessary adjustments, saving time and effort for editors and post-production teams.
- AI-Enhanced Restoration and Remastering: AI algorithms are already being used for image restoration and remastering. In the future, we may see more sophisticated AI tools that can intelligently analyze and restore old or damaged film footage, enhancing image quality, reducing artifacts, and preserving the historical value of classic films.
- AI-Powered Storytelling Assistance: AI tools may provide real-time assistance to filmmakers in developing compelling narratives. By analyzing existing story structures, audience preferences, and emotional arcs, AI algorithms could generate suggestions and insights to help filmmakers refine their scripts, pacing, and character development, ultimately enhancing the storytelling process.
These are just a few examples of potential new technologies that could emerge in the next five years, leveraging AI to bring further advancements and innovation to the film and video fields.
ChatGPT
AI-enhanced remastering could bring new life to the thousands of silent films out there. I am a big fan of Buster Keaton and I’ve seem some of his filmsThere you go, from the horse’s mouth. Honestly, a lot of it is already happening.
Posted by Michele
'%3e%3cg id='Final-Copy-2_2_' transform='translate(1275.000000, 200.000000)'%3e%3cpath class='st0' d='M7.4,12.8h6.8l3.1-11.6H7.4C4.2,1.2,1.6,3.8,1.6,7S4.2,12.8,7.4,12.8z'/%3e%3c/g%3e%3c/g%3e%3c/g%3e%3cg id='final---dec.11-2020'%3e%3cg id='_x30_208-our-toggle' transform='translate(-1275.000000, -200.000000)'%3e%3cg id='Final-Copy-2' transform='translate(1275.000000, 200.000000)'%3e%3cpath class='st1' d='M22.6,0H7.4c-3.9,0-7,3.1-7,7s3.1,7,7,7h15.2c3.9,0,7-3.1,7-7S26.4,0,22.6,0z M1.6,7c0-3.2,2.6-5.8,5.8-5.8 h9.9l-3.1,11.6H7.4C4.2,12.8,1.6,10.2,1.6,7z'/%3e%3cpath id='x' class='st2' d='M24.6,4c0.2,0.2,0.2,0.6,0,0.8l0,0L22.5,7l2.2,2.2c0.2,0.2,0.2,0.6,0,0.8c-0.2,0.2-0.6,0.2-0.8,0 l0,0l-2.2-2.2L19.5,10c-0.2,0.2-0.6,0.2-0.8,0c-0.2-0.2-0.2-0.6,0-0.8l0,0L20.8,7l-2.2-2.2c-0.2-0.2-0.2-0.6,0-0.8 c0.2-0.2,0.6-0.2,0.8,0l0,0l2.2,2.2L23.8,4C24,3.8,24.4,3.8,24.6,4z'/%3e%3cpath id='y' class='st3' d='M12.7,4.1c0.2,0.2,0.3,0.6,0.1,0.8l0,0L8.6,9.8C8.5,9.9,8.4,10,8.3,10c-0.2,0.1-0.5,0.1-0.7-0.1l0,0 L5.4,7.7c-0.2-0.2-0.2-0.6,0-0.8c0.2-0.2,0.6-0.2,0.8,0l0,0L8,8.6l3.8-4.5C12,3.9,12.4,3.9,12.7,4.1z'/%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/svg%3e)
Your Privacy Choices Notice at Collection