FFmpeg aide

Original article : https://lukaprincic.si/development-log/ffmpeg-audio-visualization-tricks

Audio Vector Scope

ffmpeg -i INPUT_AUDIO.wav -filter_complex "[0:a]avectorscope=s=480x480:zoom=1.5:rc=0:gc=200:bc=0:rf=0:gf=40:bf=0,format=yuv420p[v];  [v]pad=854:480:187:0[out]"  -map "[out]" -map 0:a -b:v 700k -b:a 360k OUTPUT_VIDEO.mp4

The code above creates a mp4 video file with a vectorscope nicely centered inside a 854×480 (480p) video. If you need a 1:1 video, just exclude the pad part:

ffmpeg -i INPUT_AUDIO.wav -filter_complex "[0:a]avectorscope=s=480x480:zoom=1.5:rc=0:gc=200:bc=0:rf=0:gf=40:bf=0,format=yuv420p[v]"  -map "[v]" -map 0:a -b:v 700k -b:a 360k OUTPUT_VIDEO.mp4

Documentation on ‘avectorscope’ filter is here: https://ffmpeg.org/ffmpeg-filters.html#avectorscope. One can play with zoom and other options to produce desired form.

Show waves

ffmpeg -i INPUT.wav -filter_complex "[0:a]showwaves=mode=line:s=hd480:colors=White[v]" -map "[v]" -map 0:a -pix_fmt yuv420p -b:a 360k -r:a 44100  OUTPUT.mp4

more options: https://www.ffmpeg.org/ffmpeg-filters.html#showwaves

Showspectrum

ffmpeg -i INPUT.wav  -filter_complex "[0:a]showspectrum=s=854x480:mode=combined:slide=scroll:saturation=0.2:scale=log,format=yuv420p[v]"  -map "[v]" -map 0:a  -b:v 700k -b:a 360k OUTPUT.mp4

Above code will create almost completely desaturated spectrum of the audio sliding from right to left. Again, there are various options to tweak, see here: https://ffmpeg.org/ffmpeg-filters.html#showspectrum-1

Histogram

ffmpeg -i INPUT.wav -filter_complex "[0:a]ahistogram=s=hd480:slide=scroll:scale=log,format=yuv420p[v]"  -map "[v]" -map 0:a  -b:a 360k OUTPUT.mp4

more options: https://www.ffmpeg.org/ffmpeg-filters.html#showwaves

Static spectrogram

Sometimes you want to just create a static image.

create a spectrogram as a single frame

ffmpeg -i INPUT.wav -lavfi 
showspectrumpic=s=hd480:legend=0,format=yuv420p 
SPECTROGRAM.png

add png to audio - you need to know the length of audio

ffmpeg -loop 1 -i SPECTROGRAM.png -i INPUT.wav 
-s hd480 -t 00:01:00 -pix_fmt yuv420p 
-b:a 360k -r:a 44100 OUTPUT.mp4

create a spectrogram as a single frame

ffmpeg -i INPUT.wav -lavfi 
showspectrumpic=s=hd480:legend=0,format=yuv420p 
SPECTROGRAM.png

add png to audio - you need to know the length of audio

ffmpeg -loop 1 -i SPECTROGRAM.png -i INPUT.wav 
-s hd480 -t 00:01:00 -pix_fmt yuv420p 
-b:a 360k -r:a 44100 OUTPUT.mp4

Above one is in two steps. More info here: https://www.ffmpeg.org/ffmpeg-filters.html#showspectrumpic

text overlays

   ffmpeg \
-i video1.mp4 -i video2.mp4
-filter_complex "[0:v:0] [0:a:0] [0:v:1] [0:a:1] concat=n=2:v=1:a=1 [v][a];
[v]drawtext=text='SOME TEXT':x=(w-text_w):y=(h-text_h):fontfile=OpenSans.ttf:fontsize=30:fontcolor=white[v]" \
-map "[v]" -map "[a]" -deinterlace \
-vcodec libx264 -pix_fmt yuv420p -preset $QUAL -r $FPS -g $(($FPS * 2)) -b:v $VBR \
-acodec libmp3lame -ar 44100 -threads 6 -qscale 3 -b:a 712000 -bufsize 512k \
-f flv "$YOUTUBE_URL/$KEY"

Print file metadata etc.

 ffmpeg -i path/to/file.ext

Convertion

Convert all m4a files to mp3

 for f in *.m4a; do ffmpeg -i "$f" -acodec libmp3lame -vn -b:a 320k "${f%.m4a}.mp3"; done

Convert video from .foo to .bar

-g : GOP, for searchability

ffmpeg -i input.foo -vcodec bar -acodec baz -b:v 21000k -b:a 320k -g 150 -threads 4 output.bar 

Convert image sequence to video

 ffmpeg -r 18 -pattern_type glob -i '*.png' -b:v 21000k -s hd1080 -vcodec vp9 -an -pix_fmt yuv420p -deinterlace output.ext

Listen to 10 seconds of audio from a video file

-ss : start time / -t : seconds to cut / -autoexit : closes ffplay as soon as the audio finishes

ffmpeg -ss 00:34:24.85 -t 10 -i path/to/file.mp4 -f mp3 pipe:play | ffplay -i pipe:play -autoexit

Common switches

  -codecs       # list codecs
  -c:v              # video codec (-vcodec) - 'copy' to copy stream
  -c:a              # audio codec (-acodec)

-fs SIZE         # limit file size (bytes)

Bitrate

-b:v 1M          # video bitrate (1M = 1Mbit/s)
-b:a 1M          # audio bitrate

Video

-aspect RATIO    # aspect ratio (4:3, 16:9, or 1.25)
-r RATE          # frame rate per sec
-s WIDTHxHEIGHT  # frame size
-vn              # no video

Audio

-aq QUALITY      # audio quality (codec-specific)
-ar 44100        # audio sample rate (hz)
-ac 1            # audio channels (1=mono, 2=stereo)
-an              # no audio
-vol N           # volume (256=normal)

Example

ffmpeg generate overlay complex filter for audio with background image

    ffmpeg -y -i input.mp3 -loop 1 -i background.png \
    -filter_complex "[0:a]showwaves=s=1280x720:mode=line,colorkey=0x000000:0.01:0.1,format=yuva420p[v];[1:v][v]overlay[outv]" \
    -map "[outv]" -pix_fmt yuv420p -map 0:a -c:v libx264 -c:a copy -shortest output.mp

https://www.youtube.com/watch?v=zKYzJ_bEJVo

ffmpeg -y -i audio.mp3 -loop 1 -i image.jpg \
    -filter_complex "[0:a]showwaves=s=1280x175:colors=Yellow:mode=line,format=yuv420p[v];[1:v][v]overlay=0:200[outv]" \
    -map "[outv]" -pix_fmt yuv420p -map 0:a -c:v libx264 -c:a copy -shortest output9.mp4

https://www.youtube.com/watch?v=1htjI7YSNZo

ffmpeg -y -i audio.mp3 -loop 1 -i image.jpg \
    -filter_complex "[0:a]showwaves=s=1280x720:mode=line,colorkey=0x000000:0.01:0.1,format=yuva420p[v];[1:v][v]overlay[outv]" \
    -map "[outv]" -pix_fmt yuv420p -map 0:a -c:v libx264 -c:a copy -shortest output1.mp4

https://www.youtube.com/watch?v=H5dCKuN9Ius

ffmpeg -y -i audio.mp3 -loop 1 -i image.jpg \
    -filter_complex "[0:a]showwaves=s=1280x175:colors=White:mode=p2p,format=yuv420p[v];[1:v][v]overlay=0:200[outv]" \
    -map "[outv]" -pix_fmt yuv420p \
    -map 0:a -c:v libx264 -c:a copy -shortest output12.mp4

Fancy Filtering Examples

https://trac.ffmpeg.org/wiki/FancyFilteringExamples

cellauto

ffplay -f lavfi -i cellauto=rule=110

Other interesting cellauto rule values: 9, 18, 22, 26, 30, 41, 45, 50, 54, 60, 62, 73, 75, 77, 82, 86, 89, 90, 97, 99, 101, 102, 105, 107, 109, ​110 (default), 124, 126, 129, 131, 133, 135, 137, 145, 146, 149, 150, 151, 153, 154, 161, 167, 169, 181, 182, 183, 193, 195, 210, 218, 225.

life

ffplay -f lavfi -i life=s=300x200:mold=10:r=60:ratio=0.1:death_color=#C83232:life_color=#00ff00,scale=1200:800:flags=16

ffplay -f lavfi -i life=s=640x480:mold=10:r=100:ratio=0.1:death_color=blue:life_color=#00ff00,boxblur=2:2

mandelbrot

ffplay -f lavfi -i mandelbrot

mirror effect

ffplay -i INPUT -vf "crop=iw/2:ih:0:0,split
[tmp];[tmp]hflip
;
 hstack"

video channel separation effect with lut filter

RGB version

ffplay -f lavfi -i testsrc -vf "split=4[a][b][c][d];[b]lutrgb=g=0:b=0[x];[c]lutrgb=r=0:b=0[y];[d]lutrgb=r=0:g=0[z];[a][x][y][z]hstack=4"

YUV version

ffplay -f lavfi -i testsrc -vf "split=4[a][b][c][d];[b]lutyuv=u=128:v=128[x];[c]lutyuv=y=0:v=128[y];[d]lutyuv=y=0:u=128[z];[a][x][y][z]hstack=4"

histogram & waveform

ffplay -f lavfi -i mandelbrot -vf "format=gbrp,split=4[a][b][c][d],[d]histogram=display_mode=0:level_height=244[dd],[a]waveform=m=1:d=0:r=0:c=7[aa],[b]waveform=m=0:d=0:r=0:c=7[bb],[c][aa]vstack[V],[bb][dd]vstack[V2],[V][V2]hstack"

vectorscope & waveforms

ffplay -f lavfi -i mandelbrot -vf "format=yuv444p,split=4[a][b][c][d],[a]waveform[aa],[b][aa]vstack[V],[c]waveform=m=0[cc],[d]vectorscope=color4[dd],[cc][dd]vstack[V2],[V][V2]hstack"

waveform

Split the waveform filter to show broadcast range of the waveform (y values between 16 and 235) in green and out of broadcast range in red.

ffplay -i http://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_480p_surround-fix.avi  -vf "split[a][b];[a]format=gray,waveform,split[c][d];[b]pad=iw:ih+256[padded];[c]geq=g=1:b=1[red];[d]geq=r=1:b=1,crop=in_w:220:0:16[mid];[red][mid]overlay=0:16[wave];[padded][wave]overlay=0:H-h"

waveform with envelope

Split the waveform filter to show broadcast range of the waveform (y values between 16 and 235) in green and out of broadcast range in red and also use envelope.

ffplay ~/matrixbench_mpeg2.mpg -vf "split[a][b];[a]waveform=e=3,split=3[c][d][e];[e]crop=in_w:20:0:235,lutyuv=v=180[low];[c]crop=in_w:16:0:0,lutyuv=y=val:v=180[high];[d]crop=in_w:220:0:16,lutyuv=v=110[mid] ; [b][high][mid][low]vstack=4"

more waveforms and vectorscope

datascope

Audio

Combine video and audio into 1 file

 ffmpeg -i video.ext -i audio.ext -c:v copy -c:a copy output.ext

aevalsrc

Making some random "musical" keys:

% cat expr
# floor(t): 0 0 0 0 0 ... 1 1 1 1 1 ... 2 2 2 2 2
#  => set a random key when floor(t) changes
if(eq(floor(t),ld(2)),
    st(0,random(4)*3000+1000));

# the next value to compare floor(t) with
st(2,floor(t)+1);

# mod(t,1) makes t always in the range [0;1) for each key
st(1,mod(t,1));

# 0.6*... + 0.4*... for "echo" effect
# exp() to mitigate the sound according to the time
(0.6*sin(1*ld(0)*ld(1))+
 0.4*sin(2*ld(0)*ld(1)))*exp(-4*ld(1))

And to test it:

ffplay -f lavfi -i "aevalsrc=$(grep -v '^#' expr|tr -d '\n'|sed 's/\([,;]\)/\\\1/g')"

showwaves and showspectrum

Given the audio file april.flac:

ffplay -f lavfi 'amovie=april.flac,asplit=3[out1][a][b]; [a]showwaves=s=640x240[waves]; [b]showspectrum=s=640x240[spectrum]; [waves][spectrum] vstack[out0]'

showspectrum

Given the multichannel audio file tearsofsteel-surround.flac:

ffplay -f lavfi 'amovie=tearsofsteel-surround.flac,asplit=2[out1][a]; [a]showspectrum=color=channel:scale=cbrt:orientation=vertical:overlap=1:s=2048x1024[out0]'

Now with different colors and scaling:

ffplay -f lavfi 'amovie=tearsofsteel-surround.flac,asplit=2[out1][a]; [a]showspectrum=color=fire:scale=log:orientation=vertical:overlap=1:s=1024x1024[out0]'

avectorscope

Given the audio file input.flac:

ffplay -f lavfi 'amovie=input.flac,asplit=2[out1][a],[a]avectorscope=m=polar:s=800x400[out0]'

showcqt

Given the audio file input.flac:

ffplay -f lavfi 'amovie=input.flac,asplit=2[out1][a],[a]showcqt[out0]'

showspectrumpic

Given the audio file input.flac:

ffmpeg -i input.flac -lavfi showspectrumpic=s=hd720 out.jpg