Clips AI
Transforms videos into focused, speaker-highlighted clips effortlessly.. [Freemium]
Last verified:
What is Clips AI?
Clips AI is an open-source Python library that automatically converts long-form videos into shorter, shareable clips. It is specifically designed for audio-centric, narrative-based content such as podcasts, interviews, speeches, and sermons. The library uses advanced NLP techniques to analyze video transcripts and identify engaging moments to clip.
Key features include automated clip finding using the TextTiling algorithm with BERT Embeddings to detect topic shifts in transcripts, dynamic video resizing that reframes videos from 16:9 to 9:16 (vertical) or other aspect ratios, speaker-focused cropping using Pyannote for speaker diarization, face detection with MTCNN and MediaPipe, and scene change detection with PySceneDetect. The library integrates with WhisperX for accurate transcription with word-level timing.
Clips AI is built for developers who need programmatic control over video repurposing. It requires Python installation with dependencies including clipsai, whisperx, libmagic, and ffmpeg. The library provides ClipFinder and Transcriber classes for finding clips, and a resize function with MediaEditor for resizing videos. A companion video editor UI is available for trimming and resizing generated clips.
Clips AI pricing
Pricing model: Freemium
Clips AI is completely free and open-source. The library is available on PyPI and GitHub at no cost. Users only need to install the required dependencies (clipsai, whisperx, libmagic, ffmpeg) which are also free and open-source. A Hugging Face access token is required for Pyannote speaker diarization but users are not charged for using Pyannote.
Clips AI pros
- Open-source and free to use
- Automated clip finding from transcripts
- TextTiling algorithm with BERT Embeddings for precise segmentation
- Dynamic resizing focuses on current speaker
- Supports multiple aspect ratios (16:9 to 9:16)
- Speaker diarization with Pyannote
- Face detection with MTCNN and MediaPipe
- Scene change detection with PySceneDetect
- WhisperX integration for accurate transcription
- Word-level timing in transcriptions
- Only a few lines of code needed
- Programmatic control for developers
- ClipFinder class for finding engaging clips
- MediaEditor for trimming and resizing
- Works with podcasts, interviews, speeches, sermons
Clips AI cons
- Requires Python programming knowledge
- Not a no-code or GUI solution
- Multiple dependencies to install (whisperx, libmagic, ffmpeg)
- Requires Hugging Face access token for Pyannote
- Only works well with audio-centric narrative videos
- May not work for non-narrative content like music videos
- Setup can have dependency conflicts
- Requires virtual environment to avoid dependency issues
Frequently asked questions about Clips AI
What is Clips AI?
Clips AI is an open-source Python library that automatically converts long-form videos into clips. With just a few lines of code, you can segment a video into multiple clips and resize its aspect ratio from 16:9 to 9:16.
What type of videos does Clips AI work best with?
Clips AI is designed for audio-centric, narrative-based videos such as podcasts, interviews, speeches, and sermons. It actively employs video transcripts to identify and create clips.
How does the clipping algorithm work?
The clipping feature leverages the TextTiling algorithm to segment long-form audio content into coherent clips using the transcript. This approach detects shifts in topics by analyzing word usage and distribution patterns. With BERT Embeddings, it provides significant improvements over the original TextTiling formulation.
How does the resizing algorithm work?
The resizing algorithm dynamically reframes videos to focus on the current speaker, converting the video into various aspect ratios. It relies on speaker diarization with Pyannote, scene change detection with PySceneDetect, and face detection with MTCNN and MediaPipe.
What dependencies are required to install Clips AI?
You need to install Python dependencies (clipsai and whisperx), libmagic, and ffmpeg. It is highly suggested to use a virtual environment such as venv to avoid dependency conflicts.
How do I transcribe a video with Clips AI?
Transcribing is done with WhisperX, an open-source wrapper on Whisper with additional functionality for detecting start and stop times for each word. You use the Transcriber class: transcriber = Transcriber() and transcription = transcriber.transcribe(audio_file_path='/abs/path/to/video.mp4').
Do I need a Hugging Face token?
Yes, a Hugging Face access token is required to resize a video since Pyannote is utilized for speaker diarization. You won't be charged for using Pyannote, and instructions are available on the Pyannote HuggingFace page.
What aspect ratios does Clips AI support?
Clips AI can resize videos to various aspect ratios, with the default being 9:16 (vertical). The aspect_ratio parameter is a tuple of (width, height), and you can specify custom ratios as needed.
Is there a GUI for Clips AI?
Clips AI itself is a Python library without a built-in GUI. However, there is a separate video-editor repository that provides a simple video trimming and resizing UI for the clips generated by Clips AI.
How do I find clips using Clips AI?
First transcribe the video with Transcriber, then use ClipFinder: clipfinder = ClipFinder() and clips = clipfinder.find_clips(transcription=transcription). The clips returned contain start_time and end_time for each clip.