# Web Audio & Video (0) Overview

Web Audio & Video Series Table of Contents

The WebCodecs API has been released for some time (since Chrome 94), making it feasible to process audio and video files or real-time video streams in browsers.
I plan to start a series introducing Web audio and video basics, and how to process video files (or video streams) in browsers.

As the author is also new to the audio and video field, this series serves as a personal learning summary. If you find any errors, please leave a comment
This series is suitable for frontend developers with some experience to enter the audio and video field. It won't delve deep into audio and video knowledge but will provide reference links when possible

# Historical Stages of Web Audio & Video

  1. The Barren Era
    Web didn't support streaming media, PCs came with CD-ROM drives, and local media players were standard software
    Users: Download movies for offline viewing
  2. The Flash Era
    Flash plugin was a browser standard, capable of playing FLV videos and RTMP streaming. Though Flash is dead, these two protocols are still alive, showing their far-reaching influence
    Users: Online video viewing gradually became popular
  3. The HTML5 Era
    Video tags can directly play MP4 videos. For existing FLV video files, JS can be used in browsers to repackage them as fMP4 and play using MSE API; FLV.js became a star project
    Users: Online video viewing became the preferred method, local players fell out of favor
  4. The WebCodecs Era
    Completing Web's missing audio and video production and editing capabilities. The capabilities are ready, but the ecosystem is not yet mature
    Users: Expected to see online editing, AI-assisted video generation, video conferencing, live streaming tools gradually becoming web-based

# Current State of Web Audio & Video

Audio & Video Workflow
Media Workflow

Encoding and decoding consume the most computing resources in the entire chain. Before WebCodecs, the Web technology stack lacked controllable codec capabilities.

The demuxing, decoding, and playback capabilities were unified and encapsulated in APIs like HTMLMediaElement and MSE, so playing audio and video on web pages is generally not a problem;
However, due to the lack of low-level data control or operation capabilities, features like video frame processing and playback buffer control are difficult or impossible to implement.

The lack of encoding capabilities has led to few video production tools on the Web; existing online editing tools heavily rely on server-side capabilities.

Without native codec capabilities, people sought various WASM versions of codecs or ffmpeg.wasm to meet basic needs;
However, they are still constrained by performance and functionality limitations, greatly restricting application scenarios.

WebCodecs is here to change this situation;
WebCodecs is the foundation for audio and video processing on the Web platform;
WebCodecs will, like HTML5, promote the application and development of audio and video on the Web platform.

This is the motivation for writing this series of articles.

If you haven't worked with WebCodecs before, it's recommended to read WebCodecs Explainer first to quickly understand the motivation and goals of the WebCodecs API

# Series Preview

  • Series Overview
  • Web Audio & Video Basics
    • Introduction to commonly used WebCodecs APIs
    • Audio & video encoding, packaging, and common formats
  • Parsing Video in Browsers
    • Video parsing, frame-by-frame processing...
    • WebCodecs decoding-related API applications
  • Creating Video in Browsers
    • Creating and exporting video files from Canvas, camera, microphone, etc.
    • WebCodecs encoding-related API applications
  • Processing Audio in Browsers
    • Complete process of capturing, processing, encoding, and packaging audio data in browsers
    • Basic audio data processing: volume adjustment, mixing, fade in/out, resampling
  • Compositing Video in Browsers
    • Adding media (video, audio, images, text) to video
    • Compositing and concatenating multiple videos
  • Image Asset Processing
    • Image processing
    • Controlling asset layers, adding animations to assets, exporting video
    • Asset animations: translation, rotation, opacity...
  • Mid-term Review
    • Personal review
    • WebCodecs value, application scenarios, case analysis

Click WebAV Tag to view the series table of contents

The materials and examples in this series come from the WebAV Project (opens new window);
WebAV attempts to provide simple and easy-to-use APIs for processing audio and video data in browsers.

You can scan the QR code to donate and provide motivation (pressure 😃 for the author
Alipay WeChat Pay

Star or Watch the WebAV Project (opens new window) to follow project progress, experience demos, and review example code;
You can subscribe to this blog site through Official Account or RSS (opens new window) to receive updates on this series.

# Appendix

Historical Stages of Web Audio & Video
Current State of Web Audio & Video
Series Preview
Web Audio & Video Related APIs
Appendix