All projects

Sonic-3D

Turning static manga panels into immersive spatial-audio video — fully local, no cloud.

Tech stack

  • Python
  • PyTorch
  • MiDaS
  • OpenCV
  • NumPy
  • FFmpeg
  • Next.js

The problem

Transforming 2D manga into an immersive audio experience normally leans on cloud APIs and GPU acceleration. The goal was a pipeline that runs entirely on a local machine — no cloud, no dedicated GPU — while staying inside a strict memory budget.

The solution

A fully local pipeline that reads a manga panel, estimates depth to place sound in space, extracts dialogue via a customized OCR stage, and synthesizes a spatial-audio video. Text grouping was improved with a Union-Find approach tuned for manga layouts, and video assembly was streamed in-memory to avoid disk bottlenecks.

Architecture

Panel → MiDaS depth estimation (spatial positioning) → customized EasyOCR with Union-Find text grouping (dialogue extraction) → spatial-audio synthesis → in-memory FFmpeg piping for final muxing. Peak memory held strictly under 8 GB throughout.

Highlights

  • +35% text-grouping accuracy via a Union-Find OCR customization
  • −40% end-to-end processing time using in-memory FFmpeg piping (no disk I/O)
  • Peak memory kept strictly under 8 GB — runs without cloud APIs or a GPU