Android中音视频合成的几种方案详析-创新互联

前言

最近工作中遇到了音视频处理的需求，Android下音视频合成，在当前调研方案中主要有三大类方法：MediaMux硬解码，mp4parser，FFmepg。三种方法均可实现，但是也有不同的局限和问题，先将实现和问题记录于此，便于之后的总结学习。下面话不多说了，来一起看看详细的介绍吧。

方法一（Fail）

利用MediaMux实现音视频的合成。

效果：可以实现音视频的合并，利用Android原生的VideoView和SurfaceView播放正常，大部分的播放器也播放正常，但是，但是，在上传Youtube就会出现问题：音频不连续，分析主要是上传Youtube时会被再次的压缩，可能在压缩的过程中出现音频的帧率出现问题。

分析：在MediaCodec.BufferInfo的处理中，时间戳presentationTimeUs出现问题，导致Youtube的压缩造成音频的紊乱。

public static void muxVideoAndAudio(String videoPath, String audioPath, String muxPath) {
 try {
  MediaExtractor videoExtractor = new MediaExtractor();
  videoExtractor.setDataSource(videoPath);
  MediaFormat videoFormat = null;
  int videoTrackIndex = -1;
  int videoTrackCount = videoExtractor.getTrackCount();
  for (int i = 0; i < videoTrackCount; i++) {
  videoFormat = videoExtractor.getTrackFormat(i);
  String mimeType = videoFormat.getString(MediaFormat.KEY_MIME);
  if (mimeType.startsWith("video/")) {
   videoTrackIndex = i;
   break;
  }
  }
  MediaExtractor audioExtractor = new MediaExtractor();
  audioExtractor.setDataSource(audioPath);
  MediaFormat audioFormat = null;
  int audioTrackIndex = -1;
  int audioTrackCount = audioExtractor.getTrackCount();
  for (int i = 0; i < audioTrackCount; i++) {
  audioFormat = audioExtractor.getTrackFormat(i);
  String mimeType = audioFormat.getString(MediaFormat.KEY_MIME);
  if (mimeType.startsWith("audio/")) {
   audioTrackIndex = i;
   break;
  }
  }
  videoExtractor.selectTrack(videoTrackIndex);
  audioExtractor.selectTrack(audioTrackIndex);
  MediaCodec.BufferInfo videoBufferInfo = new MediaCodec.BufferInfo();
  MediaCodec.BufferInfo audioBufferInfo = new MediaCodec.BufferInfo();
  MediaMuxer mediaMuxer = new MediaMuxer(muxPath, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
  int writeVideoTrackIndex = mediaMuxer.addTrack(videoFormat);
  int writeAudioTrackIndex = mediaMuxer.addTrack(audioFormat);
  mediaMuxer.start();
  ByteBuffer byteBuffer = ByteBuffer.allocate(500 * 1024);
  long sampleTime = 0;
  {
  videoExtractor.readSampleData(byteBuffer, 0);
  if (videoExtractor.getSampleFlags() == MediaExtractor.SAMPLE_FLAG_SYNC) {
   videoExtractor.advance();
  }
  videoExtractor.readSampleData(byteBuffer, 0);
  long secondTime = videoExtractor.getSampleTime();
  videoExtractor.advance();
  long thirdTime = videoExtractor.getSampleTime();
  sampleTime = Math.abs(thirdTime - secondTime);
  }
  videoExtractor.unselectTrack(videoTrackIndex);
  videoExtractor.selectTrack(videoTrackIndex);
  while (true) {
  int readVideoSampleSize = videoExtractor.readSampleData(byteBuffer, 0);
  if (readVideoSampleSize < 0) {
   break;
  }
  videoBufferInfo.size = readVideoSampleSize;
  videoBufferInfo.presentationTimeUs += sampleTime;
  videoBufferInfo.offset = 0;
  //noinspection WrongConstant
  videoBufferInfo.flags = MediaCodec.BUFFER_FLAG_SYNC_FRAME;//videoExtractor.getSampleFlags()
  mediaMuxer.writeSampleData(writeVideoTrackIndex, byteBuffer, videoBufferInfo);
  videoExtractor.advance();
  }
  while (true) {
  int readAudioSampleSize = audioExtractor.readSampleData(byteBuffer, 0);
  if (readAudioSampleSize < 0) {
   break;
  }
  audioBufferInfo.size = readAudioSampleSize;
  audioBufferInfo.presentationTimeUs += sampleTime;
  audioBufferInfo.offset = 0;
  //noinspection WrongConstant
  audioBufferInfo.flags = MediaCodec.BUFFER_FLAG_SYNC_FRAME;// videoExtractor.getSampleFlags()
  mediaMuxer.writeSampleData(writeAudioTrackIndex, byteBuffer, audioBufferInfo);
  audioExtractor.advance();
  }
  mediaMuxer.stop();
  mediaMuxer.release();
  videoExtractor.release();
  audioExtractor.release();
 } catch (IOException e) {
  e.printStackTrace();
 }
 }