Video Quality Optimization

API Reference: UI.VideoView

If your project has video recording feature, you will eventually want to upload it to some server, for various of purposes.

As the time goes, cameras on mobile devices get better and better. Also, at the same time mobile devices fall short on network and due to the fact that upload speeds might vary, you will want to compress your video at some point.

Newer devices can even record videos on 4K resolution. Even 5 second of the recording might result in ~20 MiB of data. At that point, developer needs to optimize the quality in order to make their app work as intended.

Get to Know about the Existing Video Qualities

Since Smartface is a Native Cross-Platform Framework, Smartface does not have any self-made camera application. Smartface uses System API calls to reach out the native Camera Application of the device itself.

This allows the users to natively customize their camera settings straight from Camera Application on Smartface, but video quality-wise, the options may be limited and may not be tweaked heavily.

Smartface requests specific quality patterns from the camera, however the perspective of quality might differ between different devices.

For Example, when Quality is set on low by using VideoQuality.LOW:

• Huawei devices reduce the quality to 144p.
• Xiaomi devices reduce the quality to 144p.
• Some Samsung devices reduce the quality to 1080p or do not even reduce the quality.
• Older Apple devices reduce the quality to 480p.
• Newer Apple devices reduce the quality to 720p.

The example above is subject to change over the System Updates. Do not take it as an official quality reference.

As it can be seen, different brand and devices behave differently when their native System API calls are used.

Sample Video Quality Playground

Smartface has prepared a sample project to play around video qualities on your Smartface Emulator. Check it out here on GitHub.

Change Video Quality

API Reference: Device.Multimedia.VideoQuality

Due to variety of Android devices, mutual quality tweak has only two parameters on Android:

• VideoQuality.HIGH : Does not change the quality of camera
• VideoQuality.LOW : Returns with the lowest quality possible by the device of the camera

info

Also, note that Video Quality can be changed on the Camera Options itself. As a developer, do not expect same quality over all devices since that options is configurable from the Camera Application.

Please note that CAMERA permission should be granted to record any video. Here is an example of how to add videoQuality parameter:

For further instructions about permission, refer to this documentation:

Application Permission Management

import Multimedia from "@smartface/native/device/multimedia";
import Page1Design from "generated/pages/page1";
import Application from "@smartface/native/application";

class Page1 extends Page1Design {
  constructor() {
    // Your constructor code
  }

  async recordVideo() {
    await permission.getPermission({
      androidPermission: Application.Android.Permissions.CAMERA,
      permissionText: "Camera Permission is Required",
      iosPermission: permission.IOS_PERMISSIONS.CAMERA,
    });
    Multimedia.recordVideo({
      page: this,
      videoQuality: Multimedia.VideoQuality.LOW,
      onSuccess: ({ video }) => {
        try {
          // Do stuff with video
        } catch (err) {
          console.error("onSuccessError ", err);
        }
      },
      onFailure: (e) => {
        console.log(e);
      },
    });
  }
}

caution

In order to use recordVideo properly on Android 12L (API Level 32) and below, you also need to getREAD_EXTERNAL_STORAGE permission.

Get the READ_EXTERNAL_STORAGE Permission

At the onSuccess() event of recordVideo() method, notice that the output property called video is a File instance and it is stored on the temporary storage. To reach that, Android 12L (API Level 32) and below needs the READ_EXTERNAL_STORAGE permission to do any read operation.

import Multimedia from "@smartface/native/device/multimedia";
import Page1Design from "generated/pages/page1";
import permission from "@smartface/extension-utils/lib/permission";
import Application from "@smartface/native/application";
import System from "@smartface/native/device/system";

class Page1 extends Page1Design {
  constructor() {
    // Your constructor code
  }

  async recordVideo() {
    await permission.getPermission({
      androidPermission: Application.Android.Permissions.CAMERA,
      permissionText: "Camera Permission is Required",
      iosPermission: permission.IOS_PERMISSIONS.CAMERA,
    });
    System.android.apiLevel < 33 &&
      (await permission.getPermission({
        androidPermission:
          Application.Android.Permissions.READ_EXTERNAL_STORAGE,
        permissionText: "File Permission is Required",
      }));
    Multimedia.recordVideo({
      page: this,
      videoQuality: Multimedia.VideoQuality.LOW,
      onSuccess: ({ video }) => {
        try {
          // Or you can request for Permission here. Depends on your life-cycle.
          // Do stuff with video
        } catch (err) {
          console.error("onSuccessError ", err);
        }
      },
      onFailure: (e) => {
        console.log(e);
      },
    });
  }
}

Overcome Quality Differences on Known Android Devices

As it stated previously, different devices require different quality settings. The brands that are tested and known to behave drastically behave are;

• Xiaomi
• Huawei

We need to set the quality of those brands as HIGH and the other Android devices as LOW. Expanding the example above:

import Multimedia from '@smartface/native/device/multimedia';
import Page1Design from 'generated/pages/page1';
import Hardware from '@smartface/native/device/hardware';
import permission from '@smartface/extension-utils/lib/permission';
import Application from '@smartface/native/application';
import System from '@smartface/native/device/system';

const lowQualityAndroidDevices = [
    'xiaomi',
    'huawei'
];

class Page1 extends Page1Design {
    constructor() {
        // Your constructor code
    }

    async recordVideo() {
        /**
         * We lowercase brandName string because some devices report
         * their brandName in all capital like HUAWEI
         */
        const isLowQualityDevice = lowQualityAndroidDevices.some((device) => Hardware.brandName.toLowerCase().includes(device));

        await permission.getPermission({
            androidPermission: Application.Android.Permissions.CAMERA,
            permissionText: "Camera Permission is Required",
            iosPermission: permission.IOS_PERMISSIONS.CAMERA
        });
        System.android.apiLevel < 33 && await permission.getPermission({
            androidPermission: Application.Android.Permissions.READ_EXTERNAL_STORAGE,
            permissionText: "File Permission is Required"
        });
        Multimedia.recordVideo({
            page: this,
            videoQuality: isLowQualityDevice ? Multimedia.VideoQuality.HIGH : Multimedia.VideoQuality.LOW,,
            onSuccess: ({ video }) => {
                try {
                   // Do stuff with video
                } catch (err) {
                    console.error('onSuccessError ', err);
                }
            },
            onFailure: (e) => {
                console.error('onFailureError ', e);
            }
        });
    }
}

Change Video Quality on iOS

Since iOS devices are all regulated by Apple, they offer various of quality parameters makes the Quality quite configurable. The available parameters are:

• VideoQuality.HIGH : Does not change the quality of camera
• VideoQuality.LOW : Returns with the lowest quality possible by the device of the camera
• VideoQuality.iOS.MEDIUM : Returns the medium quality set by Apple. Roughly equals to 640x480
• [VideoQuality.iOS.TYPE640X480](https://ref.smartface.io/classes/multimedia.Multimedia.html#VideoQuality: Sets the pixels to 640x480
• VideoQuality.iOS.TYPEIFRAME1280x720: Sets the iframe to 1280x720
• VideoQuality.iOS.TYPEIFRAME960x540: Sets the iframe to 860x540

Prior to observation, the lowest qualities on size are MEDIUM and LOW.

Example for reducing the quality for iOS only:

import Multimedia from "@smartface/native/device/multimedia";
import Page1Design from "generated/pages/page1";
import permission from "@smartface/extension-utils/lib/permission";
import Application from "@smartface/native/application";
import System from "@smartface/native/device/system";

class Page1 extends Page1Design {
  constructor() {
    // Your constructor code
  }

  async recordVideo() {
    await permission.getPermission({
      androidPermission: Application.Android.Permissions.CAMERA,
      permissionText: "Camera Permission is Required",
      iosPermission: permission.IOS_PERMISSIONS.CAMERA,
    });
    System.android.apiLevel < 33 &&
      (await permission.getPermission({
        androidPermission:
          Application.Android.Permissions.READ_EXTERNAL_STORAGE,
        permissionText: "File Permission is Required",
      }));
    Multimedia.recordVideo({
      page: this,
      videoQuality: Multimedia.VideoQuality.iOS.TYPE640x480,
      onSuccess: ({ video }) => {
        try {
          // Do stuff with video
        } catch (err) {
          console.error("onSuccessError ", err);
        }
      },
      onFailure: (e) => {
        console.log(e);
      },
    });
  }
}

caution

The snippet above will break on Android. Let's combine them both.

Implementation and Examples of Changing Video Quality

Let's combine the implementations on a single code snippet and go over the possible caveats.

Handle Video Quality on Both Platforms

Let's say you want to have LOW quality on Android and MEDIUM quality on iOS. Your code should be similar to:

import Multimedia from "@smartface/native/device/multimedia";
import Page1Design from "generated/pages/page1";
import System from "@smartface/native/device/system";
import permission from "@smartface/extension-utils/lib/permission";
import Application from "@smartface/native/application";

class Page1 extends Page1Design {
  constructor() {
    // Your constructor code
  }

  async recordVideo() {
    const androidQuality = Multimedia.VideoQuality.LOW;
    const iOSQuality = Multimedia.VideoQuality.iOS.MEDIUM;

    System.android.apiLevel < 33 &&
      (await permission.getPermission({
        androidPermission:
          Application.Android.Permissions.READ_EXTERNAL_STORAGE,
        permissionText: "File Permission is Required",
      }));
    await permission.getPermission({
      androidPermission: Application.Android.Permissions.CAMERA,
      permissionText: "Camera Permission is Required",
      iosPermission: permission.IOS_PERMISSIONS.CAMERA,
    });
    Multimedia.recordVideo({
      page: this,
      videoQuality:
        System.OS === System.OSType.IOS ? iOSQuality : androidQuality,
      onSuccess: ({ video }) => {
        try {
          // Do stuff with video
        } catch (err) {
          console.error("onSuccessError ", err);
        }
      },
      onFailure: (e) => {
        console.log(e);
      },
    });
  }
}

If we also want to include Xiaomi and Huawei implementation:

import Multimedia from "@smartface/native/device/multimedia";
import Page1Design from "generated/pages/page1";
import System from "@smartface/native/device/system";
import Hardware from "@smartface/native/device/hardware";
import permission from "@smartface/extension-utils/lib/permission";
import Application from "@smartface/native/application";

const lowQualityAndroidDevices = ["xiaomi", "huawei"];

class Page1 extends Page1Design {
  constructor() {
    // Your constructor code
  }

  async recordVideo() {
    const isLowQualityDevice = lowQualityAndroidDevices.some((device) =>
      Hardware.brandName.toLowerCase().includes(device)
    );
    const androidQuality = isLowQualityDevice
      ? Multimedia.VideoQuality.HIGH
      : Multimedia.VideoQuality.LOW;
    const iOSQuality = Multimedia.VideoQuality.iOS.MEDIUM;

    System.android.apiLevel < 33 &&
      (await permission.getPermission({
        androidPermission:
          Application.Android.Permissions.READ_EXTERNAL_STORAGE,
        permissionText: "File Permission is Required",
      }));
    await permission.getPermission({
      androidPermission: Application.Android.Permissions.CAMERA,
      permissionText: "Camera Permission is Required",
      iosPermission: permission.IOS_PERMISSIONS.CAMERA,
    });
    Multimedia.recordVideo({
      page: this,
      videoQuality:
        System.OS === System.OSType.IOS ? iOSQuality : androidQuality,
      onSuccess: ({ video }) => {
        try {
          // Do stuff with video
        } catch (err) {
          console.error("onSuccessError ", err);
        }
      },
      onFailure: (e) => {
        console.log(e);
      },
    });
  }
}

Convert the Video Output to Base64

API References:

• Blob
• IO.File

When you want to send the recorded video to server, most of the time it needs to be sent as base64. The steps we need to do:

• Get the video content from onSuccess
• Open the file in binary mode
• Read the file to the end
• Convert blob to base64 using Blob.toBase64 or Blob.toBase64Async

import Multimedia from "@smartface/native/device/multimedia";
import Page1Design from "generated/pages/page1";
import System from "@smartface/native/device/system";
import Hardware from "@smartface/native/device/hardware";
import File from "@smartface/native/io/file";
import Blob from "@smartface/native/blob";
import permission from "@smartface/extension-utils/lib/permission";
import Application from "@smartface/native/application";

const lowQualityAndroidDevices = ["xiaomi", "huawei"];

class Page1 extends Page1Design {
  base64Video = "";
  constructor() {
    // Your constructor code
  }

  async recordVideo() {
    const isLowQualityDevice = lowQualityAndroidDevices.some((device) =>
      Hardware.brandName.toLowerCase().includes(device)
    );
    const androidQuality = isLowQualityDevice
      ? Multimedia.VideoQuality.HIGH
      : Multimedia.VideoQuality.LOW;
    const iOSQuality = Multimedia.VideoQuality.iOS.MEDIUM;

    System.android.apiLevel < 33 &&
      (await permission.getPermission({
        androidPermission:
          Application.Android.Permissions.READ_EXTERNAL_STORAGE,
        permissionText: "File Permission is Required",
      }));
    await permission.getPermission({
      androidPermission: Application.Android.Permissions.CAMERA,
      permissionText: "Camera Permission is Required",
      iosPermission: permission.IOS_PERMISSIONS.CAMERA,
    });
    Multimedia.recordVideo({
      page: this,
      videoQuality:
        System.OS === System.OSType.IOS ? iOSQuality : androidQuality,
      onSuccess: ({ video }) => {
        try {
          // Do stuff with video
          this.convertVideoToBase64(video);
        } catch (err) {
          console.error("onSuccessError ", err);
        }
      },
      onFailure: (e) => {
        console.log(e);
      },
    });
  }
  convertVideoToBase64(file: File): void {
    try {
      const fileStream = file.openStream(
        FileStream.StreamType.READ,
        FileStream.ContentMode.BINARY
      );
      // If your video is supposed to be really big like >100 MiB, consider dividing them into chunks.
      if (fileStream.isReadable) {
        const fileBlob = fileStream.readToEnd();
      }
      if (fileBlob instanceof Blob) {
        // If the process is taking too much time, use async
        this.base64Video = fileBlob.toBase64();
        // Do stuff with the base64
      } else {
        this.base64Video = "";
      }
    } catch (e) {
      this.base64Video = "";
      console.error(e);
    }
  }
}

Important Warning

When your video size is big e.g. >200 MiB, consider dividing the binary into blocks. The moment you use FileStream.readToEnd()function, the application will write the contents to RAM. The amount of RAM is allocated to each application is limited, therefore the System might decide to either stop allocating more RAM or kill your application when you request too much RAM.

Convert the Video Output to Mp4

Since the resulting video file might differ on extension, you might also want to convert your video to an another format. Currently, only MP4 format is supported.

import Multimedia from "@smartface/native/device/multimedia";
import Page1Design from "generated/pages/page1";
import System from "@smartface/native/device/system";
import Hardware from "@smartface/native/device/hardware";
import permission from "@smartface/extension-utils/lib/permission";
import Application from "@smartface/native/application";

const lowQualityAndroidDevices = ["xiaomi", "huawei"];

class Page1 extends Page1Design {
  constructor() {
    // Your constructor code
  }

  async recordVideo() {
    const isLowQualityDevice = lowQualityAndroidDevices.some((device) =>
      Hardware.brandName.toLowerCase().includes(device)
    );
    const androidQuality = isLowQualityDevice
      ? Multimedia.VideoQuality.HIGH
      : Multimedia.VideoQuality.LOW;
    const iOSQuality = Multimedia.VideoQuality.iOS.MEDIUM;

    System.android.apiLevel < 33 &&
      (await permission.getPermission({
        androidPermission:
          Application.Android.Permissions.READ_EXTERNAL_STORAGE,
        permissionText: "File Permission is Required",
      }));
    await permission.getPermission({
      androidPermission: Application.Android.Permissions.CAMERA,
      permissionText: "Camera Permission is Required",
      iosPermission: permission.IOS_PERMISSIONS.CAMERA,
    });
    Multimedia.recordVideo({
      page: this,
      videoQuality:
        System.OS === System.OSType.IOS ? iOSQuality : androidQuality,
      onSuccess: ({ video }) => {
        try {
          Multimedia.convertToMp4({
            videoFile: video,
            outputFileName: `video-${new Date().toISOString().split(".")[0]}`,
            onCompleted: ({ video: convertedVideo }) => {
              // Do stuff with the mp4 output
            },
            onFailure: () => {
              console.error("convert error");
            },
          });
        } catch (err) {
          console.error("onSuccessError ", err);
        }
      },
      onFailure: (e) => {
        console.log(e);
      },
    });
  }
}

Warning for iOS

Due to a technical caveat, iOS devices turn the video sideways when converted to mp4. To fix that issue, refer to the code below

import Multimedia from "@smartface/native/device/multimedia";
import Page1Design from "generated/pages/page1";
import System from "@smartface/native/device/system";
import Hardware from "@smartface/native/device/hardware";
import permission from "@smartface/extension-utils/lib/permission";
import Application from "@smartface/native/application";

const lowQualityAndroidDevices = ["xiaomi", "huawei"];

class Page1 extends Page1Design {
  constructor() {
    // Your constructor code
  }

  async recordVideo() {
    const isLowQualityDevice = lowQualityAndroidDevices.some((device) =>
      Hardware.brandName.toLowerCase().includes(device)
    );
    const androidQuality = isLowQualityDevice
      ? Multimedia.VideoQuality.HIGH
      : Multimedia.VideoQuality.LOW;
    const iOSQuality = Multimedia.VideoQuality.iOS.MEDIUM;

    System.android.apiLevel < 33 &&
      (await permission.getPermission({
        androidPermission:
          Application.Android.Permissions.READ_EXTERNAL_STORAGE,
        permissionText: "File Permission is Required",
      }));
    await permission.getPermission({
      androidPermission: Application.Android.Permissions.CAMERA,
      permissionText: "Camera Permission is Required",
      iosPermission: permission.IOS_PERMISSIONS.CAMERA,
    });
    Multimedia.recordVideo({
      page: this,
      videoQuality:
        System.OS === System.OSType.IOS ? iOSQuality : androidQuality,
      onSuccess: ({ video }) => {
        try {
          Multimedia.convertToMp4({
            videoFile: video,
            outputFileName: `video-${new Date().toISOString().split(".")[0]}`,
            onCompleted: ({ video: convertedVideo }) => {
              if (System.OS === System.OSType.IOS) {
                //@ts-ignore
                Multimedia.ios._fixVideoOrientation({
                  videoFile: convertedVideo,
                  onCompleted: ({ video: orientationFixedVideo }) => {
                    // Do stuff with correct video
                  },
                  onFailure: () => {
                    console.error("fix orientation error");
                  },
                });
              } else {
                // Do stuff on Android, fixOrientation call cannot be done on Android.
              }
            },
            onFailure: () => {
              console.error("convert error");
            },
          });
        } catch (err) {
          console.error("onSuccessError ", err);
        }
      },
      onFailure: (e) => {
        console.log(e);
      },
    });
  }
}

You can call recordVideo function in the click method of a button or anything else.