Enabling WebVTT thumbnail previews

WebVTT thumbnails show previews of the stream using the time slider/progress bar.

For details of the thumbnail mosaic structure and how it is described in WebVTT, see https://docs.jwplayer.com/platform/docs.

Prerequisites

A test stream with an accompanying set of image files containing a mosaic of thumbnails and the associated WebVTT file describing the thumbnails' timing and placement within the mosaic. The thumbnails-seeking-webvtt (WebVTT Thumbnails in UnifiedExampleCode) example code project provides an example stream and image file.

Example code

To combat memory issues, the maximum number of thumbnails returned is currently limited to 150. We recommend that there should be no more than 150 Standard Definition thumbnails for any asset.

Preparation

This API footprint for this feature is straightforward; the application needs to implement the ThumbnailsDelegate protocol which provides the prepared() and failed() methods and instantiate the AssetThumnbails class.

The prepared() method will be called only when the parsing and downloading of the thumbnails is complete.
The AssetThumnbails class’ thumbnails() method returns the thumbnails, and importantly, the ownership of the memory for them is transferred with it, so it should not be called again.

For example (from ViewController.swift):

CODE

// extend the ViewController class to provide the ThumbnailsDelegate methods
  extension ViewController: ThumbnailsDelegate {
    func prepared() {
      print("Thumbnails have been prepared")
      let thumbPair = thumbnailDownloader.thumbnails()!
      thumbnailHelper = ThumbnailHelper(
        imageMap: thumbPair.thumbnailDictionary,
        cues: thumbPair.startTimes)
    }

    func failed(error: ThumbnailError, message: String) {
      print("Thumbnail error: \(message)")
      thumbnailsEnabled = false
      thumbnailsFailed = true
    }
  }

Then it is a case of calling the AssetThumbnails class’ prepareThumbnails() method supplying the URL of the WebVTT file and a reference to the instance of the delegate, for example:

CODE

  // ViewController.swift
  let thumbnailDownloader = AssetThumbnails()
  
  override func viewDidAppear(_ animated: Bool) {
    super.viewDidAppear(animated)

    // Expensive or potentially long running processes
    // should only be started
    // once the view has successfully appeared on screen
    thumbnailDownloader.prepareThumbnails(delegate: self, url: self.thumbnailURL)

Determining the thumbnail to paint

The prepared map of thumbnails stored above can be queried using the following example methods, which provide the relevant UIImage for the input progress in milliseconds.

CODE

// ThumbnailHelper.swift
  class ThumbnailHelper {
  let imageMap: [Int32: UIImage]
  let cues: [Int32]

  init(imageMap: [Int32: UIImage], cues: [Int32]) {
      self.imageMap = imageMap
      self.cues = cues
  }

  // Maps a time in seconds as a float, as returned by
  // OTVAVPlayer, to the closest subsequent thumbnail cue
  // as a time in milliseconds, as returned by the
  // ThumbnailHandler.  

  private func findNearestCue(time: Float) -> Int32 {
  // We multiply the time by 1000 here to go from seconds
  // to milliseconds
  // The thumbnails and cues are returned by the
  //ThumbnailHandler already sorted
      var index = binarySearch(inputArr: cues, searchItem: Int32(time * 1000))
      // Generally an exact cue will not be matched so we
      // will receive an insertion index
      if index < 0 {
          index = ~index
      }
      // Protect against the case where the time seeked
      // to is beyond the last available thumbnail cue
      if index >= cues.count {
          index = cues.count - 1
      }
      return cues[index]
  }

  func findFor(time: Float) -> UIImage? {
      print("Getting thumb for time \(time)")
      let nearestCue = findNearestCue(time: time)
      return imageMap[nearestCue]
  }

Presenting the thumbnail to the user

The thumbnail needs to be updated in the timeSliderDidChange function, which in the case of WebVTT thumbnails would mean calling ThumbnailHelper.findFor(time:) to get the correct thumbnail image and updating the UI.

Teardown

The AssetThumbnails class has a reset() method which must be called on destruction of the view, for example:

CODE

// ViewController.swift
thumbnailDownloader.reset()

Low memory warnings

iOS notifies apps when system memory gets low to free up resources before the system has to end them forcibly. In low memory situations while running, it is recommended that thumbnail fetching is halted and any currently held thumbnails disposed of. This can be done by clearing any references to the thumbnail map in view controllers and calling AssetThumbnails.reset() in an appropriate callback where thumbnail fetching is handled.

The following is a simple example of it being handled in a ViewController, as in the example code:

CODE

// ViewController.swift
override func didReceiveMemoryWarning() {
  thumbnailDownloader.reset()
 
  thumbnailHelper = nil
  thumbnailsEnabled = false
  thumbnailsFailed = true
}