Producing Participatory Media
Week 5 - February 17

Topics:

  • Streaming Overview
  • Network Architectures
  • Streaming Protocols
  • Media Players
  • Media Servers
  • Codecs
  • Bitrate and Bandwidth Concerns
  • Class Stream and Chat Setup




    • Streaming generally consists of an audio and/or video source fed into an encoder, transmitted to a server which then feeds it out to clients. There of course are variations on this such as streaming text where the source is not audio or video and multicast where a server may not be needed but this is the general model.

    For more information visit:
    Streaming media - Wikipedia

    Streaming Network Architectures




    Multicast
    Multicast is great in theory but in practice isn't very practical. It allows for the serving of a single stream that is replicated throughout the network and any client that chooses may "tune in". The best part is that a server need not have a large amount of bandwidth to support a large number of clients (as opposed to the unicast model). Multicast is also great for keeping clients in synch with each other. Multicast is meant for live media and is not suited for content on-demand. The biggest problem is that the network can not support a great number of multicast sources and often network administrators will not enable multicast on their networks due to bandwidth and security concerns.

    For more information visit:
    Multicast - Wikipedia
    MBone is the Multicast Backbone on the Internet that is now considered obsolete. It was a network that connected multicast enabled networks on the internet.
    Internet2 is a network open to research institutions that allows for much higher bandwidth and often has generally available multicast streams.

     


    Unicast
    Unicast offers a one-to-one relationship between the server and the clients receiving the stream. This is the model most in use today but causes problems when content is popular and the media server does not have enough bandwidth to support all of the client connections. Unicast is well suited for delivery of media on-demand but can have issues when synchronization of live content is a concern.

    For more information visit:
    Unicast Streaming



    Content Delivery Networks (CDN)
    Content Delivery Networks grew out of the need for a better quality of service for end to end streaming that could be achieved with unicast streaming alone. CDNs are generally commercial companies that have developed a network of proxy (for live streams) and caching (for on-demand media) servers. If you utilize a CDN for live streaming you will send your live stream to a server at the CDN which will then intelligently route your stream to their proxy servers based on demand. Utilizing a CDN for on-demand content requires some means to flag content to be cached on the the CDN network. Based upon popularity the CDN will replicate your content on their caching servers which then serve the content to clients. Generally CDN's have proxy/caching servers at major ISPs to be as close as possible to individual clients bypassing the major points of congestion on the internet. In a sense, CDNs are a hybrid of multicast and unicast based networks.

    For more information visit:
    Akamai
    Content Delivery Network - Wikipedia

    Streaming Protocols


    TCP - Transmission Control Protocol
    Not great for streaming audio and video as it does not accept packet loss or delay and forces server to resend packets. It is more important for live streaming audio and video to be in "real time" than to continuously fall behind trying get each and every bit through the network. Using TCP is sometimes unavoidable such as when firewalls are involved or other transmission methods are considered insecure.

    HTTP - Hypertext Transfer Protocol
    Meant for delivery of web pages and associated data (images, text and so on). HTTP utilizes TCP for delivery of information and adds a set of request methods specific to web requests (such as when a form is posted or submitted). Never the less, some streaming servers (particularly MP3 audio streams) use HTTP to deliver their media. The biggest benefit is that it works well even with the most aggressive firewall setups.

    UDP - User Datagram Packet
    Much better for delivery of live streaming audio and video. It does not have the delivery requirements of TCP and therefore packets can be delayed or damaged without causing the server to resend. Doing such allows for more efficient transfer of data over unstable networks and allows clients to stay in synch with each other while watching live streams. The biggest drawback to using UDP is that many network administrators feel that it is insecure often block UDP data on their networks.

    RTP - RealTime Transport Protocol
    Utilizes UDP as a transport mechanism but adds control connection for communication between client and server primarily for quality of service information.

    RTSP - RealTime Streaming Protocol
    A control protocol for streaming, doesn't actually deliver the data but acts as the means to setup the transport and control the flow. Utilizes the application or user selected data transport method (RTP, UDP, TCP, HTTP). Offers "VCR" capabilities for streaming media (pause, seek, etc..).

    For more information visit:
    Wikipedia - TCP/IP

    Codecs
    Codec stands for Compress/Decompress. It is the means to deliver or store high quality audio and video in less space and using less bandwidth than would normally be required.

    Video Capture and Compression Theory
    A frame of video, essentially an image, at a resolution of 640 pixels by 480 pixels with 24 bits per pixel (RGB, 255 values of Red, 255 values of Blue and 255 values of Green) would occupy 7,372,800 bits. At 15 frames per second we are looking at 110,592,000 bits per second (or 13,824,000 bytes per second or 13,824 kilobytes per second or 13.824 megabytes per second). This is waaaaaaaaaaaaaaay more than the average internet connection. Broadband connections (DSL and Cable Modems) typically allow for 1mbps (megabit per second), which more than 100 times fewer bits per second than uncompressed video. Hence the need for compression.
    Video codecs typically employ some combination still frame encoding for key frames (representing a full frame) and delta frames (current difference from the previous key frame). Specific mathematical algorithms that video codecs employ are: Discrete Cosine Transforms (DCT), Discrete Wavelet Transforms (DWT), Fractals and various combinations of them all.

    Audio Capture Theory
    Audio is captured by sampling the amplitude of incoming sound waves (amplitude (level) and frequency (time)). The sampling rate is the number of of these samples taken per second, the higher the sampling rate the more accurate the representation of the wave. Sampling rate is expressed in kHz (kilohertz) which is 1000's of samples per second. The other factor of audio capture is the number of bits per sample (quantization), 8 bit samples have 256 steps to represent a given amplitude whereas 16 bits give 65,536 steps per representation.

    The Nyquist Theorem states that an sampling rates must be at least 2 times the highest frequency to represent a wave. The higher the sampling rate the higher the frequency that can be sampled and represented accurately. This is why music utilizes a much higher sampling rate than voice, it utilizes a much wider range of frequencies.

    A sampling rate of 22.05 kHz or 22,050 samples per second at 16 bits per sample yields 352,800 bits per second (or 352.8 kilobits (kb) or 44.1 kilobytes (KB)). This is much more feasible for transmission via broadband connections but could still use some compression.

    Specific Codecs
    There is no "Best" codec... Each codec has it's plusses and minuses and each is good for its particular use.

    H.261
    Early video conferencing codec (1990) - Video conferencing requires fast encode/decode for low latency.

    H.263
    More recent video conferencing codec (1996) Used in earlier versions of Flash (MX and version 7).

    MJPEG - Motion JPEG
    Individual frames of video encoded as JPEG. Good for analog capture for non-linear video editing, every frame is in essence a keyframe and cuts can be made anywhere..

    MPEG-1
    Targeted for CDROM speeds (1.5 mbps)

    MP3
    Audio layer of MPEG-1, MPEG-1 Layer 3.

    MPEG-2
    High bitrate codec used for digital cable, broadcast and satellite as well as DVD. High royalty rates as well.

    MPEG-4
    Range of bitrates, developed for a large variety of uses including multimedia over unreliable networks such as the internet and mobile phones. Includes components for interactivity and many different audio and video profiles.

    H.264 -     also known as MPEG-4 Part 10 or AVC (Advanced Video Coding)
    Same as MPEG-4 but a newer "profile" and much more scalable than previous MPEG-4 profiles have been. The next big thing.

    AAC - Advanced Audio Coding
    Audio portion of MPEG-4

    DV
    Tape format and "light" compression codec. Great for digital video capture and editing. High bitrate.


    Other Proprietary codecs worth mentioning
    Sorenson Video 3
    DivX
    3ivx - MPEG-4
    RealVideo 10
    Microsoft Windows Media 8 (ASF)
    Microsoft MPEG-4 v2 (Not really MPEG-4, Encodes to AVI, DivX stole it)
    Q-Design Music
    On2's VP 6 Codec - Used in Flash 8

    Open Source Codecs worth mentioning
    Ogg Vorbis and Ogg Theora
    Dirac Originally developed by the BBC
    XviD
    On2's VP3 (now part of the Ogg project)

    For more information:
    Codec Comparison
    QuickTime Playback Compatibility Chart
    H.264 Begins its Ascent
    Flash Video | Optimizations and Tools
    Live TV is dead, and we're noticing the smell

    Formats
    AVI - Audio Video Interleave
    Developed by Microsoft for audio and video data utilizing VFW (Video for Windows)

    QuickTime
    Apple's proprietary audio and video format. The basis for the MPEG-4 container.
    http://www.apple.com/quicktime/

    ASF - Advanced Systems Format
    Microsoft's proprietary file format for Windows Media. Synchronized Audio/Video data.
    http://www.microsoft.com/windows/windowsmedia/format/asfspec.aspx


    Players
    QuickTime
    Windows Media
    RealPlayer
    Miscellaneous MP3 Players (stemming from WinAmp by Nullsoft

    Servers
    QuickTime/Darwin Streaming Server
    Windows Media Server
    RealServer/Helix
    Miscellaneous MP3 Servers (stemming from Shoutcast by Nullsoft (now owned by AOL))

    Encoders
    RealProducer
    Windows, Linux and Sun Solaris (there are QuickTime export plugins available and they used to have a Mac version), Free version available. Does live encoding to server as well as on-demand. Choice of bitrate, quality and so on. Has something called "SureStream" which only works on RealServer. "SureStream" encapsulates alternate quality and bitrate streams in one file. When a connection is made with a client the server chooses the best stream to use based upon the bandwidth available to the client. The RealProducer will also do variable bit rate encoding (VBR) but this requires two passes over the file and is therefore not available when doing live encoding. VBR allows for the overall delivery bitrate to remain constant but leaves the ability to have the bitrate per frame change so that frames that don't need the full bitrate give their bits to an earlier or later frame that does.

    To link off of a web page using the RealServer you need to either manually create a ram file or utilize the server's built in ramgen program. A ram file tells the browser to open up the player and hands the ram file to it. The ram file contains the true link to the stream.

    To view a real stream off of a webserver you can not use SureStream but will instead need to choose a single bitrate encode.


    QuickTime Broadcaster
    Mac only, free. Does live encoding as well as "save file to disk" for on-demand delivery.
    A meta file similar in concept to Real's ram file, called a QuickTime Media Link: qtl is used when linking to a stream from a web browser. QuickTime Media Links can be created by QuickTime Pro through the Export command or created in a text editor and given the .qtl extension.

    The contents.qtl text file:

       <?xml version="1.0"?>
   <?quicktime type="application/x-quicktime-media-link"?>
   <embed
   autoplay="true"
   src="rtsp://streaming.server.com/path/to/your/movie.mov"
   />


    QuickTime Media Links work on QuickTime 5 and above. For supporting earlier versions of QuickTime you must use a reference movie embedded on a webpage. (I will let you look that up yourselves).

    For a live stream, the server sends an SDP (Session Description Protocol) to the player which then starts requesting the live stream. This is a standard way of doing things in the MPEG-4/RTSP world.

    For more information:
     QuickTime XML Exporter
    QuickTime Media Links
    Reference Movies


    QuickTime Pro
    The QuickTime swiss army knife. Great for compressing for on-demand files and translating files for use on or off a QuickTime Streaming Server. For use on a streaming server, you need to export as a hinted file. For use on a web page it is a good idea to use fast start so that the file can be played while downloading.

    For more information:
    QuickTime Broadcaster
    Session Description Protocol RFC
    QuickTime - Preparing Media for Streaming
    QuickTime - Delivering Live Streaming Media