Ryan's Blog

The beta of Silverlight 4 has lots of neat stuff packaged with it. One of these things included is native Webcam and Microphone support. In this post I will show how to create a simple webcam viewer complete with A/V and picture capture. However, one thing not currently in the beta is streaming the video and audio to a remote destination such as a file, another client, or a media server. I have not tried to implement this in this functionality in this post because it will most likely be implemented by the time the final release comes out, so rather than “pre-invent” the wheel, I have decided to wait until Microsoft releases a version of their own. I have, however, included implementations of the AudioSink and VideoSink which can be used to pull the raw audio and video and either stream it or save it. Since the the data passed to the OnSamples () method in VideoSink is nothing more than 32bpp ARGB raw bitmap frames it is feasible to create a custom implementation that can save the video to a stream suitable for saving to a file or media server. Be sure to check out the bottom of the post for links to a screenshot of the final result and complete source code. There are also links to shader effects and an attempt at saving the streams to AVI files.

First, to house the video we create a rectangle in XAML. Note the render transform. This will flip the image about it’s Y-Axis so it appears like a mirror, which is more intuitive when viewing the video on screen.

        <Rectangle Height="254" HorizontalAlignment="Left" Margin="31,47,0,0" Name="VideoPane" Stroke="Black" StrokeThickness="1" VerticalAlignment="Top" Width="511" RenderTransformOrigin="0.5,0.5">
            <Rectangle.RenderTransform>
                <TransformGroup>
                    <ScaleTransform ScaleX="-1"/>
                    <SkewTransform AngleX="0" AngleY="0"/>
                    <RotateTransform Angle="0"/>
                    <TranslateTransform/>
                </TransformGroup>
            </Rectangle.RenderTransform>
        </Rectangle>

Next we create two listboxes to show a list of devices suitable for audio and video capture. The user needs to select an audio and video device before starting capture.

        <dataInput:Label Height="18" HorizontalAlignment="Left" Margin="109,426,0,0" Content="Video Devices" Name="label1" VerticalAlignment="Top" Width="83" />
        <ListBox Height="48" Margin="31,443,0,0" Name="VideoCaptureDevices" VerticalAlignment="Top" HorizontalAlignment="Left" Width="256">
            <ListBox.ItemTemplate>
                <DataTemplate>
                    <TextBlock Text="{Binding FriendlyName}" />
                </DataTemplate>
            </ListBox.ItemTemplate>
        </ListBox>
        <dataInput:Label Height="18" HorizontalAlignment="Left" Margin="367,426,0,0" Content="Audio Devices" Name="label2" VerticalAlignment="Top" Width="83" />
        <ListBox Height="48" HorizontalAlignment="Left" Margin="293,443,0,0" Name="AudioCaptureDevices" VerticalAlignment="Top" Width="249">
            <ListBox.ItemTemplate>
                <DataTemplate>
                    <TextBlock Text="{Binding FriendlyName}" />
                </DataTemplate>
            </ListBox.ItemTemplate>
        </ListBox>

The following will display a list of images captured when the user takes a CamShot:

        <ScrollViewer Margin="34,313,33,102" Width="505" VerticalScrollBarVisibility="Hidden" HorizontalScrollBarVisibility="Auto">
            <ItemsControl x:Name="PhotoShots" Height="106">
                <ItemsControl.ItemTemplate>
                    <DataTemplate>
                        <Image Source="{Binding}" Margin="5" Stretch="UniformToFill" Height="70" />
                    </DataTemplate>
                </ItemsControl.ItemTemplate>
                <ItemsControl.ItemsPanel>
                    <ItemsPanelTemplate>
                        <StackPanel Orientation="Horizontal" VerticalAlignment="Center" HorizontalAlignment="Center" />
                    </ItemsPanelTemplate>
                </ItemsControl.ItemsPanel>
            </ItemsControl>
        </ScrollViewer>

In our code behind we create a capture source and an ObservableCollection to hold our images captured from the webcam. We then get the list of devices and set the item source properties when the page is loaded.

        CaptureSource captureSource = new CaptureSource();
        ObservableCollection<WriteableBitmap> photoShots = new ObservableCollection<WriteableBitmap>();

        public MainPage()
        {
            InitializeComponent();
            this.Loaded += new RoutedEventHandler(MainPage_Loaded);
        }

        void MainPage_Loaded(object sender, RoutedEventArgs e)
        {
            VideoCaptureDevices.ItemsSource = CaptureDeviceConfiguration.GetAvailableVideoCaptureDevices();
            AudioCaptureDevices.ItemsSource = CaptureDeviceConfiguration.GetAvailableAudioCaptureDevices();
            PhotoShots.ItemsSource = photoShots;
        }

When we are told to start the capture, make sure the user selected the audio and video capture devices, request access to the device, and if the user allowed us access then set the sink capture sources, create the brush, start capture, and give the rectangle a drop shadow effect.

        private void StartWebCam_Click(object sender, RoutedEventArgs e)
        {
            if (VideoCaptureDevices.SelectedItem != null && AudioCaptureDevices.SelectedItem != null)
            {
                //stop the source if it's started
                if (captureSource.State == CaptureState.Started)
                {
                    captureSource.Stop();
                }

                //VideoCaptureDevice device = CaptureDeviceConfiguration.GetDefaultVideoCaptureDevice();
                VideoCaptureDevice videoDevice = VideoCaptureDevices.SelectedItem as VideoCaptureDevice;

                //AudioCaptureDevice device = CaptureDeviceConfiguration.GetDefaultAudioCaptureDevice();
                AudioCaptureDevice audioDevice = AudioCaptureDevices.SelectedItem as AudioCaptureDevice;

                //check to see if we have already been allowed access or request if we have not
                if (CaptureDeviceConfiguration.AllowedDeviceAccess || CaptureDeviceConfiguration.RequestDeviceAccess())
                {
                    captureSource.VideoCaptureDevice = videoDevice;
                    captureSource.AudioCaptureDevice = audioDevice;

                    //set the video sink's capture source
                    videoSink.CaptureSource = captureSource;
                    audioSink.CaptureSource = captureSource;

                    //create a brush and set the source to the capture source
                    VideoBrush brush = new VideoBrush();
                    brush.SetSource(captureSource);
                    brush.Stretch = Stretch.UniformToFill;

                    //start the capture
                    captureSource.Start();

                    //paint the rectangle with the video
                    VideoPane.Fill = brush;

                    //set the effect
                    VideoPane.Effect = new DropShadowEffect() { ShadowDepth = 7 };
                }
            }
            else
            {
                MessageBox.Show("Please select an audio and a video capture device to start the session");
            }
        }

The following will take a screenshot and add it to the observable collection of WriteableBitmaps:

        private void TakeCamShot_Click(object sender, RoutedEventArgs e)
        {
            if (captureSource != null)
            {
                captureSource.AsyncCaptureImage(
                    (img) =>
                    {
                        photoShots.Add(img);
                    });
            }
        }

Below are the AudioSink and VideoSink implementations. They can be extended to save the streams to whatever destination is needed:

    /// <summary>
    ///
    /// </summary>
    public class MSVideoSink : VideoSink
    {
        /// <summary>
        /// Video
        /// </summary>
        public MemoryStream VideoStream
        {
            get;
            private set;
        }

        /// <summary>
        /// Format
        /// </summary>
        public VideoFormat Format
        {
            get;
            set;
        }

        /// <summary>
        /// Capture started
        /// </summary>
        protected override void OnCaptureStarted()
        {
            VideoStream = new MemoryStream();
        }

        /// <summary>
        /// Capture stops
        /// </summary>
        protected override void OnCaptureStopped()
        {
            //post-process video data
        }

        /// <summary>
        /// Format changing
        /// </summary>
        /// <param name="videoFormat"></param>
        protected override void OnFormatChange(VideoFormat videoFormat)
        {
            //do stuff on format change, i.e. close off stream, etc
            Format = videoFormat;
        }

        /// <summary>
        /// Write the sample data to the stream
        /// </summary>
        /// <param name="sampleTime"></param>
        /// <param name="frameDuration"></param>
        /// <param name="sampleData"></param>
        protected override void OnSample(long sampleTime, long frameDuration, byte[] sampleData)
        {
            //do something here...i.e. stream it to a file, etc
            //VideoStream.Write(sampleData, 0, sampleData.Length);
        }
    }

    /// <summary>
    ///
    /// </summary>
    public class MSAudioSink : AudioSink
    {
        /// <summary>
        /// Capture starts
        /// </summary>
        protected override void OnCaptureStarted()
        {
            WaveformDataStream = new MemoryStream();
        }

        /// <summary>
        /// Post-process audio data
        /// </summary>
        protected override void OnCaptureStopped()
        {
        }

        /// <summary>
        /// Audio format
        /// </summary>
        public AudioFormat Format
        {
            get;
            set;
        }

        /// <summary>
        /// Memory stream
        /// </summary>
        public MemoryStream WaveformDataStream
        {
            get;
            set;
        }

        /// <summary>
        /// Format is changing
        /// </summary>
        /// <param name="format"></param>
        protected override void OnFormatChange(AudioFormat format)
        {
            //do stuff on format change, i.e. close off stream, etc
            Format = format;
        }

        /// <summary>
        /// Save each sample
        /// </summary>
        /// <param name="sampleTime"></param>
        /// <param name="sampleDuration"></param>
        /// <param name="sampleData"></param>
        protected override void OnSamples(long sampleTime, long sampleDuration, byte[] sample)
        {
            //do something here...i.e. stream it to a file, etc
           // WaveformDataStream.Write(sample, 0, sample.Length);
        }
    }

Below is a screenshot of the result:

Download the source code (VS2010):
SilverlightWebcamApp.zip

Edge detection (some cool effects):
http://kodierer.blogspot.com/2009/11/edgecam-silverlight-4-webcam-edge.html
http://kodierer.blogspot.com/2009/07/livin-on-edge-silverlight-parametric_4324.html

Saving streams to AVI files:
http://mtaulty.com/CommunityServer/blogs/mike_taultys_blog/archive/2009/12/10/silverlight-4-more-on-capturing-video-from-webcams.aspx