Video SDK: gameplay capture for Medal, USA
Gaming and eSports are areas we’re keen on for two specific reasons: a) our product helps developers work through lots of video and audio issues on a Windows PC, which is the world’s most important platform for gamers, and b) the evolution of both areas is leading to new ways for people to create and consume gaming content.
We’ve spoken to Pim de Witte and Zaid Elnasser— founders of Medal, who have just raised $3.5 million for a clipping service that makes it easy for users to upload short video game replays for friends to enjoy.
A vision for the new generation
Medal’s path is similar to that of Discord, who started out as a mobile gaming company, but felt that they were better off developing a tool that helps teams coordinate while playing. With their first title, Medal learned that promoting the game was a greater challenge than building it, so they pivoted to a platform for sharing moments of games between friends. Pim explains:
The main way our users wanted to help us organically grow was by sharing clips of our game. So we built a platform where people could share clips of games so that we could help game developers promote games.
The product allows you to click through moments that were upvoted by other users, thus serving as a discovery tool for people looking for new games to play:
Think about how you discover a new game. You go on Twitter, you go on Facebook, you go on VK… a lot of the times you see moments and you see videos of games and that’s what drives you to play the game… and we’ve noticed that more and more.
Medal targets non-professional, regular gamers who wouldn’t want to learn any special skills for sharing their gameplay — the desktop app provides a hotkey that simply saves 15 to 60 seconds of what you’ve just played. The clips, as Zaid explains, are mostly consumed via Medal’s iOS and Android apps:
Another reason was we wanted to take the PC gaming experience with us on the mobile — that was why we were making mobile games in the first place, and another way to do that was sharing clips. You’re not actually playing the game but you’re still kind of bringing the PC gaming experience with you on your phone; so that’s another reason we built Medal.
The product was released last May and has been growing fast: Fortnite’s recent in-game Marshmello concert was uploaded more than 4 thousand times. Pim says:
As a business our focus is essentially to be what Instagram is to capturing real life… We believe that the next generation will capture digital moments as much if not more than real moments — so we’re building a platform to enable that.
Putting MFormats to a real test
The most challenging part of Medal’s product is the desktop video capture app. As Zaid explains, the first version was built with FFmpeg:
We’ve basically been piping the API of FFmpeg and then reading the data out and it was fine at first, it worked for our initial release, but we had no audio, because we didn’t have any control over the frames as they came in and we couldn’t modify or manipulate the data in any way — so we had to start looking for something that gave us a bit more control. So we had played with building our own wrapper of FFmpeg and we looked into Media Foundation; and then I stumbled on Medialooks.
It didn’t just work out of the box. MFormats is usually used with professional video capture hardware (such as that from Blackmagic or AJA), where video frames and audio samples are produced by sources that share a common reference clock: it is clear how to process them in a way that makes sure audio and video stay in sync.
In gameplay capture, the situation is completely different: video frames are grabbed from the graphics card and audio is produced by the audio device. With MFormats it was possible to address this problem by creating frames manually. This flexibility of our architecture, the fact that MFormats gives the developer full control over each frame was the deciding factor for Medal’s team:
For us control is being able to manipulate the frames as they come in. Using FFmpeg command line we just kind of pipe them in and FFmpeg does what it wants based on your encoding options; with Medialooks we could actually manipulate the data and generate the frames ourselves so we knew what was being encoded.
This level of control was also something that helped Medal make a long-term investment into the Medialooks SDK; they didn’t want to be limited in any way by the technology that their app is built with:
We wanted full control over what we were building so that we can extend it in any way we need to in the future, and we didn’t have that with our previous solution.
While evaluating MFormats, Zaid communicated with our team to address some of his concerns and checked out our sample apps; when he got down to coding, it was within a couple of days that he had the video processing “swapped out to Medialooks”.
Since gamers use a variety of computers and play a variety of games, some of which would push the limits of even the most up-to-date hardware, performance was something Zaid evaluated very carefully:
We try to support the slowest gaming computers that we could. We wanted no impact on the user’s game so that’s how we defined it. If they can play a game without noticing that Medal was running then that was a win for us.
If not for MFormats, Medal’s team would have built their own wrapper around FFmpeg or used Media Foundation. “It just would have taken us a lot longer,” Zaid says. Obviously, using MFormats was much easier: Zaid could continue to use C# as his development language with no additional learning to be done.
The design of MFormats itself is kind of how we wanted the previous solutions to work — it just made sense in the way that if functions so it was pretty easy to grasp the overall concept.
Time to market was of utmost importance for Medal’s product. Pim reflects on how they made the decision:
We basically just calculated the amount of engineering time that it would take to build it ourselves and compared that with the license price, and the better option was just getting the license. It saved us a lot of time.
Although the app has come a long way since we started working together, there’s still a lot of work to be done: Medal’s product is quickly evolving, gaining users and becoming more and more demanding as the product is being distributed to a wider range of devices. We are excited to continue our work to contribute to this company’s vision.