If it is a non supported codec on the Xbox, it will not play. If you select a movie which is in a non supported container such as an mkv, you will notice there will now be a subtle difference on the display on the Xbox.As you can see from the above screen shot, the mkv movie has [MEncoder] written next to it.
Here is the page where I found out how to make it work. If you have a writable CD a thumb drive, use the thumb drive, because it's much easier.To view your Video content from your PS3 Media server, switch on your Xbox and scroll to the My Xbox menu. The PS3 Media Server isn't that slick just yet and offers you a list of all the folders whether they have videos in them or not.You will notice a couple of extra option here, namely Select Movies to continue to browse your videos.This signifies that by default, mencoder will transcode this movie on the fly to make it a playable format on the Xbox.From testing this on a wired network with high definition movies, although the network traffic did peak at over 90Mb/s, the quality was excellent.The roll out of these media features was not without its hiccups though as one of the Xbox One’s biggest media features, the ability to watch live TV through the console, didn’t work correctly in PAL territories until an update in March.