UPDATE III:

Added some infos how the overflow looks like.
Hope this helps the developers understand what happened. If you miss something, please contact me.
http://www.sven-tantau.de/public_files/mplayer/mplayer_details.txt



UPDATE II:

MPlayer developers patched the problem:

http://www1.mplayerhq.hu/cgi-bin/cvsweb.cgi/main/libmpcodecs/ad_pcm.c.diff?r1=1.18&r2=1.19





UPDATE:
This advisory caused a lot of trouble as the german news website heise.de linked it without checking my statements.
As there is a lot of flaming going on at the mplayer-dev-eng list, please read the postings from the developers to stay informed.
http://mplayerhq.hu/pipermail/mplayer-dev-eng/2005-August/thread.html#36551
Please read my two cents ( http://mplayerhq.hu/pipermail/mplayer-dev-eng/2005-August/036562.html ) also
as I would like to comment on some things.


-------------------------------------------------------------------------------------------

Advisory: mplayer buffer overflow

Product:          mplayer
Affected Version: 1.0_pre7 (tested), 1.0_pre6-r4 (tested), 1.0pre6-3.3.5-20050130 (confirmed)
OS affected:      Linux 2.4.* (tested), 2.6.* (confirmed), other OS not tested
Date:             24.08.2005
Author:           Sven Tantau - http://www.sven-tantau.de/
Advisory-URL:     http://www.sven-tantau.de/public_files/mplayer/mplayer_20050824.txt
Vendor-URL:       http://www.mplayerhq.hu/
Vendor-Status:    informed


Product
=======

>> man mplayer

DESCRIPTION
mplayer  is  a movie player for Linux (runs on many other platforms and
CPU architectures, see the documentation).  It plays most
MPEG/VOB, AVI, ASF/WMA/WMV, RM, QT/MOV/MP4, OGG/OGM, MKV, VIVO, FLI,
NuppelVideo, yuv4mpeg, FILM and RoQ files, supported by many
native and binary codecs.  You can watch VideoCD, SVCD, DVD, 3ivx, DivX
3/4/5 and even WMV movies, too.

...

Details
=======

For high values of the 2 bytes strf parameter in the audio header of a
video file, it is possible to overflow sh_audio->a_buffer, overwrite the
instruction pointer and execute arbitrary code.

Not sure, but I think the problem is in:

af.c: int af_calc_insize_constrained(af_stream_t* s, int len,int
max_outsize,int max_insize);

...as this function is used to calculate declen in dec_audio.c, and
declen is supposed to prevent an overflow.

Instruction pointer gets overwritten in:
libmpdemux/demuxer.c: int demux_read_data(demux_stream_t *ds,unsigned char* mem,int len);

If would like to reproduce this or write an exploit:
Get a copy of 'Animaniacs - Nations of the World.avi'.
(md5: 5ef6428a55c7b00095e2cb5554490acf sha1: 1deeb9640f9864cd5b3db04ffc9a660039a172e4)
Watch it.  :) 
Patch offset 0x12B to 0xFF. Use gdb. Have fun.


History
=======

2005-08-10 issue found by Sven Tantau
2005-08-16 vendor contacted and public disclosure
2005-08-24 no reaction from mplayer team, posting to full disclosure