I would likely attack this with the Custom Filter widget:
You could add a tag ("nosearch" maybe) to all the docs and media files you don't want to search, then use a Must Not match type in the custom filter widget to eclude them, for one approach.
You might a lso be able to target specific extensions with the custom filter widget, but I can't think off the top of my head how, exactly.
Edited to add: In 7.3 I did this by adding a Must_Not Match Query filter for "jpg" on the "extension" field:
