Developers can do their work without designers, put it out there, and the end result is that it will be, particularly to the eyes of designers, poorly designed. For the purposes of the developer, the poorly designed aspect may not be all that important, for any number of reasons. But for the designer, this becomes critical.
And the converse isn’t true. The designer can’t go out there and make improvements to the codebase without the help of the developer. Design isn’t as easy to abstract and make into reusable components the way code can. Designers, in general, have less to contribute not because they do less, but because the volume of work that designers do isn’t reusable. There’s no point to contributing non-reusable work. That isn’t what open source does.