It seems to me that there is a great deal of uncertainty about the boundaries that separate prose from poetry. Does flash fiction blur the line, cross it, dance over it chanting mocking rhymes? What do 'prose' and 'poetry' even mean?
I'll tackle the problem, but a word of warning. I'll tackle it in classical fashion, with an eye toward maximum precision, without much care for value judgments. 'Poetry' and 'prose' should have some purely descriptive value that can be used in a technical, rigorous, precise way to minimize confusion and maximize comprehension. Make of it what you will; but define it clearly.
Originally, more or less all written works were verse. I imagine that's because originally all PRE-written works were verse; it's much easier to remember a series of rhyming couplets about the great hero Jonathan Deane than to remember a series of undifferentiated paragraphs, and when you transfer the Jonathan Deane-iad to paper, future imitators will tend to pick up on its heroic couplet rhyme scheme and use of trochaic tetrameter and versify accordingly.
Now, when I say all written works were verse, I mean all. It was the industry standard. If you wanted to write about your favorite recipes for black soup (the nastylicious Spartan food of choice), you did so in verse. If you wanted to write a treatise on atoms, you did so in verse. If you wanted to write an epitaph for your dog Sneak, you did so in verse. Obviously not all verse was poetry (from the Greek poienai, to create)--not all verse was a Creation, and not all verse-makers were Creators. To an ancient Greek, poetry would be verse of artistic merit, rather than, say, a long shopping list in double dactyls.
Go to the corner store,
Fetch me a frying-pan,
Millions of eggs;
Orange juice, panda's milk,
But I insist on it
Bring me this list or I'll
Shatter your legs.
However, verse is a pain in the ass to write, and as paper became cheaper and writing became more common, the art of versifying trivial ideas was lost. Lazy writers began to write shopping lists and other more ordinary texts in no special word order, in no special meter, and such texts were called prose.
Prose had some claim to artistic legitimacy from fairly early. Herodotus, who wrote the first major history in the western tradition, wrote in prose--with no lack of either artistic merit or charm. Subsequent histories and military texts, including such classics as the Anabasis of Xenophon and the Gallic Wars of Caesar, were written in prose. However, the primary artistic form of language continued to be verse. Nevertheless, it seems likely that an excellent work in prose would be considered more of a 'making' than a flawlessly-versified shopping list.
On a very basic level, then, we are presented with a strange dichotomy (to our modern sensibilities): prose vs. verse, not prose vs. poetry. Prose and verse have very simple, common-sense definitions: words assigned to no particular scheme beyond the natural grammar of the language, and words assigned to a definite rhythmic scheme in which they operate and against which they may push. Or, more simply, prose is everything that isn't verse, and verse is words arranged in formal sequence (generally meter).
Verse has certain beauties that are difficult in prose, and was, for most of recorded history, the prestige medium; those wishing to make their mark in literature tended to produce verse more often than prose, though great prose works occasionally popped up. Yet nowadays verse is far less widely appreciated than prose, far less widely read, and far less prolifically produced. Why?
Prose has the great advantage of taking a lot less time per word to produce. An epic tale in verse may not be that long; Homer's Iliad is 145,000-odd words, which sounds like a lot until you realize that it'd run about 600 pages in paperback, shorter than a lot of Stephen King's novels. Homer was monstrously prolific (if he existed at all, and wasn't a convenient shorthand for an entire tradition); the Odyssey is 120,000; and (as of Aristotle's time, ca. 350 BCE) tradition credited Homer with just the two works (and considered two major works an incredible feat). That's just a little over 1000 pages of paperback text. Stephen King has written 56 books, with word counts ranging from 61,000 (The Girl Who Loved Tom Gordon) to 473,000 (The Stand, uncut edition) and seeming to average around 160,000 words each. That's 8-10 million words of prose in all. Our boy Stephen could never have achieved that level of productivity in verse; it's simply much slower going.
Which wasn't an issue before the printing press. Shorter works were a kind mercy to the poor monks and scribes who had to copy them out; spending more time composing fewer words with greater care was a good career move. But as it became easier and easier to put out more and more text, the pressures and incentives reversed. With sales driving the profitability of literature, rather than patronage, being prolific became an advantage; while a court poet can survive quite nicely on one epic and a lot of little occasional poems, a penny-dreadful author needs to churn out several books a year; he only gets paid once per book moved, after all.
That explains why prose, originally marginal, eventually came to predominate over verse. It does not explain why 'poetry' came to refer not to verse but to a super-set of verse and 'poetic' forms of prose. For prose poems and so-called free verse are just that: they are prose, because they are not verse; but, as they are considered to have a high degree of artistic merit and are generally not narrative, they are poetry. (Somewhere along the way, with the gradual disappearance of narrative verse, 'prose' subsumed narrative).
This seems unproductive to me. The technical distinction between prose and verse makes perfect sense. The original meaning of the word 'poetry'--a meaning which has seen some resurgence in popularity--seems sufficient. Let us call someone a poet if they create an artifact in language that has its own fundamental artistic merit, and use 'poetry' to refer to creative language art that carries artistic merit. 'Poem,' as it tends to be used colloquially, refers to a verse piece, or to a prose piece with strategic line breaks; but we may want to open up the term to refer to compositions of creative merit in either verse or prose.
"But Jonathan," says the White Rabbit, "there is a fundamental distinction between narrative writers and lyricists that goes much deeper than the distinction between prose and verse. Lyric writing does not need to be in verse to seek after its own particular, unique use of language, and 'prose poems,' 'free verse,' and verse poetry all tend to partake of that lyric idiosyncrasy!"
To which Jonathan replies with a shrug. "Use Aristotle's old division into lyric, epic, and tragic, then. Epic = narrative. Lyric = lyric. Tragic = drama--which only means, in the Aristotelian sense, characters talking in their own voices."
"This is the twenty-first century," says the White Rabbit.
"Prose is still prose, and verse is still verse," says Jonathan. "It is nothing more than a value judgement to use the word 'poetry' to refer to particular pieces of short, snappy, weird, syntactically creative prose. No technical purpose is served, only the service of recognizing the artistic merit of those prose (non-verse) pieces."
The White Rabbit, sensing Jonathan's burgeoning penchant for leporid meat, flees the scene. Jonathan, in possession of the field, says, "I only ask that we amplify this tendency and call all creative work in language by the name of poetry; then we can criticize bad novels as bad poetry." He pauses.
He should stop here, but he can't. He raises a hand in the air in a grandiose gesture and adds, "And I hope that some day we can praise the good poetry inherent in an excellent shopping list, whether in verse or prose."