Disclaimer: I am not an expert. I have only sold 3 short stories to paying markets. One of these was not SF. You are perfectly welcome to say, "Well, if she hasn't sold a novel yet, she must be talking through her hat," and move on. However, the question of what sells has been occupying my mind lately. I've even been seriously wondering whether there's any point in me continuing to write fiction, so I'm airing my conclusions here, for my own benefit as much as anything, but I hoped someone might also find my ideas thought provoking.
I had high hopes for Moving a Mountain, but it just hasn't attracted any interest despite the fact that several of my friends seem to have genuinely enjoyed it. I conclude from this that it's not badly written but it's attracting less interest from publishers than my previous novels. This means that my intuition as to what makes a good story is functioning less well then when I was younger, despite my prose writing skills getting better. As I obviously had no clue as to what would sell and what wouldn't, I needed to give the matter some serious thought. So here is the first instalment of my conclusions, such as they are.
Writing a novel is a huge gamble. It takes hours of work which means, for the aspiring amateur working full time, it's likely to take at least a couple of year's worth of spare time which will be spent writing, inventing, planning, revising and polishing. It's literally a labour of love because there's no contract and absolutely no guarantee that editor will ever deign to read the thing, let alone publish it.
What an aspiring writer desperately wants to know is, what will sell? How often have I thought, "Just tell me what you want and I'll write it! Don't keep making me play this stupid guessing game." The frustration of feeling that you're groping in the dark, writing things only to have the agent or editor say, "It didn't grab me, sorry" is a frustration known to many, me included. I thought that I needed to be able to write better. I worked away at my writing, trying to produce more vivid prose, I tried to make my characters engaging and ensure that my plots made sense. But it still wasn't enough.
Why didn't the story grab the editor? If only we knew the answer to that question! Usually it's put down to them having read one too many vampire novels that day so yet another vampire novel was doomed more or less unread. Perhaps there's some truth in that, but I'm about to suggest that it's a bit more complex.
People often say things like, "Oh, no one's buying urban fantasy any more, it's been done to death," and then someone will not only sell an urban fantasy novel, but it will become a best seller, somehow appealing to people who don't even usually read the genre.
How can this be when urban fantasy is passé? How can one out of several superficially similar novels succeed while the others sink almost without trace? If we could answer these questions, then we'd stand a much better chance of being able to write a successful novel ourselves and hence break in.
I'm assuming here that we accept that the job of an editor is to find books that will sell well and make a profit for the company. If you feel that novels are purely a means of Artistic Expression, that nasty commercial considerations are beneath you and that your Work of Exquisite Genius should not be rejected in favour of all that crass commercial stuff, then please stop reading now and click on to the next post. :)
Many people recommend reading Locus and keeping up with the latest news in SF publishing in order to see what's in and what's out. I've never had the time or inclination to do that and I'm going to be bold enough to suggest that it might not be helpful in any case. As has often been pointed out, it doesn't really help the aspiring writer terribly much to know that several new alien invasion stories are on their way because the books listed as forthcoming were accepted anything up to two years previously and have been slowly trundling through the publishing system ever since. Received wisdom (as least on places like rasfc) is that if there was a demand amongst editors for alien invasion stories (as indicated by the number of them about to hit the bookshops), it's no good burning the midnight oil and pounding the keyboard to produce one ASAP because that fad is now past and gone and what editors want now (if only you knew it!) is psychic companion animals. The conclusion they reach is that no one can guess what editors want now, don't even try to work it out, so just write what you fancy, cross your fingers and pray!
This, of course, is a counsel of despair and leads to wasted years writing stuff that never had any chance of selling. In the end, demoralisation sets in as the aspiring writer decides the whole thing is pointless and gives up. (Been there, done that.) It's especially frustrating for us older writers who achieved more positive results in the past than we're getting now, despite the fact that our writing has "improved". Certainly the prose is more vivid, the characterisation is more subtle etc. etc. So why the, "Doesn't grab me" when years ago we could at least get editors to read the whole MS based on our synopsis.
I have come to the conclusion that one property of successful SF and fantasy stories is that they resonate in some way with current concerns. I'm not saying that this is an original thought, in fact it was one of the themes of the recent TV series on fantasy and my moment of enlightenment came about while I was watching the final programme which was devoted to recent examples of the genre and Terry Pratchett in particular. Pratchett admitted in the interview that whatever was in the news at the time found its way into his current novel.
OK, well that's only to be expected because Pratchett is a satirist, but it occurred to me that it might be a wider truth and it might bear some examination.
One of the reasons it's taken me so long to gain this insight into how stories need to address the concerns of the readers is that there is a macro and micro effect. Novels, if they are to become best sellers, need to appeal to a readership beyond the hard-core genre fans. This is a macro effect. Short stories, on the other hand, have a much narrower readership and thus we're looking at the editorial equivalent of a microclimate. I think this explains why successful short story writers often have difficulty selling a novel, even when they're award winning short story writers and even when they've mastered the slightly different skills of sustaining plot and character development over 100,000 words instead of over 5,000. They are tuned into the conditions prevailing in their particular microenvironment, but they are failing to address the themes and concerns relevant to a wider readership.
I've sold 3 stories to paying magazines. The first was "Sam's First Show", a pony story for pre-teen girls bought by Pony magazine. Totally without realising it, I hit the jackpot because: the story is about a pony rescued from a sale (they had run several features about the problems of over breeding and the fate of the unwanted foals), it was about a horse show and must have arrived at exactly the time when they were planning ahead to their June edition which had a show theme, it was humorous with a positive and likeable main character and it had an upbeat ending. Because I had been reading the magazine (bought for my daughter) I had unconsciously absorbed their current concerns and though the story was dashed off one wet Sunday afternoon (rather then being carefully crafted and laboured over like my other stories that got nothing but rejections!), it hit all the right buttons and sold. Likewise my short about the editor getting more involved in a story from the slush pile than he intended sold (I suspect) because the theme of reading as escape and the editor's role in plucking the gems from the dross and making them available to readers fitted the mindset of an editor in the exciting early days of a new magazine. I very much suspect that they wouldn't buy it now and it didn't prove terribly popular with the readers (according to the poll). But it perhaps jibed with what the editor was thinking and thus it appealed.
Publishing a novel is a more costly business than just putting a story in a magazine or anthology. The novel has to appeal to a lot of readers if the publisher is to make a profit. Many books have been written about the Lord of the Rings, so I'm only going to say that it became popular in the 60s at least in part because it clicked with the prevailing mood amongst the young. (Some of us would have read it regardless because we loved fairy stories and legends and hero tales and here was a new and different one, but most people wouldn't think like that.)
The theme of a dark lord in the East suited those who wanted to see it as an analogy of the Cold War. A powerful magic ring suited those who wanted to read it as an allegory of nuclear weapons. Hippies could fall in love with the elves and the otherworldliness and the magic. I think the story was probably at its lowest popularity in the 80s. My kids and their friends weren't interested. There was nothing there to appeal to the materialistic young of Thatcher's Britain. But (yes I know it was the films), it's made a comeback now as people see other themes there, environmental concerns perhaps? Not only did the resonance with the readers make it popular in the first place, but the continued relevance keeps it popular. Once we reach the stage where it no longer has anything to say to young people, it will quietly fade away.
So I'm coming to the conclusion that theme and resonance and applicability are what makes an editor say, "I love this!" They may not even be doing it consciously, but I think they must be doing it.
So, this means that we shouldn't be asking ourselves questions like, "Are vampire novels still selling?" or "Is anyone buying sub-lightspeed colony ship stories?" we should be looking at what the newspapers and TV current affairs programmes are currently obsessed with and try to take those concerns and make sure our stories address them in some way. Whether we do it by writing about vampires or dragons or urban fey or classic rocket ships doesn't matter, I'm saying that the theme has to resonate and if it doesn't, you'll get the dreaded, "Sorry, doesn't grab me," response.
Now of course I don't just mean "be topical" because if you're too topical the fad will have come and gone before you can get your book written and submitted. Missing the boat with a topical theme that's past its sell by date is likely to be worse than just writing whatever you fancy. That's why the deep human themes like love and loyalty and betrayal and revenge just go on and on, regardless of genre and setting.
It's up to you, of course, whether you think this post is just the ramblings of an aging failed writer or a potentially interesting avenue to explore. But I'm certainly going to adopt this approach from now on. The "write what you love" has led me nowhere in terms of success, so I have absolutely nothing to lose. At worst it's just going to be a new and different way to fail. :)
Also it's not an either or. You write what you love in terms of character, setting etc. but you ensure that your story is slanted towards modern concerns, that your themes are relevant to today's readers. Most of us have more ideas than we know what to do with. Life isn't long enough to write every idea that comes our way, so this is intended as my personal guide as to what to develop and what to leave in the ideas notebook.
 I'm not saying that this will be foolproof, but at least our novel will be in the right ballpark, not 500 miles away in a different city entirely!
 My contention is that as we grow old, we lose touch with the concerns of the majority of the SF reading public who, lets face it, are likely to be considerably younger than I am. :) Once, without being conscious of it, we were writing about the things that moved or scared us and we were part of that zeitgeist. As we get older, it's likely that we are working more from our intellect rather than our emotions and might therefore be out of touch.
 This probably goes for all stories, actually, but I'm not interested in writing mainstream or thrillers or police procedurals etc.