GlaxoSmithKline has stopped a high-profile clinical trial using a novel vaccine to fight lung cancer after deciding it could not find a sub-group of patients who might benefit.

The decision comes less than two weeks after GSK said the MAGE-A3 therapeutic vaccine did not help patients with non-small cell lung cancer in the Phase III study overall, but it was still looking for improvements among other patients.

In the event, scientists analysing the trial decided there was insufficient treatment effect to pinpoint benefit in such gene-signature patients.

Vincent Brichard, head of immuno-therapeutics at GSK Vaccines, said he was “extremely disappointed” by the result.

Identifying a sub-group for whom MAGE-A3 would work had been considered a long shot by investors. Nonetheless, Citi analyst Andrew Baum said the latest setback removed “important optionality” for future sales and profits. A second trial testing the vaccine in melanoma, which also failed to help patients overall, will continue to investigate benefits in sub-populations.

The outcome of the melanoma research is expected in 2015.

Unlike traditional preventative vaccines, the MAGE-A3 treatment was designed for people with established disease, helping their immune systems to prevent the return of disease after surgery.

Other companies, including Bristol-Myers Squibb, Roche Holding and Merck, have had some recent notable successes in clinical trials of innovative drugs to boost the immune system, but GSK’s vaccine-based approach is different and has met with less success.