There’s a new Neil Young album out. That in itself isn’t much of a thing, Ol’ Neil’s been pumping them out at a steady pace for years. Unlike many of his contemporaries he’s continued into 2017 with his finger on the pulse making honest and socially conscious tunes – in 2015 he did a whole album about what scum Monsanto are. His 2016 album Peace Trail didn’t hold back on the politics either, nor did the live album he released that year, entitled Earth, which was interspersed with nature and animal sounds as an ecological statement.

None from that trio of records were bad but none were particularly good either. Hitchhiker, his latest album, on the other hand, most certainly is. In fact it’s brilliant. Ten tracks of honest and impassioned acoustic tunes, just Shakey and his tortured soul. There’s this almost haunted sound to them. But then of course they sound haunted – they were recorded 41 years ago.

Neil Young and his producer David Briggs had a habit in the mid-70s (well, Neil had a few habits at that time…) of flicking on the tape and recording a bunch of stuff on the nights of the full moon. On 11 August 1976 they did exactly that at Indigo Ranch in Malibu. This is around the time he was working on American Stars ‘n Bars, near the tail end of his most productive period. After the Ditch Trilogy, almost a year after the release of Zuma. He was reeling them off in record time.

On this particular night he and Briggsy were joined only by the actor David Stockwell, who was a mutual friend. The recording studio was buried, isolated, within a 60 acre woodland canyon. Plus if you know the history of Neil Young then you know he wasn’t really in the best frame of mind around this time either. Things were dark, he was nursing a few habits he’d later regret, divorced and grieved. These ten tracks carry every bit of that weight. As he’d say himself in his autobiography, Neil was “pretty stony” on this particular night, writing: “I laid down all the songs in a row, pausing only for weed, beer, or coke. Briggs was in the control room, mixing live on his favorite console.”

Initially he’d planned to put the session out as a complete album but the record company said no. They wanted a full-band LP, they wanted overdubs and re-recordings. So they chucked it in the can and most of the songs eventually found new homes elsewhere. Pocahontas was overdubbed and slipped into Rust Never Sleeps, as was a fiery electric take of Powderfinger and a live take of Ride My Llama. Captain Kennedy went on Hawks & Doves. The Old Country Waltz was re-recorded for Stars ‘n Bars. Hitchhiker was rewritten for Trans as ‘Like an Inca’ before it morphed into a new electric version for 2010’s Le Noise. An edited Campaigner went on the German edition of the Decade compilation while Human Highway was done with the band for Comes A Time. Hawaii and Give Me Strength had never been officially released before now.

“You ready, Briggs?” asks Neil, as the first seconds tick over on the record. What follows is far from an archival oddity. This is a vital addition to the catalogue, this is something that would’ve been raved about four decades ago had it been released as originally intended.

But creative geniuses work on a different wavelength to mere mortals. Lost albums are scattered throughout the mythology of popular music, entire records that were aborted for whatever reason. Maybe it was a dispute with the label, maybe personal differences within the band or maybe the artist simply decided it wasn’t good enough for release. Hitchhiker isn’t even Neil Young’s most famous lost album – he’s got at least two more legendary efforts that have never seen the light of day.

And there’s something mystical about that. Something about having them withheld that makes them all the more enticing, like the forbidden fruit of rock and roll. You just don’t get very much of that these days, not in the soundcloud and social media era. Although if someone was to say that Frank Ocean had an entire polished and hidden album then there wouldn’t be too many surprises. We don’t know for sure that Blonde was the same project as Boy’s Don’t Cry, and what about all these Apple Radio singles? What about the tracks that ended up on Endless, where are the polished takes of those?

Frank’s creative process is a mystery and that’s all part of the allure. Hey and Kendrick Lamar’s album of outtakes was so damn good you’d better believe he could package another one if he wanted… only difference is we don’t know it exists yet.

So with that in mind, here are a few other legendary lost albums that glimmer like Holy Grails in the minds of audiophiles and completists the world over…

Prince – The Black Album

You can bet your ass-less chaps that The Artist has potentially dozens of completely finished albums in the vaults at Paisley Park that he either never wanted to share or never got around to sharing. The Black Album is the most famous of the ones that we know about (and there are a few). Also known as The Funk Bible, it was to be the follow-up to Sign o’ the Times and the completely black cover was both a Beatles White Album reference (wouldn’t be the first, wouldn’t be the last) and also a reaction to those who said Prince had become too ‘pop’… aka too white. So he made a funk album.

And it was all packaged and ready to go when Prince had a revelation: this album was evil. It was cursed by a devilish entity called ‘Spooky Electric’, spreading its malevolence through these songs and out into the world. It’s, erm, been suggested he might have had a bad trip on MDMA around this time. Anyway he had every single copy recalled and the album release cancelled. A few hundred copies survived and are highly collectible.

Eventually The Black Album was released properly in 1994 and in 2016 it showed up on Tidal. For those seven years in between the recall and the official drop it was amongst the most bootlegged records in the world. Following his revelation, Prince then went and made Lovesexy (which included When 2 R In Love from The Black Album), which featured a naked Prince on the cover reclining on flowers. Where the unreleased album was dark and feisty (hear Bob George, wow), this one was bright and more gospel-infused. There are direct references to Spooky Electric as the devil and on the video for Alphabet St. there’s a subliminal message that reads: “Don't buy The Black Album, I'm sorry.”

But, yeah, Prince was famous for rolling through projects and he was so creative that they all had some value or inspiration to them. Here are five others worth digging into:

Camille (1986) – An eight-track record credited to his titular alter ego. Voice sped up to alter the pitch and everything. He wanted to release it without his name in the credits or anything, plus he had a film idea at one point. Nobody really knows why he cancelled it but the biggest theory is that Warner Bros didn’t want to put it out without his star power behind it. Three tracks ended up on Sign o’ the Times.

The Flesh (1985) – A live jam collection of jazz-funk instrumentals, recorded over three different studio sessions. It was never completed though because, well, the dude was too busy. When he did get around to his jazz-funk instrumental album (Madhouse) he did so with new recordings.

Dream Factory (1986) – Originally a single album for Prince & The Revolution, it ended up being the most collaborative thing that band had ever done and soon expanded into a 19-track double album. But the band broke up and it never saw the light of day (not legally, anyway). Eight tracks ended up on Sign, though.

Crystal Ball (1986) – After Camille and Dream Factory fell away, Prince regathered his spare songs and began working on this behemoth of an album. 22 songs and three records – a triple album supreme. The label told him to cut it to two albums and he did, reworking it from there into what became Sign o’ the Times. A few other songs were revised into other albums, some remain unreleased.

The Undertaker (1994) – A live in studio effort, recorded in one single take, featuring a proper rock and roll power trio format. Drums, bass and Prince’s explosive guitar. There’s a cover of Honky Tonk Women along with six other depth tracks of his own. Heavy and bluesy, he wanted to give it away free with Guitar Player magazine but Warner Bros blocked that. It got bootlegged instead.

The Beach Boys – SMiLE

Potentially the best album never made. You know that scene in Walk Hard where Dewey Cox goes super intense trying to record his masterpiece? This is largely what that was based on. The Beach Boys had put out Pet Sounds in 1966. The Beatles had just done Rubber Soul and Brian Wilson responded with his own inspiration, writing an album meant to push the boundaries of pop music. With Pet Sounds Brian helped do that… then he heard Revolver and felt that the lads from Liverpool had upped the game again.

So Brian Wilson and the band set about making SMiLE. It was to be a musical and philosophical opus, channelling the entire history of American music and purveying a strong spiritual message of happiness and “forgetting the ego”. Van Dyke Parks was brought in to help with lyrics. Production was going to be state of the art – Good Vibrations was 90 hours of audio crammed down into a three minute pop song.

But it all proved too much and deadlines passed without its completion. Brian grew frustrated and the project got more and more unwieldly as he pivoted between different concepts. Then Sgt Pepper’s came out and it dented his confidence even more. Eventually SMiLE was scrapped and in mid-1967, with their contract demanding a new album, they patched together Smiley Smile which featured several re-recordings of would-have-been-SMiLE tracks. Carl Wilson called it “a bunt instead of a grand slam” although Smiley Smile is still a pretty great album in its own right.

The years since have seen many attempted recreations of SMiLE based on the tapes that have been leaked from those many sessions, tapes which have since become legendary. The Smile Sessions were officially released in 2011 with an approximate recreation of the album on the first disc while Brian Wilson revisited them in 2004 with a set of new recordings meant to mirror his original vision, which came to be after he and Van Dyke Parks had toured on a similar concept.

Neil Young – Chrome Dreams & Homegrown

Two different albums, two completed works that never saw the light of day. Homegrown even had its album artwork ready to go and everything. That one was a mostly acoustic and a collection that showcased his country side arguably more than anything else he’d done to that point – that point being the few months on either side of New Year's 1975. Right before it was meant to be pressed, Young changed his mind on Homegrown and chose to release Tonight’s The Night instead, which itself had been sitting waiting to be released since it was pulled in ’74 – either because Young didn’t like it or the record company thought it was too emotionally heavy… which it was, but so was Homegrown. TTN is considered one of Neil’s masterpieces and rumour has it there’s an alternate version out there somewhere waiting to be released. Told you he’s full of this stuff.

As for Chrome Dreams, that one was done sometime in 1977 and supposedly set to acetate but never released. Most of the songs on it ended up on different projects, though some in different versions, while three of Chrome Dreams’ tunes were finally put out officially on this Hitchhiker record. Young made an album called Chrome Dreams II in 2007 but that one had nothing to do with this one. Supposedly Chrome Dreams will finally see the light of day soon enough, it already sort of has with one of the favourite pastimes of Neil Young bootleggers being trying to piece together what this album was meant to be.

Dr Dre – Detox

Upon leaving N.W.A., Dre dropped The Chronic which became an instant hip hop classic. He never really looked back, helping make superstars of Snoop Dogg and Eminem amongst others while producing all over the show and making bank with Death Row Records all the way. By the time he released his second album, 2001, they were calling it a return to his roots only five years after The Chronic (what, y’all forgot about Dre?). He was working on his third record as far back as 2001 and in 2008 he claimed that Detox, as it’d be called, would be his final album.

Dozens of people were said to have recorded features for it and all the word coming out from those involved was that Detox would be like nothing hip hop has ever heard. Snoop said he’d listened to finished songs in 2008 and that they were amazing but still there was no Detox. Soon after, purported tracks started leaking and bits and pieces were teased yet still nothing on the album. Eventually Dre did release his third LP in 2015… except it wasn’t Detox, it was Compton. A week before that record came out Dre officially tanked Detox on his radio show – it wasn’t good enough and it would never be completed. So… there you go. For now anyway.

By the way, we’ve also never heard the rumoured N.W.A. reunion album. Or his Breakup to Makeup collab with Snoop. Or Heltah Skeltah with Ice Cube. Or that Timbaland thing that he may or may not have done, called Chairmen of the Board. Or Voices through Hot Vessels which D.O.C. once said would be Dre’s fourth album after Detox, had either ever actually come out.

David Bowie – The Gouster

In the aftermath of Diamond Dogs and the tour of the album, Bowie underwent another of his many ch-ch-ch-changes. He went soul and The Gouster was the product. Or… it would’ve been had Bowie not withdrawn it to keep on working on it. Eventually it became Young Americans and that became one of his finest albums. The Gouster never had much of a myth life, largely because it plays like an unfinished version of YA, but last year it was released within the Who Can I Be Now? (1974-76) box set. Seven tracks, one the same as on YA (the title track) and three early alternate takes, with three others not on that album but released as bonus tracks to it in 1991 (or, in John I’m Only Dancing (Again)’s case, released as a single in 1979).

The thing is, there are three other songs from that session which have never emerged: I’m A Laser, Shilling the Rubes and After Today. Meaning basically that they bloody scammed us with the version from the box set, those grifters.

Speaking of Bowie, there’s also his 2001 album Toy which was going to feature some new songs and some reworked versions of a few obscure 1960s songs of his. The record was finished and primed for release but Bowie suggested the label was having trouble with “scheduling conflicts” and it never emerged. Soon enough he’d turned his attentions to Heathen and that was that. A version of Toy leaked online in 2011, though it may not be exactly the same as the album which was almost released.

Marvin Gaye – Love Man

By the late 1970s, Marvin Gaye’s life was a bit of a mess. He’d already been divorced once and was separated from his second wife. In channelling that divorce he’d made a rather stunning album called Here, My Dear but it didn’t sell, continuing a drought of commercial success for the legend. Let’s be honest, the drugs weren’t helping either. He was big in debt and his plan to fix it all was to record a disco album.

Love Man was prefaced by the single Ego Tripping Out and he’d done a few new ones to go with it at his LA studio but had to postpone those sessions for a cash grab tour that’d help him out with the taxman. Except the tour sucked so he cancelled it midway and the promoters (as well as some band members) sued him for breaking contract, making things even worse. He soon filed for voluntary bankruptcy.

All the time he was trying to salvage things with this new album, however he couldn’t help but let his personal chaos seep into the new tunes. Eventually he gave up and ingested an ounce of cocaine, planning on killing himself. Yeah, this one’s dark. Gaye survived and went on a small tour in Europe to help the coffers. In the end he distanced himself from his unfinished dance album and decided to get back to what he did best: sexy soul songs. In Our Lifetime? came out in 1981 and Midnight Love the year after. The latter would be his final album and, thanks to the success of Sexual Healing, the best-selling one of his career.

Years later, in 2007 to be specific, In Our Lifetime? was re-released in an expanded edition with a bonus disc featuring seven completed tracks from the Love Man sessions, plus alternates of Ego Tripping Out.

Jimi Hendrix – First Rays of The New Rising Sun

Having already conquered the guitar-based world with his first three Jimi Hendrix Experience albums, Jimi was recording no less than ever. Working with his Band of Gypsies line-up, Hendrix was taking his trademark hard psych rock into funkier territory (getting back to his R&B roots, even). The album was to be called First Rays of the New Rising Sun and he’d been working on it for a good ten months when he died, leaving it unfinished.

Most of the songs he’d done ended up on 1971’s The Cry of Love – the first in a seemingly endless line of posthumous releases – with others scattered across further muddled records. TCOL was very well received but, you know, it wasn’t the album that Jimi was prepping. So in 1997 his estate finally went about setting that right.

First Rays, the official version, copped together 17 tracks from the original master recordings and in as close to the intended sequence as could be figured. Of course, some still argue that even the estate’s imagining of First Rays wasn’t what Jimi had envisioned. And the audio quality of that one isn’t the greatest either, let alone the terrible album art. We’ll never know exactly what it would have been and that’s a shame – the direction he was going with these songs really makes you wonder what he might have done had he only lived.

Fiona Apple – Extraordinary Machine

Following the tour for her second album, Fiona Apple found herself with a bad case of writer’s block, to the point where she considered retiring from the music business. Friend and producer Jon Brion convinced her otherwise and they began work on a new record in 2002. Working at the old mansion house of silent film star Antonio Moreno, things seemed to happen pretty smoothly. Songs flowed out and the album began to take shape. A few were already getting tested at live gigs. After jumping over to London to record some string arrangements at Abbey Road, as you do, Brion reckoned the album was just about complete by May 2003. Release was pencilled in for September of that year.

Yet by September there had been delays. Apple had wanted to touch up a few tracks and the new word was that it’d come out in early 2004. That never happened either. Midway through that year a few songs leaked online to a positive response but there was no update on when the album would arrive. Brion was reported to have blamed the studio, who wanted something more radio friendly.

Ironically, if it’s true that they wanted something that’d play on the radio, a DJ in Seattle went ahead and played a (low quality) bootleg of the record, which led to even lower quality tracks flooding the net. Some fans protested at Sony about the release, with the company responding that Fiona was still at work on the album.

Extraordinary Machine finally arrived in October 2005 but by then it had been drastically reworked with production from Brian Kehew and Mike Elizondo. Two tracks were the same as on the original version but the other nine were massively altered. The official edition was widely praised by critics and Apple has maintained that the bootleg was never the finished product anyway (Brion also said it didn’t reflect what they were working towards). Still, the leak is out there if you want to compare – perhaps one day they’ll release the album that could have been from those tapes.

D’Angelo – James River

Brown Sugar and Voodoo were game changers in the whole neo-soul thing. The master behind them didn’t cope well with the fame that came with them, though. For fourteen years he disappeared. His record label cut his funding, then they cut him. There were sporadic features but work on his third solo album seemed to be slow in coming… and it didn’t help that he wanted to do it Prince-style by playing every single instrument. In fact Prince was rumoured to be guesting on it at one point. James River, it was to be called. Mark Ronson, CeeLo Green and Raphael Saadiq would also show up – as well as Questlove who’d been overseeing most of it. In 2011 Q said it was 97% done and compared it to “the black SMiLE”.

Whatever James River was supposed to be, it never arrived. Yet all of a sudden in late 2015 the news broke that D’Angelo was back and a new record was on the way: Black Messiah. Credited to D’Angelo and the Vanguard, there’s no Prince or Ronson or Green or Saadiq. If this was the same album as James River then it had undergone some hefty changes, alright. But the myth of James River ceased to matter once Black Messiah arrived because, frankly, Black Messiah is a flippin’ masterpiece.

Danger Mouse – The Grey Album

Unreleased? Not exactly. But definitely not commercially available. Jay Z had dropped some a capella takes from his Black Album in order to encourage creative people to do creative things to them. Danger Mouse took that baton and ran it all the way to the finish by mixing those vocal tracks with instrumental tracks from the Beatles’ White Album. Black + White = Grey and thus The Grey Album was born.

The results are ridiculously slick and catchy and the album got the thumbs up from Paul McCartney, Ringo Starr and Jay Z. Problem was that EMI records didn’t fancy it, not at all. Danger Mouse considered it an art project and planned a 3000-unit release but it got so much hype that it ended up all over the internet which sent EMI crazy. An online protest was held where a bunch of sites posted TGA for free download over a 24 hour period. Put it this way, you’re not likely to ever find it on the shelves down at the store but also it’s never further than a google search away.

The Niche Cache is just trying to make a difference, is all. Support the cause by smacking an ad whenever you read something righteous and every like, share, retweet counts.