Disclaimer : I am a white boy, straight outta Cambridge, UK. I do not condone the nastier, sometimes misogynistic language in this post, but in the interest of keeping it real, I have not made any effort to omit it either :b
Mobile users : you probably want to turn your screen sideways
The following was completely generated, unedited, by a rapbot (minus the title).
House Full of Bricks - D-Prime pump it up , just throw yo hands in the air this goes out to all my niggaz that don't dare i got a whole lot of diamonds in yo veins about havin thangs gold rings , and my platinum chain that's why all these bitch - ass niggaz get smoked they don't give a fuck , that shit ain't no joke nigga this goes out to all them crooked ass cops since back in the days when i was getting popped always had a good time for me to be fine i've lost my mind , and i'm still gon shine but i don't wanna see a nigga on the grind we ain't trying to be ready for war or crime check this out , a lot of people are scarred and it's my time when it comes to the stars cause i was born with a house full of bricks yeah , we can see it all in the mix but now it's hard for me to beg and feed i gotta wake up , so take away my seed
Ever since I listened to Illmatic as a youngster, I’ve loved hip hop, and the amazing creativity of (some) rap lyrics. I also love machine learning. Over a year ago, I decided I would write a machine learning algorithm that could ingest rap lyric text, then generate lyrics of its own automatically. This was hard, and I gave up. Twice. After many, many iterations, I eventually came up with a model that could produce the above. The following is a brief description of how this was achieved, the full gory technical details as well as code will be written up in a later post.
but right now i'm just trying to make it nice this is my life , you can pay the price i ain't gotta wait til it's time to take flight have a party all night , everything's gonna be alright so now do you really wanna get with me tonight it ain't no need to talk about what i write
How this was done
Generating free text using machine learning is not a new concept. There was a huge surge in interest in it on the internet about a year ago because of this amazing blog post by Andrej Karpathy. Text can be broken down into sequences, of characters, words, sentences etc. There are some particularly powerful models called recurrent neural networks, that are designed to deal with general sequences. The way these models are trained to learn about text goes like this – we give the network a short series of words (or characters) from real rap lyrics, such as
put your hands in the air , wave em like you just don't ____
and ask the model to predict what the next word in the sequence is. Based on what the model thinks compared to the truth, we update the models’ knowledge of rap language. We repeat this for many millions of word sequences, and hope that eventually the model will learn a good understanding of the grammar and syntax used in the training data.
This approach conveniently allows us to generate arbitrarily long verses of free text – we give the model a starting sentence to work from, then generate the next word. We assume this word is correct, then generate the next word, and the next.
This is a simplification – D-Prime uses a beam search approach to generate lines – we generate a new line by building up many thousands of candidate lines word by word, keeping only the most promising ones at each step. This is an idea lifted from machine translation, e.g. this paper. I also manually encoded some rules about what words could be used, to avoid repetitions etc. I also forbid the model from choosing lines that were from the real rap lyrics, we want some originality here.
Generally this kind of approach requires a lot of text to work effectively, i.e. a lot of lyrics. These I obtained the lyrics of 50,000 rap songs off the internet using a python script. I also spent a lot of time cleaning up the data, and trying to find the ‘best’ data to work from. For example, I deliberately chose verses that had a rather mediocre vocabulary, meaning they mainly used commonly used words. Incidentally, the song I found with the least mediocre vocabulary is this one, but that’s just showing off.
Rhyme and rhythm
I thought of the name DeepRhyme before I had even really started building the model. My brother pointed out ‘D-Prime’, and at that point I knew it was going to have to rhyme.
To achieve this I got the model to predict the text in reverse – that is we give it some reversed text, and ask the model to predict what word came before. I can’t speak for professional MCs, but if I need to make a rhyme, I do it like this – I have to think of the end of the line first. Unfortunately, doing this alone was not enough to make the model rhyme on its own. Instead, I forced it to. Every other line, we start the next line off (at the end), by letting the model predict the probabilities for all possible next words, and only considering those that rhyme.
So how do we know what words go together? In short- I downloaded the CMU pronunciation dictionary, and used a very basic set of hard coded rules to see if the end of words looked similar enough. This was pretty loose, but rap lyrics are famous for using very slanted rhymes, listen to Eminem or Mos Def talk about what rhymes with orange for example. I then looked at pairs of words found at the ends of lines in the actual rap lyrics themselves, and said that words rhymed if they appeared often as pairs in the lyrics, and passed my basic test. This did not work perfectly – e.g. the model thought that og rhymed with dog, and not all words were even in the pronunciation dictionary – but it worked well enough.
and i'ma let you know that my game is thick this rap shit , i do it for the bricks cause none of y'all niggas can't fuck with my click you know we ain't got no time for these tricks so don't be mad cause the whole world is fake it's a part of me , i'll do whatever i takes
Most of D-Primes rhyming is quite rudimentary, mainly just the tail rhymes we have forced. The best MCs use internal rhymes, as well as assonance and alliteration to make far more interesting syallble structure. This video is a great illustration. You can see that D-Prime does internally rhyme to a small extent, but making the model better in this regard seemed like it was going to be tough, and I didn’t pursue it.
I also tried out an ABAB rhyme scheme, but it didn’t really flow so well, I reckon rhymes have to be pretty on point for them to hold over two lines, and it’s not really that common in hip hop anyway. Here’s one of the few sections that kind of worked.
ain't no need for us to do what we want there's nothing in the world that can make me feel so sweet i be out here tryin to get my hustle on kept it real when they see me in the street cause there's no way that i've been on my grind but sometimes , i just wanna get out of control feel like i'm the only one that's on my mind and nobody wants to be a part of their soul
Rhythm, or ‘flow’, is an key differentiator of the best rap MCs. Listen to the first verse of this :
I quickly decided I wasn’t going to touch this concept with a barge pole. Even working out how many syllables are in a line is pretty tough (e.g. many of the words are not in the pronunciation dictionary). Instead I just told the algorithm to output lines that were exactly 10 words in length, including punctuation. This was the average in the training set, and lead to the most pleasing text.
Early failed attempts
Because they’re pretty funny, here’s some bars written by earlier iterations of D-Prime :
and i got a catch the streets, the streets are the streets i can tell the streets, get the streets catch your shot, i can be but i can see the part i was the streets, the streets in the streets i spit a plant, the star and the motherfuckers, but i got a bottle of the streets in the back the boots, i can take a control, i got the stars i say the back of the streets, the fucking me, i was in the stars i got a fly, i was the start, i got a back of the black start i was the more than a discoll of the streets i got a black, the streets is the streets stay back
Don’t ever say D-Prime doesn’t know where it came from.
cause i don't wanna be , no more , i'm like so i can go hard , i'm in the club i ain't got no time for a minute i can see my mama with her i can see what i can do i do my thing , i ain't been around but you didn't know , but now i know i know i ain't wanna , then i call him she said she was a bitch , she ain't gonna make it
Ain’t got no time for a minute is a great line.
yeah , i'm just tryin to catch me on the street and get caught up in my pocket full of weed you think it's time for me to feel the breeze i'ma put your ass in a pocket full of cheese
As my brother pointed out, I could probably create a gangster slang generator out of this.
D-Prime has issues with women
At the start of this post I said I hadn’t made any effort at censorship. This was a lie. If left to its own devices, D-Prime writes some fairly… coarse lyrics, often completely out of nowhere.
there's a place in my heart , i promise it will never end but it's so hard for me to be the world's best friend fuck that bitch , i love it when you suck my dick i'm bout to get caught up in the heart of a chick
The reason this happened is because these kind of lines are very common in the lyrics the model was trained on. Indeed, in the process of building this model I have read some absurdly sexist verses. This was actually causing problems, D-Prime literally couldn’t go a verse without throwing in either some sappy line to an unnamed girl (out of nowhere), or a borderline verbal sexual assault, or indeed both within the space of two lines.
D-Prime also doesn’t have a memory that lasts longer than the last two lines, which made it easy for it to get confused about what role it was supposed to be playing, for example :
ain't none of these niggas tryin to fuck with me i need a bitch , just give me some pussy and you know that i've been all over this globe cause back in the days when i was sellin dope and that's how we do it all over this world but you should know , i wanna be your girl
For this reason I took the difficult decision of completely banning D-Prime from using certain words, which helped a lot. This is a good illustration of how machine learning algorithms like this rarely act how you want them to without some supervision (see : Tay Tweets).
A couple of people I showed this to have commented on this, so I will be the first to say that D-Prime is not creative in the way that we usually think about creativity. D-Prime is trying very hard to exactly reproduce the language of the rap it has already read. Any originality in the output of the algorithm is due to lack of understanding of the language. Generating text in this manner is an interesting way of showing what the model has learned about text, but it is a bit of a parlour trick really. The model doesn’t actually understand what words mean in relation to the real world, just how the grammar of how words relate to one another in a sequence.
To bring the post full circle, for the hip hop fans out there, here is the most ‘knowing’ rhyme that D-Prime generated.
you will end up in new york state of mind cause it's all this money and the world is mine
There are two references to Illmatic in this rhyme. Coincidence? Almost certainly, but still an extremely awesome one. Now would be a fitting time to remind everyone how much better humans are at rap lyrics (link to lyrics).
Many thanks to my hip hop head brother Niall for his enthusiasm as the model progressed, for some suggestions and jokes on this blog post – and for generally being a straight up G. Play us out D.
this the type of shit that i'm lookin for bro and when it come to my hood , they call me solo cause i came a long way from the danger zone don't be hating on me when i'm in my dome i'll be right back in the heart of a ghost i'm that nigga from north , east to west coast yeah , cause you know we came to party tonight so throw your hands up in the air if it's alright we can go and get a piece of that fence you need to slow down , it don't make no sense i grab the mic and a pocket full of beats now let me show you how to make ends meet cause i know that you was down with me today but now it's time to come and get it ok don't want no money , i gotta make a mil everytime you see me in the back of my grill i'm just tryin to take a look at this ho but when i hit the flo , we make it glow yo , i remember when we used to do shows been around the world and that's just how it goes