6 replies to this topic

[Melchoire]

Hey,

I'm writing a script that lets you automatically download and save songs locally from your hypem account.

The process thus far has been difficult but I've once again run into a bit of a quagmire.

I found out that hypem assigns an ID and a Key to each song on their database. Each ID is unique to it's Key.

Doing some packet sniffing I found out that there's a page where an array full of these ID's and Key's get populated.

The page will be:
http://hypem.com/[username]?ax=1&ts=[unix-time]

I thought I was done and my script was ready to download the files but I found out that the Key's that it was parsing weren't correct.

After a little investigating it turned out that the page would return the correct song keys only if the request was being made from a web browser. That is, making an HTTP request through my ruby script returned a page with seemingly random keys.

So every time you go to that page in a web browser you'll get the right key for the song. Trying to automate it doesn't work...

My plan at this point was to stop using an HTTP client and send a raw packet over a TCP socket on port 80. So I copied exactly what the firefox request looked like and sent it over the socket....unfortunately the problem persisted.

I can't figure out how their website is distinguishing my script from a firefox client. The packets were identical when I looked at them through wireshark.

At this point I'm open to ideas any of you may have.

[artificial]

First off, provided you properly send the HTTP request, there is no effective way for that website to differentiate your web script from an actual web browser. If in doubt, send the same headers (i.e. same user agent, same referrer, etc.)

Other then that, I don't know what other advice I can give, as I've never heard of the site and wouldn't have a clue how it works.

[Melchoire]

First off, provided you properly send the HTTP request, there is no effective way for that website to differentiate your web script from an actual web browser. If in doubt, send the same headers (i.e. same user agent, same referrer, etc.)

Other then that, I don't know what other advice I can give, as I've never heard of the site and wouldn't have a clue how it works.

Yeh I'm gonna try another round of packet sniffing today and see if there was anything I missed. If that doesn't do the trick I'll have to resort to a pre-made HTTP client or something...

Edit: do you know of a HTTP tester that works on linux?

[artificial]

I use Wireshark. I believe there are packages available for Linux: http://www.wireshark.org/download.html

You can also use the Firefox plugin Live HTTP Headers to monitor simple HTTP requests made by Firefox.

[Melchoire]

I use Wireshark. I believe there are packages available for Linux: http://www.wireshark.org/download.html

You can also use the Firefox plugin Live HTTP Headers to monitor simple HTTP requests made by Firefox.

Sorry, I meant a tester as in something that lets me send HTTP headers.

I wrote my own to simulate firefox headers but in wireshark I keep getting packets that say "Continuation or non HTTP traffic"

This is what the code looks like, it's just a opening a tcp socket and printing to it.

#!/usr/bin/ruby

require 'net/http'

host = ""
file = ""
req = ""

args = ARGV.join(" ")
 
r_input = /^-f\s(.+?)\s-h\s(.+?)$/
m = r_input.match(args)

file = m[1]
host = m[2]

fin = File.open(file, 'r')
fin.each_line do |line|
  req << line
end
fin.close

unix_time =%x[date +%s]
req = req.gsub(/\n/, "\r\n")
req = req.gsub(/ts=1/, "t=" << unix_time)

tcp = TCPSocket.open(host, 80)
tcp.print(req)  

puts tcp.read

And the headers:

GET /Rodayo?ax=1&ts=1 HTTP/1.1
Host: hypem.com
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:2.0.1) Gecko/20100101 Firefox/4.0.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection: close

I tried different variations of this. With gzip/deflate, with "keep-alive" instead of close as well. But nothing did the trick.


I previously tried sniffed the packets out with wireshark and I honestly can't see anything different in the http headers. Maybe it's my code

[artificial]

Well your timestamp is wrong if you're sending 1.

Honestly, I've rarely used Ruby, so I wouldn't have much of a clue if there is anything wrong with that code. You've tried looking at the HTTP packet sent (through Wireshark or a similar program) and verifying it's correct? It looks alright. You establish the connection, you send the request, and you read the result. You probably also want to close the connection but I doubt that would have any effect on the returned result.

Are you able to paste what is returned?

edit: you're requiring net/http but you're using TCPSocket. Are you sure that library allows you to use that functionality? Try and include the 'socket' library.

Edited by Artificial, 01 May 2011 - 08:31 PM.

[Melchoire]

Well your timestamp is wrong if you're sending 1.

Honestly, I've rarely used Ruby, so I wouldn't have much of a clue if there is anything wrong with that code. You've tried looking at the HTTP packet sent (through Wireshark or a similar program) and verifying it's correct? It looks alright. You establish the connection, you send the request, and you read the result. You probably also want to close the connection but I doubt that would have any effect on the returned result.

Are you able to paste what is returned?

edit: you're requiring net/http but you're using TCPSocket. Are you sure that library allows you to use that functionality? Try and include the 'socket' library.

First, I correct the time stamp in the code, I replace it with "unix_time". But here's the weird part, in firefox even if you have the wrong timestamp you'll still get the right key I just put that in there for good measure.

The response is the same as usual, the correct page but with the wrong keys. And like I said before wireshark packets have "Continuation or non HTTP traffic" in the "Info" column.

As per your suggestion I tried "socket" but with the same result(incorrect key) =/ It was just left over from another script so i didn't bother to change it.

If you have time maybe you can try it your self. Once you send the request you'll get an HTML page. In the source there's an array called "trackList" that keeps getting populated with info. Find one of the parts where a new item is added(it'll be something like trackList[...][...].push) and try out the resulting url. The item will have an "id" and "key" field so the url would be:

http://hypem.com/serve/play/[id]/[key].mp3

if it's a song then you're doing it right, if it's a 403 forbidden then it's wrong.

Bump: finally figured this shit out

I went to the webpage in firefox. And before I entered the link for the mp3 file I deleted my cookies. And guess what.... it didn't work

Solution: manage the cookies in the http wrapper and I'm in business =D

Thanks for the help artificial. Once I'm done I'll share the script with everyone here

Edit: >_< this only gets harder and harder.

at first I saw a cookie called "AUTH" which I assumed is all you would need but then I find about this google analytics stuff like "__utma" and "__utmz" would these be used for authentication purposes?