In day two of our Google Hacking Week we are going to combine an interesting Google search query (or Google Dork) with a command line command to find AND download any file type you want.
Find the storage room in the back of the store.
Websites on the net consist of more then just webpages with information. They also links to files and folders containting interesting information like PDF’s MP3’s and more. Most of the time these files aren’t ‘visible’ when you visit a specific site but our little friends, the Google Search Bots, DO index them. All you need is the right string to find them.
- intitle: “index of” <filetypehere> <title/genre/artist>
This search query will tell Google to go look for pages with the title “index of”. These pages usually don’t contain a lot of text, but instead contain links to folders and files. Since you are looking for a specific type of file (like for example mp3’s, Pdf’s or something else) you also can add this to the query. Finally you might be looking for mp3’s of Hanna Montana or Tango’s (I don’t know what you like) : That can also be added to the search string. In the end it will look something like this.
- intitle: “index of” mp3 acdc
- intitle: “index of” pdf bookkeeping
- intitle: “index of” epub scott sigler
So using these queries you might find a real treasure-trove of files and info to download. Some of them might even be behind a login/password page (or even a pay wall) but when the web masters don’t do their homework right .. you can find the ‘good stuff’ this way.
So download them one by one ?
If you are just looking for one specific file you can use your browser to find and download it. If you want to download the ENTIRE collection of files on that page .. you need the power of a command line tool called WGET.
Wget can be found on the command line of both Linux, Mac and even Windows machines. Not all the advanced ‘switches’ we give you in this command below might work on Windows, but you can give it a try. The command is
- wget -r -l1 -H -t1 -nd -N -np -A.<.filetype> -erobots=off <url of website>
Replace <filetype> with the type of file you want to download ( .mp3, .pdf, .epub) and <url of website> with the website’s url you found using the Google search. Completed the command might look something like this.
- wget -r -l1 -H -t1 -nd -N -np -A.<.mp3> -erobots=off http://tiobiloute59.free.fr/tiesto/
The download is RECURSIVE, so it “deep dives” into all the folders. Beware : This can get you a LOT of data. So make sure you have the bandwidth and the storage capacity before you start sucking down the internet. Good Luck !
The thing with proprietary software solutions, is that they are great. Everything tends to work smoothly together right up to the point where you decide to wander off the beaten path of supplier XYZ. A couple of years ago I made the crucial mistake of pouring my entire music collection into iTunes. Now, some 10 000 songs later .. its still in there. Being totally OCD I have organized all my tracks into nice little playlists and enjoy my tunes in the “Apple walled garden”. Whether I am playing them from the Mac, sharing out the iTunes library over iTunes to my other macs, blasting them from the Airport express speakers or syncing them to my other i-Devices.
But a couple of weeks ago I could not help myself myself and crawled over the walled garden into android territory with my purchase of a Galaxy Note 2. And accessing my delicately curated iTunes library from THIS device turns out to be an near impossible task. The deep crevasse that divides me from listening to my tunes on my “droid” consists of an incompatibility to sync with iTunes (only IOS devices of course) and the total inability to get the music on my Android in an organised form. Sure I can browse the filestructure of my iTunes library and copy over files to the SD card on my Phone .. but iTunes has “reorganized” my music into folders according to artist .. not according to playlist.
Enter Tunesync. A two-part application app in the android store that saves the day. The deal is simple. Download the server part of their app and install it on your Mac that is running iTunes. Download the CLIENT side of their application and install that to your Android device. Make sure both are on the same wifi network and be amazed !
Tunesync detected my (massive) iTunes library and started indexing the playlists right away. After I selected the playlists I wanted to have on my Android it started to copy over the tracks AND the playlist order in my Androids music collection. 20 minutes later I had all the grooves I needed on my Note2. Tunesync regularly “checks” if the playlists are still up to date and “updates” them whenever I connect or start up the app. I had expected some glitches and on one occasion Tunesync had given me all my playlists .. with no tracks inside ( it erases and re-copies all the tracks on every sync instead of doing an incremental) but when I retried the sync it worked flawlessly.
Tunesync does one thing and it does it well, and the hilarious part is , it does it better then Apples iTunes-IOS wireless sync ! The app is 4.99 in the Play store and worth every dime.
Tunesync is available from the Play Store.
Its quite amazing when you stop and think that iTunes is almost 10 years old this year. Apple’s end-all-be-all monotheistic gateway for your music collection towards your iPod device, is an application that is loved and hated equally. Like a teenage mom, iTunes went from a young, innocent slender application whose sole purpose was to curate your music collection, to an over-bloated thirty-something iPod-Baby machine that had acquired more functionality (and resources) over the years. Right before Apple gave iTunes a much needed binary liposuction with version 11, iTunes was one fat mama.
But that fat mama had started herding my music collection back in its younger years, and over the course of 10 years 15000 tracks have found their way into its arms. Over 100 playlists divide these tracks into manageable chunks and … I’ll probably never get them out again. Album art, MP3 tags, comments, stars, iTunes poisons my library with proprietary metadata and decides its a better idea to arrange the songs for me instead of my own ‘one album per folder’ setup.
When I started using Linux and other operating systems more and more, I got annoyed with the fact that I could not access my iTunes music from a different OS then OSX .. and that bugged me.
Hence it was time for the great escape ! A search for an application that would export every track in every playlist that I had to a predetermined folder structure that would be compatible with parallel universes WITHOUT iTunes. After hitting “The Google” for hours on end, I decided to enter the terms “iTunes Export” and came across a brilliant little application by Eric Daugherty called .. “iTunes Exporter”.
How it works ? Simple : On Osx (sorry , its an OSX app only) close iTunes and fire up “iTunes Exporter” Select the playlists you want to export and voila : iTunes exporter creates a folder with the name of the playlist and exports the tracks (using their id3 tags as file names) together with a playlist file. The result is simple and brilliant : Your (non DRM’d) music exported to a folder tree, ready to be imported into any media system of your choice. Its handy if you iPod-iPhone or whatever iDevice ever breaks .. or you just buy an Android device.
In the words of some Scottish guy with an ax and a skirt : FREEEEDOOOOMM !!!
Links : iTunes Exporter.
By now most (if not all of you) have dabbled a little bit with Linux. Although it might look geeky and complicated at first glance, once you dip your toes into the great lake of open source it becomes a very refreshing computing experience. How do I mean ? Well : I challenge you to the following : Take a Windows Machine OR a Mac and, within 2 minutes FIND and INSTALL a free application that allows you to download a Youtube video and extract the audio to MP3. If you started off your search in Google (or in the Mac App store for that matter) you are starting to realize it is not that easy. There is a ton of ‘supposed’ freeware out there that, after installing it gives you either an app that does not completely work (unless you pay for it) or is just too complicated. Do you (as a true slider) have a Linux machine handy ? (perhaps a virtual machine or a server ) Then lets fire up the command line and give you a one line command to make that technology work for you.
Youtube-dl is a command line application that lets you download Youtube video’s and if you want to, extracts the audio from said video’s. The command is simple. On Ubuntu you can install it by hitting :
sudo apt-get install youtube-dl
After the installation is complete you need to update the app to the latest version. Also .. a couple of one liners.
sudo youtube-dl -U
You will get an error that you need to update via the GIT repositories once more. Easy peasy (copy and paste people)
sudo wget http://youtube-dl.org/downloads/2013.05.01/youtube-dl -O /usr/bin/youtube-dl
sudo chmod a+x /usr/bin/youtube-dl
Lets start Downloading.
Now surf to the Youtube video of your choice and copy the url to your clipboard.
Go back to the terminal and type :
youtube-dl <paste your url here>
Presto ! You will have the youtube video all for you ! What if you want the audio ?
youtube-dl --extract-audio <paste your url here>
Cross platform compatibility
Youtube-dl is also compatible with Macs and there is an .exe for our Windows friends. But because the command line is so easy I just leave a terminal open to my linux machine (on ANY machine i’m working on) and just copy and paste whatever cat video I would like to keep for prosperity. And youtube-dl does also support tons of other video sites like Vimeo and more !
Download of the week : Until a while ago I used an app on my Mac (I have even forgotten the name) to download any Youtube video’s I needed for the podcast or for some of my presentations. When upgrading to Lion, it decided (or rather Lion decided) it wouldn’t work anymore. Tough 🙁 I had found some alternative online sites to do it but searching for solutions like this is one of those internet ratholes like looking for free mp3 ringtones.) When twiddling around with it I had one of my favorite cross-platform download apps open in the background : Jdownloader. I use Jdownloader for multi-url links and long running downloads. Its free, its cross platform and it has this handy “url catcher” where it notices when you copy a url to your clipboard and then offers to download it. When the activity window for Jdownloader started blinking I checked out the “link grabber” window and was amazed to find that not only it had “caught” the url of the Youtube video I copied, but it offered to do something more. Jdownloader will offer to download AND CONVERT your Youtube video into several sizes like : MP3, FLV, MP4, Webm and more (depends on the video) So this tool is something that should not be missing in your application folder on ANY of your systems.
Download Jdownloader ( mac – lin – win ) here.