In day two of our Google Hacking Week we are going to combine an interesting Google search query (or Google Dork) with a command line command to find AND download any file type you want.
Find the storage room in the back of the store.
Websites on the net consist of more then just webpages with information. They also links to files and folders containting interesting information like PDF’s MP3’s and more. Most of the time these files aren’t ‘visible’ when you visit a specific site but our little friends, the Google Search Bots, DO index them. All you need is the right string to find them.
- intitle: “index of” <filetypehere> <title/genre/artist>
This search query will tell Google to go look for pages with the title “index of”. These pages usually don’t contain a lot of text, but instead contain links to folders and files. Since you are looking for a specific type of file (like for example mp3’s, Pdf’s or something else) you also can add this to the query. Finally you might be looking for mp3’s of Hanna Montana or Tango’s (I don’t know what you like) : That can also be added to the search string. In the end it will look something like this.
- intitle: “index of” mp3 acdc
- intitle: “index of” pdf bookkeeping
- intitle: “index of” epub scott sigler
So using these queries you might find a real treasure-trove of files and info to download. Some of them might even be behind a login/password page (or even a pay wall) but when the web masters don’t do their homework right .. you can find the ‘good stuff’ this way.
So download them one by one ?
If you are just looking for one specific file you can use your browser to find and download it. If you want to download the ENTIRE collection of files on that page .. you need the power of a command line tool called WGET.
Wget can be found on the command line of both Linux, Mac and even Windows machines. Not all the advanced ‘switches’ we give you in this command below might work on Windows, but you can give it a try. The command is
- wget -r -l1 -H -t1 -nd -N -np -A.<.filetype> -erobots=off <url of website>
Replace <filetype> with the type of file you want to download ( .mp3, .pdf, .epub) and <url of website> with the website’s url you found using the Google search. Completed the command might look something like this.
- wget -r -l1 -H -t1 -nd -N -np -A.<.mp3> -erobots=off http://tiobiloute59.free.fr/tiesto/
The download is RECURSIVE, so it “deep dives” into all the folders. Beware : This can get you a LOT of data. So make sure you have the bandwidth and the storage capacity before you start sucking down the internet. Good Luck !
The thing with proprietary software solutions, is that they are great. Everything tends to work smoothly together right up to the point where you decide to wander off the beaten path of supplier XYZ. A couple of years ago I made the crucial mistake of pouring my entire music collection into iTunes. Now, some 10 000 songs later .. its still in there. Being totally OCD I have organized all my tracks into nice little playlists and enjoy my tunes in the “Apple walled garden”. Whether I am playing them from the Mac, sharing out the iTunes library over iTunes to my other macs, blasting them from the Airport express speakers or syncing them to my other i-Devices.
But a couple of weeks ago I could not help myself myself and crawled over the walled garden into android territory with my purchase of a Galaxy Note 2. And accessing my delicately curated iTunes library from THIS device turns out to be an near impossible task. The deep crevasse that divides me from listening to my tunes on my “droid” consists of an incompatibility to sync with iTunes (only IOS devices of course) and the total inability to get the music on my Android in an organised form. Sure I can browse the filestructure of my iTunes library and copy over files to the SD card on my Phone .. but iTunes has “reorganized” my music into folders according to artist .. not according to playlist.
Enter Tunesync. A two-part application app in the android store that saves the day. The deal is simple. Download the server part of their app and install it on your Mac that is running iTunes. Download the CLIENT side of their application and install that to your Android device. Make sure both are on the same wifi network and be amazed !
Tunesync detected my (massive) iTunes library and started indexing the playlists right away. After I selected the playlists I wanted to have on my Android it started to copy over the tracks AND the playlist order in my Androids music collection. 20 minutes later I had all the grooves I needed on my Note2. Tunesync regularly “checks” if the playlists are still up to date and “updates” them whenever I connect or start up the app. I had expected some glitches and on one occasion Tunesync had given me all my playlists .. with no tracks inside ( it erases and re-copies all the tracks on every sync instead of doing an incremental) but when I retried the sync it worked flawlessly.
Tunesync does one thing and it does it well, and the hilarious part is , it does it better then Apples iTunes-IOS wireless sync ! The app is 4.99 in the Play store and worth every dime.
Tunesync is available from the Play Store.
Its quite amazing when you stop and think that iTunes is almost 10 years old this year. Apple’s end-all-be-all monotheistic gateway for your music collection towards your iPod device, is an application that is loved and hated equally. Like a teenage mom, iTunes went from a young, innocent slender application whose sole purpose was to curate your music collection, to an over-bloated thirty-something iPod-Baby machine that had acquired more functionality (and resources) over the years. Right before Apple gave iTunes a much needed binary liposuction with version 11, iTunes was one fat mama.
But that fat mama had started herding my music collection back in its younger years, and over the course of 10 years 15000 tracks have found their way into its arms. Over 100 playlists divide these tracks into manageable chunks and … I’ll probably never get them out again. Album art, MP3 tags, comments, stars, iTunes poisons my library with proprietary metadata and decides its a better idea to arrange the songs for me instead of my own ‘one album per folder’ setup.
When I started using Linux and other operating systems more and more, I got annoyed with the fact that I could not access my iTunes music from a different OS then OSX .. and that bugged me.
Hence it was time for the great escape ! A search for an application that would export every track in every playlist that I had to a predetermined folder structure that would be compatible with parallel universes WITHOUT iTunes. After hitting “The Google” for hours on end, I decided to enter the terms “iTunes Export” and came across a brilliant little application by Eric Daugherty called .. “iTunes Exporter”.
How it works ? Simple : On Osx (sorry , its an OSX app only) close iTunes and fire up “iTunes Exporter” Select the playlists you want to export and voila : iTunes exporter creates a folder with the name of the playlist and exports the tracks (using their id3 tags as file names) together with a playlist file. The result is simple and brilliant : Your (non DRM’d) music exported to a folder tree, ready to be imported into any media system of your choice. Its handy if you iPod-iPhone or whatever iDevice ever breaks .. or you just buy an Android device.
In the words of some Scottish guy with an ax and a skirt : FREEEEDOOOOMM !!!
Links : iTunes Exporter.
By now most (if not all of you) have dabbled a little bit with Linux. Although it might look geeky and complicated at first glance, once you dip your toes into the great lake of open source it becomes a very refreshing computing experience. How do I mean ? Well : I challenge you to the following : Take a Windows Machine OR a Mac and, within 2 minutes FIND and INSTALL a free application that allows you to download a Youtube video and extract the audio to MP3. If you started off your search in Google (or in the Mac App store for that matter) you are starting to realize it is not that easy. There is a ton of ‘supposed’ freeware out there that, after installing it gives you either an app that does not completely work (unless you pay for it) or is just too complicated. Do you (as a true slider) have a Linux machine handy ? (perhaps a virtual machine or a server ) Then lets fire up the command line and give you a one line command to make that technology work for you.
Youtube-dl is a command line application that lets you download Youtube video’s and if you want to, extracts the audio from said video’s. The command is simple. On Ubuntu you can install it by hitting :
sudo apt-get install youtube-dl
After the installation is complete you need to update the app to the latest version. Also .. a couple of one liners.
sudo youtube-dl -U
You will get an error that you need to update via the GIT repositories once more. Easy peasy (copy and paste people)
sudo wget http://youtube-dl.org/downloads/2013.05.01/youtube-dl -O /usr/bin/youtube-dl
sudo chmod a+x /usr/bin/youtube-dl
Lets start Downloading.
Now surf to the Youtube video of your choice and copy the url to your clipboard.
Go back to the terminal and type :
youtube-dl <paste your url here>
Presto ! You will have the youtube video all for you ! What if you want the audio ?
youtube-dl --extract-audio <paste your url here>
Cross platform compatibility
Youtube-dl is also compatible with Macs and there is an .exe for our Windows friends. But because the command line is so easy I just leave a terminal open to my linux machine (on ANY machine i’m working on) and just copy and paste whatever cat video I would like to keep for prosperity. And youtube-dl does also support tons of other video sites like Vimeo and more !
Links : the Youtube-DL page. and their howto for Windows and OSX users.
Download of the week : Until a while ago I used an app on my Mac (I have even forgotten the name) to download any Youtube video’s I needed for the podcast or for some of my presentations. When upgrading to Lion, it decided (or rather Lion decided) it wouldn’t work anymore. Tough 🙁 I had found some alternative online sites to do it but searching for solutions like this is one of those internet ratholes like looking for free mp3 ringtones.) When twiddling around with it I had one of my favorite cross-platform download apps open in the background : Jdownloader. I use Jdownloader for multi-url links and long running downloads. Its free, its cross platform and it has this handy “url catcher” where it notices when you copy a url to your clipboard and then offers to download it. When the activity window for Jdownloader started blinking I checked out the “link grabber” window and was amazed to find that not only it had “caught” the url of the Youtube video I copied, but it offered to do something more. Jdownloader will offer to download AND CONVERT your Youtube video into several sizes like : MP3, FLV, MP4, Webm and more (depends on the video) So this tool is something that should not be missing in your application folder on ANY of your systems.
Download Jdownloader ( mac – lin – win ) here.
Helpdesk Saturday. Being in the IT field I get a lot of phone-calls. I must say, if I don’t “accidently” leave my cell phone at home from time to time .. Saturday afternoons, rainy days, and holidays would turn into some kind of private on line help-desk sessions. It used to be even worse, being awoken on a saturday morning with by a random native of my town who was standing outside my door , computer case in his arms, demanding I fix his computer by the end of the day, and stuff like that. But as I said … its not THAT bad anymore. Some of the people I know have actually evolved from : “Calling Knightwise whenever I fracked something up” to .. “ Calling Knightwise BEFORE I frack something up.” This results in the fact that I get some phone-calls and questions on “advice” rather then “ support “I don’t mind advice calls, sometimes even think they are rather charming. People standing in an IT store, disregarding EVERYTHING the sales person has just tried to sell them, picking up the cellphone and calling ME for objective and trusted advice.
What to buy. Same thing happened last saturday : A phone-call from a niece of mine, on the one big question : What MP3 player should I buy : A creative or an Ipod. Ah … She had come to the right place of course. My house is the virtual Dagobah of portable media device advice and “ The master am I in advice about what to buy ! ” ( or some Yoda-esque quote like that) So after explaining for 20 minutes straight that the Ipod is most surely the better buy. ( I’m not pimping Apple here , I have just been so frustrated with Creative MP3 players that the last time I threw one away I surely must have launched it into orbit) My niece who is a very bright and chipper girl gave me some excellent questions and was well informed about the matter : What’s the best buy in storage-space for your money , What device has the best software, What device has the best hardware quality etc. Gingerly I answered all her questions and pointed out that the Ipod is the best buy. Creative has been struggling to make “ipod killer” after “ipod killer” but they just can”t do it. With software that is more instable then the Tjernobyl nuclear reactor, Devices that look like some five year old just had a run in with a blob of play doh and battery life that mounts up to the attention span of a ADD patient on red bull … its not something you want to by. So I say , scream, argue and imprint the line : Don’t buy a Creative : Buy an Ipod onto her synapses. Short from writing the message on the face of the moon .. I have used up all my recourses in trying to convince her. Satisfied with a job well done .. I hang up .. Flattered by the fact that people trust my advice and follow my directions before they plunge themselves into technological mayhem.
Erm .. I made a booboo. Or so I thought : Because yesterday evening .. I got a call .. From my niece. Prepping up to give her some Ipod pointers, software, websites and advice .. I was a little dumbstruck by her first line ..” Well erm .. I went out to buy an mp3 player .. And I bought a Creative”. My heart skipped a beat and for a little moment in linear time I thought that insanity must have claimed me. I seriously started doubting my communicational skills or the fact I have somehow contracted an alternate personality that had advised the purchase of such a devilish device. As she went on she explained (frustrated) that the Creative just didn”t work right. Importing movies onto it did not function properly and the software that came with it decided it would cease functioning after the first use : Result : The creative does not work. The words “ I TOLD YOU SO “ formed like a big sign in the back of my head. What part of the message : DO NOT BUY A CREATIVE had not been very clear. Did I have to implore reverse psychology ? Did I by some strange interstellar phenomenon appear to be an unreliable source of advice ? I do not know honestly. But I’m a nice guy , did NOT say I told you so , but was unable to offer any advice accept for the fact ‘ Because its CRAP” as to why the Creative MP3 player did not work. Sadly for my niece she has to dive into the manuals of the device and hope and pray it will preform correctly. Oh well.. at least i’m not like NICK BURNS !
Letter to a podcaster, letter to a friend.
This week marked the final episode of the Spacemusic.nl podcast. A show I loved and enjoyed .. Its time to step up and salute its creator.
Beaming out to eternity.
Time rushes by in an ever fasting moving world. We are always busy , always have so much to do , so much stress. As the technological progress beats its ever faster sounding drum, its sometimes hard to keep up. Yet amid this valley of fast talking people, of series of events that dance through the hurricane of progress… there is a eerie silent vortex. As I slam my buds deeper into my ears to drown out the savage society I live in.. My hand reaches out for my Ipod and I breathe in deeply before pressing the button.And then it comes. Like a wonderful ocean of peace that silently scatters the seconds into a blissful eternity. Sounds and chill-out atmospheres shatter my worries and as I peer into the remains of reality … the world slows down. Like tele-porting yourself from the heart of Beijing to a remote iceberg near patagonia .. Reality , stress , rush .. it just stops. A few seconds later I hear the sound of his voice. A voice that is warm, soothing and wonderfully at ease in this ever racing world. A voice that was strange at first , then became familiar .. and that has become the voice of one of my friends. As his carefully selected music flows through my reality the world is sugar coated with this beautiful soundtrack. And somehow my deepest thoughts and emotions greet the light of day after a dark and raging storm.As i toss pebbles into the ocean of my own consciousness , I think back to the very first time I heard this voice. Somewhere in October 2004. As music flowed around me and wrenched me from the grip of everyday life.. The voice spoke about walking through the snow. As I heard the crunching ice beneath his feet I could imagine i was there. Amid a hectic day in my life , he offered me an opportunity to step out into the virgin snow.
The first episode.
Ever since that episode, I have been hooked, addicted, in need of my next fix. Countless times his music, his voice and I chased through busy traffic, dark nights on the freeway, dull tasks at the office. But thanx to the protective chill-out screen , nothing could touch me.. Wherever I was , I chilled out and embraced the eternity between 2 seconds to contemplate what where my deepest thoughts.Or on day-trips, that me and my fiance took in the Belgian ardennes. Spacemusic would be playing as we floated through reality. Between forests, highways, slumbering towns .. The music covered it with .. .Magic. On these long drives we hardly spoke. We learned what it was to be silent , to contemplate .. to enjoy. It didn’t matter where we went.. As long as we had plenty of time to listen to TC’s tunes and appreciate the magic of the world around us.
The honor and the privilege.
I have the honor and the privilege to know the man behind the magic, I have the greater honor of calling him my friend. So today I stand between feelings , just like his music perches me on the edge of realities. I’m happy because I have faith in what the future brings. Faith in the fact that Tc’s creativity will evolve into something different.. that it will move onto greater things. But also I find sadness in the part that there will be no more trips to neverneverland, no more emergency beam-outs to chilloutland, No more spacemusic.But i understand and support the wizzard behind the curtain. I wish him the best of luck , I raise my sword in silent salute.. But most importantly .. I thank him.For the hours and hours of time he poured into this project. For the love , the passion , the joy and the emotion that flowed through these unspoken words. I thank him for the many magical moments I have experienced while listening to the music. The way he could transform a lonely commute through a rainy night , into a flight between the stars. I have and will always enjoy .. spacemusic.
Thank you Tc , you are the best .. and knowing you .. i know that whatever path you take .. you will always be.. the best.