Top Tracks of 2016

In 2016 Spotify’s “Discover Weekly” playlist made it much easier to find good new tracks and I started to break my habit of only listening to whole albums.

Here's a playlist of my 11 favourite tracks from 2016.

• Lanterns on the Lake “Beings” — Stuck For An Outline
  • “Even with my edges torn, he sees my good side.” 
  • Sublime blend of indie sensibility and postrock musicality.
• The Jezabels “Synthia” — Come Alive
  • “Allow myself more than those mothers were allowed.” 
  • Classic Jezabels: starts ever so gently, up to a pounding crescendo and down again.
• Daughter “Not to Disappear” — How
  • “To keep the pain to a minimum” 
  • Daughter often sounds like a solo project but the rest of the band make this track ascendant.
• Lucy Rose “Nebraska” — Nebraska (Public Service Broadcasting Remix)
  • “I hope it’s not too late.” 
  • Sometimes the remix makes the song. The original feels slightly passe and the pacing is wrong but this remix is haunting.
• Air Waves “Parting Glances” — Frank
  • “I’m a bad man for coming around but you invited me here.” 
  • Turns out to be a female singer but you can't tell from the vocals. Feels like one of those ‘hidden gem’ indie movies compressed into 2:51.
• Dyan “St. James ” — St. James
  • “I left with heartache and returned with rage” 
  • From some folks who actually do make music for indie movies.
• Agua Roja “Joey ” — Joey
  • “I can’t let go when I know that you are out there” 
  • Somewhere between shoegaze and surf rock. Wonderful.
• Syvia “Silent Violence” - Anxious Animal
  • “I don’t need your tough love to make me work harder” 
  • We all face that same self doubt.
• Land Lines “Plans ” — Plans
  • “There are no right decisions, only decisions we make.”
• HÆLOS “Full Circle ” — Cloud Nine
  • “Why did you leave me here?” 
  • Reminds me of Massive Attack and Faithless at their very best.
• Christine and the Queens “Christine and the Queens” — Here
  • “Here is where everything happens” 
  • So good that you won’t even notice half of it is in French.

Two thoughts on the Apple / FBI kerfuffle

1. A cell phone is (and remains) the best surveillance device ever conceived.
2. Apple is legally, morally and practically justified in opposing the order.

1.
Your carrier has a perfect record of your location 24/7 going back years. Everything you do on a smartphone uses the internet and is the opposite of anonymous.

It’s trivially easy to physically locate any person or service you interact with. Every service you use (Facebook, Google, Skype, etc.) keeps **exhaustive** records of everything (messages / searches / calls / etc. ). Your phone now backs itself up automatically (including photos).

What can you actually do on your phone that **only** happens on the phone? Almost nothing. Even a simple note will be saved to the internet.

All of this data is fully accessible to law enforcement. Carriers and services have special help desks for this purpose. Often they do not require a subpoena under a law called “Communications Assistance for Law Enforcement Act (CALEA)”.

2.
So, what exactly does the FBI want? The FBI would like to use San Bernardino as leverage to force Apple to make it easy for them to search phones again. That’s it.

Law enforcement got a free ride on searching phones due to court rulings that they are not subject to 4th amendment protection and searches do not require a subpoena.

iPhones before 2015 (and most Androids still) can be plugged in to a device (from China!) and all the data is “sucked out”. That’s very helpful if you’re the FBI but unfortunately these devices are **widely** available and heavily used by criminals.

Are you happy to keep your credit card details (ApplePay) on a phone if some thief can “suck out” those details in 2 seconds flat? Not likely.

The back door the FBI wants doesn’t exist on recent (more secure) phones, it can’t safely exist (remember the the TSA luggage key?) and is actually not even necessary. They can and do get 99% of the same data via other means.

The only way to ensure security is to never create a way to break it. Once a back door is created for law enforcement a huge wave of criminal effort will be unleashed to find, buy, or bully (think China) access to it.

The Sound of 2015

Change of format this year - 12 months : 12 albums. The albums Last.fm says I listened to most in each month.

• January : Blondfire - “Young Heart”
• 80% Metric / 20% The Cardigans
• February : Marika Hackman - “We Slept At Last”
• 80% Bill Ryder-Jones / 20% Laura Marling
• March : Dan Mangan + Blacksmith — “Club Meds”
• 33% The National / 33% David Gray / 33% Typhoon
• April : Big Data — “2.0”
• 50% Passion Pit / 40% Guest Vocalists / 10% Diplo
• May : Other Lives — “Rituals”
• 25% Shearwater / 25% Bon Iver / 25% The National / 25% Andrew Bird
• June : Florence + the Machine — “How Big, How Blue, How Beautiful”
• 100% Florence. More somber than previous albums.
• July : Tiny Deaths - “Tiny Deaths (EP)”
• 33% Beach House / 33% Cults / 33% Chad Valley (Highly Recommended)
• August : The Mynabirds — “Lovers Know”
• 33% Feist / 33% Rilo Kiley / 33% The Postal Service
• Album of the Year for me
• September : Jason Isbell — “Something More Than Free”
• 50% Drive By Truckers / 50% Ryan Adams & The Cardinals
• October : Stars — “No One Is Lost”
• 50% Beautiful South / 35% Metric / 15% Disco
• November : Grimes — “Art Angels”
• 100% Grimes. More accessible (less abstract) than previous albums.
• December : CHVRCHES — “Every Open Eye”
• Not such an instant classic as their first but excellent nonetheless.

Taken from my Last.fm stats. I’ve also made a Spotify playlist of my favorite track from each album.

The Sound of 2014

I still listen to music all day, every day during the working week. So here's my personal Top 10 from 2014 in Letterman-style reverse order.

This is available as a Spotify playlist and each track links to the album it comes from on Spotify.

10. Royal Blood - "Come On Over"
    - Best rock album released this year in my view. Of course this is the year that rock gave up.
9. Phantogram - "Black Out Days"
    - Electro pop trip-hop. Check out their back catalogue if you like this.
8. ODESZA - "Memories That You Call"
    - Ibiza style chillout via Seatle.
7. Lia Ices - "Sweet as Ice"
    - Electro folk? Mystical lyrics plus an ethereal voice with a subtle electronic sheen.
6. Bombay Bicycle Club - "Home By Now"
    - British indie at it's best. Eclectic and sublime.
5. HAERTS - "Hemiplegia"
    - An elctro Fleetwood Mac. Kind of like HAIM but better.
4. I Break Horses - "Denial"
    - Downtempo Swedish electronica, dare I say "chillout".
3. RHODES - "Raise Your Love"
    - British indie folk. Better than the much touted Hozier in my view.
2. London Grammar - "Hey Now"
    - Exceptional. The KEXP live video actually sounds better than the album.
1. The Jezabels - "The End"
    - Still the best band you've never heard of.

I'm on last.fm as joeharris76 (of course). Stop by and say hi.

Micro-review of InformaticaCloud for Amazon Redshift

I recently finished an evaluation of Informatica Cloud's platform as a candidate to take over our Redshift ETL workload.

I'm posting this 'thumbnail' for the sake of my own memory as much anything else. :)

TL;DR - It's very _close_ to being an ideal solution but, sadly, has some flaws we can't work around right now.

Good:

• Very straightforward agent based architecture. Very simple to install and consume.
• Reliably syncs/replicates *simple* data from source DBs via flat files and cleans up after itself.
• Performs simple and complex joins on the data source. This is a *major* weakness for competitors as they can't do this.
• All functionality is wizard driven and probably consumable by line of business users.

Bad:

• Cannot create SQL based transforms without uploading PowerCenter mappings. This is too much to ask of LOB users.
• Extracts land as plain text on the agent host before gzipping. This is a *very* slow approach. They should pipe directly to gzip.
• Logging is *extremely* verbose. Every single row is written to the success or error. You'll quickly create many GB of logs.
• Upsert operations update *every* matching row in Redshift. This is handled as a delete plus and insert within Redshift, which means we then need a very slow vacuum to reclaim space.
• Finally, the deal breaker, cannot handle `timestamp with timezone` data types. In 2014. Redshift cannot handle these either but I need the ETL platform to be able to work with them.

The Sound of 2013

I listen to music all day, every day during the working week.

Here's my personal Top 10 from 2013 in Letterman-style reverse order.

10. Drenge - Bloodsports
    - 2 man garage punk. Helmet reincarnated.
9. Young Galaxy - Pretty Boy
    - Indie electro pop from Montreal. The whole catalog is excellent.
8. Blackbird Blackbird - It's War
    - Chemical Brothers meets Morrissey with added Ibiza chill. I was hooked after 10 seconds.
7. Hooray For Earth - No Love
    - Indie rock fused with electronica. The Happy Mondays on a downer in Brooklyn.
6. Mr Little Jeans - Oh Sailor
    - Electro pop with a backing choir, also check out the cover of 'The Suburbs'.
5. Say Lou Lou - Julian
    - Swedish electro pop, high-style and ethereal vocals. Should be taped to the cover of Vogue.
4. Other Lives - Take Us Alive
    - Oklahoma folk-tronica. Discovered via the amazing skate video 'India Within'
3. The National - Graceless
    - It's not 'Fake Empire' but it's pretty damn good.
2. Daughter - Youth
    - Haunting folk. Bleak and beautiful.
1. The Jezabels - Endless Summer
    - Best band you've never heard of. INXS plus Texas with a bit of the Smashing Pumpkins on the side.

I'm on last.fm as joeharris76 (of course). Stop by and say hi.

Migrating from SQL Server to Redshift

An overview of the work required to migrate our large SQL Server data warehouse to Amazon Redshift. We faced and resolved a number of issues, particularly around the different Unicodes required by each database.

Migration to Redshift from SQL Server from joeharris76