A "spectrum_enable_enhanced_vpc_routing" parameter has appeared in Redshift.

Redshift Spectrum finally supports Enhanced VPC routing

What seems like an age ago, I spotted a setting on one of our Redshift clusters that suggested Enhanced VPC routing support for Redshift Spectrum might be on the way.  After waiting a while, and waiting some more, and then waiting some more, it seems that Amazon have finally released this into the wild, and Redshift Spectrum now works with clusters that have Enhanced VPC routing available!

As of Build 1.0.4349 or Build 1.0.4515, this functionality will be available in Redshift.  It hasn’t made it into the official announcements yet, but it has popped up on the Redshift forums here: https://forums.aws.amazon.com/ann.jspa?annID=6197 Continue reading…

in-browser query editor for Redshift (close-up)

AWS releasing in-browser Query Editor for Redshift

One of the things that I really like about Google BigQuery is the ability to write queries right there in the web browser without having to install a hefty IDE.  Sure, there are times when having the full power of something like JetBrains DataGrip comes in handy (source control integration, customisation, formatting), but sometimes you just want to dive in and write a quick query without any messing around.  Amazon did this for Athena, which was really handy, but strangely never did so for Redshift…until now! Continue reading…

My Top 4 Games of 2017

I did it last year, so it’s time to do it again.  Actually, it was time to do it again about six months ago but life got in the way.  Here are my top games from 2017. Caveat: This list is only from games I’ve played myself in my limited available time.  There are plenty of other great games that I just haven’t got round to yet. Continue reading…

AWS Lambda function with SQS as an event trigger

AWS Lambda can now be invoked directly from SQS

While quietly perusing Twitter this evening, I happened to notice one from the official AWS account with a link to a blog post from Amazon tech hero Randall Hunt describing the newly available capability for AWS Lambda: SQS as an event source!

This is functionality that I, personally, have been wanting for a while now.  While Simple Notification Service (SNS) is absolutely brilliant for a fan-out architecture, and provides immense flexibility with a wide range of supported subscriber types, controlled, serverless polling of SQS wasn’t really a viable option.  While you *could* run a Lambda for a few minutes doing long-polling on SQS, and then terminate before exhausting the 5-minute execution duration cap, it really felt a bit dirty.  To properly implement a queue-polling architecture, you really had to deploy an application on EC2, which meant managing servers etc.  Not that there’s anything wrong with that of course, it just seemed like there was a big glaring hole in the Serverless model.

Native SQS to Lambda event integration though really patches this omission and then some.  Randall’s blog post explains it in full, but it seems like Amazon have implemented some really nice intelligent scaling mechanisms to adjust Lambda concurrency (up to a defined limit) in response to queue depth.  This should really help constrain costs and ensure consistent throughput regardless of spiky traffic.

I’ve not yet had a chance to observe this in the wild though, so best tested for your workload before betting the farm on it, but this looks like yet another long-awaited piece of functionality that Amazon have knocked out the park.  Have you explored SQS as an event source for Lambda yet?  Any observations or gotchas so far?  Let me know in the comments below!

Hello World function on the AWS Lambda console

6 Tips for developing with AWS Lambda Functions

AWS Lambda functions are getting a lot of press right now, with “serverless” being the topic de jour. We’ve been using it extensively and it’s incredibly flexible and useful, although not suitable for every situation. It really does remove a lot of obstacles for the average developer. In saying that, there are a few tips that I’d like to share based on our experiences with Lambda. Many of these may be no-brainers, but it’s always worth sharing these things as you never know when they’ll be useful.

Continue reading…