You’re not a real Data Engineer if you use no or low code tools
You’re not a real Data Engineer if you use no or low code tools

I tend to read a lot of posts online to keep up with developments in the data industry. This usually consists of keeping an eye on the data engineering subreddit, people I follow on Twitter (no, I’m not calling it “X”, that’s stupid), my LinkedIn network and data topics on Medium. I find this gives me a good blend of opinions and content to keep up to date with things (even if my reading list on Medium is getting wildly out of hand). One thing I find interesting is there seems to be a sense of elitism from some in the community who are quite vocal about what they perceive a Data Engineer to be (or not to be).

...

Yo ho, yo ho, a pirate’s life for me!
Gaming Yo ho, yo ho, a pirate’s life for me!

What does a pirate say on their eightieth birthday? “I’m eighty” (“aye, matey”….geddit?)

Ahem, having picked up Xbox Game Pass a couple of months ago, I finally decided to give Sea Of Thieves a shot. I played some of the beta over a year ago and quite enjoyed it, but was running solo, so didn’t get the full experience and filed it away under “might get to it at some point”. Some of the DoD Storm crew have been playing pretty regularly and raving about it, so I thought I’d jump in and give it another shot.

read more
Is AWS retiring the ds2.xlarge Redshift node type?
Development Is AWS retiring the ds2.xlarge Redshift node type?

It’s rampant speculation time. AWS have released a number of nice features for Redshift over the last few months, from maintenance improvements like auto-vacuum and auto-analyze, to time savers like the in-browser query editor and new cluster configuration recommendation tool, that helps you find the right cluster configuration for your needs. It’s these two features that lead me to my wild assumption in the title, for one main reason - neither of them support the ds2.xlarge node type.

read more
Thoughts on the Anthem demo
Gaming Thoughts on the Anthem demo

This weekend I’ve been spending a little time on the Anthem demo/beta. That is, eventually, once the initial networking issues were resolved. It’s part of the reason for these betas, especially for online persistent games like Anthem, The Division, or Destiny, but the scale of the networking issues suggests that either someone completely underestimated the number of players, or made some very poor technical choices.

Regardless, I managed to get into the demo and spend some time flying around in my Iron Man-esque Javelin suit, and here’s what I made of the experience.

read more
My top games of 2018
Gaming My top games of 2018

It’s that time of year again where we’re recovering from eating and drinking too much, and looking forward to the new year filled with exciting opportunities and a chance to do better. Who knew that an arbitrary point in the Earth’s orbit round the Sun had such power? I’ve definitely neglected the blog this past year or so (I’ve been a little more active on my dev blog, but only just) and I’m contemplating just migrating all the content from this site over to picnicerror.net and focusing my efforts on one site. Anyway, I digress. I think my last post on this site was looking back (very belatedly) at my favourite games from 2017, so it seems kind of fitting that my next post is this one.

read more
Amazon Redshift now supports Elastic resize
Development Amazon Redshift now supports Elastic resize

One of the major pain points for me with Amazon Redshift has always been the coupling between storage and compute. Competitors like Snowflake and Google’s BigQuery offer independent compute and storage, which means easier (and quicker) scaling in times of increased load. Redshift’s main drawback in the scalability sense has been that it can take up to 24 hours to resize your cluster (during which it’s in read-only mode), meaning there’s a lot of pressure to get your cluster configuration spot on before you go into production. Redshift’s provision of elasticity is just not up to par with most of Amazon’s other services. While Redshift Spectrum helps with this, it’s not a solution to the issue of scalability for an existing cluster.

In the lead up to re:Invent, Amazon last night dropped a load of really neat announcements (server-side encryption for DynamoDB as standard, SSE support for SNS), among which was the reveal of Elastic resize for Redshift. As an aside, if this is the stuff they’re announcing now, there should be some really nice announcements at re:Invent.

read more
Redshift Spectrum finally supports Enhanced VPC routing
Development Redshift Spectrum finally supports Enhanced VPC routing

What seems like an age ago, I spotted a setting on one of our Redshift clusters that suggested Enhanced VPC routing support for Redshift Spectrum might be on the way. After waiting a while, and waiting some more, and then waiting some more, it seems that Amazon have finally released this into the wild, and Redshift Spectrum now works with clusters that have Enhanced VPC routing available!

As of Build 1.0.4349 or Build 1.0.4515, this functionality will be available in Redshift. It hasn’t made it into the official announcements yet, but it has popped up on the Redshift forums here: https://forums.aws.amazon.com/ann.jspa?annID=6197

read more
AWS releasing in-browser Query Editor for Redshift
Development AWS releasing in-browser Query Editor for Redshift

One of the things that I really like about Google BigQuery is the ability to write queries right there in the web browser without having to install a hefty IDE. Sure, there are times when having the full power of something like JetBrains DataGrip comes in handy (source control integration, customisation, formatting), but sometimes you just want to dive in and write a quick query without any messing around. Amazon did this for Athena, which was really handy, but strangely never did so for Redshift…until now!

read more
My Top 4 Games of 2017
Gaming My Top 4 Games of 2017

I did it last year, so it’s time to do it again. Actually, it was time to do it again about six months ago but life got in the way. Here are my top games from 2017. Caveat: This list is only from games I’ve played myself in my limited available time. There are plenty of other great games that I just haven’t got round to yet.

read more

About

picnicerror.net is a personal blog where I post various ideas, thoughts and discoveries through both my day to day work in marketing technology and general hobbies and interests.

Know More

Tag

Social