You’re not a real Data Engineer if you use no or low code tools
You’re not a real Data Engineer if you use no or low code tools

I tend to read a lot of posts online to keep up with developments in the data industry. This usually consists of keeping an eye on the data engineering subreddit, people I follow on Twitter (no, I’m not calling it “X”, that’s stupid), my LinkedIn network and data topics on Medium. I find this gives me a good blend of opinions and content to keep up to date with things (even if my reading list on Medium is getting wildly out of hand). One thing I find interesting is there seems to be a sense of elitism from some in the community who are quite vocal about what they perceive a Data Engineer to be (or not to be).

...

6 Tips for developing with AWS Lambda Functions
Development 6 Tips for developing with AWS Lambda Functions

AWS Lambda functions are getting a lot of press right now, with “serverless” being the topic de jour. We’ve been using it extensively and it’s incredibly flexible and useful, although not suitable for every situation. It really does remove a lot of obstacles for the average developer. In saying that, there are a few tips that I’d like to share based on our experiences with Lambda. Many of these may be no-brainers, but it’s always worth sharing these things as you never know when they’ll be useful.

read more
SQS vs SNS for Lambda Dead Letter Queues
Development SQS vs SNS for Lambda Dead Letter Queues

Serverless computing and event-driven functions are what it’s all about at the moment. But what happens when the event trigger fires, and your process then encounters an error? How do you recover from this given the event has since passed and may never happen again? This is a common question in AWS when working with their serverless, event-driven Lambda Functions.

Fortunately, AWS lets you define Dead Letter Queues for this very scenario. This option allows you to designate either an SQS queue or SNS topic as a DLQ, meaning that when your Lambda function fails it will push the incoming event message (and some additional context) onto the specified resource. If it’s SNS you can send out alerts or trigger other services (maybe even a retry of the same function - although watch out for infinite loops), or any combination of the above, given its fanout nature. If it’s SQS you can persist the message and process it with another service.

So let’s look at both options in a little more detail.

read more
Is AWS about to enable Redshift Spectrum with Enhanced VPC Routing?
Development Is AWS about to enable Redshift Spectrum with Enhanced VPC Routing?

AWS is knocking it out of the park at the moment with loads of new services and features coming out every week. Indeed, it can be hard to keep up with the degree of change. But, while working on one of our Redshift clusters today we spotted a potential scoop that would remove a key blocker for one extremely useful service, Redshift Spectrum.

Up until now it’s only been possible to use Spectrum if you don’t have Enhanced VPC Routing enabled on your Redshift cluster. There are so many benefits to using Enhanced VPC Routing (reduced data transfer cost, control, security) that it’s hard to see why anyone wouldn’t be using it, especially if you move data between Redshift and S3 a lot.

But we spotted a new parameter being applied to one of our clusters when we made some maintenance changes to a parameter group. There’s now a parameter named spectrum_enable_enhanced_vpc_routing showing, which hints that Amazon may be about to remove this crucial limitation.

read more
Three Guardians and a Baby – Destiny 2 and parenthood
Gaming Three Guardians and a Baby – Destiny 2 and parenthood

I recently read Keza MacDonald’s article on Kotaku about struggling to get into Destiny 2 now that she has a nine month-old baby, and was able to sympathise somewhat. We’ve just had baby number two (now approaching 2 months old), and I caved and bought Destiny 2, which is (as expected) proving tricky to play while dealing with an infant.

But I was a little surprised to read some of the hatred directed at the article and author (okay, mostly on Twitter). Here’s someone writing an article about her experiences with a notoriously grindy game that she likes, but is coming to terms with the fact that she may not be able to explore all its content due to other commitments. It’s an opinion piece.

read more
Redshift connectivity officially announced for Power BI Service
Data & analysis Redshift connectivity officially announced for Power BI Service

Last year, Microsoft added a preview connector enabling Power BI to query Amazon Redshift. This wasn’t publicised as an “official” data source, and took some steps in order to be able to even see the connector in Power BI Desktop. Crucially, you could only use this connector in Power BI Desktop, not when workbooks are deployed to the cloud. Yesterday, Microsoft announced the connector is now available within the Power BI Service, which means that workbooks containing Redshift data connections can now be deployed to the cloud. I’ve been working a lot with Redshift over the past year or so, and Power BI’s still my go-to data-viz solution, so I’m delighted to see the this announcement, as it means that Redshift-based workbooks can now be shared with others via powerbi.com.

read more
Reboots: heresy or homage?
Gaming Reboots: heresy or homage?

Hollywood is obsessed with reboots. Total Recall, Star Trek, Big Trouble in Little China (I’m gutted about that last one)! It’s wide open for debate as to whether all these reboots are a good thing, or whether people should stick to creating original stories. This practice has bled over to the gaming industry as well in recent years.

read more

About

picnicerror.net is a personal blog where I post various ideas, thoughts and discoveries through both my day to day work in marketing technology and general hobbies and interests.

Know More

Tag

Social