I ventured on a technical journey that has proven more difficult than I anticipated. On the cusp of success, I reflect on what has transpired.
I wanted to issue Amazon Redshift commands from AWS Lambda. I wanted to do this with .NET Core, as it would elegantly fit into a solution written in the same.
The issues I came across:
I went through each of these issues one by one, identifying them and working through solutions. In the end, I settled on trying to get a solution deployed and working using Npgsql in .NET Core.
This presented many problems. I got everything working locally first without issue. I used a service account to connect to Redshift using SSL, created a transaction, and issued UPDATE, INSERT and DELETE statements.
However, when I tried to publish to AWS, the build failed.
It turns out that the package manifest for Npgsql 3.2.X was not correct, and there were some version mismatches in the System.Data namespace (amongst others). To get around this, I deprecated my project to .NET Core 1.0 and the version of Npgsql to 3.1.0. Alas, it published.
But on testing, it failed.
But the error was more promising - a timeout when connecting to Redshift. As I suspected, there were network issues connecting from AWS Lambda to the Redshift instance. I was close. So I enabled VPC access to the Lambda via Security Groups and worked through some DNS resolution issues in the VPCs.
And it worked.
The satisfaction was short-lived, as I ended up writing to Kinesis Firehose (and that wrote to Redshift) by the time the solution went to production.
But I was happy to prove it could be done with the tools at hand.
Posted on Jun 05, 2018
An upset alien, Stinky, Bunny Wabbers, Tom Cat, Skeets, a snake and some stick...
Electric trees stand out against a cold purple sky speckled with holes and a star...
He shines bright and glows despite being black and white. He sips earl grey in the mornings and...
Adapted from a photo of the Thai resort town, we look out over the water at an island as the sun...
Also inspired by the afroed master Bob Ross, I executed this winter scene a bit better than the...