Iain Samuel McLean Elder

AWS UK Community Summit 2024

2024-10-04T20:00:00+00:00

On 2024-09-26 I attended the AWS UK Community Summit in Manchester.

Of course it rained. Everyone warned me it would! I’m used to it living in Glasgow. It turns out Manchester isn’t even in the top 20 rainiest UK cities. I wonder why it gets such a wet reputation.

My picks from the schedule were mostly from the Architects track, with some Builders and Leaders content too.

Damien Jones: Building and Automating Serverless Auto Scaling Data Pipelines in AWS
David Boyne: Avoiding the big ball of mud with serverless event driven architectures
Sheen Brisals: How do you measure your enterprise’s readiness for serverless?
Yan Cui: Money-saving tips for the frugal serverless developer
Vadym Kazulkin: Detect operational anomalies in Serverless applications with ML-based Amazon DevOps Guru
Sarah Hamilton: Operating Quality Event-Driven Serverless Architectures

Full schedule:

Serverless: in 5 out of 6 talk titles, it looks like an unofficial theme of the event! It’s a broad term that refers to managed services that allow you to build robust solutions at a higher level of abstraction. You think about events and workflows rather than servers, storage, and requests. That stuff is obviously still important, but for most use cases there is no need to stray from the default behavior that services such as S3 and Lambda provide. I’m enthusiastic about serverless, because the less generic infrastructure I have to manage, the more I can focus on delivering value.

Sarah’s talk was my favorite because she changed my mind and settled a doubt. It was a secondary point in her argument, but Sarah made me see the light on AWS’s CloudFormation Development Kit (CDK).

A complex CloudFormation template may describe resources with patterns of interdependencies. Writing this all out by hand may be tedious and validating its correctness can be worse. If you’re tempted to parse the template into an object model to check its properties programmatically, you may as well go further and use a whole object model to generate the template. That’s what the CDK is: a programmatic object model for CloudFormation resources that also renders templates.

AWS maintains the CDK so we trust the correctness of the object model and the template renderer. Then what remains for us to test the correctness of our templates is to test properties of an object model.

Sarah’s revelation is what made this report take so long to write. After the conference I put her lesson into action. Last week I speedran a course in TypeScript and CDK to solve a problem for which bare CloudFormation offers ugly solutions. I hope to share more about that later.

The rest of the article shares my main takeaways and dumps links or resources for further research.

Damien Jones, a.k.a. the Amazon Web Shark, kicked us off with Building and Automating Serverless Auto Scaling Data Pipelines in AWS.

“Big Data” is often described by four V’s: Volume, Variety, Velocity, and Veracity. Damien added a 5th V: Value: Is it important or useful? Does it have a monetary value?

Damien told us the pros and cons of the building blocks he uses for his pipelines: S3, Lambda, Athena, Glue, Step Functions, and EventBridge.

I hadn’t seen Glue in action before. I like its dashboards that present the data quality checks.

Damien’s recorded demo of his pipelines was slick. I prefer recorded demos because controlling the screen doesn’t distract from the communication. He overlayed a timer, highlights, and blurring to guide the audience’s attention to the right place.

Damien Jones on LinkedIn
The Amazon Web Shark blog
Community Sessions repo on GitHub
IBM defined a 5th V back in 2016
Open Broadcaster Software (OBS) for recording
CapCut for video editing

David Boyne followed with Avoiding the big ball of mud with serverless event driven architectures.

The Project Paradox: Why do we take the biggest decisions at a time when we have the least knowledge?

Without conscious design, every project inevitably graduates toward a complex, unpleasant mess.

Event-Driven Architecture (EDA) and Domain-Driven Design (DDD) give us patterns to control the complexity, but we need better tools to make them more discoverable.

Links and resources to learn more:

David Boyne on LinkedIn
Boyney.io
More about the Project Paradox
Big Ball of Mud article by Brian Foot and Joseph Yoder in 1999
EventCatalog David’s project to make EDA more discoverable
Wing Lang, David’s other project, a programming language for the cloud
EventStorming: a workshop format and book by Alberto Brandolini for collaborative exploration of complex business domains
Postel’s Law: “Be liberal in what you accept, and conservative in what you send”
Event-Carried State Transfer: a pattern that allows clients to update without needing to call back to the source system
Events tell story. Tables tell state.
Backwards compatibility, extra fields, parallel versions
David mentioned some patters: Open Host Service, Anti-Corruption Layer, Conformist
Domain-Driven Design: A Practitioner’s Guide has a glossary to describe these
Azure Cloud Design Patterns also describes them
AWS Cloud design patter, architecture, and implementations also describes them
CloudEvents, a common specification for event data
AsyncAPI, industry-standard asynchronous API definitions

Sheen Brisals asked How do you measure your enterprise’s readiness for serverless?

Serverless is not magic. You need to understand your use case, skills, and requirements. You need to assess its efficiency, cost, and trustworthiness.

Afraid of vendor lock-in? AWS is not your vendor but a partner.

I asked Sheen what he thought about David Heinemeier Hansson publicly quitting AWS for bare-metal servers because it’s cheaper, because he doesn’t need instant scaling. Sheen said that you need to look at the total cost of ownership and innovation.

Sheen Brisals on LinkedIn
Book: Serverless Development on AWS: Building Enterprise-Scale Serverless Solutions by Sheen himself
Blog: Introducing: The Serverless Book! by Sheen himself
Book: Learning Systems Thinking by Diana Montalion
Talk: How to shift to a serverless mindset by David Anderson (various recording on YouTube)
Book: The Flywheel Value Effect also by David Anderson
Book: Domain-Driven Design by Eric Evans
Book: Building Microservices by Sam Newman
Article: We have left the cloud by David Heinemeier Hansson
Taco Bell Migrates Ecommerce to AWS Serverless with Trk10, Cuts Cost by 90%

Yan Cui took us up to lunch with Money-saving tips for the frugal serverless developer.

Yan gave a rapid fire of common costly mistakes and how to solve them. After each, a refrain: “I don’t mind spending money, but I want value!” All components need a purpose and a return on investment.

Favorite 1: CloudWatch Logs ingestion is expensive. If you use Lambda extensions and a telemetry API to bypass CloudWatch logs, CloudWatch Logs may still charge you for ingesting the default messages that every Lambda execution writes. Block all writes using an IAM policy.

Favorite 2: Avoid using the RequestResponse (synchronous) calls for Lambda-to-Lambda, because you will pay twice for the execution time. For secondary tasks that aren’t in the critical path, use Event (asynchronous) invocation.

I asked Yan whether I can avoid paying for Lambda to wait for a slow API response by using a Step Function, which waits for free. If the API was a webhook, then maybe a Step Function would make sense, but if it was a synchronous HTTP request/response, then Lambda or something needs to keep the connection open to receive the response.

More resources:

Yan Cui on LinkedIn
The Burning Monk, Yan’s blog
Slide deck shared 2024-05-18 with all the examples
The CloudWatch Book by Tobias Schmidt and Alessandro Volpicella
Lambda advanced logging controls allow you write JSON. “Not great, better than nothing.”
Laws of the Frugal Architect
Configure direct end-user access to DynamoDB via identity tokens to avoid API gateway and Lambda costs. Not for fainthearts!
Momento cache charges per use (compare to ElastiCache, which charges per uptime)

Vadym Kazulkin showed how to Detect operational anomalies in Serverless applications with ML-based Amazon DevOps Guru.

We learned what problems the Amazon DevOps Guru solves and saw it in action.

DevOps Guru is a time-saver over standard CloudWatch. It takes the drudgery out of making alarms for every operational metric. It applies machine learning to Config, CloudWatch, X-Ray, and CloudTrail to provide faster operational insights.

It categorizes notifications as “reactive” and “proactive”. Its proactive mode overlaps with Security Hub. Whereas Security Hub checks security configurations such as public access and encryption, DevOps Guru adds value by checking operational configurations such as read capacity or detailed monitoring.

Vadym generated capacity notifications using a load testing tool called hey, a replacement for Apache Bench, or like a rapid-fire curl.

Sarah Hamilton told a story about Operating Quality Event-Driven Serverless Architectures.

She opened with an analogy, a restaurant called Sync’n’Serve that struggles to coordinate its chefs, waiters, and customers as its popularity grows. I like talks that start with stories.

Sarah shared her opinions on testing, with reference to the “square of balance” popularized by Sheen’s book. The square describes a trade off among testing, observing, recovering, and delivering.

Sarah prefers unit tests over integration tests and end-to-end tests. The further you get from each component, the more it costs to perform a test because of its complexity, and the less it tells you about exactly where a problem occurs and who is accountable. Unit tests of course can’t catch everything, so she prefers to cover only the critical paths using the more complex test types.

Speedy reversal of a bad change is worth more than complete test coverage. Bugs and errors are inevitable, so we need to be ready to detect (observe) them and handle (recover from) them, and we need to keep delivering to stay relevant in the market.

Sarah Hamilton on LinkedIn
Sarah’s blog: Bridge Integrity: Integration Test Strategy for EventBridge Based Serverless Architectures
Luke Hedger’s site (co-author of Sheen’s book)
Edsger Dijkstra: testing show the presence, not the absence, of bugs.
Use Composite Alarms to avoid overload and fatigue. One is enough to prompt action.
Unit Testing doesn’t check inter-service contracts. Have each team and domain publish a schema that the others consume.

Thanks to all the Summit team for delivering an exciting and satisfying full day of AWS learning and networking. Joe Park, Edmund Craske, Robin Ford, Phil Horn, Tom Misiukanis, and the rest who I’ve yet to meet, you all deserve to be proud of what you did for the community this year.

Looking forward to more, bigger, and better next year!

Edinburgh Python Meetup August 2024

2024-09-10T20:00:00+00:00

Python Meetup Edinburgh August 2024

On 2024-08-27 I was the opening speaker at the reboot of the Edinburgh Python Meetup. Thanks to Mark Smith’s team for masterminding the event and inviting me to such a delightful evening.

For me, starting out as a public speaker, this just the right size of event. About 40 people, some familiar faces, and a relaxed atmosphere. They warmed us with pizza and beer and general mingling in the open-plan kitchen for about 30 minutes before we all shuffled over to the seating and the projector to talk about snakes.

I had the honor of opening with What year was that? Dating music posters with Python.

My open-source work using Python in the AWS Community in theory gives me a lot to talk about, but to do it justice I wanted to more prep time than I was prepared to give myself in the last weeks of summer. This topic was simpler to prepare and a gentle opener that was well-received by Python newcomers and oldtimers alike. What follows is a bare-bones explanation of the solution. I’ll share more detail and insights from my research in a future blog post.

“Saturday 2nd August. What year was that?”

A Sunday afternoon WhatsApp rant from my Dad about the Creamfields 98 lineup inspired me to explore a modern answer to this question.

After a lot of Googling, a scan of a paper-based perpetual calendar helped him solve the problem. It’s a fine solution. A 200-year range fits on the back of a phone book, needs no electricity, and solves the problem with some linear lookups. But it’s tedious, and doesn’t get me invited to Python talks.

Can we solve the problem faster with today’s digital calendars? Almost. The user interface for repeating events is designed to solve problems such as “every Monday” or “every 4th Tuesday”. Google Calendar’s designer regrettably did not foresee my need to configure “Every Saturday 2nd of August”.

The underlying standard that enables calendar interoperability is the IETF’s iCalendar protocol. It’s easy to express using recurrence rule syntax:

RRULE:
    FREQ=YEARLY;
    BYDAY=SA;
    BYMONTH=5;
    BYMONTHDAY=2

And we can use the rrule class from Python’s dateutil module to evaluate it:

from datetime import date
from dateutil.rrule import rrule, YEARLY, SA

dads_birthyear=date(year=1954, month=1, day=1)

today=date.today()

creamfields_rule=rrule(
    freq=YEARLY,
    byweekday=SA,
    bymonthday=2,
    bymonth=5,
    dtstart=dads_birthyear,
    until=today,
)

print([match.year for match in creamfields_rule])

[1959, 1964, 1970, 1981, 1987, 1992, 1998, 2009, 2015, 2020]

You’ll always get a range of years. You need some cultural context to figure out which one it must be. A music festival attracts performers at the top of their game, so knowing when a band was big helps you to narrow it down.

Mark Smith followed to confess and apologize for Stupid Things I’ve Done With Python.

Mark has a folder on every computer called “stupid Python tricks”. It’s full of stuff that Python allows you to do, even if you probably shouldn’t.

Things like ish, which abuses the minus operator and Boolean values to make this code for testing “trueishness” into valid syntax:

if 'Yup' == True-ish:
    print 'True-ish!'

The serious point he makes behind the fun and weird code is to understand Python’s Data Model.

Python implements the basic operators using specially named methods on objects (for example __sub__ for left-side subtraction and __rsub__ for right-side subtraction). As with all programming, a solid understanding of the fundamental concepts makes it easier to reason about the higher-level behavior, and easier to write expressive code.

Pathlib in the standard library overrides the division operator via __truediv__ and __rtruediv__ methods to allow path construction using syntax like this:

>>> Path("usr") / Path("local") / Path("bin")
PosixPath('usr/local/bin')

It inspired me to deepen my understanding of the Python language. I’m a confident Python hacker and have middling skills, but I struggle to explain what self really is or what fundamentally defines an object. Mark promises that all the answers are in the data model doc, so add that to the reading list!

No one recorded us, but YouTube shows Mark delivering this talk elsewhere, most recently at PyCon Italia 2022.

Thanks to TrustPilot for hosting, TravelPerk for sponsoring the event, to Neil Miller for giving us a opening pep talk,and to Mar Doig for taking the photos!

After topping up on beer and pizza at TrustPilot HQ, we drifted into Milnes on Rose Street for some more pricey pints and joyously nerdy chatter. For me it was great way to reintegrate into Edinburgh’s tech scene.

Looking forward to monthly meetups going forward. September is already fully booked! Hope to see you there soon.

How to find expiring RDS certificates across an AWS organization

2024-07-29T16:00:00+00:00

Most AWS regions have a looming deadline (2024-08-22) to rotate an SSL/TLS certificate in AWS RDS/Aurora. An RDS instance that keeps using the old certificate beyond this date may stop working because clients may refuse the expired certificate.

As the administrator an AWS organization, how do I identify instances across the organization that need attention?

The AWS News Blog article Rotate Your SSL/TLS Certificates Now - Amazon RDS and Amazon Aurora Expire in 2024 explains the problem well. To summarize:

A DB instance created before 2024-01-25 by default uses the certificate rds-ca-2019
The rds-ca-2019 certificate expires on 2024-08-22
Since 2022, AWS lets you use certificates valid for 40 years (rds-ca-rsa2048-g1) and 100 years (rds-ca-rsa4096-g1 and rds-ca-ecc384-g1)
The issue affects plain old RDS instances and plain old RDS clusters
The issue does not affect Aurora Serverless v1
The issue does affect Aurora Serverless v2

The article isn’t clear on why Aurora Serverless v1 is an exception, so I had to read around to find out. From the documentation I learned the distinguishing features.

An Aurora Serverless v2 cluster uses the special DB instance class db.serverless
An Aurora Serverless v2 cluster, like a plain old RDS cluster, uses the provisioned engine mode
An Aurora Serverless v1 cluster uses no DB instances
An Aurora Serverless v1 cluster uses the serverless engine mode

Sources: CreateDBCluster EngineMode, Migrating to Aurora Serverless v2.

Confusingly, the DBInstanceClass documentation doesn’t list db.serverless at all.

To to find certificates that need rotation, I need to look just at the DB instances. The certificate isn’t a property of the cluster, but of the instances in the cluster.

Now that I understand the theory, I’m ready to identify the instances that need attention.

The main article shows how to do this in one account, but I have hundreds. How do I find all the problem DB instances across my organization?

Because I’ve set up AWS Config with a configuration aggregator across my organization, I have a quick way to solve this using the advanced query interface.

I read the resource schema in GitHub to discover that the AWS::RDS::DBInstance type has a property called configuration.cACertificateIdentifier. Watch out for the weird capitalization!

Use a query like this to list all the DB instances across your organization.

SELECT
  accountId,
  awsRegion,
  resourceName,
  configuration.cACertificateIdentifier
WHERE
  resourceType = 'AWS::RDS::DBInstance'
ORDER BY
  accountId,
  awsRegion,
  resourceName;

The first three columns uniquely identify and locate each instance in the in the organization. The last column is the property I need to check.

You’ll get a result set shaped like this:

accountId	awsRegion	resourceName	configuration.cACertificateIdentifier
111111111111	eu-west-1	demoinstance1	rds-ca-rsa2048-g1
222222222222	eu-central-1	demoinstance2	rds-ca-rsa2048-g1
333333333333	eu-central-1	demoinstance3	rds-ca-2019

The example result allows me to split the instance into two groups, ignore and inform.

I ignore demoinstance1 and demoinstance2. They need no more attention because they use one of the long-lived certificates.

I inform the responsible team about demoinstance3 because it uses the certificate rds-ca-2019 that is about to expire. Now they can plan a remediation.

In this article I’ve discussed the upcoming deadline to rotate a SSL/TLS certificate for RDS/Aurora DB instances. You’ve seen where to get more information to understand the problem both from the AWS News blog and from the RDS service documentation. Finally you’ve seen how to use AWS Config’s organization aggregator and advanced query feature to report on the certificate configuration across all the DB instances in the AWS organization.

Geomob Edinburgh June 2024

2024-06-29T12:00:00+00:00

Geomob Edinburgh June 2024

The Edinburgh Geomob met again on Tuesday the 25th of June 2024 at Codebase, in the shadow of Edinburgh Castle. Since the first session in March was so engaging, I had been looking forward to this one even more. Thanks always to Gala Camacho for fostering a relaxed and welcoming community.

Geospatial tech is for me not a career but a hobby. Despite (or maybe because of) that it’s the tech meetup that I get most excited about. Every visit expands my horizon of the interesting problems that are out there and how open data and tools help solve them. Some of the technical details and jargon go over my head, and that’s fine. My own niche of security and cost engineering in AWS could be just as confusing to someone not immersed in it.

Maps are as much about understanding how we relate to each other as they are about physical measurements.

Just as important as the talks themselves is the mingling. It’s all about sharing stories and tech gossip! From one attendee I learned of the Scottish Tech Army, a not-for-profit organization that brings together NGOs and volunteer techies to help the third sector improve its understanding and use of technology. He lent his skills to a project that assists newly visually impaired people with audio cues to maintain their mobility and confidence.

The rest of the post sums up my notes and thoughts sparked by the three talks.

Steven Kay kicked off the session with his explorations of Orthogonal Strip Maps.

The London Underground map is probably the most famous modern example of a schematic map. It throws out almost everything you might expect a map to show. It ignores real curves and distances between stops. But with one notched line it answers the most important traveller’s question: where can I go from here?

It’s obvious in the 21st century, but its realization took a creative leap in the 20th, and builds on the idea of an orthogonal map which goes back centuries.

The earliest orthogonal example Steven showed was a page from Paterson’s Itinerary (Volume 2, maps 113-114), published in 1785. It shows the road route from Cambridge to Ely, and optimizes the layout for a traveller marking progress from one place to the next.

It draws the route from top to bottom, split into two vertical strips to give more space to details along each half of the route. You read down from Cambridge on the left and up to Ely on the right. Most maps put north at the top. This one has a compass to show that the route runs south-south-west. It draws and names settlements close to the route, but the side roads to reach them are faint. The main route shows mile markers to estimate travel time and plan rests. It retains curves in the road and nearby rivers that may serve as navigational aids.

A more recent example is Daniel Huffman’s 2015 linearized map of Lake Michigan. He illustrates the basic idea:

See Daniel’s article for the complete map. It’s beautiful and reveals how much humans organize around water. He gets into more history of such maps, and deep into the why and the how of his own effort.

I won’t attempt to reproduce the math needed to draw Daniel’s map, because even he admits that he no longer understands it (a friend helped). Although Steven’s own equations were much simpler, he got a similar result, and it’s still pleasant to look at.

Steve’s basic approach is to take a strip of the map on each side of the route and lay them out in a straight line.

The V-shaped artifacts indicate where a right-angle turn has been flattened out, as if diagonally cutting an L shape and laying each part flat.

I imagine Daniel’s more complex equations are what avoid such artifacts in his version. Also Lake Michigan doesn’t have any sharp corners.

One of the audience asked about scaling each segment of the displayed route to the expected speed, so that you may have greater levels of detail for areas where you spend more time (slower parts would have bigger segments). The speaker thought it might be worth exploring until another in crowd countered: “Do you really want to see that much of Princes Street!?”

Apart from the visuals, the takeaway here is that it was all possible using data from OpenStreetMap (with a little manual cleaning) and open-source tools such as QGIS for the mapping and Blender for some animations.

One more miscellaneous detail: The Ordnance Survey published an open data product called OS Open Zoomstack that now competes with OpenStreetMap. I wonder which use cases it would be better fit for and how compatible it would be with OSM. There’s a whole other research topic!

Chris Fleming next gave us a retrospective of 20 years of OpenStreetMap: Why we’re still mapping.

Chris is a local legend of open mapping. Always enthusiastic, warm, and welcoming to newcomers like me, and always finding new ways to make OpenStreetMap (OSM) more useful for people.

And how useful is OSM for people in 2024? Chris suggested Strava, Instagram, Pokémon Go, and Cycle Travel. (That last one was new to me, but it looks great!)

I laughed when Chris told how one Pokémon Go player added a pond to his back garden in OSM so that he could catch more water pokémon. It’s such a popular game hack that the OSM Wiki now gives tips to do it responsibly, considering the needs of other map users.

In the early days Chris turned to Linux User Groups to find early evangelists and volunteers, because they already understood the value of open source development.

The first UK “mapping parties” brought people together to map city centers. To help divvy up the work they would split a city’s neighborhoods using a “cake diagram”. The Mapping Cake Diagram for the Manchester Mapping Party 2006 is even pleasingly circular.

Between 2007 and 2011 Yahoo allowed OpenStreetMap to trace its satellite images for maps. The OSM Wiki lists several achievements that would have been almost impossible at the time without it. The agreement continued until Yahoo shut down its mapping unit.

By time I had started contributing to OSM in 2013 through the Humanitarian OpenstreetMap Team (still active now), I recall that Bing had a similar arrangement.

By 2012 OSM had mapped 95% of Scotland’s roads. At this point Chris showed us some screenshots of Edinburgh’s city center to compare 2013 and 2024. Although 2013’s maps seem crude and sparse now, the Scottish team deserved to be proud to get that far.

I think Chris used the OSM Comparator from the Mapping Edinburgh Toolkit. Today its view of Edinburgh looks like this:

By now the map was complete enough to support analyses in other fields such as a survey of medical occupations in 1909 and 1910.

Since then the level of detail has only increased. Almost all the street addresses are numbered. In one area someone even mapped out all the plant pots and trees. At some point you have to stop and wonder how much detail is too much!

But then, it all depends on the use case. What might be mapjunk for one person may be a critical navigational aid for another. Chris like me is sighted, and so we can’t always anticipate how visually impaired people may think about the world. I may have thought that tracking the kerb heights of all pavements was makework for bored mappers. Then Chris shared the story of a blind OSM user who asked for more of that data, because it helps him to walk around new city streets more safely.

Since I discovered OSM in 2013, smartphones have taken over the world. A new generation of mobile-first mapping tools such as Every Door and Street Complete are now available in our pockets at all times to fill in the small details that you don’t get from satellite images or GPS traces. When I tried Every Door in my local area, it set up 100 “quests”, such as identifying a road surface or the denomination of a place of worship. Geodata comes from the real world, and these tools make it easier to collect while you’re out there.

The Edinburgh OSM user group meets monthly. The OSM wiki shares details of the next events.

Tristan Goss from Earthwave increased the scale with cs2eo.org, which provides intersection processing for glaciological satellite altimetry.

The polar ice is melting at a rate of 1.2 gigatons per year.

We know this because of CryoSat-2, a European Space Agency (ESA) satellite dedicated to measuring polar sea ice thickness. It launched in 2010 with a 3-year mission. 14 years later, it’s still sending 1TB per month of measurements.

Tristan’s team uses that data to build a website called cs2eo, which serves “High performance geospatial-temporal queries for Earth Observation”. It allows you search over regions of the polar ice caps within a time range to get all the relevant ice height measurements captured by the satellite.

If you sketched all the approaches the satellite makes over the poles, you would draw lots of lines in random directions getting close to a central point. Only some intersect. The cleverness that Tristan’s team brings is to correlate these “strips” of data into something coherent for analysis.

Tristan looks forward to the new mission CRISTAL (Copernicus Polar Ice and Snow Topography). When the audience asked which instruments he would like a new satellite to have, he wished for one magic satellite with colocation of all the sensors! He also admitted it may cost more than the GDP of a small country, so it depends on how far the mission budget stretches.

The generalization of this use case is point cloud analysis. Tristan would like to correlate the data with other big-scale sources where each point lies kilometers from the next. Earthwave’s Specklia service exists to enable this for more use cases.

Funding for the polar project is hard to get because so few people live there. So if you want to support Tristan’s work, move to Svalbard.

Thank you to the Hanging Bat, Diagonal and OpenCage for hosting the geobeers and geobanter.

Over a pint of Leith Pils I heard about a Discord server for the Scottish tech community. I wonder, is it the one run by the Scottish Technology Club? That Club site looks like a great source for local tech news and events. They also have a mentoring program. Either way, I need to join and find out.

If any of this sounds fun, come to the next Geomob, probably in September or October. Watch the Geomob website for updates.

Report from AWS Summit Berlin 2024

2024-05-20T00:00:00+00:00

The AWS Summit in Berlin 2024 was another great opportunity for learning and networking. As an official AWS event, of course it began with a flashy keynote to bring good news. And it was Big News.

Stefan Höchbauer, Managing Director of AWS Germany and Europe Central, announced the AWS Sovereign Cloud. It’s the EU GovCloud. It may be the foundation of the next 100 years of utility computing. I’m excited about it, even though I won’t get to work with it (only EU residents may operate it), because it allows all European governments to use the same AWS technology that its private companies do, even for the most restricted and regulated workloads.

As I was only able to attend the second day of this two-day affair, I missed the representative of the Brandenburg local government making the same announcement. I bet they are pleased with the billions that AWS will invest to make it happen!

The two-day format was my only major complaint about the Summit, as it made planning to meet with other community builders more difficult. Some attended on Wednesday, others on Thursday. But I understand why they did it like that. The venue was so full that the Community stage was tucked into the corner of a side building that was hard to find and looked like it was in the middle of being rebuilt or knocked down! I was pleased to learn at the end of the keynote that the next host city will be Hamburg, one of Europe’s largest ports, with a fine maritime mercantile tradition. An excuse to visit more of Germany!

I had hope to write more about all the talks I enjoyed, but life got in the way. So the rest of this post shares my notes and thoughts on my favorite of the day.

Sana Shah, Cloud Operations Lead at DEMICON, opened the Community Stage with GenAI Deep Dive: Build a GenAI assistant for a women-in-cloud initiative.

This was probably the best session of the day because how it changed my thinking. I was skeptical before of generative AI services and this changed my mind. Sana maintained a confident delivery to share a clear use case and a practical solution.

Sana started the AWS Women’s User Group in Berlin. Running a user group is a lot of work. If you also need to fit it around the day job, then you need all the help you can get.

Sana showed us how to configure the AWS Bedrock service to act as her personal assistant so that it would answer her questions about the user group and about the global trends in the employment gender gap. She configured the service to combine the basic language skills of a base model (I think it was Anthropic’s Claude) with information from curated documents about the user group and a report from an NGO report about the global trends (if I remember more about these documents I’ll update here).

It turns out that this is an example of retrieval-augmented generation (RAG). The “retrieval-augmented” part just means making the generated responses more useful by getting parts of the answer from a given document.

Educational too was her summary of the economics of engineering solutions using generative technology.

Prompt engineering ($): Query the base model directly and add any extra data you may need into every query.
*Retrieval-augmented generation (RAG) ($$)): Configure extra inputs (databases, documents) that the system may consult if the base model has no useful answers.
Pre-train ($$$): I’m still not sure what this is. Sana’s focus was on RAG, assuming that this stage is complete.
New model ($$$$): Make a new base model to rival Claude or ChatGPT. Very few people in the world have resources to do this.

What’s the difference between including a PDF in the prompt and making it available via RAG? Query time simplicity and consistency. My analogy in SQL: You can write an SQL query that contains all the source data using CTEs with VALUES clauses or SELECT ... UNION ALL SELECT ... segments. But if you write this more than twice you probably want to just store it in a table so that you can just reference the table directly the next time. Where the BedRock system goes further is that it will automatically consult the configured RAG sources.

I would have loved to see what the model answered to a question that fell outside the use case, such as “How is the weather today in Berlin?”. Does the base model know anything about Berlin, or anything about sunshine, or a Thursday in Spring?

Sana ended by showing that Bedrock can not only respond to questions, but it can perform actions on our behalf. A Bedrock Agent is basically a way to invoke a command using a text chat interface instead of a command line interface or a graphical user interface. So Sana instructed her agent to send an email via name and address prompting to the MC, who proudly showed off her new message thanking her for her awesome contributions!

Huge thanks to the “Förderverein AWS Community DACH” for running the Community Stage that made possible this event-within-an-event. The “Förderverein” acts as a “group of user groups” in Germany and promotes the interests of the AWS community across Germany, Switzerland, and Austria. They ran the event smoothly. I haven’t seen them all in action yet, but I believe that it’s one of the best-organized AWS user communities. There’s a lot we can learn from them.

Geomob Edinburgh March 2024

2024-03-30T21:00:00+00:00

The Edinburgh Geomob met for the first time on Tuesday the 26th of March 2024 at Codebase, just off Lothian Road. Thanks to Gala Camacho for hosting.

Of all the technical communities I’ve encountered, the geospatial data community is the most inclusive, diverse and welcoming. Maybe that’s why I keep coming back despite it being for me more of a hobby than a career. Gala’s closing remarks helped me to understand why I feel this way: “Geo comes from the real world.”

To collect or to make use of geodata you have to get off the internet, get out the house, and get into the real world. Maybe I’m reading too much into that, but it makes sense to me.

The Geomob is the first tech meetup I’ve attended in Edinburgh for almost a decade, and to get back in the mood it was just right. A relaxed half hour of mingling, remembering old faces, and snacking before talks began. 15 minutes for each speaker, enough to tell a story without overwhelming. Done after an hour with plenty of time for more chat over afterbeers.

Looking forward to the next one on Tuesday the 25th of June 2024.

The rest of this post shares my thoughts on the three talks.

Robert Coup from Koordinates presented “Kart: Practical versioning for geodata”.

Robert said few geodata projects have a robust versioning system. Maybe they use one static copy of the data for the lifetime of the project, or they make changes willy-nilly and end up with files named my_geodata_2024-03-30_edited_version3_FINAL.shp.

As a software engineer Robert is used to versioning all his program files using tools such as Git. Kart builds on top of Git versioning with features that make it easier to work with geodata. Robert didn’t give us a demo, so my understanding of Kart remains theoretical. I hope to see it in action to answer some questions.

I asked, “Why not just work directly with Git?” If you track changes to your geodata in a Git repo you’ll already be in a better place. One consideration is size and selectivity. Some of Robert’s customers store several terabytes of data and work on different sections at different times. Basic Git is clumsy for handling large files. Kart supports this use case with a spatially filtered clone.

The spatial filter sounds like a compelling feature. I wanted to know more but missed the chance to ask more details at the geobeers. Does it work like a geoindex over Git’s own index? Does it partition the source file into smaller spatial chunks?

In 2017 the Windows source code contained 3.5M files and when added to a Git repo used 300GB of space. At that scale, many Git commands would take 30 minutes up to hours and a few would never complete. So to make it usable Microsoft extended Git with a Virtual File System that avoids downloading all the data. Does Kart reuse any of that work?

Andrew Bladon from Esri presented “How mapping technology can optimize route planning for climbers”.

Andrew loves climbing mountains, less so when it’s wet and windy. It’s not just unpleasant to be outside but dangerous on such difficult terrain. It might be sunny somewhere nearby, but checking all the possible locations and local conditions is too much for one person.

After enough miserable ascents and cancelled plans, Andrew did something about it. He made a mobile app that he described as a “conditions navigator”. I’d call it a “good weather” map. It comes preloaded with Andrew’s favorite peaks, pulls in weather forecasts from OpenMeteo, and builds on Esri’s own mapping technology to apply traffic-light coloring to the peaks. Red: stay away; Yellow: only the brave; Green: perfect conditions, get up there!

I loved this talk because of how personal it was. We learned a lot about Andrew and what drove him to build this app. He told a great story and showed some beautiful pictures! In a way, this was just an excuse for a mountain climber to show off his beautiful clear-day summit photography. And it’s welcome because we have geodata to thank for it.

The audience asked whether it could be used for other outdoor activities such as camping. Andrew agreed although he’d need to reload the app with different points of interest or include a feature that allows different users to have their own points.

When I tried it for my own trip this weekend, I found the app on GitHub, and you need a lot of developer tools to get it up and running. I’d love to see this up on the Google Play Store or the Apple App Store so more people can try it.

Read Andrew’s own story at the Esri Developers blog, “Creating an application to identify climbing locations using a weather forecast API”, and see the views that make it all worthwhile.

Moral Masuoka from Beautify Earth (BE) presented “Art Walls: Mapping potential points of connection through murals”.

BE is part mural catalog, part agency for talent and real estate. The company’s mission is “street art connections”: connecting people with street art across the US and UK.

The company’s transatlantic scope is due to Moral’s personal journey from Los Angeles (LA) to Edinburgh. Each place challenges our ideas of what a city should be. While Edinburgh may beat LA for walkability, LA has a lot more art going up on its walls.

We toured Moral’s favorite murals through the BE mural map, the Jersey City Mural Arts Program (JCMAP) powered by CANVS, and Street Art Cities.

Maybe it was just a fluke, but it surprised me in LA how clean the walls next to the murals were! I compared it to central Barcelona’s Raval neighborhood, where if you don’t paint the shop shutters then some vandal is going to do it for you. So you might as well pay an artist to do it for you and make it look nice.

I made a note from Moral’s slide deck: “Seeing North Edinburgh through the eyes of the Internet”. This appears to have been an event promoted by Edinburgh Council in February 2023 hosted by the Data Civics team at the Edinburgh Futures Institute exploring North Edinburgh via Instagram, Twitter, Google Earth, street photography and community archive.

I asked how BE designs for two competing use cases, that of the graphic designer who wants the mural to look great on a digital display and that of the touring visitor who needs to find the physical original. They’re still figuring it out as they find the best way to run a sustainable business while serving the needs of both groups.

Anything that may brighten up our drab public spaces and encourage civic pride is welcome. Today the BE site shows just a few murals in UK, so I hope the venture takes off and gives us a whole-island catalog of street art to enjoy.

After the talks there was a quick word from the sponsor Geovation. It’s a sort of business incubator for geospatial startups. I think they get funding from the Scottish government and the UK Ordnance Survey. I’m grateful to them for enabling such an entertaining evening.

Finally we poured ourselves into the Hanging Bat pub to compare notes and mingle with the community over geobeers. The pub is on Lothian Road, perfectly placed to relax with a pint before getting the bus home.

Print start and end timestamps for each Bash command

2024-03-30T05:20:00+00:00

Today I learned to make Bash print start and end timestamps for tracking execution times.

To be honest the first result looks ugly, but it’s an achievement. Read on to learn how I did it.

$ lsb_release -a; sleep 3
----------------------------------------------------------------------------------------------------------------------
2024-03-31T06:07:20+02:00
----------------------------------------------------------------------------------------------------------------------
LSB Version:	core-11.1.0ubuntu2-noarch:printing-11.1.0ubuntu2-noarch:security-11.1.0ubuntu2-noarch
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.6 LTS
Release:	20.04
Codename:	focal
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-03-31T06:07:23+02:00
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
$

It’s bugged me a for while that Bash doesn’t display the times a command starts and ends.

It bugs me most when I run a command and later realize I need to track its duration for performance measurements or the execution timestamps for auditing.

I knew the variable PS1 sets the prompt string. And I knew how to embed a command so Bash would rerun it time to display the latest output. I could use that to display the end time of the previous command, but without the start time it’s useless.

While searching for something completely different in the Bash manual, I stumbled across the missing piece PS0. Its value is expanded like PS1 and displayed by interactive shells after reading a command and before the command is executed.

Now PS0 can print the start time and PS1 can print the end time. That provides automatic execution timestamp auditing and allows computing the duration.

Its discovery carried me away. I wanted to know how to draw a horizontal rule to make a clear saparation between the injected output and the command’s normal output. I googled bash draw a horizontal line for good results. Patrick DeYoreo shows how to draw a line between Bash commands. Vivek Gite shows how to repeat a character ‘n’ times in Bash, which amounts to the same thing. Below I show how to adapt both solutions as a Bash function.

Patrick’s insight is to use the COLUMNS variable to know the terminal width, or how many characters (columns) fit in one line of text, and so draw exactly enough characters. It uses a loop to print each line segment and increments a line counter to stop after printing a segment for each column.

function horizontal_line {
    local line=0
    while ((line++ < COLUMNS)); do
      printf "${1:--}"
    done
    printf '\n'
}

Vivek’s insight is to trick the printf builtin into drawing a line. The format specifier contains the line segment. The trick is %.0s which I think means “print 0 characters of the argument” and to pass a sequence of arguments as long as the terminal width. It draws the same line.

function horizontal_line {
    printf -- "${1:--}"'%.0s' $(seq $COLUMNS)
    printf '\n'
}

I prefer Patrick’s solution because it’s easier for me to understand.

My own contribution is ${1:--}. It’s the first parameter, the character to draw the line with. The default character is -. It needs braces to use the syntax :- that separates the parameter name 1 from the default.

After defining the function here’s how I set PS0and PS1:

PS0='$(horizontal_line; date --iso=seconds; horizontal_line)\n'
PS1='$(horizontal_line "!"; date --iso=seconds; horizontal_line "!")\n$ '

(Probable bug: For some reason the second horizontal_line call doesn’t put a baked-in newline and so first line of the command output fused to the prompt. The workaround is to put another literal newline at the end of the prompt.)

The next step is to write the timestamp and rule in one horizontal line to save space. Maybe the horizontal_line function needs a parameter to set the length, or a new function takes a string and a horizontal position to overwrite the line.

Download and analyze Amplify Hosting access logs

2024-03-13T13:00:00+00:00

Amplify Hosting has no ready-made analytics

AWS Amplify Hosting is good for running a static site, but it doesn’t do analytics.

That means it doesn’t offer easy insights into visitor traffic or behavior.

It does share access logs from which you can build your own analytics.

Acquiring the access logs is a complicated process, and I learned so much while figuring it out that I need to write a series of articles to explain it all.

In this first article I describe a basic solution and provide some missing documentation for the Amplify Hosting service.

In a future article I’ll present a complete solution to download all available access logs and show the queries I use to analyze my own site’s performance.

Read access logs in the console … or don’t

The short story here is you don’t use the console. The experience is so bad it almost made me give up on Amplify. Cloudflare Pages was about to win my business.

In the AWS console you go to the Amplify app, then go to the monitoring page, and then to the access logs tab.

Then you get frustrated because you realise the view is close to useless for analysis.

15 records per page? Give me 100 at least.

Only 4 columns? The full format contains over 30 fields.

It’s as if the service team ran out of time to finish this feature or they don’t want you to use it. Am I missing something?

It gets worse when you click “Edit time range”.

You can access logs for any two week period starting with the date and time provided below. If you wish to access older logs, you will need to setup a Lambda function that runs every two weeks and stores the results in S3.

Did AWS just tell me to write a distributed map reduce function in Erlang?

Download the access logs with the GenerateAccessLogs API

So what can you do? If you’re like me, you’ll try to write a Bash script to automate around the problem.

The Amplify user guide’s Monitoring page reassures us that Amplify Hosting retains all access logs for an app until you delete the app, and repeats the console’s limitations. It doesn’t refer to any API to collect the logs.

The API exists. Use GenerateAccessLogs to reproduce the console’s functionality, with all its limitations. In a future article I’ll show how to automate around it to build the log export feature you wish existed. Here I’ll show the basic solution.

You call the API with your app ID, your domain name, and the access time range in ISO format.

aws amplify generate-access-logs \
--app-id dddamplifyapp1 \
--domain-name example.org \
--start-time '2024-03-09T00:00:00' \
--end-time '2024-03-10T00:00:00'

Amplify Hosting responds with a signed URL to an object in an S3 bucket. I redacted the sensitive values in the signed URL’s common parameters.

{
    "logUrl": "https://
               aws-amplify-prod-eu-west-1-access-logs
               .s3.eu-west-1.amazonaws.com
               /dddamplifyapp1
               /example.org
               /Sat%20Mar%2009%2022%3A41%3A50%20UTC%202024
               ?X-Amz-Security-Token=...
               &X-Amz-Algorithm=AWS4-HMAC-SHA256
               &X-Amz-Date=20240313T213714Z
               &X-Amz-SignedHeaders=host
               &X-Amz-Expires=3599
               &X-Amz-Credential=ASIA...%2Feu-west-1%2Fs3%2Faws4_request
               &X-Amz-Signature=..."
}

The Amplify Hosting service owns the log file bucket. It appears to generate the object on demand.

You have an hour to request the URL before the credentials expire. If you request within that time, S3 responds with a chunk of the access log. Keep reading to see how to request it with curl.

The Amplify Hosting log format has quirks

Amplify Hosting is a layer above a range of AWS services including CloudFront, AWS’s content distribution network (CDN) that serves the site content.

Amplify Hosting creates a CloudFront distribution dedicated to your app. You have no direct access to the distribution because Amplify Hosting hides it from you. CloudFront provides Amplify Hosting with its access logs.

But the format that Amplify Hosting shares via the GenerateAccessLogs API isn’t identical to CloudFront’s standard log format. Amplify Hosting has small differences.

The key format is <app_id>/<domain>/<timestamp>.

After URL-decoding the example timestamp is Sat Mar 09 22:41:50 UTC 2024. That’s a weird format. Why is the year apart from the month and date? Why not use the ISO format?

The timestamp marks the first time you request the chunk. Days later if you request the same chunk you get the same key.

The log file format contains not tab-separated but comma-separated values (CSV).

Despite those differences I’m able to analyze the logs with any tool that reads the CSV format.

The S3Stat and GoAccess tools promise ready-made analytics for CloudFront logs. I haven’t yet tested these so I don’t know whether they handle Amplify’s variant.

Now you can automate the log collection

To fetch just one chunk of the logs you complete three main steps:

Make the GenerateAccessLogs request
Extract the logUrl value from the response
Download the log file at the URL

You can automate all these steps in Bash with a command like this:

curl "$(
  aws amplify generate-access-logs \
  --app-id dddamplifyapp1 \
  --domain-name example.org \
  --start-time '2024-03-08T00:00:00' \
  --end-time '2024-03-09T00:00:00' \
  | jq -r '.logUrl'
)" \
> logs.csv

First the aws command runs. This is the same as before.

| jq -r '.logUrl' passes aws’s output through the JSON processor jq to extract the logUrl value from the JSON response. The -r switch writes the “raw” value, without the quotes that normally surround a JSON string.

"$(...)"' is a command substitution. The output I just described is collected into a Bash string and used as a parameter for what comes next as if it were a literal.

So you can imagine that at this point the command becomes the much simpler:

curl "https://..." > logs.csv

curl requests the content at the URL and just writes it out.

> logs.csv saves the content to a log file with the standard CSV file extension.

Now you just have a CSV file, so you can use your favorite tool to analyze these.

My current favorite tool is DuckDB. My love for DuckDB deserves its own article. Here I’ll just show the commands to do the simplest analysis.

Read the CSV into a SQL table and start the DuckDB CLI.

duckdb -cmd 'CREATE TABLE logs AS SELECT * FROM logs.csv;'

In SQL, count the requests in the access log.

SELECT COUNT(*) AS request_count FROM logs;

Avoid common errors from GenerateAccessLogs

The API has no default date range. If you omit one, it gives an error.

aws amplify generate-access-logs \
--app-id dddamplifyapp1 \
--domain-name example.org

An error occurred (BadRequestException) when calling the GenerateAccessLogs operation: Unable to complete request for the given time range, reduce time range and try again

The API fails if you request more than two weeks. Here I request one month.

aws amplify generate-access-logs \
--app-id dddamplifyapp1 \
--domain-name example.org \
--start-time '2024-01-01' \
--end-time '2024-02-01'

An error occurred (BadRequestException) when calling the GenerateAccessLogs operation: Time range must be no greater than 2 weeks

You’re lucky if you can get a full two-week response. The API may fail with what looks like a timeout. The error text is the same as when you omit the time range.

aws amplify generate-access-logs \
--app-id dddamplify1 \
--domain-name example.org \
--start-time '2024-01-01' \
--end-time '2024-01-15'

An error occurred (BadRequestException) when calling the GenerateAccessLogs operation: Unable to complete request for the given time range, reduce time range and try again

I find it’s more reliable to use a range of just one week.

That means a single chunk contains even less data. That’s why you’ll want to read the next article in the series, where I show you how to download all the chunks!

AWS can do better here

I don’t understand why Amplify Hosting puts that straitjacket over the CloudFront logs.

Please, AWS, consider making the console experience more like that of AWS Config Advanced Query or even Athena. Those services show all the available columns and more than 15 rows.

Even better, consider making the CloudFront access logs directly accessible so I don’t need to spend my evenings and weekends figuring out a custom solution for this! Even if I had to pay for the storage or the requests, it would be a preferable solution, because I could go straight to Athena to derive insights from my data.

Conclusion

In this article I explained showed you how to download and analyze Amplify Hosting access logs.

I started by explaining that Amplify Hosting provides no analytics features. I showed how it’s impractical to use the console to read the access logs and, how it’s difficult to download more than two weeks of log data.

I introduced the GenerateAccessLogs API that allows you to collect the access logs using the CLI, explained the format of the response, and explained the format of the log file and how it varies from the CloudFront standard log file format.

I showed how to use Bash to automate the collection of a single chunk of the access logs and how to analyze the result using DuckDB. I shows some common errors and what you need to do to avoid them.

Before I finished, I shared my opinion on how AWS can improve this experience from the Amplify Hosting service.

Finally I promised in the next article to show you how to go beyond the basic solution to download all the access logs.

Read Denis Gladkikh’s solution

After I started writing this, I found Denis Gladkikh’s blog series on Analyzing AWS Amplify Access Logs. It identifies the same problem with Amplify, proposes a similar solution, gives a tour of the different analytics tools you can use on the logs. I recommend you read these too!

The blog is back

2024-03-12T21:00:00+00:00

The blog is back! 2024 is the year that I take my writing more seriously so that I can contribute more effectively to the community.

I love writing code and hacking on open source projects. But code can be a bit impenetrable for the people unfamiliar with the problem it solves.

So this year I want to spend more time writing articles to support and explain the open source work that I do and to highlight interesting problems that I solve at work.

Just today someone contacted me to ask for advice about AppSync and DynamoDB. It’s not my speciality, so I recommended other AWS Community Builders to contact to ask for advice based on what they have written about those topics:

Benoît Bouré has a whole AppSync series on his blog
Johannes Koch published a tutorial to build a serverless app using AppSync and DynamoDB
Seth Orell writes about why his first pick for a database is DynamoDB
Robert J Berger’s Omnyway stack uses Amplify

Although I’ve not worked directly with any of them yet, I was able to recommend them based on their writing. I want people to be able to do the same for me! So I need to start writing again so that people can see what problems I like to work on and what I can help them with.

Why haven’t I published more? I take a lot of technical notes at work, notes that I could adapt into a lot of technical articles. But it would take time to generalize from confidential customer data and use cases or redact other things I cannot share publicly.

10 years ago I was single and more energetic and so I had time explore miscellaneous topics on my old blog. My favorite article from my previous blogging life is Now I’m married with dogs and so I have to manage my time more carefully.

This year I hope to involve my customers in the learning-in-public process and work with them to write about the problems we solve in a way that they are proud of sharing online, either on my own blog or on their company blog if they have one.

Another reason that seems more silly but did psychologically block me was the difficulty of knowing who reads what I write. My block has no comments, and the Amplify Hosting backend I use has no built-in analytics.

Amplify’s missing analytics almost made me tear up the architecture and start again. But last weekend I hacked together a script that downloads the access logs and uses SQL to find the most requested pages, the top referrers, and other basic insights.

That analytics solution for Amplify Hosting will be the topic for the next article!

I’ll add comments too, using utteranc.es, a comment manager backed by GitHub issues. I using GitHub to host my code and manage my projects, so I’ll try it for comments too.

Hope to see you round here more often!

How to install binaries from GitHub

2023-09-11T00:00:00+00:00

How to install binaries from GitHub

Ways to install binaries from GitHub Releases:

Ways to release binaries to GitHub Releases:

actions/create-release (Archived, but references active projects)
softprops/action-gh-release