Mapper with varying schema




How do I handle ingesting logs/messages from different IoT devices emitting in different record layout?

Is there a design pattern or language framework that allows me to handle it so that 1) development is relatively easy 2) I can add more devices in future without major change.

I am ingesting logs on DynamoDB so yes I have flexible schema that I can persist but logs I am getting are in totally different schema.

I was thinking about writing mappers but that solution doesn’t scale well. I will have to manage multiple mappers.

I do have a choice to move to gcp if that has better options but I am looking for high level framework

Cross posting this to r/aws r/python r/googlecloud and r/java

submitted by /u/tempread1
[link] [comments]

Original article: Mapper with varying schema
Author: /u/tempread1