How do I handle ingesting logs/messages from different IoT devices emitting in different record layout?
Is there a design pattern or language framework that allows me to handle it so that 1) development is relatively easy 2) I can add more devices in future without major change.
I am ingesting logs on DynamoDB so yes I have flexible schema that I can persist but logs I am getting are in totally different schema.
I was thinking about writing mappers but that solution doesn’t scale well. I will have to manage multiple mappers.
I do have a choice to move to gcp if that has better options but I am looking for high level framework
Cross posting this to r/aws r/python r/googlecloud and r/java
submitted by /u/tempread1
Original article: Mapper with varying schema