Ian Butler

Tech Lead, Senior Engineer II

Writing code for over 20 years Ian has worked on everything from API development to data engineering and currently he is a Tech Lead for the Data Intelligence team. Ian currently does a lot of work with distributed systems, search, data processing and analytics. His current passion project is writing a full text distributed search engine using existing databases like Postgres and MySQL as the data store.

Talk:
High Throughput Data Processing with GenStage

Level:

Data processing is a part of many disciplines in technology and as the amount of data you need to process grows so do the techniques you need to be able to safely and speedily ingest that data. Join us as we go through the building blocks Elixir offers to build different ingestion topologies that handle things like backpressure, partitioning, retries, and batching.

Talk objectives:

  • Make listeners aware of tools like GenStage
  • Show how GenStage can be used to build high throughput data processing systems with useful features
  • Educate why Elixir is uniquely suited for this type of data processing system
  • Provide examples of real-world use cases where this has been done (including one’s I’ve built)

Target audience:

  • Engineers working at companies who need fast, effortlessly scalable, online data processing systems that can handle pretty large throughput with reasonable resource usage.
  • Data Engineers looking to dip into Elixir.