Chapter 1: Prototyping and Environments
Chapter Goal: Create a prototype script to read a single sensor value
No of pages 25
Sub -Topics
1 Introduce the example of collating data from a large number of other
machines
1.1 Possible usecases of this pattern include log aggregation, server
monitoring, IoT, monitoring of customer servers, etc
1.2 We’ll use raspberry pis with a mix of server monitoring and
very basic sensors like temperature sensors. There’ll be no IoT
specific setup or detail, it’s just that this is easier for people to
follow along with without inventing another system being
monitored.
1.3 There’ll be plenty of extra context here for how to apply the
ongoing example if you do already have a system that needs
monitoring.
2 Prototyping using jupyter and nbconvert
3 Use pipenv to set up dependency environment
3.1 Note that by introducing pipenv before setuptools we’re
preempting the confusion about the right way to do dependency
and environment management.
Chapter 2: Testing, Checking and Linting
Chapter Goal: Progress the prototype to a series of reliable functions that
can be tested
No of pages: 30
Sub - Topics
1 Testing with PyTest (especially fixtures and MUT style)
2 Type hinting and checking with mypy
3 Linting with flake8 and autoformatting with black
4 pre-commit and commit hooks
5 GitHub CI integration for easier contributions
Chapter 3: Packaging Scripts
Chapter Goal: Create an installable package that gives a single script to
read the sensor value
No of pages : 30
Sub - Topics:
1 setup.py and setuptools when it comes to packaging (not pip /
setup.py for environment management, that’s in chapter 1)
2 Namespace packages
3 Console entrypoint
4 argparse
Helpful aside: Package name conflicts, installing from GitHub releases,
release hashing, wheels
Chapter 4: From Script to Library
Chapter Goal: Extend the package to allow reading of multiple sensors
through the command line
No of pages: 20
Sub - Topics:
1. Abstract Base Classes
2. Second sensor value
3. argparse subcommands
Chapter 5: Alternative Interfaces
Chapter Goal: Make the script functionality available as a HTTP
microservice
No of pages : 40
Sub - Topics:
1 Simple API servers using flask
2 Plugin architecture using entrypoints
3 Dynamic dispatch
4 Serialisation considerations with custom classes (like units from pints
package)
Chapter 6: Speeding Things Up
Chapter Goal: Discuss optimisation strategies, what the tradeoffs between
async and different types of caching are. We’ll use caching
here, but async later
No of pages : 25
Sub - Topics:
1 asyncio vs lru_cache vs redis vs sqlite etc
2 Use of timeit
3 File operations using context managers
Chapter 7: Aggregation Process
Chapter Goal: Create a new package, read configuration files, do a basic
HTTP loop
No of pages : 25
Sub - Topics:
1 cookiecutter
2 Config files (configparser vs json vs yaml)
3 Requests library
4 More depth in pytest usage
Chapter 8: Asynchronous Programming
Chapter Goal: Understand the event loop, especially async for loops,
demonstrate how it’s a good fit for the aggregation process
No of pages : 40
Sub - Topics:
1 Defining asynchronous functions
2 Using the event loop
3 Syntactic sugar for loops and iterators
4 Async tasks vs await
5 async executors
Chapter 9: Asynchronous Databases
Chapter Goal: Understand async executors, using sqlalchemy and JSONB
No of pages : 30
Sub - Topics:
1 sqlalchemy (and why pandas isn’t a good fit here)
2 JSONB format and schemaless
3 aiofile, asyncpg and usability/speed tradeoffs
Chapter 10: Viewing the Data
Chapter Goal: Creating Jupyter notebooks and using matplotlib
No of pages : 35
Sub - Topics:
1 Calling async functions from Jupyter Notebooks
2 Binding function calls to ipywidgets for interactive reports
3 Examples of matplotlib
4 GeoJSON
Chapter 11: Fault Tolerance
Chapter Goal: Extending ABC interfaces and efficient use of iterables for
large HTTP responses
No of pages : 20
Sub - Topics:
1 Using __subclasshook__ effectively
2 Chunked responses vs framing
3 JSON deserialisation of partial data and efficient data transfer
Chapter 12: Callbacks and Data Analysis
Chapter Goal: Using generators, iterators and coroutines for data
analysis, async timeouts
No of pages : 30
Sub - Topics:
1 Iterator based filtering
2 Coroutine based plugins, for example a coroutine that pulls historical
data and compares it to the current value to decide if an alarm should
be raised
3 waitfor and executor timeout considerations