Heads up! I've decided to self-manage my blog. I'm leaving this post live as to not "break the internet" but feel free check out the post on my new blog here! blog.ericyd.com/loop-supreme-part-6-workers..
This is part 6 in a series about building a browser-based live looper
Part 6: Workers and AudioWorklets
Move away from using
setInterval on the main thread. Attempt to use a Worker to keep time, so interruptions on the main thread don't cause timing delays.
There were a bunch of changes in this iteration.
The first thing I did was added a "Start" component, which is really just a button that gets access to the user's media devices, and initializes the
AudioContext. I was getting really frustrated because
AudioContexts are not supposed to be instantiated until a user performs an action in the app. But wrapping the
AudioContext in a piece of state or a ref was causing all sorts of annoying issues. I wrote up some more justification for this choice in the code. I'll explore options to remove this in the future since it's kind of annoying, but for now this solved a lot of problems. One unexpected side effect was that I was able to remove almost all the custom functionality in
AudioRouter. I realized that most of the methods in that context were written to avoid annoying null checks on the
AudioContext interface. Since the
AudioContext was not guaranteed to be non-null, it removed the need for most of this context's functionality. In fact, I may remove the context entirely at a later point and just pass the
MediaStream props directly, since the component tree is fairly small in this app.
I also decided to change the
MetronomeProvider to a standard component, and pass it's props directly. Since I don't anticipate needing a deep component tree, this simplified some implementation and made it a bit more clear to follow the data flow.
By far the most significant refactor in this update was moving the clock and recorder to a
AudioWorkletProcessor, respectively. Previously I was using
setInterval on the main thread, along with a
MediaRecorder that updated some component state to store the buffer of recorded audio. This was not working well; I experienced lots of dropped samples and really bad timing differences, beyond normal latency issues. After doing some research, it seemed that moving the audio processing and time keeping off the main thread was the way to go. It is possible that I will need to revisit this yet again; online resources indicate that using the built-in clock on the
AudioContext is the most accurate way to schedule audio events. This makes sense to me. However, the pitfall for Loop Supreme is that we need to know when a loop is starting and stopping. There is no way (that I'm aware of) to pre-schedule dispatched events / messages through a
AudioWorklet, to notify other components to begin or stop recording. Of course, it may end up that the recorder and clock can live in the same
Worker, and the recording can be initiated by a message. This would potentially solve all my problems, but I think there are more important things to figure out before I go down that route.
Overall, I'm really pleased with the current progress. I still have some latency issues to figure out and some fine tuning, but overall things are working pretty smoothly.
setIntervalis not available in
AudioWorkletProcessor. Originally I was going to use
AudioWorkletProcessorsfor everything, but since I am trying to use
setInterval, I needed to use a standard
Worker. This ended up being OK because registering an
AudioWorkletProcessoris async whereas registering a
Workeris sync, which is a little easier to use in a single component.
Workers / AudioWorklets with TS/React is a bit of a pain. It turns out it is possible to use TS with a Worker, and it works great in the Webpack dev server. However, building the app seems to run into an issue; I believe I'll need to eject the CRA bootstrap to get access to the webpack config so I can configure a custom resolve hook.
Workers and AudioWorklets are really cool! Loop Supreme already relied pretty heavily on an event-based pattern to coordinate beats and loops, and listening to events from the worker thread is a really natural way to achieve this, with better performance too! This eliminated the need for a custom EventTarget implementation
State of the app
From a user perspective, not much has changed. Some styling has been updated, but the basic click-track and recording track is identical.
Recording does work and the timing seems better, though there is still pretty significant latency between recording and playback. I'll need to see how to fix this so "playing in time" is possible
- I decided to stop time logging because this took a lot of time and research. My previous time logs were very off the cuff, so when I lost track of total time spent I decided it wasn't worth it any more. Suffice to say - this one took pretty long 😅