SDK v2.3

SDK v2.3 was just tagged (api-v2.3.0), and with it come some new safety features as well as helpers for USB and USB-MIDI:

Exceptions (out of memory)

tl:dr: Use try/catch around all large memory allocations.

Plugins can throw and catch exceptions across the plugin/host boundary now.
If your plugin is allocating memory, and the allocation fails because we’re out of memory, your plugin will get a std::bad_alloc exception which you can (and should) catch and respond to gracefully.

Example:

std::vector<float> sample_data;

try {
    sample_data.resize(sample_size); 
    sample_loaded = true;
} catch (std::bad_alloc &) {
    Gui::notify_user("Could not allocate memory for the sample. Choose a smaller file.", 2000);
    sample_loaded = false;
}

If you do it like above, then if the MetaModule runs out of memory, your module will just pop up a notification to the user, and the patch will continue being played. No memory is leaked, nothing crashes.

On the other hand, if you don’t catch failed allocations, like this:

std::vector<float> sample_data;

sample_data.resize(sample_size); 
sample_loaded = true; // BUG: need to check if allocation succeeded

In the above example, if there’s not enough memory to resize sample_data, then the patch will be stopped, the user will get a generic “The module crashed” notification, and then different things happen depending on the context:

  • If the module is already in a playing patch, the module will be in a possibly corrupted state (future firmware may try to remove the module or the plugin)
  • If the failed allocation happened in the module’s constructor when the user attempted to add the module to the patch or open a patch with that module in it, then the module will not be added. Any dynamic memory already allocated by the module will be leaked.
  • If the failed allocation happened in plugin init(), then the plugin will be removed.

Clearly, the first example with try/catch is much better because memory will not be leaked, and your module can take alternative actions or at least inform the user what’s going on. In the second example case, there’s a chance that memory was leaked, and a chance the module is in a corrupted state and will crash if the user tries again.

USB and USB-MIDI Queries

Added USB query functions (see System API):

  • System::get_usb_connection_status()
  • System::get_usb_device_name()
  • System::get_usb_midi_rx_cable() and System::get_usb_midi_tx_cable(): look
    up a MIDI port of the attached device by cable number, so a module can
    list the ports by name, filter incoming messages based on cable number, and/or
    transmit messages over a particular cable.
  • System::get_usb_midi_in_jack_info() and System::get_usb_midi_out_jack_info():
    low-level information from the device’s descriptors.

Example for filtering MIDI RX by a cable name:

auto status = System::get_usb_connection_status();

for (unsigned c = 0; c < status.num_midi_rx_cables; c++) {
    auto cable = System::get_usb_midi_rx_cable(c);
    if (cable.name.contains("Kontrol DAW")) {
        my_port = c;
    }
}

if (auto msg = midi.pop_message()) {
    if (msg->usb_hdr.cable_num == my_port)
        process(*msg);
}

Fixed a non-working MIDI classes:

  • Fixed MidiInput and MidiOutput API: some headers were missing, making them
    unusable in v2.2.0 and v2.2.1
  • Added docs for MIDI classes: docs/midi.md

Test Plugins

In the SDK, in the test/ folder are some example plugins that are meant to test and demonstrate usage. Feel free to poke around or run them on hardware.

  • Implemented test plugins:
    • tests/exceptions-test tests OOM/bad_alloc and more
    • tests/usb-info-test` dumps everything the USB query API reports to the console.
    • tests/midi-test converts incoming MIDI to gate/pitch CV and gate/CV
      back to outgoing Note On/Off, with an LED to verify MIDI rx unpatched. Also
      demonstrates filtering based on cable number.
2 Likes

great stuff, this is going to be so useful!

I also saw on the firmware update…

  • Console device: Can function as a CDC console for debugging or reading logs

could you give us a little more info on this?
does this mean we could add some trace info to plugins, it could be really useful during plugin development.

a bit OT, but related to how I dev plugins…

perhaps my ‘biggest’ (exaggerated, first world) workflow issue, is the transfer of new modules.
so on other systems… as they are linux based, its
a) compile on dev machine
b) scp module to device
c) restart host on device (which auto loads last patch)
d) I can see console log, as I have an ssh on device.

ok, so I think console logs gives me (d)

b/c iirc, officially (due to web if limitations), I have to remove sccard, copy, reinsert, restart mm.

I think unofficially,
I can use the web interface to copy, then restart mm
hmm, thinking about it , perhaps, I can use curl to copy plugin file? not tried that!

do you think theres a way to streamline this?
either already, or with a change?
perhaps a ‘reboot’ command (to reload force module reload etc)

Im not too worried, about 100% correct e.g. due to data rate limitations, if it works 90% time thats fine… more about wear n’ tear, but constant power cycling or rack, and sd card remove etc. -
and ofc, just a quick workflow, that I can do from the command line :wink:

ofc, i understand I can use the simulator, but honestly, most of the time I find dev on vcv desktop, then just doing ‘checks’ on the mm device is enough.

Ah, it’s in the firmware but not user-selectable :frowning:
I have some big improvements to the console logging in the a dev branch, making it async so you can call a simple printf() inside an audio hot loop (like printf("Hit threshold\n") not so much printf("%s\n", big_string.c_str()))

Here it is:
https://github.com/4ms/metamodule/releases/download/firmware-v2.3.0-console/metamodule-firmware-v2.3.0-console-firmware-assets.zip

it’s the same as v2.3.0 official but with that option available in the preferences. Note, you can’t use USB MIDI at the same time as the USB console. Not yet anyhow…

might work? I don’t see why not if it’s a small file. hmmm…

I wonder if we can get something working over the USB console. at 115200 baud … err a 256kB plugin would take ~22 seconds to transfer. Not great but if it’s only 64kB, it’s fairly fast. Maybe we can increase the baud rate, I don’t actually know, I haven’t played with the USB console a lot.

There could be a console command to reload the plugin that was just sent. Right now I have just one console command “c” enables color (different colors for each core) and “m” (for mono) turns it off.

I agree it could be done faster. I end up doing most module dev in firmware, so it’s a 5-6 second firmware reload via JTAG when I make changes.

1 Like

that’s not an issue, if you’d prefer to release a ‘dev/debug firmware’ alongside normal firmware, so that it can prevent debug code being in release version thats fine.

though convention, usually makes this an onus on the developer of the 3rd party software, and may be easier on your side to not have to release different firmware.

thats cool, I don’t need much, Im used to formatting by own data if/when required, I just end up with my own small logging functions which get noop’d for release.
similarly, its very rare to put stuff in audio loop…and I can always, get around this by redirecting it to UI thread…

I guess the main thing is, I found it a bit of a faff to draw onto the UI for debugging, and its a small screen so there you can’t really print much to it either…
but my ‘needs’ are pretty limited :wink:

limitations are fine, I can fall back to other strategies in those cases.

There could be a console command…
yeah, anything thats easy to expose like reload would be really useful…
perhaps the console logging could be a command too?
ie. keep dev preferences out of the UI, as they aren’t really needed there?

~22 seconds to transfer.
yeah, thats fine… thats still faster than removing sdcard, putting in a reader, copying, then putting back into mm :wink:
… and importantly, less wear n’ tear and faffing…

JTAG, yeah, Im just not keen on having this hanging out of my rack,
its a bit overkill for module dev in my mind (as I can test most functionality on desktop)
for sure, I do understand for ‘full on dev’ of firmware where attaching a debugger is useful, its the way to go.

Thinking more about this, I’m not sure why it didn’t occur to me earlier: I used to have a MSC device mode in firmware where attaching it to a computer would display the internal file system (which is where plugins live)

Maybe there can be a dev mode where you can have a USB cable attached to the computer for console use, then you enable “dev drive” mode (press a certain key in the console?) and a drive shows up on your desktop. Then just copy the .mmplugin to the drive like a normal file. The drive is in RAM, so it’ll be fast to copy. Unmounting the drive exits the mode and loads the .mmplugin (unloading a matching brand slug if it’s already loaded). While “dev drive” mode is enabled, the firmware can block GUI access to the internal virtual drive so we don’t have any multiple writer issues.

3 Likes

I have this working. Not extensively tested yet. This is the PR:

Here’s how it works: There’s a toggle in preferences for Developer Mode. When this is on, and you plug into a computer, it allocates an 8MB FATFS virtual drive that the computer should see and automatically mount. You also get the the console and a MIDI device (it’s a composite USB device class).

The basic workflow is to copy the .mmplugin file to the FATFS drive and then eject it. The MetaModule sees the eject event, scans it for .mmplugin files, and tries to install everything it finds. If an already-installed plugin has the same name, then it uninstalls that one first.
Once it’s done, it removes the .mmplugin files from the drive and re-mounts it (or at least, toggles the USB device off and on again which should make the computer “see” it as a unplug/plug and re-mount the drive).

There’s bound to be issues with how different OS’s deal with Eject vs. Unmount events, so using the console is the more robust workflow (and it’s scriptable):slight_smile:
There’s not really a true CLI prompt in the console yet but it does listen to what you type and will respond to some commands now (case-insensitive):

For instance, typing “help” and pressing enter gives:

Commands:
  install  install any .mmplugin files on the developer drive,
           then restart the USB MSC device service
  eject    stop the USB MSC device service
  mount    start the USB MSC device service
  status   report the developer drive state
  col      console color on
  mono     console color off (default)

The basic console-based workflow to update a plugin would be:

# MacOS:
diskutil mount DEV_MM 
cp plugin.mmplugin /Volumes/DEV_MM
diskutil unmount /Volumes/DEV_MM 
echo "install" > /dev/console-device

# Linux:
mount -L "DEV_MM"
cp plugin.mmplugin /mount/point
umount /mount/point 
echo "install" > /dev/console-device

There’s also a nice script that does this automatically and prints out meaningful error messages and retries a reasonable number of times, etc. But it just boils down to the same commands: mount, copy, unmount, send install command. The mount isn’t often required, but it doesn’t hurt.

Let me know what you think of that workflow, if it would work for you or if you see anything to improve (or just want to try it and see!)

2 Likes

Oh nice! it was already getting a bit tedious to go through the routine of: take sd-card, put it in adapter, plug into pc, copy file, eject, plug into MM, unload plugin, rescan plugins, load plugin, load patch - ad infinitum.

This should at least reduce the number of steps quite a bit, but I can imagine that the stability/UX can potentially be a bit flakey.

The workflow seems pretty doable to me. The automatic mount/unmount saves replugging at least, so I think it’s a sensible approach (and it’s only for dev-mode anyway).

Am happy to give it a try when you think it’s ready.

sounds great! cool that its a composite device too !

does the dev mode also enable the console logging? (as discussed above)