Structured Multi-Track Accompaniment Arrangement
via Style Prior Modelling


Authors: Jingwei Zhao, Gus Xia, Ziyu Wang, and Ye Wang

Abstract: In the realm of music AI, arranging rich and structured multi-track accompaniments from a simple lead sheet presents significant challenges. Such challenges include maintaining track cohesion, ensuring long-term coherence, and optimizing computational efficiency. In this paper, we introduce a novel system that leverages prior modelling over disentangled style factors to address these challenges. Our method presents a two-stage process: initially, a piano arrangement is derived from the lead sheet by retrieving piano texture styles; subsequently, a multi-track orchestration is generated by infusing orchestral function styles into the piano arrangement. Our key design is the use of vector quantization and a unique multistream Transformer to model the long-term flow of the orchestration style, which enables flexible, controllable, and structured music generation. Experiments show that by factorizing the arrangement task into interpretable sub-stages, our approach enhances generative capacity while improving efficiency. Additionally, our system supports a variety of music genres and provides style control at different composition hierarchies. We further show that our system achieves superior coherence, structure, and overall arrangement quality compared to baselines.

Our structured arrangement paper is accepted by NeurIPS 2024. For more technical details, see our paper, GitHub repo, and Colab tutorial.


Accompaniment Arrangement Demo

Our system specializes in multi-track accompaniment arrangement for whole pieces of music. Given an input lead sheet, it first establishes the rough whole-song structure with a piano accompaniment and then orchestrates the piano into a complete multi-track arrangement. Besides supporting customizable (controllable) band instrumentation, this two-stage approach aligns with musicians' creative workflow. Our demos are presented with three tabs as follows: 1) the multi-track arrangement, 2) the piano arrangement, and 3) the input lead sheet.

[Demo-01] Can You Feel the Love Tonight

[Demo-02] All I Want for Christmas is You

[Demo-03] Over the Rainbow

[Demo-04] Boogie Woogie Bugle Boy

[Demo-05] Always True to You in my Fashion


Orchestration Demo

The second stage of our system can also work individually as an orchestrator. Given a piano accompaniment score, it specializes in piano-to-multi-track arrangement for whole pieces of music, where customizable (controllable) band instrumentation is supported. Our orchestration demos are presented with two tabs as follows: 1) the multi-track arrangement, and 2) the input piano score.

[Demo-06] The Moon Represents My Heart

[Demo-07] The Smurfs Theme

[Demo-08] Toki no Nagare ni Mi o Makase

[Demo-09] Suddenly Miss You so Bad

[Demo-10] Floral Sea


More Demos


This site leverages the jekyll template and uses html-midi-player for MIDI visiualization and Kontakt Factory Library 2/BandLab for MIDI-to-audio synthesis.