Welcome! I am Tongyu Lu! 我是路通宇!

About Me

• Currently working at Singapore University of Technology and Design (SUTD) as a Ph.D student at ISTD
• Graduated from ETH Zürich as a master student in Electronic Engineering and Information Technology program
• I am interested in music informatics, including automatic music transcription, music generation and computational musicology
• I am an amateur composer

My photo in Sapporo

Projects

• MelodySim, a similar-music dataset augmented from Slakh2100 dataset, along with a triplet neural network that fits well on this dataset. (Github|Paper)

• DiscoDiff, a coarse-to-fine latent diffusion model architecture that generate coarse RVQ code as the first step and predicts the fine RVQ codes as the final step. (Github|Paper)

• A piano hammer-string-soundboard physics-based model simulation. (Github)

• MIDI melody track identification (Github)

• Note-level automatic music transcription , a model-based AMT pipeline which has high performance on transcribing linear instruments. (AMT) by Single-chord Fourier Approximation (Github)

• Jazzify, an open-source rule-based computational musicology system, which is able to detect chord symbols and keys given chord sequence, and generate jazz-style comping. (Github)

Music Work Selections

I heve kept composing since 2017, and I started learning Jazz since 2021. My works are mostly Jazz (all styles including swing, ballad, latin, funk etc.) and Classical (chamber music, piano solos, orchistral works), along with some adaptations. Here are some selections. You can find my music collections in my YouTube Channel, Bilibili Personal Page, SoundCloud Personal Page or Netease Music (网易云音乐) Personal Page

• Waltz Unfrozen, a jazz waltz; Links Youtube | Bilibili

• Overload Mode, a fusion funk; Links Youtube | Bilibili
  Lu Tongyu · Overload Mode

• Sonata in e-flat minor, a piano Sonata in funk style; Links Youtube | Bilibili
  Lu Tongyu · Sonata in e-flat minor

• Jazz Trio: Cold Clean Autumn; Links: Youtube | Bilibili
  Lu Tongyu · Cold Clean Autumn

• Piano Work: Etude in F Major - Ripple; Links: Youtube | Bilibili
  Lu Tongyu · Etude in F Major - Ripple

Harmonic Constraint in Music Spectrogram Reconstruction

It is assumed that readers have known the concepts about harmonic (overtone) structure of musical notes. And readers are recommended to know the non-negative matrix factorization (NMF) algorithm. But anyway, I tried to briefly elaborate it in this article.

This article gives a brief review of harmonic constraints on timbre dictionary. Such harmonic constraints are useful in helping non-negative matrix factorization (NMF) algorithm to converge into a musically meaningful result, which could enable automatic music transcription (AMT) and music source separation (MSS). Noticing the disadvantages of the traditional NMF algorithm and hard-harmonic-constraints, this article proposes a dB-trick for NMF which could loosen the non-negativity constraints, and a soft-harmonic-constraint method based on regularization which gives more freedom to the parameter compared with the existing hard-harmonic-constraints.

more >>

A Startup for Physics-based Piano Modeling (Theory and Simulation)

Pre-requisites:

1. Knowing basic approaches to partial differential equations (PDE)
2. Knowing discretization and $z$-transform.

I have been studying the physics-based modeling for accousitc piano since I was in ETH (around 2022). When I am in SUTD, I took a course “Science of Sound” taught by Prof. Chen Jer-Ming, and refined the simulation myself. However, when I happen to try out Pianoteq, I realized that it is no more necessary for me to push forward this project alone.

But anyway, I will share my piano modeling note (mainly the basic setups of piano hammer-string-soundboard model) to everyone that is interested.

more >>

Backpropagation - More than Chain Rule

Pre-requisites:

1. Knowing multi-variable differentiation;
2. It is recommended that readers have known the concepts of directed graphs.

This is my lecture note of the natural language processing (NLP) lecture in ETH Zurich. Lecturer: Ryan Cotterell. This lecture tells you what backpropagation algorithm really is.

more >>

Neural Network Theory Lecture Note 1 - Approximation Theorem

Pre-requisites:

1. It will be helpful to know some basic concept about functional analysis, e.g. linear functional, normed space, Hahn-Banach Theorem; anyway they will be explained in my note
2. Basic form of single-layer neural network.

After the happy summer vacation, I have to attend the master program at ETH Zurich. This is my lecture note for my first course, Neural Network Theory, at ETH.

Lecturer: Prof. Helmut Elbrächter

This first note is about the capacity of a single-layer neural network in approximating an arbitrary function whose domain is in ${\rm \pmb{R}}^d$.

more >>

REPET Algorithm - Background Music Separation Using Auto-correlation

Pre-requisites:

1. Knowing the discrete Fourier transform (DFT);
2. Knowing the auto-correlation calculation for discrete series.

This is a note for Zafar Rafii’s REpeating Pattern Extraction Technique (REPET) algorithm in its basic form. REPET is a super-simple algorithm which could separate human voice from accompanied music, although it is effective only in typical cases when background music is highly repetitive.

more >>

Beta Regression vs. Logistic Regression

Pre-requisites:

1. Knowing the Maximum-likelihood criterion;
2. It is recommended that readers have known the concepts and basic applications of Beta distribution.

This article tells you a method to modeling a regression problem whose output is bounded and supposed to be countinuous.

more >>

Pytorch - A Seemingly Indifferent 1-D Dimension Leading to Model Failure

This is a reflection on PyTorch: be careful with the shape of tensor, even for differences between shape[B,1] and shape[B].

Have you encountered such a problem when training a neural network: you are sure that your model is perfectly correct, and your training procedure is also correct. But you cannot get the expect result. Your model outputs seem blurred, and the loss curve does not show obvious descent. What is wrong?
Have you encountered python warning like this? UserWarning: Using a target size (torch.Size([12])) that is different to the input size (torch.Size([12, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.

more >>

From Beta Distribution to Conjugate Distribution

Pre-requisites:

1. Knowing the Bayes theorem;
2. It is recommended that readers have gone through elementary training about statistics.

This article tells you the basic ideas behind the Beta distribution, and its basic applications. In detail, we could use Beta distribution to model random varibales which represent proportions or probabilities.

In this article, you could understand why Beta distribution is called “distribution of distributions”, and get an intuition about the concept “conjugate distribution”.

more >>

(Chinese) - 记录一个儿时的游戏-"攒" (A Biography for A Game in my Childhood-"Zan")

This article contains two Chinese articles about my childhood: a Chinese introduction for game "Zan" (see also http://www.wikibin.org/articles/bor-bor-zan-2.html), and a Chinese article regarding my view of points on this game. more >>

PyTorch - Use create_graph to Compute Second-order Derivative

by lucainiaoge

---

This article assumes that readers have been familiar with the concept of computation graph.

Can we use computation graph to calculate arbitrary-order derivative? The answer is yes, we can. But how to leverage it using PyTorch, and how to understand the whole thing?

more >>

An Introduction to Deep Clustering

by lucainiaoge

---

Toy Example

Suppose we have a series of vectors $V=[v_n],n=1,2,…,N$, where $v_n\in {\rm R^K}$. We want to cluster together similar vectors, what will we do?

more >>