osu! web editor – Playback

I’m finally back to kick some tail. rarelyupset.com/osu-web-editor/

Two months ago I started trying to make beatmaps in osu! (What’s that? https://www.youtube.com/watch?v=BFP2Rzwyvxc). I enjoyed it, nobody ever saw my work, yadda yadda I thought the editor could use an update. One’s actually coming by the end of 2017, so this might be moot.

One month ago I decided React.js was probably worth learning. What’s something I could use React for, I wondered to myself.

Now I have a barebones beatmap player, and I want to talk about how it works. Much of my implementation so far draws from opsu!

Structure

React is “a JavaScript library for building user interfaces”, and it does this with “components”, which in two words are special <div>s.

In its current form, the highest-level component is the <Editor>, which contains the <TopBar>, <BottomBar>, two <Sidebar>s, and the <Playfield>. They all have their own subcomponents, but that’s the gist of it.

Components hold information in states, and can pass data down to their children – for example, the <Editor> holds the whole beatmap file and passes hit objects to the <Playfield>, which then renders them as <Slider>s and <Circle>s. React also makes it easy to drop the <Editor> into a potentially higher <Window> to switch between the <Editor> and a <TextEditor>.

One last key thing about React components: They don’t have to be HTML elements. I (ab)use this to use SVG to draw the beatmap.

Drop .osz

osu! beatmaps are usually distributed as .osz files, which are zip archives. In them you’ll find an .mp3 of the song, a .osu file that is the actual beatmap, and more that I haven’t handled yet.

hikari@vultr-cheap:~$ wget http://osu.mengsky.net/api/download/345600
hikari@vultr-cheap:~$ file 345600
345600: Zip archive data, at least v2.0 to extract

Dropping files is straightforward, and because some poor soul has already written JSZip, so is extracting the .osz. Right now I blindly load the first .osu file in the archive and parse and load the .mp3 file from there.

Play

The implementation of playback is naive but working. An <audio> element plays, and every 16ms (60fps) a setInterval() checks the currentTime of the <audio> element and feeds that to the <Editor>, which passes the information to the two unimplemented timelines and the <Playfield>.

<TopBar
    currentTime={this.state.currentTime}
    objects={this.state.currentDiff.hitObjects}
    timingPoints={this.state.currentDiff.timingPoints}
/>;
<Playfield
    currentTime={this.state.currentTime}
    objects={this.state.currentDiff.hitObjects}
/>;
<BottomBar
    currentTime={this.state.currentTime}
    timingPoints={this.state.currentDiff.timingPoints}
/>;

<Playfield />

The <Playfield> is an SVG document. I’m using SVG because it’s fast enough for the number of elements that’ll be visible, easy to use, and plays nice with React – meaning click events. Paper.js is another tool I’m considering.

By setting the document’s viewbox="-40 -40 592 464", I match my grid to osu!’s coordinates, a 512 x 384 area. I also add 40-unit borders for large circles placed at the edge of the field.

In playback, <Playfield> filters away objects that aren’t the current point in playback and draws the objects that are visible.

const visibleObjects = this.props.objects.filter((object) => {
    if (object.objectName === 'circle') {
        return (object.startTime > VISIBLE_START) && (object.startTime < VISIBLE_END);
    }
    else {
        return (object.startTime < VISIBLE_END) && (object.endTime > VISIBLE_START);
    }
}).map((object) => { // Map these objects to object types
    switch(object.objectName) {
    case 'circle':
        return <PlayfieldCircle ... />;
    case 'slider':
        return <PlayfieldSlider ... />;
    case 'spinner':
        return <PlayfieldSpinner ... />;
    }
});

Circles and Spinners

Spinners were the easiest to implement – the word [spinner] shows up. Hit circles were similarly simple – hitcircle.png shows up at the circle’s xy-position. The challenge in rendering a beatmap is the sliders – notes on which the player clicks and holds, following a path… a <path>.

Sliders

osu! sliders are stored as a number of control points defining a path and then the length the slider goes along the path. While the given length can extend longer than the defined path, in practice this never happens. There are three kinds of sliders: Two-point sliders make a straight line, 3-point sliders make a circular arc going through the three points, and 4+ points define a Bezier slider.

Linear sliders are easy:

case 'linear':
    d = 'M ' + this.props.points[0].join(' ') + ' L ' + this.props.points[1].join(' ');

Bezier sliders are only a bit harder. A Bezier slider can be composed of multiple segments, separated by “red points”. Because SVG doesn’t support nth-degree Bezier paths,  I approximate them by sampling from the parametric definition (straight from jsbin).

for (let i = 0; i <= 1; i += samplingIncrement) {
    dStr += ' L ' + Curve.bezierPointAt(i, segment).join(' ');
}

static bezierPointAt(t, slider_points) {
    if (slider_points.length === 1) { return slider_points[0]; }
    let newpoints = [];
    for (let i = 0, j = 1; j < slider_points.length; i++, j++) {
        newpoints[i] = Curve.linearInterpolate2d(t, slider_points[i], slider_points[j]);
    }
    return Curve.bezierPointAt(t, newpoints);
}

Circular arcs are the most involved. The quick overview is finding the circle’s center and the angles of the arc, then sampling the arc.

To find the circle’s center, I find the midpoints between the points on the arc. From these midpoints, I make that are perpendicular to the lines between the points on the arc. Calculating the perpendicular vectors stems from y = mx + b being perpendicular to -my = x + c. The center of the circle is found a t the intersection of these vectors. I took the calculation from StackExchange.

let midpointA = Curve.midpoint(slider_points[0], slider_points[1]);
let midpointB = Curve.midpoint(slider_points[1], slider_points[2]);
let vectorTowardCenterA = [slider_points[0][1] - midpointA[1], midpointA[0] - slider_points[0][0]];
let vectorTowardCenterB = [slider_points[1][1] - midpointB[1], midpointB[0] - slider_points[1][0]];
let centerPoint = Curve.parametricIntersection(midpointA, vectorTowardCenterA, midpointB, vectorTowardCenterB);

Knowing the center, angles are easy to find using the points’ positions relative to the center and atan2. I handle the case where the arc crosses π/-π, determine the direction of rotation from start angle to end angle, and sample along the arc.

for (let i = 0; i <= 1; i += 0.05) {
    dStr += ' L ' + [centerPoint[0] + radius * Math.cos(angleStart + i * arcLength), centerPoint[1] + radius * Math.sin(angleStart + i * arcLength)].join(' ');
}
I messed up oops

Conclusion

And that’s it, just about everything I have done so far. It’s by no means complete, but it’s enough to go in a few different directions from here. My priorities are:

  • The editing part: It is, after all, the whole point
  • Separating drawing and audio: Later I’ll need to add hitsounds – a noise that plays whenever a note is hit. Playing at the right time is critical, and simply checking at 60Hz might not be enough
  • Optimizing drawing: Using SVG’s quadratic and cubic Bezier paths and circular arcs when possible, for pretty curves and less calculation in JavaScript; not recalculating every slider on every frame
  • Adding menus/controls

I’ve done a horrible thing: Javascript logistic map generator

Why?

I’m in a math class. We were supposed to write a program in any language that would pop out the bifurcation diagram of the logistic map (http://en.wikipedia.org/wiki/Logistic_map).

How?

I’m not a programmer, or maybe I’m just a bad one. I didn’t want to install Mathematica on my laptop and I wasn’t familiar with any graphics libraries in Python or Java. So what programming language am I familiar with that I could do graphics in?

<rarely_upset> because javascript is the only language I can make graphics in

<Acid`> oh

<Acid`> oh.

Any programming language could do the math part. Start at a lot of initial conditions, iterate a lot of times, keep the last x iterations to find periodic stable solutions.

I chose a target resolution, mapped my target region to the image (x in [0, 1), r in [2.75, 4) to 1920×1080), and went at it.

<@rarely_upset> wikipedia links to java applets, but I don’t think anyone has ever done this in javascript

My “2D graphics engine” would be creating a blank div the size of my target image (e.g. 1920×1080) and then populating it with 1px divs. For the last x values from every initial condition, I added an rgba(0, 0, 0, 0.05) div at the closest pixel.

Chrome crashed. Too much memory. That was the first thing I learned from this exercise: 256 last values * 1920 * 1080 = way too many divs.

<Acid`> I don’t think javascript was the right decision

In my next iteration, I kept a 1920 x 1080 matrix of integers, adding 1 at each point when an output value landed there. Then, for every cell in the matrix with a value greater than 0, I made a matching div with a darkness of min(255, value). This cut my div count down to about 500k. Chrome survived, but it still took forever. I needed to make the most of my CPU.

<Acid`> I think you need to sit down and think about what you’ve done

Using web workers I split the math into threads. This sped things up a bit, but with in-depth profiling (watching my CPU usage in Task Manager) I could see that most of my time was consumed in having poor Chrome render everything.

Enter Robert Eisele’s pnglib.js. Chrome had a much easier time with a single png than half a million divs.

What now?

The project currently lives on Dropbox: at https://rarelyupset.com/chaos

Knock yourself out.

<@rarely_upset> do you know anything about running things on gpu

<@rarely_upset> …specifically javascript

<Acid`> …

<Acid`> That’s illegal