A Year of Data

A few months ago, I hit the milestone of a full year’s worth of data collected by my server monitoring tools. I started a writeup at the time, which I’m finally revisiting in hopes of actually posting it. (I’ve also learned that I seem to be capturing exactly a year of historical data, which is unfortunate; I’d perhaps naively assumed that the powers-of-two backoff in storage frequency would give me an infinite time window, albeit eventually degrading to a single sample for the oldest time slice.)

First, a quick overview; here is a year’s worth of smoothed temperature data (the average-per-interval is plotted, whereas my usual “is something wrong” daily charts are plotting the maximum in a given interval).

Smoothed 1 Year Temperatures

The temperature curve is about what you’d predict for a computer that lives in a poorly-insulated room in Pittsburgh. I lost my outside temperature correlation data (RRDWeather gave up on me a while ago – maybe weather.com changed their policies, or just their web service API, and I never really did anything beyond sending an e-mail to the developer). However, I’m quite confident that they’re well aligned; anyone who lived here this year can pick out, say, that late warm spell we had at the end of November. The one thing that concerns me a little is the decreasing separation between the board and the drive temps; that makes me wonder if I’ve obstructed airflows a little (leading to more indirect heating of the drives). Still and all, there are no major deviations here, and a year later I’m basically in the same temp bracket that I was last July, which is somewhat soothing (and the whole point of establishing this baseline data).

One of the ongoing puzzles from this server-monitoring data has been the voltage data. It has a variety of odd characteristics, including daily and seasonal cycles, as well as a fair amount of apparent noise. I had thought they’d quieted down a lot when I installed a uninterruptible power supply (with supply-voltage smoothing). I still think that’s true for most of the lines, particularly the 3.3V and 1.5V rails, which are actually providing power to the chips. However, there’s still a significant variation on the 5V and 12V rails. Looking at this data for the past month, you can clearly see a daily cycle on the 12V rail through Weeks 24 and 25:

Voltages June-July 2010

In the chart here, you can also see a “reboot discontinuity” at the beginning of week 23; from memory, I think this was a kernel update.

An Aside Into Passive Monitoring

The occasional almost-vertical drops in voltage (noticeable in this chart at the reboot and the end of week 26) are strongly correlated with system usage (for instance, the CPU utilization chart); a co-worker has suggested that this would be an interesting approach to intrusion detection. I might get more into this in a later post, but history suggests otherwise, so please tolerate a brief excursion to look at another set of charts, in this case the history of the DNS daemon running on the server. First, the last 28 hours of data:

DNS Data - Last 28 Hours

I haven’t spent as much time exploring the reporting mechanism for this chart, so things don’t add up as nicely as some of the others, but the intent here is that the thin blue line records the number of requests, which is then partitioned into success or failure cases. I’m not sure if the open gap (between the explained results and the number of queries) is due to failures other than NXDomain (non-existent domain), or if this is simply an artifact of differing time scales and reporting methods. I’m also not sure about the sampling time frame – this is obviously “queries per time period”, but without knowing the granularity, the absolute scale (area under the curve / total queries per hour) is meaningless. However, the basic view of local network activity is pretty compelling, as long as interpreted strictly in relative terms.

For a moment of background: Calliope, the server, performs both DHCP and DNS duties. Except for a couple of local hosts, DNS is simply forward and cached from Verizon’s DNS provider. The interesting bit here is that my local server provides DNS with zero “time to live”, so clients will immediately ask again if they need to resolve the same host repeatedly. This means that the record of DNS activity on the server is a reasonably accurate record of internet usage across all hosts in my network, even the uninstrumented ones like the PlayStation etc.

In this chart, you can see a midnight spike, probably primarily cron jobs and the like. More interestingly, you can see my flurry of surfing in late afternoon, trying to figure out whether the T was a viable way to see the city fireworks, followed by several dark hours when we were off watching things go boom. Then this morning you can see machines coming on for the first time (checking for updates, etc), a period of quiet coffee drinking (lately I’m reading “The Worst Journey in the World”, rather than surfing, over coffee), and then a steady rumble of activity as I fired up my main box and started working on this post.

The weekly and monthly charts have similar stories to tell, but not much additional interest; overall activity tends to peak on the weekends, and during the week there’s a definite shift to activity later in the day (when I get home from work), but this is all predictable (though useful from an intrusion-detection point of view). Before returning to the earlier question of voltage excursions, I’ll just post one more DNS chart, this time showing the past year’s activity:

DNS Data - Past Year

For the most part, the patterns have disappeared into noise, though there’s still a definite periodicity; I wonder if my cycle of amusements (computer vs books vs television, and so forth) is really that predictable? There’s also a spike around Christmas, from shopping and travel planning, and an interesting nadir in March, which I think I can explain: this was a period of high intensity at work, and just before I started working on a consulting side-project (which explains the above-average usage for the following months). In any event, I find this data kind of interesting, though the real fun would be in correlating it against a diary or other activity measures to see if it’s really a valid proxy for how much time I spend on the computer.

Back to the Voltage Fluctuation

Anyway, returning to the voltage dilemma, we saw a month’s worth of data previously; here is the data for the full year:

Motherboard Voltages: Past Year

It’s not the easiest chart to read, I admit. In all cases the plots are deviations from “nominal voltages”. Most of the traces are really rock solid, varying only a few millivolts if that. However, there’s clearly something going on with the 5V and 12V lines: they vary together, and they have a clear seasonal component. That pattern from October and November is particularly interesting, if you remember that both those months had sharp cold spikes followed by surprisingly mild weather.

I’ve probably given away the game by now, but I’ll walk through it anyway. Since statistics are pretty tricky on these datasets (the data points are actually averages over varying intervals, and I suspect that does evil things to the normal distribution assumption), I decided to satisfy my curiousity with a visual correlation instead. To do this, I plotted a key temperature curve (reported motherboard temperature) on the same graph as the 5V and 12V deviation plots, then inverted the scale on the temperature (so valleys are the highest temperatures) and iteratively adjusted constant and scale until they lay in the same general region. (I probably could have also just inverted temp and thrown it onto a secondary axis, but this was easier.) As a result, the vertical units for the voltage lines are millivolts of deviation from their spec voltages, and for the temperature they’re completely synthetic units. Note that the chart is kind of mis-titled: for the reasons noted above (“I’m not good at stats”), this is not a direct plot of correlation, but simply a superposition of the two datasets (suitably massaged). We already know that the 12V line and the 5V line are pretty well aligned, so look for how well the green line tracks the pattern established by the voltages:

Temperature / Voltage Analysis - Past Year

Looking at this chart, as has been clearly foreshadowed, we have a strong “correlation” over the course of the year. My explanation for the basic pattern here is the system fans. There are both 5V and 12V fans in the system. Since the server is relatively underutilized (only accessible by me and my wife, and typically for light duty), the usage of these fans is the dominant change in power usage over time.

There is clearly some overshoot in both directions, in warmest days of summer and the coldest days of winter. I invoke two explanations for this. In summer, these are NOT independent measures – the fans are working to cool the motherboard. I would infer that temperatures were actually higher in August than July, but the fans kicked in to keep the motherboard temperature roughly constant across those two months.

This rationale doesn’t seem to work to explain the winter overshoots, since the fans only work to push the temperatures in one direction. There are two scenarios here. One is that we over-compensate with heating when it’s coldest outside; this generates a noisy graph on a yearly scale (since the temperatures are varying widely over the course of the day, and the fan load correspondingly). A second explanation would be that the fans are essentially idle through the winter, and as a result they cease to be the dominant driving force on particularly the 12V line. This allows the daily noise of varying utilization to show through, instead of being drowned in the all-day-long load of the fans.

Unfortunately, I lack the data to disambiguate these last two scenarios. Most of the fans are controlled by an off-board fan controller (sitting in a drive bay, but not providing instrumentation data back to the host computer). The BIOS-controlled fans should report their speed, but I’ve never been able to get good data out of them; I suspect that at heart the problem is that this motherboard came from a pre-packaged HP “media computer”, and they may have saved a few pennies by cutting out some of the monitoring tool support. At some point I’ll upgrade my dev box, and transplant its rather more enthusiast motherboard to the server. That’ll be a setup chore to be sure, but perhaps I’ll be able to get some direct information. In the meantime, having such a long baseline allows me to be pretty confident in my indirect conclusions. (Note that this is the whole point of this post: if I point-sampled this data at any time throughout the year, I really wouldn’t be able to draw any conclusions at all. However, a year’s worth of data tells a pretty compelling story, even without formal stats to back it up.)

Sampling Outside the Box: 1-Wire and Friends

I have some longer-range thoughts or daydreams, time and money permitting (primarily time). I spent quite a while looking for a fan controller which ”would” report its sensor readings to the host. Apparently they exist, but the only examples I was able to find that really fit my needs were these beasts from Matrix Orbital. While extremely in their own way, I really don’t need an OLED screen displaying weather etc on my server box, when the whole point is that I don’t see or hear the server anyway. However, some of the tech specs on this monster mentioned that it can gather and report data from any “1-wire” sensor source. That set me off on a whole other trail.

It turns out that “1-wire” is a Texas Instruments physical and logical protocol for simple, low-power sensor networks. It looks to be primarily designed for industrial applications, but there is a hobbyist community as well (particularly for weather stations, it seems, although I found a fascinating story of a buried moisture sensor driving an automated sprinkler system – now that’s gardening! [Update: Okay, I think the hobby-built CO2 sensor beats the automated sprinklers; no link because I noticed belatedly that it's deep in an "Advanced Marijuana Cultivation" forum - I'm still bemused by how Google ends up cross-cutting conversations!]). There are USB “host adapters” available, as well as a pretty wide range of sensor units (and raw adapters which simply sample a voltage and report it) – the net result seems to be that you can monitor pretty much anything you want, if you obey the topology and run length constraints and don’t mind writing some software to interrogate and record.

So my new dream is to expand my coverage significantly. This project started as a way to monitor the health of my server (and particularly to spot dead fans) without being able to see or hear it. However, there’s no reason I can’t use the same infrastructure to monitor our living space as a whole. I’d love to track ambient temperatures in various parts of the house, to see where the insulation is failing (and prove out my intuition that it really is ”cold” in the dead space under my computer desk). Weather metrics are less near and dear to my heart, but it could be amusing to have a windspeed sensor or track rainfall. Moisture sensors in the basement could give a approximation of Home Heartbeat, without the vendor lock-in and high prices. I haven’t found a good AC sensor yet, but it seems like it would be possible to do some approximation of the “Smart Grid” idea (realtime electricity-usage tracking) without having to wait on Duke Light to figure it out. Even further down the road (and after building some trust in the system), it could be combined with an X-10 network for real closed-loop environmental control – far better than simply putting everything on a timer, or “home automation” which requires a user at a web browser to actually do anything. Before I dive into anything that big (and potentially annoying if done wrong), I’ll probably start with simpler closed-loop systems. For example, there’s no reason why my nightly cron tasks can’t wait for system idle, instead of running at midnight – that’s when I get some of my best work done! Some small-scale experiments with this sort of feedback loop would teach me a lot about how to control yoyoing and all the other pitfalls of feedback-based machine control, without annoying my wife.

A Note About Tools

That means there are going to be a lot of LaTex-created images in this section. Sorry, but it’s a lot less painful than trying to write out math in raw HTML. (Okay, writing this inspired me to search for LaTex to MathML translators, and of course the W3 itself maintains quite a nice list. Oh well, I promise I’ll look into that for next time.) In the meantime, I’m using the wiki built into Trac to write this post, with the TracMath plugin to take inline LaTeX blocks and render them as PNGs. This means that I just type

#!latex
$$N_{i,k}(v)={{\left(
  T_{k+i} - v\right)\,N_{i+1,k-1}(v)}\over{T_{k+i}-T_{i+1}}} +
  {{\left(v-T_{i}\right)\,N_{i,k-1}(v)}\over{T_{k+i-1}-T_{i} }}$$

and I get

Well, okay, I don’t write that, because I can’t remember LaTeX math syntax to save my life. (I got pretty good at it in college, but it’s been a while.) Anyway, it’s hard to parse by eye, and I’m very prone to swap a u for a v, which can be unfortunate when you’ve just carefully drawn a distinction between them.

Instead, I’m using Maxima, a free Computer Algebra System. It’s stranger to pick up than Mathematica, but the price is right (and it’s quite powerful once you get used to its ways). It seems to be really flaky on recent Ubuntu releases, which is unfortunate, but I’m primarily using 5.17.0 for Windows and it works like a champ. It has a very nice tex function, which will spit out the TeX formatting for any given expression.

So in practice, I work the math in Maxima, have it convert to TeX, then paste that into the post. (Sometimes I tweak it slightly, for example to pull out a leading “1/2″ rather than leaving the whole expression over 2 as Maxima seems to prefer. But for the most part, the expressions in the text are directly out of Maxima.) I have some ideas of writing a Trac plugin to close the loop and directly evaluate and typeset Maxima expressions in the text, but I haven’t got it all worked out yet: the equations in a piece generally build on each other, so you need enough lifecycle awareness to start and maintain a separate Maxima state for each page in flight. Also you probably need some non-displayed helper definitions and intermediate values, so you need a separate bit of syntax to indicate what should be rendered into the page. I think the fact of the existence of the PageOutlineMacro means it’s possible, but the fact that the PageOutlineMacro actually re-parses the whole page means it’s not necessarily easy. (I’d also still have to solve creating and maintaining enough state that the various individual blocks could fetch their expression / formatted results from the initial gather-and-run macro.) Anyway, for the moment I’ll do it the hard way.

Back to the Math

I’m going to run through a concrete k=3 (quadratic) example with 6 control points (so n=5), and claim that this demonstrates a general solution for k=3. I’m pretty sure it does, but it’s a lot easier to keep track of the indexes with a concrete number. Or so I claim.

So, the knot vector T should be the sequence [0, 0, 0, 1, 2, 3, 4, 4, 4]. Recall that indices are zero-based, and that N_i,1 is defined as

Given this, the N_i,1 terms will start being interesting at i=2, where N_2,1= 1 for 0 <= u < 1. From then on, each N_i,1 is non-zero on N_i,1= 1 for i-2 <= u < i-2+1, up through the last value we need at N_5,1. Using Maxima to evaluate the raw B-spline equation

for our knot vector T (again, this is the k=3 case), we get a whomping huge expression in terms of the control points P, the N_i,1 “switches”, and of course v. I won’t reproduce the whole thing here (fully expanded, there are 33 terms, and even my new wider columns can’t handle that), but here’s the coefficient of the second control point P₁ for flavor:

Explicit Equations for Each Segment

The good thing is that we don’t ever really need the whole equation at once. Only one of the N_i,1 terms will be 1 at any given time — this is what produces the curve segments. By setting N_2,1 through N_5,1 to 1 in turn (and the other N_i,1 terms to 0) we can produce the four curve segments. In turn, these are

Maxima Script

For anyone who’s following along at home, here’s the Maxima to get our results so far:

N(N1, T, i, k, v) := if k = 1 then N1[i] else
(if T[i+k-1] = T[i] then 0 else (v- T[i])* N(N1, T, i, k-1, v) / (T[i+k-1] - T[i]))
 +
(if T[i+k] = T[i+1] then 0 else (T[i+k] - v) * N(N1, T, i+1, k-1, v) / (T[i+k] - T[i+1]));

BSpline(P, N1, T, n, k, v) := sum(P[i] * N(N1, T, i, k, v), i, 0, n);
T5Vals : [0, 0, 0, 1, 2, 3, 4, 4, 4];
T5[i] := T5Vals[i+1];

BSpline5_3(v) = BSpline(P, N1, T5, 5, 3, v);

makeN1Helper(curveSeg, n, k) := makelist(if j - k + 3 = curveSeg then 1 else 0, j, 0, n-1);
BSpline5_3_Seg(seg, P, u) := BSpline(P, makeN1Helper(seg, 5, 3), T5, 5, 3, u);
for seg : 1 step 1 thru 4 do disp(
  segment[seg] = facsum(ratsimp(
    BSpline5_3_Seg(seg, P, v)), P[seg-1], P[seg], P[seg+1]));

Note that Maxima lists are one-based, hence the helper function T5 to preserve the zero-based indexing in the original definitions. Since N_i,0 is never referenced, I can get away with a list starting at index 1 (though I admit this is made more confusing by the choice of a zero-based loop variable. Oops.) Also note that BSpline5_3 wasn’t actually defined as a function there — that was just an equality, to make Maxima print it out nicely. Other than that, this is pretty much what it looks like. Oh, and notice that the recursive N definition has to avoid a couple of divide by zero cases to implement the “where 0/0 is defined to be 0″ escape hatch.

Reparamaterization

Okay, those equations work but they look pretty ugly (and they’re different in each segment, but we’ll work on that in a minute). It took me a while to sort this out, but I sort of tipped my hand back in the “Definitions” post where I introduce w and j, so I’ll just repeat the equation from there: v = u (n-k+2) = j+w-1. Remember that j is the segment number ranging from 1 to n-k+2, which is exactly the subscript of segment in the explicit equations above. So we substitute w-1 in the first equation, 1+w-1 in the second and so forth. Using the Maxima definitions above, this is just evaluating the BSpline5_3_Seg function above at seg+w-1 rather than v:

for seg : 1 step 1 thru 4 do disp(
  segment[seg] = facsum(ratsimp(
    BSpline5_3_Seg(seg, P, seg+w-1)), P[seg-1], P[seg], P[seg+1]));

(the facsum bit just rearranges the terms) yielding

Now we’re getting somewhere. The middle two equations are identical except for shifted subscripts on P. The first two and last two are different, but we can work that out in a minute. First let’s generalize the uniform version and check our claim that

works for those middle terms. This will also give us a peek at the what’s going on with the end segments.

B3[uniform](P, j, w) := (-P[j]*(2*w^2-2*w-1)+P[j+1]*w^2+P[j-1]*(w-1)^2)/2;
for seg : 1 step 1 thru 4 do disp(
  difference[seg] = ratsimp(
    BSpline5_3_Seg(seg, P, seg+w-1) - B3[uniform](P, seg, w))
);

yielding

What we’re computing here is the difference between the “real” equation for each of the four segments and what we get from using the uniform equation for all of the segments (ignoring the segment number).

Doesn’t look immediately promising, but there are two good things:

• the two middle segments are as predicted by the uniform function above
• the two end segments’ differences only involve two control points

We know that P₁ and P₄ are used in the middle two segments as well as the end segments, so we can’t do anything to them. However, the two end points (P₀ and P₅) are only used in the end segments. So what it we feed some different value (not the current P₀, but some combination of P₀ and P₁) into the uniform equation? Can we make the result be the same as evaluating the “real” B-spline function on the original control points?

In the uniform version of segment 1, we replace P₀ with a new variable x, then set it equal to the real first segment of the k=3 B-spline:

We then solve for x hoping we can eliminate all the w terms:

rightAnswer : ratsimp(BSpline5_3_Seg(1, P, 1+w-1));
uniformWithFreeP0 : ratsubst(x, P[0], B3[uniform](P, 1, w));
solve(uniformWithFreeP0 = rightAnswer, x);

and we get (drumroll, please)

(Okay, that didn’t really need a LaTeX block, but it really did take me quite a while to figure out, and I was inexplicably happy when I found it (literally: one very really reason for this whole series is to explain to my wife why I was grinning so much that week). If I’d happened to have run into that formula up front I would have spared myself a great deal of frustrating math trying to figure out how to make sense of what I was seeing. Of course, I also wouldn’t have learned a lot about splines, so maybe it was better this way.)

Repeating the process for the last control point (in this case P₅), we get x = 2 P₅-P₄:

rightAnswer2 : ratsimp(BSpline5_3_Seg(4, P, 4+w-1));
uniform2WithFreeP0 : ratsubst(x, P[5], B3[uniform](P, 4, w));
solve(uniform2WithFreeP0 = rightAnswer2, x);

This is a reassuring result — the equations are all symmetric, so it should be possible to reverse the control points and produce the same curve. Therefore, it’s nice to see that this formulation for the end-points is also symmetric.

So the practical upshot of this is that we now have a way to evaluate all the segments of a B-spline, even the “weird” end segments, with the same uniform equation. To do this, we have to massage the control points a little. Given the input sequence

( P₀ , P₁ , … , P_n-1 , P_n )

we need to make a copy, so we can feed the uniform function the sequence

( 2 P₀ – P₁ , P₁ , … , P_n-1 , 2 P_n – P_n-1 )

instead. That doesn’t seem like much of a price to pay for freedom (or at least for being able to evaluate all segments of all k=3 B-splines curves with the same straight-line evaluation code).

The same general approach works for k=4 curves (cubic B-splines). Because the cubic blending functions involve more points in each segment, we have to replace both P0 and P1 with free variables, and solve for them simultaneously (using the known equations for the first two segments, instead of just the first segment), but it’s the same logic and it works quite nicely. I haven’t tried to generalize a formula for any value of k, mostly because quadratic and cubic B-splines are far more common than other types.

This took much longer to post than I’d planned, but I hope it was worth the wait for someone. I can absolutely guarantee that this isn’t an original result, but I haven’t actually seen it in Wikipedia or MathWorld or the usual suspects, so I feel entitled to be slightly smug.

So I’m trying out a new “theme”, as a quick-and-dirty way to avoid having to actually, you know, design a visual look for this site. Which would be nice, but unfortunately said theme apparently doesn’t handle definition lists (<dl>, <dd> and <dt>), which are my recent favorite way to avoid dealing with tables. (I stole this affection from a blog post I read sometime, but since it no longer appears in the top page of Google results for “definition list”, I’m deeming it an orphan work and mine for the appropriation.)

Anyway, it looks like there is some CSS work in my future, probably mining this site pretty heavily. (I continue to be delighted to live in a world where you can search on any given HTML tag and be pretty much guaranteed to find a nicely organized page of CSS techniques, rants against rampant misuse (generally a good source for ideas you hadn’t yet thought of), and generally a rich literature for those of us in the “I care about CSS for exactly as long as it takes to get my font sizes consistent” camp).)

Given my frequency of updates, you can therefore look forward to many months of inappropriately-sized definitions and other offenses against visual design.

(I actually do have another article in the B-spline Series, wherein I get to look all clever (had I not previously tipped my hand about my various confusions) by presenting a rather nice uniform version of k=3 and k=4 B-splines based on preconditioning the control points. There will be more LaTeX math! More empirically-derived constants which I pass off as proven! It’ll be great.)

function evaluateBSplineWithConditional(u) =
v = u * (n-k+2)
segment = floor(v)
w = fractional_part(v)
if(segment <= 1) then
  do weird thing(segment, w)
else
  do uniform thing(segment, w)

(I may have got the indexing slightly wrong, but I think this works (and is a good illustration of why I was being so pedantic about u, v, and w above — it’s not complex math, but it’s easy to mess up the parameterization).) Also note that I’m only looking at the weirdness at the beginning of the curve; properly the conditional would need to check whether the segment was at either end of the curve. This is a simplified version to support the discussion, not an actual implementation.

Endpoints are often exceptional cases; why not just define the evaluation with that exceptional case and move on? All the more so since my attempted trick of “pre-conditioning the data” (replacing the original sequence of control points with a sequence with repeats at the beginning and end) didn’t work? (An important background assumption to the rest of this discussion is that this function is going to be run many many times, since we need a lot of points on that curve to make it look like a smooth curve when we draw it on-screen, or when we send the points to a CNC router to cut metal, or whatever we’re doing with it. If we only needed to do it once, this would probably be the right solution.)

The Problem with Branching

The answer basically has to do with computer architecture and the way it has evolved over the past couple of decades. Once upon a time, when I was programming on my father’s hand-me-down Heathkit Z-80, this would have been a perfect solution — each time we needed to evaluate a B-spline, we’d simply decide if we were in the normal or weird section of the curve, and calculate accordingly. The problem is that as we poured billions of dollars into CPU fabrication techniques, we got more and more transistors, and needed some way to take advantage of them in order to make things run faster. One major solution is “pipelining”, which essentially can be described as doing many things at once. Rather than waiting for each step to complete before we start on the next step, the CPU will get a lot of things started at once, and pull them all together when needed. (Perhaps I’ve been watching too much Iron Chef, but this seems like partitioning work out to a sous chef.) In the extreme case of the P4 architecture, the pipeline was up to 20 stages long. A much more technical discussion can be found at http://arstechnica.com/old/content/2001/05/p4andg4e.ars but the basic takeaway is that for a modern CPU to run fast, it needs to be able to do many things at once.

A conditional (the “if / then” in the pseudo-code above) wreaks holy hell on this scheme. Before the CPU can get started on “do uniform thing” or “do weird thing”, it has to know which one it needs to do. This is one cause of a “pipeline stall”, since instead of doing 20 things at once, we have to wait for the conditional to be evaluated (through all 20 stages) before doing anything. This is absolutely brutal on performance.

Modern CPUs actually try to work around this problem as well, using “speculative execution” and “branch prediction” — basically, they keep track of what happened last time we ran this code, and assume that the same thing will happen this time. If they guess right, when the conditional is finally evaluated they can just act as if they knew it all along. If they guess wrong, they have to throw away all the speculative work and start over on the correct path. This works, but it’s hugely complex, and it makes performance less predictable and more dynamic (because performance becomes directly tied to the accuracy of the branch prediction). In our case, things would run fast through the first curve segment, when the “weird” path is consistently followed. This would be followed by a period of degraded performance when it kept guessing “weird” but the real answer was “uniform”. At some point, the branch prediction unit would be retrained to guess “uniform” instead, and performance would recover until we hit another “weird” segment. In short, speculative execution and branch prediction are nice things to have, but no substitute for “straight-line”, branch-free code.

This whole preference for a single codepath is even more dramatically true when we talk about “SIMD” (Single Instruction, Multiple Data) parallelism. This approach to performance showed up on CPUs in the MMX and SSE instruction sets, and is one of the ways to really increase performance on modern silicon. Essentially, rather than evaluating points on the spline one at a time, we could stack up a list of points to be evaluated side-by-side, then grind through them four (or however many) at a time. See http://arstechnica.com/old/content/2000/03/simd.ars for a more responsible description. The challenge here is that “Single Instruction” bit — you have to be doing the same thing to your four data points, not looking at each one individually and deciding whether to give it the “uniform” or “weird” treatment. This is true of the SIMD instructions in modern Intel and AMD CPUs, and even more so of “shader programs” running on graphics cards’ GPUs. In fact, early shader programs didn’t even have conditional instructions; everything had to be straight-line code. More recent cards allow you to branch on the GPU, but at a potentially fearsome cost to performance (see http://forums.nvidia.com/index.php?showtopic=107865 for instance).

Separate Batches

So we’re all agreed that branching is Bad. What can we do?

One approach is the “hoist the conditional” out of the inner loop. This somewhat compromises the nice clean interface of the evaluateBSplineWithConditional function, but often be a performance win. To describe this approach, we need to introduce another level of the program, namely the function which was calling the evaluateBSplineWithConditional function. Assume that we’re trying to fill an array with evenly-spaced points on the B-spline, in order to render them or drive a CNC router or whatever. Using the original definition we’d have something like

function naiveFillArrayWithSpline(array, numPoints)
  dU = 1 / (numPoints - 1)
  for(i = 0; i < numPoints; i++)
    u = dU * i
    array[i] = evaluateBSplineWithConditional(u)

The naiveFillArrayWithSpline function would really work on any parametric curve function that expects a u ranging from 0 to 1. Instead, we can give it some knowledge about the end-conditions of B-splines, to produce a version without any conditionals in the main loops:

function batchedFillArrayWithSpline(array, numPoints)
  numSegments = n-k+2
  pointsPerSegment = numPoints / numSegments
  dW = 1 / pointsPerSegment
  for(i = 0; i < pointsPerSegment; i++)
    w = dW * i
    array[i] = do weird thing (1, w)
  for(j = 1; j < numSegments - 1; j++)
    for(i = 0; i < numPoints; i++)
      w = dW * j
      array[j*pointsPerSegment + i] = do uniform thing(j, w)

(Again I’m ignoring the weirdness at the end of the curve, which would require another “do weird thing” loop. I’m also playing fast and loose with the array indexing, assuming that numPoints divides evenly into numSegments, and so forth — this is another strawman, in other words.)

This is a much uglier function than naiveFillArrayWithSpline. It also has a lot of specific knowledge about B-splines embedded in it; you couldn’t reuse this for, say, making an array filled with sin(u). However, it has the huge virtue that there are no conditionals embedded in the inner loops. In fact, there are no conditionals at all — by changing the problem from “evaluate a B-spline at a given value of u” to “fill an array with evenly-spaced values of a B-spline” we’ve been able to significantly improve the performance characteristics.

This version still has flaws, however. One is a software engineering issue: we might need both this version and the evaluateBSplineWithConditional function, if we need to be able to fill arrays efficiently but also need to be able to evaluate arbitrary points (perhaps searching for an intersection). Maintaining two functions with different structure, but supposed to produce the same result, is a good recipe for maintenance pain. (In a prior job, we had exactly this situation — and in fact the single-value version did disagree with the bulk version, which was an on-going source of confusion.)

There’s an argument to be made that software engineering pain is the price of doing business, and perhaps a different language or some code generation approach would let us keep a single version of the source code while still producing these two versions for actual production use. There are two technical flaws (or two aspects of the same flaw), though, that mean this version is sub-optimal even ignoring maintenance considerations.

From the point of view of running this code on a CPU, the issue is code size. Instead of a single, small codepath we now have a rather larger function to evaluate. The on-disk size of a program is pretty much irrelevant in the modern world, but the actual in-use code size still matters because of caching. The CPU has a limited (and often surprisingly small) instruction cache on-die; it’s vastly faster to execute code from that cache than it is to load code from main memory. While it’s unlikely that the method above would actually be larger than the instruction cache, it’s definitely larger than a uniform version, which means it would occupy more of the instruction cache. If, say, the actual rendering loop has to be pushed out of the cache in order to load our spline evaluation method, then we’re paying a definite performance cost for code size (although one that’s very difficult to rigorously detect and characterize).

In the case where this code would run on the GPU, the cost is somewhat easier to see. Rendering commands are transferred from the CPU to the graphics card in “batches”, where everything in a batch is using the same shader programs, the same constants, and so forth (only the vertex data changes between data points). The GPU-based analogue of the method above would be to render the spline in three batches: one for the first “weird” segment, one for the uniform segments, and one for the final “weird” segment. The number of batches is very often a limiting factor in naive rendering approaches; magnifying the batch count by a factor of 3 is a quick way to become batch-limited. In these circumstances it can even be possible for the performance wins of GPU-native evaluation to be completely lost in the face of the communications overhead.

Arithmetic “Conditionals”

If the cost of the computations is low enough, and the penalty for branching is high enough, there can be another answer. Instead of having the program explicitly branch, or separating the different conditions into different loops / different batches, we can simply do both branches. The obvious problem is that we’ll have two output values, but we need to produce exactly one (the right one). To solve this problem, we introduce a “condition variable” which is 1 in one conditional branch and 0 in the other. This is typically a “natural” operation because in C (and C-like languages) boolean variables are internally stored as an integer which is 1 for true and 0 for false. In this idiom, the core of the function becomes something like

function doBoth(segment, w)
isWeird = (segment <= 1)
isNormal = 1 - isWeird

weirdResult = do weird thing(segment, w)
normalResult = do uniform thing(segment, w)

answer = weirdResult * isWeird + normalResult * isNormal

This seems incredibly counter-intuitive as a speed-up, but it can actually be a win if the “weird” and “uniform” paths are both rather cheap to evaluate, and the cost of separate batches is high enough. A nice advantage of this approach is that we’re back to having a single method which can be used as the core of an array-filling loop and also used to evaluate “one-off” points.

In practical terms, I would guess that on the CPU, this approach is likely to be slower than the batchedFill method above, while on the graphics card it would be closer. (This reflects the fact that CPU instruction cache pressure is probably less critical than the communications overhead of sending batches to the GPU, combined with just how fast the GPU is at evaluating math — in the GPU case, the bar for “rather cheap to evaluate” is lower, and the cost of separate batches is higher). I haven’t done any benchmarking at all to test whether the B-spline functions actually fit these assumptions on the hardware I have available.

Tradeoffs and Real Hardware

The last paragraph hints at why all of these approaches are kind of unfortunate. It’s very possible (in fact, likely) for the “right” answer to vary with different graphics cards, different CPUs, and different connections between CPU and GPU. For instance, within a given generation of graphics cards, the cost of evaluating functions tends to drop (as the cards get faster), but the cost of submitting a batch is typically fixed (by the interface between CPU and graphics card, which experiences much slower evolution). However, for on-board graphics (where the GPU is on the motherboard or even integrated into the CPU itself) the cost of batch submission may be almost zero, while these kinds of GPUs are typically computationally weak (so the cost of evaluating the functions would be higher).

One very common solution to this sort of dilemma is to optimize for performance on your hardware, and figure it’s reasonably representative of your customers. If there’s any validity to that assumption, this may be a reasonable approach, but it’s fraught with peril; if nothing else, the optimal solution will change over time (with new generations of hardware), so your program is unlikely to age well. It can also be embarrassing, if customers upgrade their hardware and notice that your program doesn’t take any advantage of their expensive new graphics card.

To really do this kind of thing right, across heterogenous hardware and with an eye toward future-proofing your program, you would need to “do it all”, with dynamic strategy selection. For each performance-critical routine, provide all three variants listed above, then benchmark on the customer hardware to figure out which is fastest in this particular computer. This approach actually can work, and is really used by some very performance-sensitive applications. (For instance, the ATLAS project http://math-atlas.sourceforge.net/ does this for certain key linear algebra computations. Likewise, the “md” RAID system on Linux boxes will test several versions of the XOR routine at runtime to determine which is fastest. In both of these cases, a very small but very performance-critical part of the code is being optimized.) This approach is very expensive in programmer and testing time, and easy to get wrong if you don’t have a wide range of hardware available to validate your performance assumptions.

Sometimes you can instead leverage the work of other projects. For example, many problems can be coerced into problems of linear algebra, and you can then take advantage of the optimally-tuned ATLAS functions to solve them. There is typically overhead in expressing a problem this way, but it can be a win in time-to-market and coding complexity. Unfortunately, this isn’t a mechanical solution, and it isn’t applicable in all cases. In the case of B-splines, it would be possible to express the “do weird thing” and “do uniform thing” sections in terms of matrix multiplication, but it would not be easy to capture the general “do this or do that” problem as I’ve set it up here.

So the long and the short of it (after this very long digression) is that conditionals are a pain no matter how you slice it. Vastly better is to be able to precondition the data — that is, feed a fixed formula something clever to make it do what you want, rather than using two different formulas. (You can tell that I got it to work, or think I did, from my phrasing. If B-splines were inextricably different in their end segments, I’d be writing a very nice summation about how conditionals are hard but sometimes unavoidable, and the next section would be about how the real clever is solution is to have several B-splines to draw, and draw all the middles of all the curves in one batch, and all the end-segments in another batch. Which is actually pretty clever, now that I think of it. But it’s definitely not the way this post is going.)

For a typical background you can look at MathWorld or Wikipedia. These are pretty representative of the descriptions you’ll find on the internet in casual searching: technically correct, with nicely formatted equations, but not a whole lot of actual help if you’re trying to understand the things. In particular, the recursive definition of the basis functions are very clever but incredibly unintuitive as far as what they mean. (I also notice that the links share my own indecisiveness about how to title-case “B-spline”.)

A much more useful writeup can be found in the “B-spline Curves” section in these excellent class notes. This is a nice treatment with lots of pretty pictures, although I think the author makes things somewhat more complex by staying with the raw 0 to 1 parameterization throughout (so the elements of the knot vector end up as fractions rather than integers). Nothing wrong with it, just a different treatment (and I suspect one motivated by uniform representation across many kinds of curves, rather than my focus on uniform representation for all segments of the B-spline to ease high-volume computation).

(I should say at this point that I’m torn about how to present things. It’s very tempting to start with what I know now, tell a nice clean story, and maybe then offer an amusing anecdote about how even I didn’t quite get it at first… but that’s exactly what the textbook I was working from did (well, except for the amusing anecdote), and it didn’t serve me very well. So instead I’ll let the game tell the story, and just lay it out as it happened to me.)

First I have to explain why I was motivated to take a shortcut, which is a mixture of raw intellectual laziness and premature optimization. Starting with the intellectual laziness, here’s the basic definition of a B-spline. First, some definitions and terminology (Note that wherever I say something condescending like “the casual reader could be confused”, I really mean “I was confused for about a day”):

u

The parametric input variable. In general usage, u definitionally ranges between 0 and 1 — this defines the extent of any parametric curve (one where x=f(u) and y=g(u)). However, spline literature, or at least the text I’m working from, is pretty casual about redefining u to work over various different ranges as convenient. To avoid confusion, I’ll use u in the traditional 0..1 sense, and v and w to track these other ranges.

P_0..n

The list of control points. This is probably the most important input to the B-spline curve. Note that indexing is zero-based, and there are n+1 control points. Note that P is a list of vectors, and their dimensionality wholly determines the dimensionality of the output. In other words, if P is a list of (x,y) points, then the position of a B-spline defined on P at a given u will also be an (x,y) point. Likewise if P is defined in 3-space, that is (x,y,z) then the resulting curve is also defined in 3-space.

n

One less than the number of control points.

k

An input to the B-spline curve, this determines the degree of the polynomials used for each curve segment, as well as various other things. Most of this post considers the case of k=3.

degree

Generally: the highest exponent appearing in a polynomial: the degree of u²+3 u + 1 is 2. Specifically, for the B-spline segments, degree = k-1 (so my working example with k=3 produces a quadratic on u).

v

A variable in the range [ 0 , n-k+2 ]. This range is mapped to the 0 to 1 range of u in the obvious way: v = u (n-k+2)

T_0..n-k+2 T_0..n+k+1

[Edit 11/03/09 - there are n+k+2 total knots (numbered 0 through n+k+1), even though the highest number appearing in the knot vector is n-k+2.] The knot vector. Note that the knot vector’s indexing is zero based. In the case we’re considering (uniform end-interpolating B-splines) it consists of a list of integers between 0 and n-k+2. The first k entries are 0, then the numbers 1 through n-k+1, then k copies of n-k+2. For example, the knot vector for a 6 control point (n=5), second degree (k=3) B-spline would be T = { 0, 0, 0, 1, 2, 3, 4, 4, 4 }. Note that these are not quite indices into the control point array (for one thing, they wouldn’t reference all n+1 control points). These values actually are in the same parameter space as v, above, and basically they serve to knock out terms in the basis functions (when the same value appears repeatedly in the knot vector).

N_i,k(v)

A B-spline basis function, parameterized on i and k. Defined in two parts, as

and

where 0/0 is defined to be 0. This was the intimidating expression that inspired me to take any shortcut presented. In retrospect, I should have noticed that the denominator would be 1 in both terms unless you’re near an end of the curve, in which case one and then both terms will go to 0 (thanks to the slightly odd definition of dividing by zero). The rest of the time, the N_i,1 terms will act as switches, picking a set of terms from each basis function which will be non-zero in the current section of the curve. Also, note that in much of the literature, this function is written as a function of u, but we’ve already made the jump to the v range as described above, so I replaced the u’s with v’s.

B_k(v)

The actual B-spline curve, given a list of control points p and a degree-controlling k. This function is sometimes referred to as p(u), but this is easily confused with the list of control points. Defined as

I looked at all that, winced hard, and skipped ahead to where they’d solved the uniform version, giving the nice-ish equation for a single segment of

or in matrix form,

This was what I wanted anyway, an invariant equation where I could just feed it different parameter values and it would produce an x and y. To prevent completely following the path of my own confusion, I’ll point out here that this function is actually defined in terms of

w

Yet another parameterization. In this case, w ranges from 0 to 1 within each curve segment. This is a reinterpretation of the v parameter as the sum of an integer part (the segment number) and a fractional part (w).

j

The curve segment number, not to be confused with the i which goes from 0 to n in the definition of B above. To tie these various interpretations of parameter together, v=u (n-k+2)=j+w-1. j ranges from 1 to n-k+2. Note that j is one-based, NOT zero-based (hence the -1 offset when mapped from the 0 to n-k+2 range). This is because at the end of the curve, j=n-k+2 and w=1 — effectively, w is adding one to the range of j (so we need a -1 correction to get the ends to line up right). At the beginning of the curve, w=0, so j must be 1 or we’d again go out of range. A more intuitive interpretation of this is that j is the count of intervals, so this is a pretty standard fencepost correction (http://en.wikipedia.org/wiki/Off-by-one_error).

In terms of these renamed variables, the uniform equation would be

So hurray, there was an easy and uniform equation after all; I could either pass in a different window of three control points for each segment, or I could pass in the segment number in addition to the segment-relative parameter (w) and look up the right control points.

The first problem I encountered was that the curve created this way doesn’t actually interpolate the end points. That is to say, when v=0 the curve isn’t at p₀. Here’s where I really took a shortcut. Instead of backing off and trying to figure out what was going on, I noticed two things:

• The knot vector sort of looks like index indirection lookup table. I think this is an especially Computer Science-ish assumption — adding a layer of indirection is a pretty standard way to make non-uniform processes look uniform (see below for a more detailed discussion of what I mean by “uniform”), and a table of integers looks a lot like a list of indices. Particularly because the knot vector repeats its first and last values.
• If you feed the B_uniform function the points [p₀ , p₀ , p₁] (pretending that the control point list had multiple copies of the end-points), and evaluate it at v=0, it does actually evaluate to p₀.

I was actually working with k=4 for these initial experiments; I only dropped back to k=3 in an attempt to simplify things to figure out what was going on. The effect I saw is magnified at higher values of k — in the cubic k=4 case, each B_uniform is mixing together 4 control points instead of 3. To get it to interpolate the end-point required two additional segments to get from the last “normal” segment (on [ p₀ , p₁ , p₂ , p₃ ]) to actually end up at p₀. The last normal segment, evaluated at u=0, gives us 1/6 (P₂+4 P₁ + P₀). When we evaluate on [ p₀ , p₀ , p₁ , p₂ ] as the first “weird” term, at u=0 we get 1/6 (P₁+5 P₀). It’s not until we’re looking at [ p₀ , p₀ , p₀ , p₁ ] that we actually get p₀ at u=0.

In short, I got a little seduced by algebraic numerology — of course if you have a function which mixes 4 points, and you replace enough of those four points with copies of the same point, you’ll end up with a function which returns that point no matter what you give it. (This is the final reductive case, where all four points are actually copies of p₀ .)

The actual problem here is that we’ve wandered pretty far from the original definition of a B-spline. In particular, we have too many curve segments — v is ranging from something like -1 to n-k rather than 1 to n-k+2. (The key difference from the “real” B-spline is that the proper B-spline doesn’t actually have a “normal” segment on [ p₀ , p₁ , p₂ , p₃ ] — instead, it’s already started going into special cases.)

The practical upshot of this is that we’re spending far too much of the parameter space near the end-points of the curve, leaving only a small portion for the whole interior of the curve. This could be disastrous for rendering, where it’s very nice to sample at evenly-spaced intervals of u and assume that this will lead to generally evenly-spaced vertices in 3-space: with this scheme we’d end up with a great many sampling points at the beginning and end of the curve, and a paucity in the interior.

Even this difficulty could be overcome, by introducing yet another reparameterization to redistribute the incoming u values more evenly over the length of the curve. However, there was no getting around the problem that the actual 3-space shape of the curve was different than my reference implementation (a 3D modeling tool). I’m always willing to entertain the possibility that the reference is wrong and I’m right (perhaps too willing), but at least I needed to understand what was going on here.

That much I knew coming in. For various reasons, I wanted to write a program to evaluate B-splines, with a particular eye to rendering them in 3D. This seemed like it was going to be a very easy task; instead, I learned a lot along the way.

A Motivation for B-Splines

Ignoring the math for a moment, let’s talk about the basic idea of B-splines. There are quite a few ways to fit a smooth (roughly meaning “at least twice differentiable, and with at least the first derivative being continuous across the function) curves to a set of control points. (And in fact there are a lot more if the curve doesn’t even have to pass through the control points!)

The simplest approach is to observe that you can always fit a quadratic (degree = 2) polynomial to any three points, or a cubic (degree = 3) to any four points, and so forth. I don’t think anyone uses this approach in serious modelling or drawing software, but it’s kind of the benchmark against which other approaches can be compared. One key benefit, which I’ll discuss below, is that this approach gives you a single function across the entire length of the curve, making them naturally uniform. There are a few problems, however:

• They get uglier as you have more and more control points, since the degree of the polynomial rises along with the control point count
• More fatally, they wiggle. This is most easily seen in a picture:

Polynomial interpolation demonstrating "wiggle"

A much better family of functions (for geometric modeling purposes) are the Bézier curves. These very useful curves are used in many drawing programs. These curves don’t actually go through the control points, except for the first and last ones, but instead use the control points as “handles” (you can think of them as pulling the curve toward them with springs, rather than having the curve attached to them). Bézier curves exhibit the “convex hull property”, meaning (casually) that if you draw lines connecting the control points, the curve will be contained within those lines. In other words, they don’t have the overshoot problems of simple polynomial interpolation. The fact that they don’t go through the control points is disconcerting at first, but isn’t fundamentally a problem for many design and modeling tasks.

Here’s a Bézier curve through the same control points:

A Bézier curve through the same control points

By now you can probably guess what the nice properties of B-splines are. They act a lot like Bézier curves, having the convex hull property and interpolating only the first and last control points, with the key difference that a change to a control point will only change a limited portion of the curve. To accomplish this trick, B-splines are made up of a number of separate curve segments, rather than a single high-degree polynomial. These segments are cleverly constructed so that when they meet, both segments are curving the same way (their first derivatives are the same). In fact, B-splines have an extra parameter, k, which controls both the polynomial degree of these segments and how continuous they are when they meet (how many derivatives are equal at the intersection point). In common usage, k is 3 or 4, leading to either quadratic or cubic curve segments. (Lower values of k aren’t “curves” in the common sense; k=2 produces straight lines connecting the control points, and k=1 is the disjoint set of control points, not connected at all. Higher k values lead to a “stiffer” spline, as the requirement of equal second and third and fourth derivatives forces the curve closer and closer to a straight line.)

As an example, here’s a k=3 B-spline (each segment is a quadratic curve) through the same control points as the curves above. As before, the control points are shown as blue circles. Additionally, the segments are each drawn in different colors, and the boundaries between segments are marked by red squares:

A k=3 B-spline curve through the same control points

B-spline Curve with segments rendered from u=-0.1 to u=1.1

Notice that we have the nice non-wiggling behavior of the Bézier curve, combined with a much narrower range of influence for each control point. The control points only influence nearby segments, meaning that this curve “flattens out” much more quickly than the Bézier version.

Last post I promised the backstory behind the pretty pictures. As a fair warning, there are no pictures this time around, just lots of discussion of Linux monitoring tools and their integration.

As a second caveat, I’m not always positive whether a given tool is vital for the monitoring per se, or whether it was just something I used to set the stage and figure out what I could monitor. That being said, I’ll do my best to reconstruct the history, with the help of Aptitude logs and FSVS diffs.

A very brief plug for a package I like: fsvs is basically Subversion-for-config-files in my usage, though I understand it  can be used as a bare metal deployment system. Well worth looking into. (And I just discovered that apparently it stands for “Fast System VerSioning”. Huh. I always assumed the “FS” was for “File System”, but I admit I never thought much about the “VS”… shame they had to force the acronym with intercaps.)

There are three distinct layers to the software stack, if you follow the data from original collection all the way to final presentation. I’ll start with the data providers.

Sources for Health Data

For some of these I’ll link to the project’s home page, but in all cases these were installed from the Debian package repositories. I’m using the most recent versions for Debian Lenny, aka stable. If it’s relevant, you can check which version exactly I’m using via the Debian search page.

cpufrequtils

I used these to set up CPU frequency scaling policy; can’t remember if this is also used directly in data gathering. The key point here is using kernel freq management instead of a userspace daemon. I don’t think it necessarily matters from a monitoring point of view, but it’s much tidier and should be more efficient.

hddtemp

Monitors hard drive temperatures as reported by SMART-capable drives (otherwise known as “just about any SATA or IDE hard drive, these days). I can’t find a good homepage for this tool (it’s not the one at hddtemp.com, which appears to be Windows-based and a bit full of itself). As I recall, the main purpose (for this use) is to provide a network daemon (localhost only, thank you) which the data gathering can query on demand. I explicitly disable the periodic monitoring option, which would be redundant (and rolls my logs).

smartmontools

I use this package to enable SMART on all drives and monitor for catastrophic changes — in that case, it sends an e-mail directly to my phone. I also use it to schedule automatic self-tests of the drives (scattered across typical idle time). I don’t remember if this tool is required to make the pictures, but it’s a key part of the peace of mind story.

lm-sensors

(A confession is in order here. I accidentally lied to a co-worker about this one; I don’t actually use sensord at all. I did at one point, but it was just logging to syslog, which kept the drives awake at all times but wasn’t easy to skim for trends. Instead, I use a common monitoring package for all values I’m tracking, as described below. Sorry if I threw you off-track, Eric!)

This very useful package reads motherboard sensors for things like CPU and VRM temperatures, voltages, and fan speeds. There’s a command line tool for detecting and configuring your sensors (some config file editing is required, but it’s a pretty straightforward process). The sensord companion package can monitor for thresholds and log sensor values, but as noted above I prefer to use a common tool for that.

RRDWeather

This is a neat little tool for fetching weather data from weather.com’s web service and storing it in an RRD database. This tool is a little unlike the others, in that it’s directly storing samples, rather than providing a service which can be queried on demand.

The rest of the monitored values are either built into the kernel, or provided directly by the system being monitored (such as Apache’s server stats).

Data Storage

The key part here, and the one that got me started on all of this in the first place, is RRDTool. The RRD stands for “Round Robin Database”. It has no built-in data collection apparatus, nor any GUI or web-based support for graph creation and rendering (though I believe there’s a small example CGI script). Rather, it simply provides a command-line mechanism for creating databases, a command-line or programmatic interface for adding data samples, and a command-line tool to generate charts given a chart specification.

The neat thing about RRD is that the databases can cover long periods of time, with fine near-term temporal resolution, while only consuming a fixed amount of disk. In rough terms, the trick is to progressively degrade the resolution: store the last day or so at 5 second intervals, say, but the past week only at 1 minute intervals, and so on until far enough back you might only save only value per day. This is a very clever trick, and solves one pernicious problem of long-baseline logging: when the data gathering itself becomes the problem you’re trying to detect.

There are some interesting consequences of this scheme. The most important to the user is that you have to define, at database creation time, exactly how you want to combine values (to come up with a single value to represent a minute, starting with samples every five seconds). From an alternative point of view, you need to figure out what charts you want to draw, and be sure to save that data. Fortunately, you can specify multiple aggregations functions, but you pay for it in increased disk usage for a given time period. For example, you might decide that you only care about the maximum recorded temperature in a period; in that case the database will only store maximums once it starts coalescing. For something like voltages, where any deviation up or down is bad, you might go further and record average, min and max. (I imagine this could also make a nice “envelope” chart, showing all three at once.) The only real problem is that you can’t decide retroactively that you wish you were recording some other aggregate function, since it’s already thrown that information away.

A second consequence of this scheme is more subtle, and didn’t immediately occur to me. Simple graphs of these values will behave as expected. However, when you start getting cute and graphing calculated values (such as my stacked CPU utilization chart), you can see anomalies where the line breaks out of the bounds of the actual physical measurement. For example, the stacked line for idle plus system plus user time hit a whopping 300%, despite the theoretical limit of 200% (one hundred per “virtual core”). The problem is that I’m gathering maximums over a period of time, then stacking them together. If, at one point during the five minute period, idle was at 100% (er, I mean 200%), and at another time the kernel was using 200%, then each max would be 200% and the stacked line would be twice as high as it logically could be. Makes perfect sense when you think about it, but it took me longer to catch on to than I’d care to admit. I suspect that this effect would only show up for certain aggregation functions and certain combinations of them (maybe average would always sum to unity?), but I wouldn’t care to bet on it without doing more math than I’m prepared to do for this post.

Oh, one other thing: I posted exclusively PNGs in the “pretty pictures” post, to increase the odds that everyone would be able to see them correctly. The tool, however, is perfectly happy rendering to SVG, PDF, or EPS as well. This is good news if you want to annotate or print the resulting charts. I haven’t played with the other formats much, so I can’t speak to details like how the SVGs look under significant resizing.

Data Collection

As noted above, RRDTool provides data storage and rendering, but defers to others to actually provide the data. There’s a remarkably rich ecosystem here — of course you can code your own with the scripting language of your choice, but there are also tools for gathering data from a wide variety of sources. You can find a partial list at the “RRDWorld” page.

Most of the full systems (as opposed to task-specific utilities) are focused on large-scale monitoring; the kind of task where one computer is dedicated almost exclusively to data gathering and presentation, watching an enterprise LAN. Very nice, and presumably useful if you’re into that sort of thing, but muchly overkill for my needs. Also, these suites seem to mostly assume SNMP as a common monitoring language, so I’d need to bridge all those disparate data sources to that format.

A second obvious option would be writing a script to gather exactly the data I want. This would be a cool project, would let me talk to any data source with a reasonable textual interface, and would give me the satisfaction of doing it all myself (er, except for the data storage and rendering). The trouble is I’m just not up to it. I mean, I could write the script perfectly well, but I’d be cutting corners — I probably wouldn’t properly daemonize it, making startup and shutdown tricky, and the time I’d save by working in Perl would be returned with interest in the time I’d spend trying to get CPU utilization down. After all, this isn’t a run-once data-munging script; this would be polling multiple times a minute, potentially spawning quite a few processes each time to gather reports. What I really need is a dedicated program, ideally written in an efficient language (trading programmer time for CPU cycles), and even more ideally able to talk to most of these data sources in their native APIs (to avoid repeatedly launching the command line interfaces).

Fortunately for me, this program has already been written: collectd. At least at the time I made the choice, this was a dedicated, narrow-focus app, written in C, and emphasizing low impact for high frequency monitoring. (Since then they seem to have added an embedded Perl interpreter and gone to a more modular architecture, but so far the core values seem to have remained intact.) In the newer version in Lenny, all data gathering (and even the RRD backend) is plugin based. However, there are native plugins for all the sources listed above (as well as quite a few kernel and application-native sources), so I’m not running a boatload of Perl fragments every few seconds.

Configuration is slightly tricky: collectd.conf is an Apache-style semi-XML format, which took some getting used to. A few of the counters still don’t seem to do exactly what I want; in particular I haven’t found a setup for disk traffic monitoring that deals gracefully with my RAID5 and LVM partitioning scheme. However, on the whole I’ve been very happy with the tool, and I’m quite willing to believe that my remaining problems are user error.

Oh, one word of warning: collectd’s preferred RRD setup changed significantly in the upgrade from Debian Etch to Lenny. Unfortunately I missed this at the time, and when I finally caught on to it there wasn’t an easy way to merge my newly collected data and the historical baseline. I still have the old files archived, but this pointed up a particular risk in upgrading data collection tools when you’re trying to maintain a baseline. I’d probably be a lot more upset if I’d been looking at the loss of a year or so of data, so I’m grateful I learned the lesson at the cost of January through March. They were boring months anyway, at least from a server health point of view.

Graph Definition and Rendering

As may have been obvious from the discussion of calculated values, RRDTool has a pretty robust scheme for defining composite values. It’s a sort of funky RPN syntax, but that’s a nice reminder of old HP calculators. This actually happens inside a chart definition, where you can load data from one or more files, then combine the values as desired, before finally drawing various styles of stacked or filled lines. In my limited experience, the RPN syntax is manageable, but the overall file format is a bit of a handful (and begs for a templating engine for related charts). Since I needed a tool anyway to bridge graph requests into Apache (and ideally to manage little things like caching and updating them), I decided to try some web-based front ends.

I started out using Torrus (linked from the RRDWorld page cited above). The version in Debian Etch was SNMP-centric, and offered its own data gathering services, but I could disable all that and just point it at a directory tree of RRD files (which is what collectd produces). There was some Perl wrangling to produce the charts I wanted (since collectd tends to store one datapoint per file, and I wanted both CPUs on the same chart, for example), but the end result was quite workable. I got a hierarchical view of all available data sources, with auto-generated charts for each, as well as the couple of hand-tuned ones. Configuration was a little heavyweight (modifying Perl files as well as XML config files), but the end result was quite usable.

As you may guess from the past tense, we have since parted ways. When I upgraded to Debian Lenny, the new collectd hierachy wasn’t as close to what Torrus expected. Torrus had also changed; the “plain old directory tree” option was now barely supported, in favor of a streamlined workflow for directly querying your SNML sources. Wonderful if that’s what you’re doing, but in my case it’s not. So, faced with the option of proliferating scripts to try to glue these tools together, I looked for another solution.

That’s when I started using drraw. It follows the lead of RRDTool in being completely ignorant of the semantic domain: it’s simply (and wonderfully) a tool for creating and displaying charts generated from RRD datasets. I briefly missed the auto-generated charts from Torrus, but I found myself making much more data-dense charts as I set out to recreate them. Every single chart on my “dashboard” homepage is at least displaying multiple datasets, and all but one of them are using more or less complex calculated values as well. (The exception that comes to mind is the memory usage chart, which instead uses the RRD-native stacking functionality, as well as using base-1024 instead of base-1000 for scaling.)

So far, several months in, I’ve been very happy with drraw. Its URLs are a little ugly, and if it weren’t for FSVS I’d need a solution for backing up any important chart definitions, but the flexibility is great: it’s trivial to make ad hoc modifications to a chart to test a theory or reveal a trend, then abandon them or update the saved definition if they work out.

In Closing

I have the utmost respect for anyone who’s made it this far. I’d be happy to share configuration details with anyone who wants to try this approach (or a modified version); I’d also love to hear from anyone with better ideas. I certainly haven’t done an exhaustive survey of the available software; this is pretty much satisficing rather than optimizing. My standards are pretty high in terms of data fidelity, and at least moderate in terms of system overhead, but beyond that I wanted to get this monitoring up and running, then get back to my programming projects.

Next time I’ll try to introduce my latest long-running hobby project, a pure .Net (and Mono) implementation of the UPnP MediaServer protocol stack, and maybe sketch some of the twists and turns between having an interesting idea and actually having working software. Thanks for reading.

For a while now I’ve been meaning to post something about Calliope, my (Debian) Linux server at home. I’ve got a nice draft somewhere about all the useful services I’ve got running, why I chose the packages I did, and so forth. At the end of the day, though, the basic take-away is “pick some services you want to run, install them, and Google till you get them to work”. (I’ll probably post it someday, though perhaps not after giving it such a heart-warming buildup.)

Anyway, the more I think about it, the most distinctive part of Calliope isn’t the  actual services she provides, but rather the tools I’ve assembled for keeping an eye on her. This isn’t like a desktop box, where I’m sitting at the console or the machine is turned off. Instead, most of my interaction is indirect — listening to music on the PS3, or looking at files shared through Windows networking or Apache, or even just acquiring a network address through DHCP. The common thread in all these cases is that I’ll notice abrupt failure, but I’m not directly logged in to see notifications or status messages.

One traditional solution has always been e-mail. I send myself nightly notifications of backup success, mostly because it gives me a warm fuzzy feeling to know that my Subversion repository is safely in at least two places (and on a recent trip, I found it a surprisingly reassuring touch of home), but fundamentally I’m not prepared to page through long logs or status reports on my phone (where I read most of my e-mail). In fact, I’m not prepared to do that even on my desktop or netbook, unless I already know there’s a problem (and remember, theonly readily visible symptoms of problem are “slow” or “failed to connect”).

So to summarize, I’ve got an uninvolved admin staff (myself), who wants things to Just  Work, and doesn’t want to have to explain to his wife that the internet is down right now because he was  in the middle of a project when he got bored with it. Fortunately, I’ve got Debian-stable as a pretty damn rock solid baseline to build from, so most problems will be the result of misconfiguration, user error, or mechanical failure. (The last is also a challenge, since the box is out in the hallway with the cats’ litter box — not a lot of foot traffic to notice things like fan failure or hard drive Squeaks of Doom.) Oh, and because I don’t do Linux server admin for a day job, I’m not necessarily going to be able to distinguish between dire-sounding-but-routine conditions and actual symptoms, since I lack a good intuitive baseline.

Anyway, enough exploration of the problem space. (Well, never enough, but at some point requirements analysis turns into plain old griping.) I actually have a solution that’s working pretty well for me so far, and is probably the biggest difference between this box and previous Windows servers I’ve set up.

Dashboard Health Charts

At the beginning of the post, I showed the charts I see when I turn on my computer in the morning to check weather and news; I’ll repeat them here. These are abbreviated versions of the full dataset, just enough to give me a quick picture of anything that’s going drastically wrong.

(In real usage, these are a 2×2 grid, but that breaks the margins here.) From the top, we see percentage of free space on various hard drive volumes, CPU utilization, memory usage, and system temperatures, all over the last 28 hours.

These particular charts show a pretty boring day. You can see the CPU spike at midnight from cron jobs, and Friday evening when I was actually using the machine. I can’t remember off-hand the reason for the circa 6 am spike, but it’s not unexpected. Temperatures are drift a little higher than usual, but that’s echoing the outdoor temps (the lower grey line on that chart).

Some technically inclined readers may be thinking that 10% utilization is awfully high for an idle box; what’s not shown on this chart is that the CPU freq is governing itself down to 300 MHz when it’s idle. Sometime I need to get some empirical evidence on whether this is a better trade (for power saving) than keeping the freq up and having the box strictly idle more often. A better chart would probably multiple the utilization percentage by the CPU frequency scaling; maybe I’ll try that one of these days, but I suspect that the current setup has much the same effect as a logarithmic chart — pulling the “10% at 300 MHz” and “100% at 2.4 GHz” samples closer to the middle, which makes for finer relative detail, even as it destroys absolute comparability.

Anomalies

In summary, in the time it takes me to decide if I’m checking weather or CNN, I’ve already got a quick gut sense that Calliope is basically healthy and happy. For comparison, here’s a (somewhat simulated) chart of what I saw at on 6/8, or rather what I would have seen if I hadn’t been just back from a trip and catching up on my jetlag. Imagine that the bars are the same 5 minute-ish accuracy seen above; I’ll discuss the progressive degradation later. (Because I’m working quickly, this is also a full chart with legend, rather than the abbreviated version I see on the daily dashboard.)

Anyway, here we see an approximately 4 hour long spike in CPU utilization, at a time when I certainly wasn’t doing anything interactively. For a cross check, here’s the temperature chart for the same period:

Temperature Spike

(I’m somewhat tipping my hand about being able to drill into this information, what with the legends and all, but pretend you’re just seeing the dashboard version.) We see a significant spike in CPU temperature over the same period, as well as a small rise in overall system temperatures. I’ll refrain from posting the drive freespace and memory charts, since they didn’t really show any correlated activity. (Which in itself says something: this probably wasn’t a massive spam run or anything, since I would expect to see a lot of memory usage if a user process was taking up that kind of CPU time for four hours.)

The first time I saw this pattern, I was a little panicked. There was some good news, though:

• I knew this wasn’t routine, since I’d been seeing these charts for weeks and knew what to expect
• I had good data on the start and end of the spike, letting me go straight to the right part of the log files
• I had some idea of what hadn’t happened, based on the lack of signal in the disk and memory charts

To cut to the chase, it turns out that as configured on Debian, mdraid (the Linux RAID subsystem) does a full resynch of all RAID-ed drives on the first Sunday of every month. (Basically, this is reading the data and checksums from the three drives involved and making sure that they actually agree; sort of a sanity check that what’s ending up on disk is the same data that we’re sending.) Once I figured that out, I decided that I approved, and now I expect that spike every month. (In fact, I’d be concerned if it weren’t there.) Which is exactly the point — having an easily interpreted set of charts in my face every day, along with longitudinal data to provide a baseline, lets me get on with my life with some expectation that the server is doing the same.

Historical Baseline

As I say, I see these charts most days, which means I’m accumulating a baseline behind my eyes. However, Calliope also records longitudinal data for posterity. (As you saw above in the “RAID spike” charts, the temporal resolution progressively degrades; as promised, I’ll eventually talk about this.) The disk space charts are pretty boring over time; they mostly serve as a record of kernel upgrades and significant data transfers (like getting our wedding videos onto the RAID drive). CPU utilization is a little more interesting:

CPU Utilization - Past Month

The periodic nature of background nightly and weekly tasks is clearly visible, as is the periodicity of my own usage. You’ll note that peak usage is over the theoretical 200% cap (100% per CPU). This is because the chart is individually taking the max value of the various utilization types in the aggregation time span, then stacking them. It’s not technically correct, but it serves its purpose. I could plot averages or minima, either overlaid on the same chart or as separate charts, but then I’d probably have to use lines instead of the pretty bars (which I think help for the quick visual read).

For prettiness (at least if you’re into that sort of thing), it’s hard to beat the memory chart. There’s not much drama here, but that’s the good news: a long term growth in the red region would indicate a persistent memory leak, and I’d have to build some system to kill and restart the offended processes (or just reboot periodically).

Basically what you see in the memory chart is the reboot frequency, when everything resets to zero. Beyond that, the dark green is free-free, while the bright green is disk cache. So mostly you can see how quickly the system ends up reading four gig from disk that the kernel decides is worth caching. That white line down the middle is the result of missing samples; as I recall that’s when I shut the machine down to move it into the hallway, and incidentally hook it up to an uninterruptible power supply.

For the last of the “core” charts, here are two longer-term temperature graphs (past week and past month):

There’s a nice periodicity here. The grey ambient line is really outside temperature (I don’t remember if it’s at PIT or Allegheny County Airport), so it’s not local temp but it’s a decent proxy for general climate trends. Someday I’ll get an in-house temperature sensor (probably 1-wire, that stuff sounds pretty neat — I didn’t even know I had gardening automation needs!), but for the moment this is good enough. I think it would be possible to calculate the thermal “half-life” (I don’t know the formal term; I mean the notionally constant time for the difference between interior and exterior temperatures to be halved) of our house, or at least that hallway, based on the lateral displacement between exterior temps and their reflection in the case-internal temperatures. Maybe someday I’ll get around to it, and maybe offset the two data streams to align them. (Or get really fancy and strip out the correlation term, leaving normalized case internal temps… but that’s probably a bad idea because the magic smoke doesn’t care about whether it’s weather-caused or activity-caused.)

Other Datasets

So far I’ve talked a lot about those four charts. There’s a good reason; I chose them because they should reveal several classes of resource exhaustion problems, and serve as pretty good proxies for overall system activity. In fact, as a friend at work pointed out, the temperature data is probably a pretty good stealth- / spy-ware detection scheme: lots of software will try to hide from system logs, but at some level you can’t do anything with a compromised box without generating some heat (and it would be peculiarly paranoid piece of software that spent a lot of effort on spoofing the CPU thermo-sensors).

In any event, those are the charts I look at every day, and they tell me a lot. However, they are not the only datasets I’m gathering. A few examples:

DNS Queries and Results - 1 Week

This chart shows DNS (Domain Name Service — mapping names like “www.cnn.com” to a numeric address) activity and success or failure. I mostly keep this one around as a monitor not on Calliope itself, but rather on the rest of the network. Many types of malware will at some point have to do some name resolution, and some will result in an absolute flood of DNS queries. Likewise, there’s a chance that this would detect someone connecting to my wireless network and piggybacking on my internet connection. Right now all looks quiet; you can see pretty clearly that I do most of my computer work in the middle of the evening.

I’ve also got similar charts for disk and network adapter activity, but they’re remarkably uninteresting — with at most two users connecting, and very little cache contention, they only register above threshold values if I’m streaming video or the like. The data gathering tool I use to collect ambient temperature information is hitting weather.com, and incidentally collects wind speed, humidity and barometric pressure; vaguely interesting, but nothing that’s not already available on the web. At various times I’ve gathered stats on Apache utilization, but again the load is mostly too low to even show up on the default scales (“queries per second” is a little ambitious when you hit a site maybe 50 times a day).

There is one important set of health stats which I record, even though I don’t look at them too often: the reported motherboard voltages of the various power supply lines. This is a great example of why gathering longitudinal data is so important; honestly, I couldn’t tell you what level of deviation I’d expect on the 1.5V line (and when I look at instantaneous stats on my desktop using SpeedFan, all I can say is “yeah, that’s kind of near 1.5V…”). However, with a baseline from when the computer was known to be working, I can pretty easily spot changes in the trend or disturbing variations. Here are three such charts. First, the latest 28 hour window:

To get everything on the same chart, I’ve roughly normalized the voltages (by subtracting off the nominal voltage). Some of these are so stably off-center that I suspect my correction terms may be wrong (for example, is Vcore really 1.5 volts or is it perhaps supposed to be 1.525V?), but I’m really just trying to get them all in the same area so I can compare deltas. (Similarly, one could argue that I should be showing percentage deviation rather than absolute, but that would require me to actually make sure what the design-spec voltages really are.)

My primary take-away from this chart is that the jitter is really quite small in absolute terms; the 12V line is noisiest over time (probably partly due to varying fan loads), and even then it’s only drifting in a range of 0.022V.

To see the correlation with the other data, and see if these values change under load, let’s examine that same 28 hour interval with the RAID sync that we looked at above:

Motherboard Voltages - RAID Sync time period

Wow, that’s a nice correlation with the temperature and CPU load charts. The 12V line drops most in absolute terms, probably driving the fans, but we can see that everyone took a hit. Not out of spec, I don’t think, but enough to make me wonder if in the long term I might want a little more power supply now that I’ve jammed this box full of disk drives.

For the final voltage chart, let’s look at all data available (four months worth):

Motherboard Voltages - 4 month period

Okay, there’s a pretty clear change in the middle of this one. Fortunately, I think I know the reason; recall the Memory Utilization chart for the same period, when I mentioned that I’d moved the box and also put it on a UPS? By my guess, this is a pretty graphic depiction of the benefits of active voltage regulation rather than running off raw wall power. (I think it also moved to a different circuit, so it’s possible that this is really showing the benefits of not being on the same circuit as the microwave oven. I guess I could have controlled for that, testing with the UPS on the same circuit first, but we really wanted the server out of the living room.) Again we see the benefits of historical data: I couldn’t have told you what an acceptable or expected level of jitter was on the +5V line, but I can surely see that it has stabilized a great deal.

Coming Attractions

I’m realizing that this post has gotten a lot longer than I originally thought, so I’m going to break my promise. I won’t discuss the implementation of this (although there are some strong hints in the watermarking on the charts); instead I’ll use that as incentive to post sooner than six months from now. I hope this has been somewhat interesting, and perhaps suggested some ways to get more value out of self-monitoring stats than taking a quick look at SpeedFan and saying “Yup, that’s a reported voltage all right!”

Several months ago, my Vista box stopped being able to go to sleep. It tries to, everything spins down for a moment, but then it immediately restarts and comes back up in the “just awakened” state. I can even hear it happening from another room (when the machine tries to sleep automatically), because the fans go full-on for a moment (under BIOS control) before Windows takes over and they idle back to their normal quiet state. (I’ve got a new case, the Antec 900, which I love — vastly quieter than my old server tower, which had five 80mm fans and sounded like a jet taking off, all the time — but it’s pretty loud when that big ol’ fan on top spins up to full speed and the two 120s over the drives max out.)

Anyway, I just ignored it for a long time, since the common solutions from Google didn’t help, and I couldn’t figure out what was triggering it. I tried the obvious suspects, like disabling Wake On LAN (though I actually kind of want Wake On LAN, but it was worth a try), but they didn’t help. Looking through Event Viewer didn’t give any hint as to why the machine had woken up; as far as I can tell, Vista thinks it’s doing what I told it to, so isn’t logging it.

But then a comment on a blog led me to the “powercfg” command line tool. Apparently this was shipped in XP and 2003, but at that point it appears to have only supported power scheme management. However, the Vista version has a -lastwake option, which “Reports information about the event that woke the computer from the last sleep transition.” Exactly the information I was missing! So it turned out that, for some reason, my keyboard (a wireless USB Logitech job) was waking the machine up. Disabling the keyboard’s ability to wake up the computer (via Device Manager) was sufficient to let it go to sleep and stay there.

This is still a somewhat unfortunate solution, since I would kind of like to be able to wake the computer up without reaching down for the power button, but given the choice I’ll take being able to sleep at all. I’m also losing network connectivity on wake, which is a well-known problem with nVidia chipsets. But I’m going to bask in the newfound sleepiness of my machine for a few days before attacking that one.

Segmented Cylinders

Last post, I’d gotten a unit cylinder to render halfway decently, after some compromises (and throwing more triangles at the thing). After that, I had to translate and rotate them where I wanted them. That should have been a trivial step (“I have a unit vector along the x axis, please map it over here”), but turned out to be more trouble than it should have been. (I won’t tell the full story, but here’s a quick runthrough.) The Matrix.CreateWorld method looked very promising indeed, but kept squashing out any z component (for reasons which became clear later; at the time I was passing the z unit vector as my “up vector”). After some hunting, and reading far more than I ever wanted to about quaternions, I fell back to a simpler approach: I want to overlay the 0-1 line on the x axis onto some line p0 to p1. Clearly, I need to translate by p0 and scale by |p1-p0| (that’s supposed to be the length of the line, if my pseudo-typography isn’t coming through). In almost all cases, I also need to rotate; from basic vector math, the axis of rotation will be the cross product i × (p1-p0) (I can’t get the little hat over the i, but that’s the unit vector in the x direction). The amount of the rotation will be the angle between the vectors: acos(i • (p1-p0)). (I’m particularly fond of UK-based sites, since they tend to refer to “maths”. Juvenile, I know.) This all works fine except for the case when p0 and p1 actually lie on the x axis already; in that case, the cross product goes to all zeros, which was what was flattening out my matrices from CreateWorld (at least I think so; I never went back to test the theory). So I ended up with a scale, a conditional rotation, and then a translation. I was sad about the conditional — even though modern shaders can handle conditionals, it still feels like a mark of shame. There’s probably some way to be clever about this; after all, any axis of rotation will do just fine if you’re rotating zero degrees. But I was in a hurry to get something rendering again, so I gave up for the moment.

Anyway, this was enough to get me from oddly-lit unit cylinders all the way up to this:

Sandworm: Curve as discrete cylinder segments

In this case, I’m rendering the first Hermite basis function, since I had code lying around to calculate it. With 20 samples, the curve looks quite smooth (all things considered), but it’s discontinuous, and gets into worse trouble near the ends. (In areas of high curvature, the abruptly changing normals at lines of overlap look distinctly articulated, rather than like a smooth curve.) So the next step was to get at least C0 continuity, and try to do something about the normals.

Beveled Cylinders

At this point I was on a roll with matrices. Nothing feels as much like a triumph as finally succeeding at a seemingly trivial task that took you all Thanksgiving afternoon (between moments of wincing at the Lions). I decided I wanted the cylinders to join along the plane which passes through the actual endpoint p_n, and whose normal is the average of the incoming and outgoing vectors, as seen in the following (atrocious) Paint diagram:

The incoming and outgoing position vectors determine the plane

The plan is to virtually slice the cylinders along that plane. (Note that the average of the incoming and outgoing position vectors was chosen so that both cylinders would be sliced at the same angle; otherwise the diagonal sections wouldn’t match up.) This means that approximately half the vertices will be placed farther along the position vector than they were in the sandworm version, and half will fall short of their positions there. This approach preserves the cross-section of the cylinder — at any point, a cut normal to the long axis of the cylinder would show a perfect circle or a segment of one — but potentially allows arbitrary position error at the endpoints. In particular, if two segments meet at a very narrow angle, the outermost points may have to be projected a very long way in space to get to that common plane. (There are alternative strategies, as I consider later on; first I just wanted to get one of them to work. Besides, meeting at such a narrow angle is a pretty good indicator of insufficient tessellation in the first place; either it is just a nasty curve, or we’re sampling too sparsely in an area of high curvature.)

I should stop at this point to note the obvious: I could quite simply guarantee C(0) continuity by emitting multiple cylinder segments as a continuous strip, rather than transforming a single cylinder to a series of locations. I didn’t want to start with that approach, however, because I wanted to have a good justification for why both segments should share those endpoints, rather than simply force majeure. I also liked being able to play with different matrices for each end of the cylinder, which would have been trickier if they were all passed in a single batch. (Not insurmountably so, but I wanted to focus on the geometry, not on instancing techniques.)

Anyway, I had a clear description of where I wanted the cylinder endpoints to end up. Now the trick was to get them there. The two ends are necessarily independent, since the meeting angles at opposite ends of a segment have nothing naturally in common (although now that I think about it, that would be an interesting tessellation heuristic; even then it would probably be an upper limit, rather than a fixed value). I’d just been reading about geometry instancing, trying to figure out how I was going to deal with tens of thousands of these cylinders running around, so the natural approach was to pass an array or arrays of shader parameters. The x coordinates of the cylinder vertices are all 0 or 1, so I can use them to index the arrays, and then be freely able to specify as many parameters as I need for each end independently. (I was a little surprised at using a floating point coordinate as an array index, but it works fine, and seems to be the standard approach.)

So now the ends of the cylinders are decoupled. I’m trying to project the vertices of each end onto a (different) plane, so the natural thing to do is to pass an array of two matrices, one for each end. Matrices are good at projections, after all. Trouble is, the projection that comes to mind would all project the points along the normal to the plane of projection — an orthographic projection operates this way. But I want to project along the cylinder and intersect the plane wherever that happens, rather than taking the shortest path. (Note that projecting along the plane’s normal would end up distorting the cross section of the former “cylinder”; in most cases I think it would turn into an extruded ellipse instead of a circle.) But just as I started trying to work out the math from scratch (and old linear algebra texts), something made me look at the static constructors available on the Matrix class again. Lo and behold, there’s the CreateShadow method: “Creates a Matrix that flattens geometry into a specified Plane as if casting a shadow from a specified light source.” Using the long axis of the cylinder as my “light source” direction, and the plane described above as the projection plane, this fits the bill perfectly! (There are some follow-on problems, but first pretty pictures from when I was still happy this was working.)

A smoothly-connected curve with 10 samples

This looks pretty good for only 10 samples. In a larger version you can see the specular highlight “bleed” a little along the knuckle of the vertex in the bottom right, but this is at least partly due to the abrupt change in the angle per pixel rate, as I talked about in my previous post.

Here is a wireframe version, where you can quite clearly see the endcaps of the cylinders being extended and contracted around the sample points:

Wireframe cylinders, showing effects of projection matrix

All is not altogether wine and roses, however. There are fundamental problems when the local radius of the curve is tighter than the radius of the cylinder: the “tube” starts to overlap itself and develop strange buckles and creases. There also seems to be an additional problem, because I’m not sure that that by itself would explain all the oddities in this badly-formed sine wave:

Sine wave showing self-intersection, positional error

Sad to say, this test also demonstrates the problems of arbitrary positional error which I mentioned earlier; the massive “hoods” at either end don’t do a great job of reflecting the shape of the curve (and have strangely messed-up normals). I think part of the problem here is that I locked the free ends of each curve (where there’s only one segment, hence no average plane) to the vertical axis. This looks very pretty on curves like the Hermite function above, and makes for good screenshots of the [0..1] region, but I suspect from this picture that I’d do better to use the one segment I do have, rather than an arbitrary fixed value.

An Afterword: Normals

An issue with this approach is how it handles normals. I intentionally used the average of the as-tessellated position vectors, not the analytical tangent, to choose the plane which is the locus of the endpoints. The goal was to improve quality when the as-sampled and analytical tangents diverge. First, consider a sine wave which is only sampled at the ±1 points. In each case, the analytical tangent is zero. However, we necessarily have to travel vertically to get to the next sampled point. Using the analytical tangent would tend to produce a shearing effect on the cylinder, while the as-sampled approach gives a nicely angled corner. As a second example, consider the same sine wave, but only sampled when it crosses the axis. Every sample is zero, but using the analytical tangent would have the seams slanting back and forth to no particular gain.

The trouble, of course, is that my normals are getting flattened into the same plane as the vertices; this effectively gives me OpenGL-style “computed normals” as the average of the adjoining segments. (Note that I’m not 100% sure where my normals are actually ending up; this description agrees with what they look like, but I’ve been burned by that before. The point of confusion is what effect I’m producing by transforming my (non-homogenous) normal vectors by the upper-left 3×3 matrix, after multiplying in the shadow matrix. It sure looks like they’re getting projected into the plane, but I don’t want to swear to it.)

To go a step further, I would want to calculate how much the analytical tangent at the point deviates from the as-sampled tangent, and use that correction factor as the basis for an additional rotation of the normals (potentially rotating back out of the shared plane). I have not yet pursued this approach because the whole strategy is assuming the availability of the previous and next point in the tessellation, and I haven’t yet decided how well that will scale to serious instancing. (I did play with it, passing the un-shadow-matrixed translate/scale/rotate matrix as a third matrix in the array, and got some tidy-looking curves looking like they were made from segments of PVC pipe, but that wasn’t really the goal.)

However, before I get into trying to solve that puzzle, I want better control over what I’m doing. I should probably shell out the $99 for nVidia’s shader debugger, but I’m much more interested in being able to write my shader code in F#, to evaluate on either the GPU or the host processor. I suspect that this exercise will be a good chance to get back to my functional programming roots, give me a pretty deep understanding of F# mechanics, and also serve as a good introduction to the nuances of shader evaluation.