Jekyll2022-11-02T14:56:49+00:00https://427.org.uk/feed.xml427.org.ukRandom musings and out of date thingsBen BrownPure Data Objects Written in C2021-01-24T00:00:00+00:002021-01-24T00:00:00+00:00https://427.org.uk/2021/01/24/pure-data-objects-written-in-c<p>I’ve been using <a href="https://puredata.info/">Pure Data</a> for a month or two now to spice up the visuals when I stream live, and I’ve found it pretty fun but occasionally frustrating. Most of my frustration has been around finding objects (or combinations of objects) that do what I want to do. For whatever reason I’ve strugged to find specific things online so there’s a good chance that what I’ve done in the rest of this blog has been for naught, other than a learning experience for me.</p>
<p>The main thing I wanted was something I could use to switch between scenes I have set up. Each one has one or more <code class="highlighter-rouge">gemhead</code> objects that can be turned on or off. I had rigged something up using an <code class="highlighter-rouge">hradio</code> object and a <code class="highlighter-rouge">select</code>. Each outlet of the <code class="highlighter-rouge">select</code> outputs then went to a <code class="highlighter-rouge">1</code> message, and every other output went to a <code class="highlighter-rouge">0</code> one. Both of these messages then go in to a <code class="highlighter-rouge">toggle</code> object which is linked to all the <code class="highlighter-rouge">gemhead</code> objects for that scene. This gets very cumbersome, for example I have seven scenes I want to use, each one of these needs a <code class="highlighter-rouge">0</code> and <code class="highlighter-rouge">1</code> message, and each <code class="highlighter-rouge">0</code> message has to be connected to every <code class="highlighter-rouge">select</code> output except the one that corresponds to that scene. Here is a picture of the patch:</p>
<p><img src="/assets/pd_select8_before.png" alt="A Pure Data patch with a lot of spaghetti connections looking generally messy" /></p>
<p>Yesterday I had a look at this excellent git repo, <a href="https://github.com/pure-data/externals-howto">HOWTO write an External for Pure Data</a>, and saw that it would be pretty simple to write an object in C, which I then did. And then I wrote a few more.</p>
<p>The object I created to help me with this problem is one I’ve called <code class="highlighter-rouge">select8</code>. It takes the place of the <code class="highlighter-rouge">select</code> object in this patch and has 8 outlets. Maybe one day I’ll change it to use a creation argument for the number of outlets but 8 will do for now. What it does is pretty simple. It takes in a float (such as that provided by the <code class="highlighter-rouge">hradio</code> object) and sends a <code class="highlighter-rouge">1</code> out of the corresponding outlet. The important change from the built in <code class="highlighter-rouge">select</code> output is that it also sends a <code class="highlighter-rouge">0</code> out of all the <em>other</em> outlets. This means that not only do I not need to have a <code class="highlighter-rouge">0</code> and a <code class="highlighter-rouge">1</code> message for every scene, I don’t need to connect the other outlets to each <code class="highlighter-rouge">0</code>. This has cleaned it up a lot, as you can see from the screenshot below:</p>
<p><img src="/assets/pd_select8_after.png" alt="A Pure Data patch which is much cleaner looking than the previous picture, though still with room for improvement" /></p>
<p>Now that I’d got bitten by the bug, I wrote some more objects. The next one was <code class="highlighter-rouge">switch8</code>, which I wanted to use with <code class="highlighter-rouge">pix_video</code>. You can only have one <code class="highlighter-rouge">pix_video</code> object for a given device and I wanted to be able to route it through different render chains, so this is what I use. It takes anything at all in to it’s first inlet, and sends it out of one of the eight outlets. Which outlet is in use is determined by a float you send to the second inlet. Like <code class="highlighter-rouge">select8</code> the number of outlets is fixed, maybe a future version will be more flexible but that’s more than enough for me right now.</p>
<p>I’ve also written <code class="highlighter-rouge">onchange</code> and <code class="highlighter-rouge">on1</code>. These both take in floats, and emit bangs. <code class="highlighter-rouge">onchange</code> emits a bang when the input value is changed, <code class="highlighter-rouge">on1</code> when the input value is just <code class="highlighter-rouge">1</code>. I wrote these because of a side-effect I noticed after I modified <code class="highlighter-rouge">select8</code> to advance to the next outlet when it receives a bang. I discovered that despite me using the explicit <code class="highlighter-rouge">outlet_float</code> function that would also be picked up as a bang, so I needed a way to filter these. I initially wrote <code class="highlighter-rouge">onchange</code> which almost did what I wanted, but it was really <code class="highlighter-rouge">on1</code> that was the final piece of the puzzle.</p>
<p>That’s all I written for now, I’m sure I’ll write more in future. I’ve made them all freely available under the Clear BSD License over on gitlab: <a href="https://gitlab.com/benofbrown/pd-objects/">https://gitlab.com/benofbrown/pd-objects/</a></p>Ben BrownI’ve been using Pure Data for a month or two now to spice up the visuals when I stream live, and I’ve found it pretty fun but occasionally frustrating. Most of my frustration has been around finding objects (or combinations of objects) that do what I want to do. For whatever reason I’ve strugged to find specific things online so there’s a good chance that what I’ve done in the rest of this blog has been for naught, other than a learning experience for me.Streaming Pure Data GEM Video Via OBS2020-12-28T00:00:00+00:002020-12-28T00:00:00+00:00https://427.org.uk/2020/12/28/streaming-puredata-gem-video-via-obs-on-linux<p>Recently I’ve been doing a bit of streaming with my modular synth and wanted something to make the video side a bit more interesting for anyone watching it. After a bit of searching I discovered <a href="https://puredata.info/">Pure Data</a> (aka <em>Pd</em>), and the <a href="https://gem.iem.at/">GEM plugin</a> for it, which adds graphical elements you can create and manipulate in Pd, so I thought I’d give it a go.</p>
<p>Getting audio in was pretty simple, I already use <a href="https://jackaudio.org/">JACK</a> for my audio routing, so all I had to do was configure Pd to use JACK as well, and connect everything up in <code class="highlighter-rouge">qjackctl</code>. Then in my patch I created an <code class="highlighter-rouge">adc~</code> object which I could then route the left and right channels from my mixer out from.</p>
<p>After getting the <code class="highlighter-rouge">gem</code> package installed and tweaking the settings of Pd to actually load it I managed to get a quick demo up and running fairly quickly and gazed in wonder at the cube spinning in front of me. And then I hit my first problem. I stream at 1280x720 resolution, so wanted to set the GEM window to that size. Looking it up I saw that I could send it a <code class="highlighter-rouge">dimen 1280 720</code> message before creating it and that would change the size of the window. That worked really well, until I tried to make my cube bigger, and straight away I noticed that no matter what I did it would get clipped out where the old window was. So although the window was bigger, I couldn’t put anything in the new space.</p>
<p>This took quite a long time to figure out, and in fact I never did figure it out. I ended up having trouble getting the vanilla Pd distributed with Debian to run with the plugins you need to be equivalent to Pd-extended and ended up installing something else entirely, <a href="https://puredata.info/downloads/purr-data">Purr Data</a>. Not only did it give me access to the objects I needed, it also fixed the annoying window problem, so I’d say <strong>Step 1: Install Purr Data</strong>.</p>
<p>Now I had a few cubes spinning around in the GEM window, I wanted to get those to show in <a href="https://obsproject.com/">OBS Studio</a>, which is a really excellent program you can use to combine various audio and video sources in to a stream which you can push to YouTube, Twitch etc etc etc. My first attempt was to use the XWindow capture source in OBS. This worked, but not very well. I don’t know if it’s my window manager or just X11 but it was very flickery, which is no good at all. The next part of this blog is the whole reason I’m writing it, because it took me a stupidly long time to figure out and get working.</p>
<p>So far I’ve been enjoying Pd, but I’ve found resources online a bit lacking. It turns out the help in Pd (or at least Purr Data) itself is pretty good, though looking for things online didn’t really give me much joy, though I did find the <a href="https://github.com/umlaeute/v4l2loopback">v4l2loopback</a> Linux kernel module. If you’re using Debian like me, then you can get this easily by installing the <code class="highlighter-rouge">v4l2loopback-dkms</code> package. I then loaded the module and set it up to create a device at <code class="highlighter-rouge">/dev/video10</code>. After some failed attempts I found this combination of options to work best:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo modprobe v4l2loopback devices=1 video_nr=10 card_label="OBS Cam"
</code></pre></div></div>
<p>Now, the label bit was copy and pasted from the <code class="highlighter-rouge">v4l2loopback</code> wiki, though the example there is using it as a destination for OBS rather than a source, so although the label isn’t strictly accurate for what we’re doing my OBS scenes are now configured to use it so I’m not changing it now.</p>
<p>To add it as a source in OBS it’s really very simple, you just add a <code class="highlighter-rouge">Video Capture Device (V4L2)</code> source and choose <code class="highlighter-rouge">OBS Cam</code> in the drop-down it comes up with. Now all we need to do is get GEM to send the video output to it.</p>
<p>Again I didn’t find that much info about doing this online, though <a href="https://lists.puredata.info/pipermail/pd-list/2015-04/109878.html">a thread on the Pure Data Mailing List</a> pointed me in the right direction.</p>
<p>The first thing I learnt is that the way to send the video to a <code class="highlighter-rouge">v4l2loopback</code> device is to use the <code class="highlighter-rouge">pix_record</code> object with the file to record to set to the device our <code class="highlighter-rouge">modprobe</code> command created earlier, <code class="highlighter-rouge">/dev/video10</code>. This object however expects a picture, not an object like our cube. So to get video of our cube to be sent via <code class="highlighter-rouge">pix_record</code> we need to render each frame as a picture, and map that as a texture on to another object, which is then sent to <code class="highlighter-rouge">pix_record</code>. To do this we use <code class="highlighter-rouge">pix_snap</code> to take a snapshot of what we’ve rendered, then <code class="highlighter-rouge">pix_texture</code> to use it as a texture, before finishing with a <code class="highlighter-rouge">rectangle</code> object. In order to preserve the aspect ratio of the original frame this needs to be <code class="highlighter-rouge">rectangle 16 9</code>. I translate this object so it’s outside of the view in the Gem window using <code class="highlighter-rouge">translate 1 0 0 3</code>, though you can leave it where it is if you want, it just looks a little odd. Finally this is sent to the <code class="highlighter-rouge">pix_record</code> object. We then send messages to that object to start and stop recording.</p>
<p>If anyone wants to use this method, I’ve attached an example patch <a href="/assets/gem-record-example.pd">gem-record-example.pd</a> that should get you up and running, here’s the gist of it:</p>
<ul>
<li>Install <a href="https://puredata.info/downloads/purr-data">Purr Data</a></li>
<li>Install <a href="https://github.com/umlaeute/v4l2loopback">v4l2loopback</a> - the <code class="highlighter-rouge">v4l2loopback-dkms</code> package on Debian derived distros</li>
<li>Create a v4l2loopback device</li>
<li>Make your scene in Purr Data with <code class="highlighter-rouge">gemwin</code>, <code class="highlighter-rouge">gemhead</code> etc</li>
<li>Add another <code class="highlighter-rouge">gemhead</code> with a higher priority number</li>
<li>Add a <code class="highlighter-rouge">trigger a b</code> to that, and connect both outouts to <code class="highlighter-rouge">pix_snap</code> set to capture the whole window <em>n.b. the docs say it defaults to doing this but that didn’t seem to work for me</em></li>
<li>Add a <code class="highlighter-rouge">pix_texture</code> to that, then <code class="highlighter-rouge">translate 1 0 0 3</code>, then a <code class="highlighter-rouge">rectangle 19 6</code>. This is where the capture will be applied as a texture, it doesn’t need to be visible in the window, which is what the <code class="highlighter-rouge">translate</code> is for.</li>
<li>Add the final object, <code class="highlighter-rouge">pix_record</code> to the end of that chain. To start sending video to the device send the message <code class="highlighter-rouge">codec v4l2, file /dev/video10, auto 1, record 1</code>. To stop sending it send <code class="highlighter-rouge">record 0</code>.</li>
<li>Open OBS, and add a <code class="highlighter-rouge">Video Capture Device (V4L2)</code> source and choose <code class="highlighter-rouge">OBS Cam</code>.</li>
<li>Use OBS as normal, you should now be able to see the scene you creted in Gem, without flickering.</li>
</ul>Ben BrownRecently I’ve been doing a bit of streaming with my modular synth and wanted something to make the video side a bit more interesting for anyone watching it. After a bit of searching I discovered Pure Data (aka Pd), and the GEM plugin for it, which adds graphical elements you can create and manipulate in Pd, so I thought I’d give it a go.In C, in C - Part III2020-09-06T00:00:00+00:002020-09-06T00:00:00+00:00https://427.org.uk/2020/09/06/in-c-in-c-part-3<p>In <a href="/2020/08/15/in-c-in-c.html">Part I</a> I set about writing a program in C to perform Terry Riley’s piece “In C”. In <a href="/2020/08/23/in-c-in-c-part-2.html">Part II</a> I added some basic envelope generation so I could play distinct notes, however strictly speaking I could only play <em>one</em> distinct note by the end of it. In this latest session I’ve worked on playing multiple notes.</p>
<h2 id="slight-refactor">Slight refactor</h2>
<p>Before I did that I addressed something I wasn’t happy with. I didn’t like the name I’d given some of the structs, particularly the <code class="highlighter-rouge">note</code> and <code class="highlighter-rouge">notes</code> structs. <code class="highlighter-rouge">note</code> wasn’t a complete note, but a component that would be built up to form what played a note, and <code class="highlighter-rouge">notes</code> didn’t hold a series of musical notes as you might expect. So I renamed <code class="highlighter-rouge">note</code> to <code class="highlighter-rouge">timbre</code>, and <code class="highlighter-rouge">notes</code> to <code class="highlighter-rouge">instrument</code>, as I think that makes more sense. An instrument is made up of timbres and plays notes. That means I now needed an actual <code class="highlighter-rouge">note</code> struct to define such a note, and a new struct I’ve called <code class="highlighter-rouge">phrase</code> to contain multiple notes. They’re all pretty simple:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>struct timbre {
float freq_ratio;
float amp;
int (*func)(float, int, const ao_sample_format *);
};
struct instrument {
int total;
struct timbre *timbres;
float current_amp;
float (*attack)(float);
float (*release)(float);
};
struct note {
float freq;
float sustain_level;
int sustain_time;
enum note_state state;
};
struct phrase {
int current;
int total;
struct note *notes;
};
</code></pre></div></div>
<p>Keen eyed observers might notice that some items that were in <code class="highlighter-rouge">notes</code> before it became <code class="highlighter-rouge">instrument</code> have not been moved over to the <code class="highlighter-rouge">instrument</code> struct, as they are now more relevant to the note, these are <code class="highlighter-rouge">freq</code>, <code class="highlighter-rouge">sustain_level</code>, <code class="highlighter-rouge">sustain_time</code>, and <code class="highlighter-rouge">state</code>.</p>
<p>I’ve also simplified the attack and release functions to just reference the current amplitude, I don’t think I’ll need more than that.</p>
<h2 id="parts">Parts</h2>
<p>The next thing I’ve done is create a very basic struct, <code class="highlighter-rouge">part</code>, which describes a musical part. This ties together an <code class="highlighter-rouge">instrument</code> to the <code class="highlighter-rouge">phrase</code> it will play:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>struct part {
struct phrase *phrase;
struct instrument *instrument;
};
</code></pre></div></div>
<p>It’s this new struct that will replace the old <code class="highlighter-rouge">notes</code> one in the <code class="highlighter-rouge">state</code> struct that gets passed to both the threads.</p>
<h2 id="rendering-changes">Rendering changes</h2>
<p>The biggest change is in the <code class="highlighter-rouge">render_buffer</code> function, there’s also a slight tweak to the <code class="highlighter-rouge">render_instrument</code> function it calls as the frequency has been separated from the instrument and moved to the note information I’ve had to add it to the function prototype and reference it in the function directly.</p>
<p>Most of the changes in <code class="highlighter-rouge">render_buffer</code> itself is to change the references to the state, which was in the <code class="highlighter-rouge">notes</code> struct (now renamed <code class="highlighter-rouge">instrument</code> to reference the currently playing note, whihch is signified by the <code class="highlighter-rouge">current</code> index of the new <code class="highlighter-rouge">phrase</code> struct, which it accesses through the <code class="highlighter-rouge">part</code> struct that is now passed to it in place of <code class="highlighter-rouge">notes</code>. This was getting a bit out of hand so for convenience I added a couple of macros, <code class="highlighter-rouge">CUR_NOTE</code> and <code class="highlighter-rouge">CUR_STATE</code>, to signify the note we’re playing and its current status.</p>
<p>Next I just changed the handling at the end of the loop, so if there is another note in the <code class="highlighter-rouge">phrase</code> it will get played, if not we will signal to <code class="highlighter-rouge">play_buffers</code> that there wasn’t anything else for it to do. Here’s the function as it is currently, with those two defines in place:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#define CUR_NOTE part->phrase->notes[part->phrase->current]
#define CUR_STATE CUR_NOTE.state
void render_buffer(struct buffer *buffer, struct part *part, const ao_sample_format *format) {
int sample;
int generated = 0;
for (int i = 0; i < format->rate; i++) {
if (CUR_NOTE.sustain_level > 0) {
sample = render_instrument(part->instrument, CUR_NOTE.freq, i, format);
} else {
sample = 0;
}
switch (CUR_STATE) {
case wait:
CUR_STATE = attack;
/* fall through */
case attack:
if (part->instrument->attack == NULL) {
CUR_STATE = sustain;
break;
}
part->instrument->current_amp = part->instrument->attack(part->instrument->current_amp);
if (part->instrument->current_amp >= CUR_NOTE.sustain_level) {
/* compensate for overshoot */
part->instrument->current_amp = CUR_NOTE.sustain_level;
CUR_STATE = sustain;
}
break;
case sustain:
if (CUR_NOTE.sustain_time-- <= 0) {
CUR_STATE = release;
}
break;
case release:
if (part->instrument->release == NULL) {
CUR_STATE = finished;
break;
}
part->instrument->current_amp = part->instrument->release(part->instrument->current_amp);
if (part->instrument->current_amp <= 0.01) {
CUR_STATE = finished;
}
break;
case finished:
break;
}
sample *= part->instrument->current_amp;
if (sample > 32768) sample = 32768;
buffer->data[4 * i] = buffer->data[4 * i + 2] = sample & 0xff;
buffer->data[4 * i + 1] = buffer->data[4 * i + 3] = (sample >> 8) & 0xff;
generated += 4;
if (CUR_STATE == finished) {
if (part->phrase->current < part->phrase->total - 1) {
part->phrase->current++;
continue;
}
break;
}
}
buffer->generated = generated;
}
</code></pre></div></div>
<h2 id="adding-notes">Adding notes</h2>
<p>I’ve also added a convenience function to add notes to a phrase, simply called <code class="highlighter-rouge">add_note</code>. It looks like this:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>void add_note(struct phrase *phrase, float freq, float sustain_level, int sustain_time) {
phrase->total++;
phrase->notes = realloc(phrase->notes, phrase->total * sizeof(struct note));
if (!phrase->notes) {
perror("add_note: realloc");
exit(EXIT_FAILURE);
}
phrase->notes[phrase->total-1].state = wait;
phrase->notes[phrase->total-1].freq = freq;
phrase->notes[phrase->total-1].sustain_level = sustain_level;
phrase->notes[phrase->total-1].sustain_time = sustain_time;
}
</code></pre></div></div>
<p>And here it is in use with my current test:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>add_note(state.part->phrase, 440, 1.0, 10000);
add_note(state.part->phrase, 440, 0.0, 10000);
add_note(state.part->phrase, 220, 1.0, 10000);
</code></pre></div></div>
<p>I mentioned in the previous blog that rests would be easily implemented by using a note with 0 sustain, which is exactly what I’ve done here with the second note. So this program now plays two notes with a rest in between, all 10000 samples long (so roughly ¼ of a second).</p>
<h2 id="next-steps">Next steps</h2>
<p>I’m fairly happy with progress so far, but next I want to be able to play multiple phrases at a time, and also change from one phase to another. I think I’m going to have to tweak how the attack and release phases interact with the sustain time, otherwise I could easily end up with everything getting horribly out of time with each other, but I don’t want it to end up completely rigid, so I think I’ll just see how it turns out.</p>Ben BrownIn Part I I set about writing a program in C to perform Terry Riley’s piece “In C”. In Part II I added some basic envelope generation so I could play distinct notes, however strictly speaking I could only play one distinct note by the end of it. In this latest session I’ve worked on playing multiple notes.In C, in C - Part II2020-08-23T00:00:00+00:002020-08-23T00:00:00+00:00https://427.org.uk/2020/08/23/in-c-in-c-part-2<p>In <a href="/2020/08/15/in-c-in-c.html">Part I</a> I set about writing a program in C to perform Terry Riley’s piece “In C”. At the end of it I mentioned the need to change volume over time so I can play distinct notes, so that’s exactly what I’ve focussed on in the latest bit of coding I’ve done.</p>
<h2 id="envelopes">Envelopes</h2>
<p>In the synthesiser realm we use envelopes to change volume over time. There’s generally four parts of the envelope, the <em>Attack</em>, <em>Decay</em>, <em>Sustain</em> and <em>Release</em>. When you play a note on a keyboard for example, the envelope is triggered, which starts with the <em>attack</em> phase. This takes the volume from 0 up to it’s maximum level, and will either do this quickly or slowly depending on how it’s been set up. You could even skip the attack entirely and start the note at full volume. Next is <em>decay</em>, which decreases the volume over time, until it reaches the <em>sustain</em> level.</p>
<p>The note then stays at the <em>sustain</em> level until the key is released, which starts the <em>release</em> phase, which in turn reduces the volume over time. By tweaking these four parameters we can change the feel of the sound fairly dramatically.</p>
<p>So far I’ve simplified this a bit by skipping the <em>decay</em> phase, so I’m creating what’s known as an <em>ASR</em> envelope, or <em>Attack/Sustain/Release</em>.</p>
<h2 id="extending-the-notes-struct">Extending the notes struct</h2>
<p>In order to do this I’ve added some fields to the <code class="highlighter-rouge">notes</code> struct. Here’s how it was in the previous post:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>struct notes {
int total;
struct note *notes;
};
</code></pre></div></div>
<p>And here it is now:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>enum note_state {wait, attack, sustain, release, finished};
struct notes {
int total;
struct note *notes;
float freq;
enum note_state state;
float sustain_level;
int sustain_time;
float current_amp;
float (*attack)(float, int, const ao_sample_format *);
float (*release)(float, int, const ao_sample_format *);
};
</code></pre></div></div>
<p>Quite a lot more stuff added in there. I’ve also added an <code class="highlighter-rouge">enum</code>, <code class="highlighter-rouge">note_state</code>, which we’ll use to reference what phase the note is in, and the corresponding field <code class="highlighter-rouge">state</code> will be used to keep track of this.</p>
<p>Frequency was previously set on a per-<code class="highlighter-rouge">note</code> (badly named struct!) basis, I’ve changed this now to be set at this higher level, and have the <code class="highlighter-rouge">note</code> struct instead use a multiple of the frequency set in this expanded <code class="highlighter-rouge">notes</code> struct. It also used to be an <code class="highlighter-rouge">int</code>, but I’ve changed it to be a <code class="highlighter-rouge">float</code> now.</p>
<p><code class="highlighter-rouge">sustain_level</code> is the maximum amplitude this note will reach, with <code class="highlighter-rouge">1.0</code> being the loudest. <code class="highlighter-rouge">sustain_time</code> is how long the note will be held after the attack phase is complete. At the moment it’s in samples which isn’t that friendly to set, but makes it pretty easy to handle in the code.</p>
<p><code class="highlighter-rouge">current_amp</code> is for tracking the current volume of the note, so we know when to change to the next phase.</p>
<p><code class="highlighter-rouge">attack</code> and <code class="highlighter-rouge">release</code> are pointers to functions which are called in the <em>attack</em> and <em>release</em> phases respectively.</p>
<p>I’ve thrown together some quick attack and release functions to use for this proof of concept:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>float attack_linear(float current, int pos, const ao_sample_format *format) {
if (current == 0.0) {
current = 0.0001;
}
return current * 1.001;
}
float release_linear(float current, int pos, const ao_sample_format *format) {
return current / 1.0001;
}
</code></pre></div></div>
<p>It’s pretty basic stuff right now and don’t use the <code class="highlighter-rouge">pos</code> or <code class="highlighter-rouge">format</code> variables that are available to them - I imagine they will be useful in more complex functions though.</p>
<h2 id="how-it-works">How it works</h2>
<p>It’s pretty simple really, though I did get it wrong a few times before getting it working. When we render the buffer that will be played we check the status of the note. If it’s <em>wait</em> we change it to <em>attack</em>, then call the attack function and update the current amplitude, which we multiply the value we derive for that sample by. When the amplitude hits the sustain level we move to the <em>sustain</em> state, and start to decrement <code class="highlighter-rouge">sustain_level</code>. When that reaches 0 we go in to the <em>release</em> state and set the amplitude with the release function until the amplitude reaches 0 (or close enough to 0) when we move to <code class="highlighter-rouge">finished</code>. We then set the flag on the buffer to tell the <code class="highlighter-rouge">play_buffers</code> function that all the data has been processed and it can stop running after its current loop. In the finished program it will play more than one note, so will only set this flag when all notes have reached the <code class="highlighter-rouge">finished</code> state.</p>
<p>I’ve also added the ability to skip the <em>attack</em> or <em>release</em> phase by setting the function pointers to <code class="highlighter-rouge">NULL</code>.</p>
<p>The main change is in the <code class="highlighter-rouge">render_buffer</code> function, where I’ve added a switch statement to handle the changing of states, and to multiply the result of <code class="highlighter-rouge">render_notes</code> by the amplitude value:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sample = render_notes(notes, i, format);
switch (notes->state) {
case wait:
notes->state = attack;
/* fall through */
case attack:
if (notes->attack == NULL) {
notes->state = sustain;
break;
}
notes->current_amp = notes->attack(notes->current_amp, i, format);
if (notes->current_amp >= notes->sustain_level) {
/* compensate for overshoot */
notes->current_amp = notes->sustain_level;
notes->state = sustain;
}
break;
case sustain:
if (notes->sustain_time-- <= 0) {
notes->state = release;
}
break;
case release:
if (notes->release == NULL) {
notes->state = finished;
break;
}
notes->current_amp = notes->release(notes->current_amp, i, format);
if (notes->current_amp <= 0.01) {
notes->state = finished;
}
break;
case finished:
break;
}
sample *= notes->current_amp;
</code></pre></div></div>
<p>Another change is that the main loop in <code class="highlighter-rouge">handle_buffers</code> will now keep going until all the notes have finished rather than the arbitrary four iterations it did before:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> for (;;) {
pthread_mutex_lock(&mutex);
state->active_buffer ^= 1;
render_buffer(&(state->buffers[state->active_buffer]), &(state->notes), &(state->format));
if (state->notes.state == finished) {
state->buffers[state->active_buffer].last = true;
if (send_ready) pthread_cond_broadcast(&data_ready);
pthread_mutex_unlock(&mutex);
return NULL;
}
if (send_ready) {
pthread_cond_broadcast(&data_ready);
send_ready = false;
}
pthread_cond_wait(&read_ready, &mutex);
pthread_mutex_unlock(&mutex);
}
</code></pre></div></div>
<h2 id="next-steps">Next steps</h2>
<p>Now that I’ve figured out playing individual notes my next step will be sequencing them somehow. Rests should be easy, after all they’ll just be notes with a 0 sustain level, though I should probably do something to skip calling the note generation function in this instance to avoid wasting CPU cycles. Overall I’m more confident that I’ll get this finished than I was after the previous post, and I’ve enjoyed figuring this stuff out so far.</p>
<p><em>Update - See <a href="/2020/09/06/in-c-in-c-part-3.html">Part III</a></em></p>Ben BrownIn Part I I set about writing a program in C to perform Terry Riley’s piece “In C”. At the end of it I mentioned the need to change volume over time so I can play distinct notes, so that’s exactly what I’ve focussed on in the latest bit of coding I’ve done.In C, in C - Part I2020-08-15T00:00:00+00:002020-08-15T00:00:00+00:00https://427.org.uk/2020/08/15/in-c-in-c<p>Recently I’ve been listening to Terry Riley’s groundbreaking composition <a href="https://en.wikipedia.org/wiki/In_C">In C</a>. Riley wrote this piece in 1964, and since then there’s been many, many recordings of it.</p>
<p>It’s an unusual piece in that it consists of 53 short musical phrases, which performers play in order, repeating them as many times as they feel appropriate before moving on to the next one. The entire score fits on one side of A4 paper, but performances generally take between 45 minutes and an hour and a half.</p>
<p>Over the years there have been versions of it with many different instruments and a variety of ensembles, ranging from traditional acoustic instruments to electronic synthesisers and computer software.</p>
<p>Another of my hobbies is writing in the <a href="https://en.wikipedia.org/wiki/C_%28programming_language%29">C programming language</a>. So I got to thinking, “Why I don’t I try writing a program in C, that will perform In C”?</p>
<p>Now, I imagine I’m not the first person to try this, but it seems almost impossible to search for this on the internet and so far I’ve not found anything matching this. Also I want to do this as a bit of a challenge, so I don’t want to get inspiration from anyone else, other than basic usage of certain libraries.</p>
<h2 id="the-story-so-far">The story so far</h2>
<p>I had a quick look for a library that would enable me to make sounds from code written in C, and stumbled on <a href="https://www.xiph.org/ao/doc/">libao</a> which seems sufficiently low-level. Here’s a brief example snippet from their docs:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>for (i = 0; i < format.rate; i++) {
sample = (int)(0.75 * 32768.0 *
sin(2 * M_PI * freq * ((float) i/format.rate)));
/* Put the same stuff in left and right channel */
buffer[4*i] = buffer[4*i+2] = sample & 0xff;
buffer[4*i+1] = buffer[4*i+3] = (sample >> 8) & 0xff;
}
ao_play(device, buffer, buf_size);
</code></pre></div></div>
<p>This is populating a buffer with the samples required to reproduce a sine wave at the given frequency and then play it.</p>
<p>Sine waves on their own aren’t super interesting to listen to, but by combining them at different amplitudes you can get more interesting tones, which is known as <a href="https://en.wikipedia.org/wiki/Additive_synthesis">Additive Synthesis</a>. So the first thing I did after that was to make a way for these sine waves to be combined easily.</p>
<p>This is a pretty brute force approach but it works well so far. I have a struct which will define a note, another struct to hold a list of these notes, and a function to add a sine wave to this list of notes:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>struct note {
int freq;
float amp;
int (*func)(int, int, const ao_sample_format *);
};
struct notes {
int total;
struct note *notes;
};
int sine_wave(int freq, int pos, const ao_sample_format *format) {
return (int)(32768.0 * sin(2 * M_PI * freq * ((float) pos / format->rate)));
}
void add_sine(struct notes *notes, int freq, float amp) {
notes->total++;
notes->notes = realloc(notes->notes, notes->total * sizeof(struct note));
if (notes->notes == NULL) {
exit(EXIT_FAILURE);
}
notes->notes[notes->total-1].freq = freq;
notes->notes[notes->total-1].func = sine_wave;
notes->notes[notes->total-1].amp = amp;
}
</code></pre></div></div>
<p>The function <code class="highlighter-rouge">sine_wave</code> will be called many many times as we populate the buffer, <code class="highlighter-rouge">pos</code> here represents the sample number and at 44.1 kHz there’s a lot of them for each one second buffer. To combine these sine waves, we simply need to multiply the output of the <code class="highlighter-rouge">sine_wave</code> function with the amplitude for this note, then add these values for each note together, e.g.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>for (int n = 0; n < notes->total; n++) {
sample += notes->notes[n].amp * notes->notes[n].func(notes->notes[n].freq, pos, format);
}
</code></pre></div></div>
<p>Algorithms aren’t my strong suit so I’m hoping modern processing power will ensure I’ll always be able to generate that second of sounds before it needs to be played.</p>
<p>The next challenge I’ve had is to be able to run a function to populate the buffer at the same time the old buffer is playing. As <code class="highlighter-rouge">ao_play</code> blocks I’ve needed to use threading, which is something I struggled with in the past and was no exception here.</p>
<p>It’s taken me several attempts and a bit of head scratching but I’ve finally got there today.</p>
<p>First we launch the function that will play the buffers in a new thread:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pthread_t threads[2];
int rc = pthread_create(&threads[0], NULL, play_buffers, (void *) &state);
if (rc != 0) {
perror("pthread_create: handle_buffers");
exit(EXIT_FAILURE);
}
</code></pre></div></div>
<p>Now this function will need to wait for the first buffer to be created, so it waits for that condition to be met:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pthread_mutex_lock(&mutex);
pthread_cond_wait(&data_ready, &mutex);
pthread_mutex_unlock(&mutex);
</code></pre></div></div>
<p>This has to get to the <code class="highlighter-rouge">pthread_cond_wait</code> <em>before</em> the buffer function sends the related broadcast, so I put in a very brief delay:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>struct timespec t = {.tv_sec = 0, .tv_nsec = 100000};
nanosleep(&t, NULL);
</code></pre></div></div>
<p>And then we launch the function to populate the buffer:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rc = pthread_create(&threads[1], NULL, handle_buffers, (void *) &state);
if (rc != 0) {
perror("pthread_create: handle_buffers");
exit(EXIT_FAILURE);
}
</code></pre></div></div>
<p>This function then fills one of the two buffers, and on the first run signals the <code class="highlighter-rouge">play_buffers</code> function to play it. After the first run it will then wait for <code class="highlighter-rouge">play_buffers</code> function to be ready, as we’ve already established that will always take longer to run. As I’m still in the proof of concept phase I’ve just made it loop 4 times before finishing:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>for (int i = 0; i < 4; i++) {
pthread_mutex_lock(&mutex);
state->active_buffer = i % 2;
bfreq /= 2;
add_sine(&state->notes, bfreq, 0.1);
render_buffer(&(state->buffers[state->active_buffer]), &(state->notes), &(state->format));
if (i == 3) state->buffers[state->active_buffer].last = true;
if (i == 0) pthread_cond_broadcast(&data_ready);
pthread_cond_wait(&read_ready, &mutex);
pthread_mutex_unlock(&mutex);
}
</code></pre></div></div>
<p>After the <code class="highlighter-rouge">play_buffers</code> function has established which buffer it should be playing at this point in time it releases the lock so the filling of the buffer can begin again whilst the current buffer is playing:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>for(;;) {
pthread_mutex_lock(&mutex);
active = state->active_buffer;
buffer = &(state->buffers[active]);
pthread_cond_broadcast(&read_ready);
pthread_mutex_unlock(&mutex);
ao_play(state->device, buffer->data, buffer->generated);
if (buffer->last || buffer->generated < buffer->size) {
return NULL;
}
}
</code></pre></div></div>
<h2 id="next-steps">Next Steps</h2>
<p>Next I’m going to look at changing the volume over time, so distinct notes can be played. After that I need some way to schedule upcoming notes so they can be triggered at the time, and also span buffers if necessary.</p>
<p>I honestly don’t know if I’ll get this project finished but it’s been fairly fun so far, and now I’ve got the threading stuff sorted out I’m more confident that I’ll get there eventually.</p>
<p><em>Update - See <a href="/2020/08/23/in-c-in-c-part-2.html">Part II</a> and <a href="/2020/09/06/in-c-in-c-part-3.html">Part III</a></em></p>Ben BrownRecently I’ve been listening to Terry Riley’s groundbreaking composition In C. Riley wrote this piece in 1964, and since then there’s been many, many recordings of it.Fresh new style2020-06-28T00:00:00+00:002020-06-28T00:00:00+00:00https://427.org.uk/2020/06/28/fresh-new-look<p>After many years (<em>almost 7!</em>) I’ve finally ditched the default jekyll style and created something a bit different.</p>
<p>Frontend development isn’t my strong suit, so I’ve kept it pretty simple. The colour scheme is one I’ve been using on my terminal and on my desktop for a long time, <a href="https://ethanschoonover.com/solarized/">Ethan Schoonover’s Solarized Dark</a>.</p>
<p>There’s no JavaScript or images, not a lot of CSS, uses built in fonts, and it seems to work pretty well on my phone and my laptop so I’m happy with it.</p>Ben BrownAfter many years (almost 7!) I’ve finally ditched the default jekyll style and created something a bit different.Real config on staging, featuring nginx and dante2020-06-27T00:00:00+00:002020-06-27T00:00:00+00:00https://427.org.uk/nginx/dante/linux/2020/06/27/real-config-on-staging-ft-nginx-and-dante<p>I’m a firm believer in keeping staging as close to production as possible. It just makes sense to me, there’s too much room for error if things are different, and you’re not really testing what you think you are.</p>
<p>Now this can get a bit tricky when it comes to configuration files, especially when they contain hostnames, doubly so if you’re using HTTPS, which of course you are aren’t you?</p>
<h3 id="platform-overview">Platform Overview</h3>
<p>OK, “platform” might be stretching it for my current setup. I look after the technical side of a pretty niche website, <a href="https://www.ninehertz.co.uk/">Ninehertz</a>. It’s been around for a long time now, and still gets a decent amount of traffic but nothing too taxing.</p>
<p>For the staging side I use a virtual machine. Like a lot of people these days I manage it with <a href="https://vagrantup.com/">Vagrant</a> using the <a href="https://libvirt.org/">libvirt</a> backend.</p>
<p>The site itself is served by <a href="https://nginx.org/">nginx</a>.</p>
<h3 id="tls-complications">TLS Complications</h3>
<p>If we’re going to use the exact same config on live and staging, then we’re going to need to use TLS (fka SSL), and if we’re going to use TLS, we need certificates for it. Now, we don’t really want genuine private keys on staging, access to those should be restricted, so we’re going to make some dummy certificates ourselves.</p>
<p>We could simply use self-signed certificates, but then we’d get annoying warnings in the browser. So what I do is create my own CA certificate.</p>
<p>This is really easy to do, and completely safe as long as you follow some precautions:</p>
<ul>
<li><strong>Use a different browser profile when you import the CA</strong> - I really can’t stress this enough.
<ul>
<li>Firefox has a profile manager you can use to create a separate profile, create one with a meaningful name and import the CA to that.</li>
<li>Chrome has the <code class="highlighter-rouge">--user-data-dir</code> argument which does a similar thing. A shell alias makes this easy to use.</li>
</ul>
</li>
<li><strong>Only add this CA to the system approved list on your VM</strong> - don’t do this on your host machine.</li>
</ul>
<p>First we create a private key for the CA:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openssl genrsa -out CA.key 4096
</code></pre></div></div>
<p>Next the CA itself:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openssl req -x509 -days 365 -key CA.key -subj '/CN=dummy CA cert' -out CA.crt
</code></pre></div></div>
<p>Adjust the subject and days to taste, though I recommend making it obvious that this isn’t a real CA in the subject. 365 days is a year, so it will expire after that time, though it’s just a case of re-running the command to renew it.</p>
<p>Now we need to create a key and a Certificate Signing Request for the server to use. For this example we’ll use <code class="highlighter-rouge">www.example.com</code> and will create a cert that’s also valid for <code class="highlighter-rouge">example.com</code>:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openssl genrsa -out server.key 2048
openssl req -key server.key -new -subj '/CN=www.example.com' -out server.csr
</code></pre></div></div>
<p>Keep hold of these two files, <code class="highlighter-rouge">server.key</code> and <code class="highlighter-rouge">server.csr</code>. The key will be needed to use the certificate we create and the CSR can be used to renew the cert later.</p>
<p>Now to sign the certificate request to generate a certificate:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>printf 'basicConstraints=CA:FALSE
keyUsage=nonRepudiation, digitalSignature, keyEncipherment
subjectAltName=DNS:www.example.com,DNS:example.com' | \
openssl x509 -req -days 365 -in server.csr -out server.crt -CA CA.crt -CAkey CA.key \
-set_serial 1 \
-extfile -
</code></pre></div></div>
<p>Some things to note:</p>
<ul>
<li><code class="highlighter-rouge">set_serial 1</code> - this sets the serial number. When the certificate expires and you create a new one, make sure you set the new serial to something higher than this value.</li>
<li><code class="highlighter-rouge">subjectAltName</code> etc, this is what allows the cert to be used for more than one hostname. It’s also required for the site to display in Chrome without warnings.</li>
</ul>
<p>Now you have a brand new <code class="highlighter-rouge">server.crt</code> and <code class="highlighter-rouge">server.key</code> you can use in the <code class="highlighter-rouge">nginx</code> config in place of the real certs, just use the same paths for them so the config remains the same.</p>
<h3 id="approach-1-hostfiles">Approach 1: Hostfiles</h3>
<p>The simplest way to approach this problem is to simply add entries to <code class="highlighter-rouge">/etc/hosts</code> or the equivalent on your host OS, to point records to your VM.</p>
<p>Downsides to this are:</p>
<ul>
<li>You can’t look at live and staging at the same time to compare things</li>
<li>You can easily forget you’ve done that and confuse yourself later down the line</li>
<li>Your test site can access third party resources, some of which you only want live to be able to access</li>
</ul>
<p>For these reasons I decided I needed something better - a proxy.</p>
<h3 id="approach-2-socks-proxy-via-ssh">Approach 2: SOCKS proxy via SSH</h3>
<p>You may or may not know this already, but <code class="highlighter-rouge">ssh</code> includes a pretty usable SOCKS proxy, you just invoke it with <code class="highlighter-rouge">-D</code> and a port number, then use that port as your proxy. Combined with adding hosts records redirecting to the loopback device this worked great.</p>
<p><em>But hang on! You said you’d moved away from using the hosts file!?</em></p>
<p>Indeed I had, but my main problem was editing <code class="highlighter-rouge">/etc/hosts</code> on the host OS, and this change is in the VM. As clients can use the SOCKS proxy to resolve hostnames too this is a nice solution, you know when you’re using the proxy that you’re hitting staging.</p>
<p>I used this approach for a little while, but I wasn’t completely happy with it for the following reasons:</p>
<ul>
<li>You have to run the <code class="highlighter-rouge">ssh</code> command every time, which gets annoying</li>
<li>It still doesn’t control access to third party resources</li>
</ul>
<p>With these concerns in mind I decided I needed a SOCKS proxy that would start automatically when the VM started, and had some access control. After some searching I found…</p>
<h3 id="approach-3-dante">Approach 3: dante</h3>
<p><a href="https://www.inet.no/dante/">Dante</a> is a free SOCKS server, developed by Inferno Nettverk A/S. It’s included in Debian, my OS of choice, so was simple to install. It also supports allowing and denying access to resources, so I quickly set up some rules. I knew it would resolve the hostnames I was interested in to the local loopback address, so I just added a rule to allow traffic to that interface and deny all other traffic.</p>
<p>This worked really well, I didn’t need to <code class="highlighter-rouge">ssh</code> in any more, but something looked a bit strange. The fonts weren’t right.</p>
<p>I knew straight away why this was, Ninehertz uses Google fonts, which were now being blocked. For a while I lived with that, but I got curious and started thinking of a better way.</p>
<h3 id="approach-4-nginx-proxy_pass-for-specific-hostnames">Approach 4: nginx proxy_pass for specific hostnames</h3>
<p>My next approach was to add specific hostnames to the VM’s hosts file and direct them to the local nginx instance, with a small config snippet using <a href="https://nginx.org/r/proxy_pass">proxy_pass</a> to redirect the traffic. I’d also create certs for these hostnames, signed with my CA. With a bit of <a href="https://www.ansible.com/">Ansible</a> this was pretty easy to do so I got it up and running fairly quickly.</p>
<p>This worked great for Google Fonts, but I hit another problem when I loaded a page with an embedded <a href="https://soundcloud.com">SoundCloud</a> embed, which was trying to fetch things from loads of subdomains. This wasn’t going to scale nicely at all - I really needed a way I could use wildcards.</p>
<h3 id="approach-5-nginx-stream-proxy_pass-with-dante-redirecting">Approach 5: nginx stream proxy_pass, with dante redirecting</h3>
<p>I had a bit of a think and came up with what I thought was a cunning plan. I’m a bit of an nginx enthusiast and like to stay up to date with new features etc, even if I don’t really use them. One such feature popped in to my head, <a href="https://nginx.org/r/ssl_preread">ssl_preread</a>.</p>
<p>Combined with the <a href="https://nginx.org/r/map">nginx map</a> directive, this allows us to read the target hostname from the TLS handshake (amongst other things) and route the traffic to different ports depending on the values it finds. We can even use wildcards here, which makes it even easier.</p>
<p>This uses the <a href="http://nginx.org/en/docs/stream/ngx_stream_core_module.html">nginx stream module</a> which operates at a low level, so we don’t even need to think about TLS handshakes, we can now forward the TCP traffic unencrypted to the hosts we want to allow.</p>
<p>For other hosts we’ll just forward it to a dummy HTTPS server, also in nginx, with an invalid certificate.</p>
<p>All I need to do now, is capture all requests to HTTPS on dante to this nginx stream server. Now we can’t use port 443 here without changing our own http server config, which would defeat the point of this, so we run the stream server on a different port.</p>
<p>Looking at the dante documentation it looked really easy to redirect this traffic to our arbitrary port, but when I put the config in place, the danted service failed to start. I’d missed a key part of the documentation, from <a href="https://www.inet.no/dante/doc/1.4.x/config/redirect.html">Redirection</a>:</p>
<blockquote>
<p>Some additional functionality is however offered as modules that can be purchased separately. The redirect module can be used to modify traffic by redirecting it to a different location than the client requested.</p>
</blockquote>
<p>OK, so dante can do it, but not without a proprietary module. This was a little disappointing, but if that’s their model then that’s fair enough.</p>
<h3 id="finished-setup-nginx-stream-proxy_pass-dante-and-iptables">Finished setup: nginx stream proxy_pass, dante, and iptables</h3>
<p>Finally I arrived at the setup I’m using today. The final piece in the puzzle was to simply use <code class="highlighter-rouge">iptables</code> to redirect all http and https traffic to the local nginx instance, with the exception of traffic from nginx itself as that would cause a loop.</p>
<p>Here’s the nginx configuration for the stream module:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>map $ssl_preread_server_name $backend {
hostnames;
.ninehertz.co.uk 127.0.0.1:443;
fonts.googleapis.com $ssl_preread_server_name:443;
fonts.gstatic.com $ssl_preread_server_name:443;
.soundcloud.com $ssl_preread_server_name:443;
.sndcdn.com $ssl_preread_server_name:443;
default 127.0.0.1:8001;
}
resolver 127.0.0.1;
server {
listen 8000;
proxy_pass $backend;
ssl_preread on;
}
server {
listen 8002;
proxy_pass 127.0.0.1:80;
}
</code></pre></div></div>
<p>I should probably mention that I use <a href="https://www.powerdns.com/">PowerDNS</a> on this box and on live as a local resolver, that’s what <code class="highlighter-rouge">resolver 127.0.0.1</code> is referring to. We need to set a resolver up here so nginx can look up the hostnames it needs to proxy the external traffic.</p>
<p>Most of the work here is done in the <code class="highlighter-rouge">map</code> section. <code class="highlighter-rouge">$ssl_preread_server_name</code> is the server name passed in the TLS negotiation. If it matches the hostnames listed and it’s one of the sites that are being staged on the VM then the variable <code class="highlighter-rouge">$backend</code> is set to the loopback device. If it’s a site we don’t host, but want to allow access to <code class="highlighter-rouge">$backend</code> is set to the original hostname.</p>
<p>We are using the <code class="highlighter-rouge">hostnames</code> option in the map (which needs to be listed before the list of values we’re checking) so we can use the special leading <code class="highlighter-rouge">.</code> form to match a domain and any subdomains at the same time. e.g. <code class="highlighter-rouge">.ninehertz.co.uk</code> matches <code class="highlighter-rouge">ninehertz.co.uk</code>, <code class="highlighter-rouge">www.ninehertz.co.uk</code>, <code class="highlighter-rouge">foo.bar.ninehertz.co.uk</code> etc.</p>
<p>Finally <code class="highlighter-rouge">default</code> is a special record which is used to set a default value, which will be for anything we want requests to to fail. These will be redirected to another nginx server block in the http scope, which is running on port 8001.</p>
<p>The first server block simply listens for connections on port 8000, and passes them to the <code class="highlighter-rouge">$backend</code> set by the map. We need to have <code class="highlighter-rouge">ssl_preread on</code> here or the special <code class="highlighter-rouge">$ssl_preread_server_name</code> variable won’t get set.</p>
<p>The next server block is for handling any requests on port 80, it just passes them all to the loopback address.</p>
<p>Here’s the config of the server that non-matching results are forwarded to:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>server {
listen 8001 default_server ssl;
ssl_certificate ssl/default.crt;
ssl_certificate_key ssl/default.key;
return 403 "No\n";
}
</code></pre></div></div>
<p>This is very simple, we just have a cert signed by our CA and it’ll return 403 for all requests.</p>
<p>Dante wise it’s pretty simple, we pass anything going to to ports 80 or 443, and reject everything else.</p>
<p>Finally, here’s the iptables rules to make it all work:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>iptables -t nat -A OUTPUT -m owner --uid-owner nginx -j ACCEPT
iptables -t nat -A OUTPUT -d 127.0.0.1/32 -j ACCEPT
iptables -t nat -A OUTPUT -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 8002
iptables -t nat -A OUTPUT -p tcp -m tcp --dport 443 -j REDIRECT --to-ports 8000
</code></pre></div></div>
<p>It simply accepts all requests from the <code class="highlighter-rouge">nginx</code> user, all requests to the local loopback, and redirects any remaining requests to the default https port of 443 to port 8000 on the loopback device.</p>
<p>And a nice feature of this approach is that we no longer need to touch <code class="highlighter-rouge">/etc/hosts</code> on either our host machine or the VM.</p>
<p>I hope you might find this useful, if you have any feedback drop me a message on any of the platforms listed at the bottom of each page.</p>Ben BrownI’m a firm believer in keeping staging as close to production as possible. It just makes sense to me, there’s too much room for error if things are different, and you’re not really testing what you think you are.Upgrading to Jessie with MD RAID2013-12-15T10:18:15+00:002013-12-15T10:18:15+00:00https://427.org.uk/debian/jessie/md-raid/2013/12/15/upgrading-to-jessie-with-md-raid<p>The other day I finally got around to upgrading my desktop PC from <a href="http://www.debian.org/">debian</a> wheezy to jessie. On the whole it was a painless upgrade, and consisted of:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>apt-get update
apt-get -y dist-upgrade
sed --in-place -e 's/wheezy/jessie/g' /etc/apt/sources.list \
/etc/apt/sources.list.d/*
apt-get update
apt-get -y dist-upgrade
reboot
</code></pre></div></div>
<p>Unfortunately at this point things went a little wrong, as the new kernel couldn’t see my MD RAID devices. <code class="highlighter-rouge">/proc/mdstat</code> was empty, and I got an error about the root device missing. After a bit of research I discovered that the kernel wasn’t waiting long enough for the RAID to become active, which was easily fixed by changing <code class="highlighter-rouge">GRUB_CMDLINE_LINUX_DEFAULT</code> in <code class="highlighter-rouge">/etc/default/grub</code> to <code class="highlighter-rouge">"quiet rootdelay=1"</code> and running <code class="highlighter-rouge">update-grub</code>.</p>
<p>Once that was fixed it was just a case of removing the nvidia drivers I was using and installing the <em>nvidia-legacy-304xx-driver</em> package instead. Just installing the package didn’t work as X11 kept trying to use the non-legacy drivers which don’t work with my ancient card, but removing the <em>nvidia-kernel-319.72</em> package sorted it out.</p>
<p>Overall the experience of jessie has been good, it seems to boot a bit quicker, and nothing else seems to have been broken in the upgrade. Much easier than squeeze to wheezy which ruined apt when I tried it a few years ago!</p>Ben BrownThe other day I finally got around to upgrading my desktop PC from debian wheezy to jessie. On the whole it was a painless upgrade, and consisted of:New Site Again2013-07-12T12:10:24+00:002013-07-12T12:10:24+00:00https://427.org.uk/jekyll/update/2013/07/12/new-site-again<p>I’ve finally got rid of wordpress and replaced it with <a href="http://jekyllrb.com/">jekyll</a>. Now I need to learn to use markdown!</p>Ben BrownI’ve finally got rid of wordpress and replaced it with jekyll. Now I need to learn to use markdown!Installing Solaris 10 from a Debian Server2009-06-03T08:40:07+00:002009-06-03T08:40:07+00:00https://427.org.uk/2009/06/03/installing-solaris-10-from-a-debian-server<p>I recently rescued an old Netra T1 105 from work, and initially installed Debian on it, without much trouble. However, I’m a contrary type so I thought, why not install <a href="http://www.sun.com/software/solaris/10/index.jsp">Solaris 10</a> on it instead?</p>
<p>Now installing Solaris is pretty easy, as long as you already have a Solaris server in your network, or a DVD drive. As this Netra doesn’t have a CD let alone a DVD drive, I always knew I would have to install it over the network. And as I didn’t have a Solaris box in my network, I would have to install it from Debian. Now this took some trial and error, and a LOT of googling, but I finally got it done.</p>
<p>Firstly, I already had RARP and DHCP on my box, so I thought I’d set them up first. If you’ve not got them, you’ll need to apt-get them:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>apt-get install rarpd dhcp3-server bootparamd
</code></pre></div></div>
<h1 id="rarp">RARP</h1>
<p>RARP is very simple, and just assigns an IP to a given MAC address. Once installed you just need to put the MAC and ip in /etc/ethers, e.g.
<code class="highlighter-rouge">de:ad:be:ef:f0:0f 192.168.0.10</code></p>
<h1 id="dhcp">DHCP</h1>
<p>Now in the DHCP config at /etc/dhcpd.conf I added:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>host greenmachine {
hardware ethernet de:ad:be:ef:f0:0f;
fixed-address 192.168.0.10;
option host-name "greenmachine";
}
</code></pre></div></div>
<p>Then just restart dhcpd.</p>
<h1 id="installer-script">Installer script</h1>
<p>Now it gets a little bit fiddlier, as we need to run the solaris setup script on the debian box to prepare it for the net install. Get hold of a Solaris ISO image (or physical DVD if you are old fashioned) and mount it on /media/cdrom. Before we run the script we need to do some fudges so the script runs OK on debian. I found <a href="http://alex.charrett.com/technical/linux-jumpstart.html">this site</a> really useful for getting the script running and sorting out bootparamd, which comes in a bit later.</p>
<p>First a couple of symlinks. Needless to say, you need to be root for this. In fact you should be root for all of this.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ln -s /bin/tar /bin/bar
ln -s /bin/sed /usr/bin/sed
</code></pre></div></div>
<p>Next we need to write a couple of dummy scripts that the installer uses:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cat <<EOF > /bin/adb
#!/bin/bash
read HEX
HEX=0x${HEX%%=d}
printf "%d\n" $HEX
EOF
cat <<EOF > /bin/mach
#!/bin/bash
uname -p
EOF
chmod +x /bin/adb /bin/mach
</code></pre></div></div>
<p>Now to create the target for the installation media. On my box at the time I had a large /share, so that’s what I used:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mkdir -p /share/jumpstart/{installer,config}
</code></pre></div></div>
<p>Now run the installer:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cd /media/cdrom/Solaris_10/Tools
./setup_install_server /share/jumpstart/installer
</code></pre></div></div>
<p>It’ll take a while to run but should complete successfully. Now we will remove our scripts/symlinks from before so we’re not leaving mess around the system.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rm /usr/bin/sed /bin/adb /bin/mach /bin/bar
</code></pre></div></div>
<h1 id="nfs">NFS</h1>
<p>Next comes NFS, as our target server will need to be able to get to the files. Now first of all, the Sun box will try to use nfsv4 to connect, however it’s unlikely you’re going to have v4 set up, so if you don’t use v4 you should disable it. This is fairly simple, edit /etc/init.d/nfs-kernel-server and add “–no-nfs-version” to the “–exec $PREFIX/sbin/rpc.nfsd” line, so it looks like this:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>--exec $PREFIX/sbin/rpc.nfsd -- --no-nfs-version 4 $RPCNFSDCOUNT
</code></pre></div></div>
<p>Now we need the exports to /etc/exports for our target. Here I’ve allowed my whole subnet but it’s up to you to decide how to do it.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/share/jumpstart/install 192.168.0.0/24(ro,no_root_squash,async,no_subtree_check)
/share/jumpstart/config 192.168.0.0/24(ro,no_root_squash,async,no_subtree_check)
</code></pre></div></div>
<p>Now restart nfs-kernel-server:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/etc/init.d/nfs-kernel-server restart
</code></pre></div></div>
<h1 id="bootparamd">BOOTPARAMD</h1>
<p>Bootparamd is used to give the installer information about where to get it’s files from and stuff like that, my config is below. Once you change anything in the config you need to restart the bootparamd process with /etc/init.d/bootparamd restart.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># /etc/bootparams
greenmachine root=192.168.0.1:/share/jumpstart/install/Solaris_10/Tools/Boot/ \
install=192.168.0.1:/share/jumpstart/install \
boottype=:in \
rootopts=192.168.0.1:rsize=8192,wsize=8192
</code></pre></div></div>
<h1 id="boot-image">Boot image</h1>
<p>Now we just need to put the boot image into the tftp directory, which from checking /var/log/daemon.log seems to be /srv/tftp by default.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cp Tools/Boot/platform/sun4u/inetboot /srv/tftp
</code></pre></div></div>
<p>Now the target will request a file with the same name as it’s IP address, but there’s a catch, it needs to be in hex. It’s pretty simple to work out though, and easily scripted:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cd /srv/tftp
IP=192.168.0.10
HEXIP=$(echo $IP | tr . ' ' | xargs printf %02x | tr a-z A-Z)
ln -s inetboot $HEXIP
</code></pre></div></div>
<h1 id="configtar">config.tar</h1>
<p>One last and important thing to do, is to set up our config tarball. This tells the target system how to use it’s disks and what packages etc should be installed. This comprises of several files.</p>
<h2 id="rules">rules</h2>
<p>There is an example rules file on the Solaris DVD, in Solaris_10/Misc/jumpstart_sample. I just copied it to my config location (/share/jumpstart/config) and added the following line:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>any - - profile -
</code></pre></div></div>
<p>Now I’m not even sure the rules file is strictly necessary, however it is normally used to generate the rules.ok file, which is what I will do next.</p>
<h2 id="rulesok">rules.ok</h2>
<p>This is pretty simple, just run the following:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cd /share/jumpstart/config
sed -e '/#/d' -e '/^\s*$/d' rules > rules.ok
chksum=$(sum rules.ok | awk '{print $1}');
echo "# version=2 checksum=${chksum}" >> rules.ok
</code></pre></div></div>
<h2 id="profile">profile</h2>
<p>Next we write the profile, which says what should be installed and how the disks should be used etc. Here is a minimal one below:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">install_type</span> <span class="n">initial_install</span>
<span class="n">cluster</span> <span class="n">SUNWCrnet</span>
<span class="n">cluster</span> <span class="n">SUNWCssh</span> <span class="n">add</span>
<span class="k">package</span> <span class="n">SUNWgss</span> <span class="n">add</span>
<span class="k">package</span> <span class="n">SUNWgssc</span> <span class="n">add</span>
<span class="k">package</span> <span class="n">SUNWbash</span> <span class="n">add</span>
<span class="k">package</span> <span class="n">SUNWwgetr</span> <span class="n">add</span>
<span class="k">package</span> <span class="n">SUNWwgetu</span> <span class="n">add</span>
<span class="n">partitioning</span> <span class="n">explicit</span>
<span class="n">filesys</span> <span class="n">mirror</span> <span class="n">c0t0d0s0</span> <span class="n">c0t1d0s0</span> <span class="m">5120</span> <span class="p">/</span>
<span class="n">filesys</span> <span class="n">mirror</span> <span class="n">c0t0d0s1</span> <span class="n">c0t1d0s1</span> <span class="m">512</span> <span class="n">swap</span>
<span class="n">filesys</span> <span class="n">mirror</span> <span class="n">c0t0d0s3</span> <span class="n">c0t1d0s3</span> <span class="m">5120</span> <span class="p">/</span><span class="n">opt</span>
<span class="n">filesys</span> <span class="n">mirror</span> <span class="n">c0t0d0s6</span> <span class="n">c0t1d0s6</span> <span class="m">5120</span> <span class="p">/</span><span class="n">usr</span>
<span class="n">filesys</span> <span class="n">mirror</span> <span class="n">c0t0d0s7</span> <span class="n">c0t1d0s7</span> <span class="m">5120</span> <span class="p">/</span><span class="n">home</span>
<span class="n">metadb</span> <span class="n">c0t0d0s4</span> <span class="n">size</span> <span class="m">8192</span> <span class="k">count</span> <span class="m">4</span>
<span class="n">metadb</span> <span class="n">c0t1d0s4</span> <span class="n">size</span> <span class="m">8192</span> <span class="k">count</span> <span class="m">4</span>
<span class="n">filesys</span> <span class="n">mirror</span> <span class="n">c0t0d0s5</span> <span class="n">c0t1d0s5</span> <span class="n">free</span> <span class="p">/</span><span class="n">share</span>
</code></pre></div></div>
<p>This will install a very basic installation with SSH, bash and wget. It will use both disks as a RAID1 device, with 5GB /, /usr, /opt and /home slices, and the remaining free space in /share.</p>
<h2 id="sysidcfg">sysidcfg</h2>
<p>This file has some other config options, such as DNS servers, network and user config. Mine is below:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>name_service=DNS{domain_name=kyussnet.local name_server=192.168.0.254 search=kyussnet.local}
network_interface=hme0{dhcp protocol_ipv6=no}
system_locale=en_GB
timezone=Europe/London
root_password=kFkZmCjlJFIz6
nfs4_domain=dynamic
timeserver=localhost
security_policy=NONE
terminal=vt100
</code></pre></div></div>
<p>The root password there is the 3DES hashed form of “letmein”. Of course I have changed it since ;)</p>
<p>Now tar all these files up and put them somewhere your target can get to it via http. I used an apache instance running on the Debian server for this, so it was pretty simple:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tar -cf /var/www/config.tar .
</code></pre></div></div>
<h1 id="installation">Installation</h1>
<p>Now on to the actual installation. You need to be able to connect to the target via serial. I used my <a href="http://www.amazon.co.uk/High-Speed-USB-Serial-Adapter/dp/B000CDMXG4/ref=sr_1_1?ie=UTF8&s=electronics&qid=1238864531&sr=8-1">KeySpan USB to Serial adapter</a> that I have from work, and a Cisco RS-232 to RJ-45 rollover cable, which I connected to the LOM port on the back of the netra. After powering up the netra, I sent it a break so I got the OpenBoot “ok>” prompt.</p>
<p>At the “<code class="highlighter-rouge">ok> </code>” prompt, type in the following:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>boot net -v - install http://192.168.0.1/config.tar
</code></pre></div></div>
<p>Now the URI at the end is for the tarball we made earlier, it might be different on your network. The important thing is that you use the IP address and NOT a DNS hostname, as DNS will not be configured at the time it fetches the tarball. Now just sit back and relax, within an hour or so you should have a fully installed system you can SSH to.</p>
<h1 id="additional-notes">Additional Notes</h1>
<h2 id="disk-labels">Disk labels</h2>
<p>Solaris systems recognise two different disk labels, SMI and EFI. Unfortunately you can’t install to EFI disks, but you can relabel them to be SMI. The first time I tried this installation it bombed out when it got to dealing with the disks and left me at a sh prompt. To fix it I ran:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>format -e
</code></pre></div></div>
<p>Choose each disk in turn and the option to label it. This should work fine, it will warn about data loss but we don’t care do we? Nothing’s on there yet.</p>
<p>Once you’ve done that you can re-run the installer without having to reboot with:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/usr/sbin/install.d/pfinstall /tmp/install_config/profile
</code></pre></div></div>
<h2 id="duplex">Duplex</h2>
<p>By default it seems that the network devices were set to half duplex, or Solaris is just terrible at autonegotiation. To force it into full duplex, run the following and reboot:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cat <<EOF>>/etc/system
set hme:hme_adv_100fdx_cap=1
set hme:hme_adv_100hdx_cap=0
set hme:hme_adv_10fdx_cap=0
set hme:hme_adv_10hdx_cap=0
set hme:hme_adv_autoneg_cap=0
EOF
</code></pre></div></div>
<h2 id="md5-passwords">MD5 passwords</h2>
<p>By default Solaris 10 uses 3DES to encrypt passwords, which is old, crusty and insecure. To beef up the security a bit you can use MD5, which is a better choice. Open up /etc/security/policy.conf and change the “<code class="highlighter-rouge">CRYPT_DEFAULT=__unix__</code>” line to “<code class="highlighter-rouge">CRYPT_DEFAULT=1</code>”.</p>
<p>Once you’ve done that, then you’ll want to change the passwords for any users you’ve set up with the “passwd” command.</p>BenI recently rescued an old Netra T1 105 from work, and initially installed Debian on it, without much trouble. However, I’m a contrary type so I thought, why not install Solaris 10 on it instead?