Structure of .E2PAT files

[Xanadu]

Hi folks,

I did some analysing to figure out the structure of the .E2PAT files. I've got most of it figured out. But I'm still missing a clue about motion records.

I'm working in MAX4LIVE so I've coded the structure in Javascript files. Here I'm presenting my first findings. I suppose there are a few bugs left, but I hope I will get some other guys going on this.

At the moment I can read an .E2PAT file, but there's no editing yet. I was able to change a global value, save my changed file and it read back into my electribe 2. So I assume there's no checksum preventing the loading of edited files.

The overall structure is E2Pattern.

Note that the globals are from byte 0x110 to 0x145.

Here's the code to load it and access it's parts:

Code: Select all

E2Pattern = function (filename) {
	this.fullname = filename;

	var f = new File(this.fullname, 'read');
	
	this.filename = f.filename;
	this.folder = f.foldername;
		
	this.length = f.eof;
	
	this.pattern = f.readbytes(this.length);
	
	f.close();
	
	const partOffset = 0x900;
    const partSize = 0x330;
	const sequenceOffset = 0x30;
	const stepSize = 12;

	this.parts = [];
    for (var i = 0; i < 16; i++)
    {
		this.parts[i] = new E2Part(this.pattern, partOffset + i * partSize);
    }

	this.sequences = [];
    for (var i = 0; i < 16; i++)
    {
		this.sequences[i] = [];

		this.sequences[i].bars = [];
		this.sequences[i].steps = [];
	
		for (var b = 0; b < 4; b++)
	    {
			this.sequences[i].bars[b] = [];
			
	        for (var s = 0; s < 16; s++)
	        {
				var step = new E2Step(this.pattern, partOffset + i * partSize + sequenceOffset + (b * 16 + s) * stepSize);
	            this.sequences[i].bars[b][s] = step;
				this.sequences[i].steps[b * 16 + s] = step; 
	        }
	    }
    }
}

E2Pattern.prototype = {
    get FullName  () { return this.fullname; },
	get FileName  () { return this.filename; },
	get Folder    () { return this.folder; },
	get Size      () { return this.length; },

	get Pattern   () { return this.pattern; },
	
	get Name      () 
		{
			var s = "";
            for (var i = 0x110; i < 0x120; i++)
            {
                s += String.fromCharCode(this.pattern[i]);
            }
            return s;
        },
		

	get BPM       () { return this.pattern[0x122] + 0x100 * this.pattern[0x123]; },
	get Swing     () { return this.pattern[0x124]; },
	get Length    () { return this.pattern[0x125]; },
	get Beat      () { return this.pattern[0x126]; },
	
	get Key       () { return this.pattern[0x127]; },
	get Scale     () { return this.pattern[0x128]; },
	get ChordSet  () { return this.pattern[0x129]; },
	get Level     () { return 127 - this.pattern[0x12a]; },
	set Level     (value) { this.pattern[0x12a] = 127 - value; },
	// Unknown bytes 12b - 130
	get GateArp   () { return this.pattern[0x131]; },
	get MFXType   () { return this.pattern[0x13d]; },
	get Alt_13_14 () { return this.pattern[0x144]; },
	get Alt_15_16 () { return this.pattern[0x145]; },
		
	get Parts     () { return this.parts; },
	get Sequences () { return this.sequences; },

	SaveAs : function (filename) {
		outlet(0, '=== WriteE2pat');

		outlet(0, this.folder + "/" + filename + ".e2pat");
		
		var f = new File(this.folder + "/" + filename + ".e2pat", 'write');
		
		outlet(0, 'Open:', f.isopen);
		outlet(0, 'Name:', f.filename);
		outlet(0, 'folder:', f.foldername);
		
		outlet(0, 'Length loaded bytes', this.pattern.length);
		
		f.writebytes(this.pattern.slice(0, 10000));
		f.writebytes(this.pattern.slice(10000, 16641));
		
		outlet(0, 'length:', f.eof);
	
		f.close();
	}
}

The parts start at 0x900. They are 0x330 bytes long. The first 0x30 bytes are the parameters. Then there are 64 steps. The structure of the steps is 12 bytes long.

Here is the code for E2Part:

Code: Select all

E2Part = function (pattern, offset ) { 
	var that = this;
	
	this.loc = pattern;
	this.offset = offset;
}

E2Part.prototype = {

	get Offset () { return this.offset; },
	get Size () { return 0x30; },

	get Laststep () { return this.loc[this.offset + 0]; },
	get VoiceAssign () { return this.loc[this.offset + 2]; },
	get PartPriority () { return this.loc[this.offset + 3]; },
	get MotionSeq () { return this.loc[this.offset + 4]; },
	get TriggerPadVelocity () { return this.loc[this.offset + 5]; },
	get ScaleMode () { return this.loc[this.offset + 6]; },
    // 7 
	get Oscillator () { return this.loc[this.offset + 8] + 0x100 * this.loc[this.offset + 9]; },
	// 10
	get EditOsc () { return this.loc[this.offset + 11]; },
	get FilterType () { return this.loc[this.offset + 12]; },
	get Cutoff () { return this.loc[this.offset + 13]; },
	get Resonance () { return this.loc[this.offset + 14]; },
	get EGInt () { return this.loc[this.offset + 15]; },
	get Modulation () { return this.loc[this.offset + 16]; },
	get LFOSpeed () { return this.loc[this.offset + 17]; },
	get LFODepth () { return this.loc[this.offset + 18]; },
    // 19
	get Attack () { return this.loc[this.offset + 20]; },
	get Decay () { return this.loc[this.offset + 21]; },
    // 22 23
	get Level () { return this.loc[this.offset + 0x18]; },
	get Pan () { return this.loc[this.offset + 0x19]; },
	get AmpEG () { return this.loc[this.offset + 0x1a]; },
	get MFX () { return this.loc[this.offset + 0x1b]; },
	get GrooveType () { return this.loc[this.offset + 0x1c]; },
	get GrooveDepth () { return this.loc[this.offset + 0x1d]; },
    // 0x1e - 0x1f
	get IFX () { return this.loc[this.offset + 0x20]; },
	get Fx () { return this.loc[this.offset + 0x21]; },
	get InsertFxAmount () { return this.loc[this.offset + 0x22]; },
    // 0x23
	get Pitch () { return this.loc[this.offset + 0x24]; },
	get Glide () { return this.loc[this.offset + 0x25]; },
    // 0x26 - 0x30
	get sequenceStart () { return this.offset + 0x31; }
}

The structure of each step is quite simple. I'm not sure if HasNotes and Chord are the correct meanings of the bytes I gave these names.

Code: Select all

E2Step = function (Pattern, Offset ) { 
	var that = this;
	
	this.pattern = Pattern;
	this.offset = Offset;
}	

E2Step.prototype = {
	get Loc      () { return this.pattern; },
	get Offset   () { return this.offset; },

	get Size     () { return 12; },

	get HasNotes () { return this.pattern[this.offset + 0]; },
	get Gate     () { return this.pattern[this.offset + 1]; },
	get Velocity () { return this.pattern[this.offset + 2]; },
	get Chord    () { return this.pattern[this.offset + 3]; },
	get Note1    () { return this.pattern[this.offset + 4] - 1; },
	get Note2    () { return this.pattern[this.offset + 5] - 1; },
	get Note3    () { return this.pattern[this.offset + 6] - 1; },
	get Note4    () { return this.pattern[this.offset + 7] - 1; }
	// 8 - 11
}	

Here some code to get things going.

Code: Select all

var outlet_file = 0;
var outlet_global = 1;
var outlet_part = 2;
var outlet_sequence = 3;

var outlet_max = 4;

// pat is a byte_array with the e2pat file in it 
var pat;
var pat2;
 
function read_e2pat(filename)
{
	outlet(outlet_max, '=== Create E2Pattern'); 
	outlet(outlet_max, 'file', filename);
	
	pat = new E2Pattern(filename);

	for (var i in pat) {
  		outlet(outlet_max, "property", i, pat[i]);
	}
	
	get_file()

	get_global_parameters();

	
	for (var i = 0; i < 1 /* 16 */; i++)
	{
		get_part_parameters(i);
		get_sequence(i);
	}

	const START = 0x800;
	const LEN = 0x200;
	
	hexDump(pat.pattern, 0x100, LEN);
	hexDump(pat.pattern, START, LEN);

	var step = pat.Sequences[0].bars[0][0];
	
	//hexDump(pat.pattern, step.Offset - 0x30, LEN);
	//seqDump(pat.pattern, step.Offset);
	
	outlet(0, 0x330, "0x" + d2h(0x330, 4), 0x30 + 64 * 12, "0x"+ d2h(0x30 + 64 * 12, 4));
	outlet(0, 0x900, "0x" + d2h(0x900, 4), 0x330 * 16, "0x"+ d2h(0x330 * 16, 4));
	outlet(0, pat.length, "0x" + d2h(pat.legth, 4), 0x900 + 0x330 * 16, "0x" + d2h(0x900 + 0x330 * 16, 4));
}

function read_e2pat_2(filename)
{
	outlet(outlet_max, '=== Create E2Pattern 2');
	outlet(outlet_max, 'file', filename);
	
	pat2 = new E2Pattern(filename);
	
	for (var i = 0; i < pat.pattern.length; i++)
	{
		if (pat.pattern[i] != pat2.pattern[i])
			outlet(outlet_max, "0x" + d2h(i, 4), pat.pattern[i], "0x" + d2h(pat.pattern[i]), pat2.pattern[i], "0x" + d2h(pat2.pattern[i]));
	}

}

function get_file()
{
	outlet(outlet_file, "FullName", pat.FullName);
	outlet(outlet_file, "FileName", pat.FileName);
	outlet(outlet_file, "Folder", pat.Folder);
	outlet(outlet_file, "Size", pat.Size, "bytes");
}

function get_global_parameters()
{
	outlet(outlet_global, "Name", pat.Name);
	outlet(outlet_global, "BPM", pat.BPM);
	outlet(outlet_global, "Swing", pat.Swing);
	outlet(outlet_global, "Beat", pat.Beat);
	outlet(outlet_global, "Length", pat.Length);
	outlet(outlet_global, "PatternLevel", pat.Level);
	outlet(outlet_global, "MFXType", pat.MFXType);
	outlet(outlet_global, "Key", pat.Key);
	outlet(outlet_global, "Scale", pat.Scale);
	outlet(outlet_global, "ChordSet", pat.ChordSet);
	outlet(outlet_global, "GateArp", pat.GateArp);
	outlet(outlet_global, "Alt_13_14", pat.Alt_13_14);
	outlet(outlet_global, "Alt_15_16", pat.Alt_15_16);
}

function get_part_parameters(partNumber)
{
	var part = pat.Parts[partNumber];
	
	outlet(outlet_part, "Part", partNumber);
	outlet(outlet_part, "Oscillator", part["Oscillator"]);
	outlet(outlet_part, "Laststep", part["Laststep"]);
	outlet(outlet_part, "VoiceAssign", part["VoiceAssign"]);
	outlet(outlet_part, "PartPriority", part["PartPriority"]);
	outlet(outlet_part, "MotionSeq", part["MotionSeq"]);
	outlet(outlet_part, "TriggerPadVelocity", part["TriggerPadVelocity"]);
	outlet(outlet_part, "ScaleMode", part["ScaleMode"]);
	outlet(outlet_part, "EditOsc", part["EditOsc"]);
	outlet(outlet_part, "FilterType", part["FilterType"]);
	outlet(outlet_part, "Cutoff", part["Cutoff"]);
	outlet(outlet_part, "Resonance", part["Resonance"]);
	outlet(outlet_part, "EGInt", part["EGInt"]);
	outlet(outlet_part, "Modulation", part["Modulation"]);
	outlet(outlet_part, "LFOSpeed", part["LFOSpeed"]);
	outlet(outlet_part, "LFODepth", part["LFODepth"]);
	outlet(outlet_part, "Attack", part["Attack"]);
	outlet(outlet_part, "Decay", part["Decay"]);
	outlet(outlet_part, "Level", part["Level"]);
	outlet(outlet_part, "Pan", part["Pan"]);
	outlet(outlet_part, "AmpEG", part["AmpEG"]);
	outlet(outlet_part, "MFX", part["MFX"]);
	outlet(outlet_part, "GrooveType", part["GrooveType"]);
	outlet(outlet_part, "GrooveDepth", part["GrooveDepth"]);
	outlet(outlet_part, "IFX", part["IFX"]);
	outlet(outlet_part, "Fx", part["Fx"]);
	outlet(outlet_part, "InsertFxAmount", part["InsertFxAmount"]);
	outlet(outlet_part, "Pitch", part["Pitch"]);
	outlet(outlet_part, "Glide", part["Glide"]);
	
	/*
	for (var prop in part) 
	{
  		outlet(outlet_max, "    PART", partNumber, "property", prop, part[prop]);
	}
	*/
}

function get_sequence(partNumber)
{
	outlet(outlet_sequence, "part", partNumber);
	for (var b = 0; b < 4; b++)
	{
		for (var s  = 0; s < 16; s ++)
		{
			var step = pat.Sequences[partNumber].bars[b][s];
			outlet(outlet_max, "    SEQUENCE", "Part", partNumber, "BAR", b, "STEP", s, "Gate", step.Gate, "HasNotes", step.HasNotes, "Chord", step.Chord, "Velocity", step.Velocity, "Notes", step.Note1, step.Note2, step.Note3, step.Note4);
			outlet(outlet_sequence, "step", s, "Gate", step.Gate, "HasNotes", step.HasNotes, "Chord", step.Chord, "Velocity", step.Velocity, "Notes", step.Note1, step.Note2, step.Note3, step.Note4);
		}
	}
}

function d2h(d, padding) {
    var hex = Number(d).toString(16).toUpperCase();
    padding = typeof (padding) === "undefined" || padding === null ? padding = 2 : padding;

    while (hex.length < padding) {
        hex = "0" + hex;
    }

    return hex;
}

function hexDump(bytearray, start, len)
{
	outlet(outlet_max, "hexDump", bytearray, start, len);

	for (var i = start; i < start + len; i += 16)
	{
		var s = d2h(i, 4) + '  ';
		for (var j = 0; j < 8; j++)
		{
			s += d2h(bytearray[i + j]) + ' ';
		}
		s += ' ';
		for (var j = 0; j < 8; j++)
		{
			s += d2h(bytearray[i + j + 8]) + ' ';
		}
		s += ' ';
		for (var j = 0; j < 16; j++)
		{
			var b = bytearray[i + j];
			if (b <0x20>= 0x7f)
			{
				s += '.';
			}
			else
			{
				s += String.fromCharCode(b);
			}
		}
		outlet(outlet_max, s);
	}
}

function seqDump(bytearray, start)
{
	var len = 12 * 64;
	outlet(0, "hexDump", bytearray, start, len);

	for (var i = start; i < start + len; i += 12)
	{
		var s = d2h(i, 4) + '  ';
		for (var j = 0; j < 12; j++)
		{
			s += d2h(bytearray[i + j]) + ' ';
		}
		outlet(outlet_max, s);
	}
}

function test()
{
	outlet(outlet_max, 'bingo', 5);
	pat.Level = 5;
	pat.SaveAs("P3");
}

Of course this is no way a finished work, but I hope I will get some others working on this to. Any comment is appreciated.

Cheers,

Xanadu

[Poumtschak]

Wow. Just wow.

(but why javascript?)

[Xanadu]

Hi Pountschak,

Good question.

First I started coding this in C#. But then I realized:
- There are no nice dials and knobs in C#
- There is no easy MIDI connections in C#
- There's no integration with Ableton Live clips and MIDI clips in C#

And I want to be able to make the editor to communicate with the electribe over MIDI and import MIDI clips from Ableton or a any other storage. The Electribe 2 has a an export function to Ableton, so I want to be able to adapt that if I see any possibility.

I've been coding in Max4Live now for 4 or 5 years. It supports a lot of music oriented stuff that is not available in a general programming language. But the visual language is not friendly for dissecting this big data structure.

Max4Live has Javascript builded in. It also has Java and C, but these API's have steep learning curves. So I decided to go for Javascript:
- Javascript supports rich data structures
- It allows me to use Max4Live and it's music oriented user interface

Can you suggest a better programming environment?

Cheers,

Xanadu

[Poumtschak]

So you're building an editor, without waiting Korg to provide one?

So I decided to go for Javascript:
- Javascript supports rich data structures
- It allows me to use Max4Live and it's music oriented user interface

Can you suggest a better programming environment?

Hell no! Perish the thought!
Just having a hard time deciphering the data structure from the code you posted.

But from your explanation, I understand that javascript makes sense.
It could even be ported to Reaper, since it has built-in javascript as well.
http://www.reaper.fm/sdk/js/js.php

Anyways, nice work!

[Xanadu]

So you're building an editor, without waiting Korg to provide one?

The Electribe 2 is the first Electribe I own. I've searched the internet for editors on previous Electribes. I did only find third party editors. Perhaps KORG has the best record in the world for providing great low budget gear, but not for supporting it.

Besides that, I like to go to the nitty gritty to understand what's going on.

My first goal is to isolate parts and sequences, save them to disc and to combine them into new E2Pats.

[cntrlchng]

Awesome!

try disassembling the fw update file!

[bingkingbo]

wowowo, so cool!! great work

[256K]

the only thing my puny brain can decipher from this is that this is f*ck*ng exciting! i can't wait until you amazing tech guys reverse engineer this machine and we get access to its guts!

[Poumtschak]

First I started coding this in C#. But then I realized:
- There are no nice dials and knobs in C#
- There is no easy MIDI connections in C#
- There's no integration with Ableton Live clips and MIDI clips in C#

Hello again,

Have you considered using the Ctrl framework?
http://ctrlr.org/

Much like the good old mixermaps on Cubase (yes, I'm that old), but multi-platform, VST/AU or standalone.

We could start by using the MIDI implementation chart (i.e. CC) of the electribe2 to manage patches per part with a graphical UI.

[Xanadu]

Hi Poumtschak,

I did not not know of Ctrlr until Today. Certainly I wil give it a try. I've been using Ableton since 2008 and Max4live since the first day it was released. So this combination has become my default platform with great possibilities for scipting / programming.

I will be able to publish the editor as a device in Ableton, but also as a stand alone executable for Windows, but not for Apple.

Today I wrote code to save a part, a sequence and a part with it's sequence. The next step is to be able to load these files back into a pattern. This gives the basics for a librarian of patches (parts) and sequences. I expect to publish the first version as a work in progress on MaxForLive.com next weekend.

The MIDI chart of electribe 2 is quite limited. It only supperts the dials, but not the big jogwheels on the upper row. Motions are not exported either.

I think I will be able to use MIDI to edit vaues in the e2pat files fron the electribe itself.

Previewing of sequences is a possibillity too.

Cheers,

Xanadu

[Poumtschak]

The MIDI chart of electribe 2 is quite limited. It only supperts the dials, but not the big jogwheels on the upper row. Motions are not exported either.

You're right about the jogwheels.
But from the MIDI Implementation Chart, it seems there's a Korg exclusive messages (SysEx) support, that should cover these, and - most likely - a lot more.
Its documented for all the previous models in the electribe series
http://www.korg-datastorage.jp/Software/MIDIImp/
We just have to wait for an official release for the current version.

Congrats on what you've achieved so far.

[Xanadu]

I've not been able to get more response from Sysex than this:

Code: Select all

OUT: 240 66 127 10 247
SYSEX: 240 126 0 6 2 66 35 1 0 0 1 3 0 0 247

I suppose the last 1, 3, 0, 0 codes is the firmware version.

Has anyone found additional Sysex codes?

I can load previous saved parts and sequences now. When I save the patched E2PAT file it load into my electribe!

@Poumtchak: do you have Ableton Live with MaxForLive? Of course there's a hefty price tag. Reaper as almost free en is cltrlr is freeware. I can compile my editor into a Windows executable that could be distributed for free. But I need som assistance to compile it for Apple.

Cheers,

Xanadu.
[/img]

[256K]

DUDE YOU ARE AMAZING FOR ALL THIS WORK. I cant wait till you post it for us laymens to use

[Poumtschak]

I've not been able to get more response from Sysex than this:

Code: Select all

OUT: 240 66 127 10 247
SYSEX: 240 126 0 6 2 66 35 1 0 0 1 3 0 0 247

Which translates in hex into :

Code: Select all

SYSX: F0 42 7F 0A F7
SYSX: F0 7E 00 06 02 42 23 01 00 01 03 00 00 F7

What's this Request message you're using?

Code: Select all

F0 -Identify sysex
42 -Korg manufacture's ID
7F -?
0A -?
F7 -EOX (end of exclusive) 

Mine does not respond to that.

It does not respond to the standard SysEx ID Request either:

Code: Select all

F0 7E 7F 06 01 F7

It works with my nanoKEY2 though:

Code: Select all

000010D3  MOX  2     F0  Buffer:     6 Bytes   System Exclusive      
SYSX: F0 7E 7F 06 01 F7
000010DE   1   2     F0  Buffer:    15 Bytes   System Exclusive      
SYSX: F0 7E 00 06 02 42 11 01 01 00 03 00 01 00 F7

Too bad I dont' have much time these days to dive into this. I'm puzzled.

[leehu]

Sysex manual is on the way - was due at the end of Jan so hopefully not much longer...