Any docs about the PCG file format?

[pandel]

Hello everybody!

Haven't been here for a while... but now I'm looking for some kind of documentation about how data is arranged in a PCG file.

I'd like to try to develop some kind of library software by myself and the neccessary file format documentation seems to exist, because some people out here have done so before.

Does anybody know (perhaps especially Daz or Dan), if there is a way for me to get some more descriptive documention that is needed for such a project?

Regards,
Holger

[danatkorg]

Hello everybody!

Haven't been here for a while... but now I'm looking for some kind of documentation about how data is arranged in a PCG file.

I'd like to try to develop some kind of library software by myself and the neccessary file format documentation seems to exist, because some people out here have done so before.

Does anybody know (perhaps especially Daz or Dan), if there is a way for me to get some more descriptive documention that is needed for such a project?

Regards,
Holger

I'm sorry, but there is no documentation for the format. As far as I'm aware, any apps which access or create PCG files (such as those that Daz hass created) have used the system exclusive format as a guide. Unfortunately, as of software version 1.3, the PCG format has diverged somewhat from the system exclusive format. I wish that I could have a better answer for you.

[Charlie]

Dan, I don't understand: What does "diverged" mean?

[X-Trade]

What does "diverged" mean?

It means split away, separated out, parted...
For example at points on a railway track, the two tracks diverge.

I think Dan is saying that, since OS 1.3, the PCG format is no longer strictly a sysex based file - it now contains bits of data in other formats too, for whatever reason.

[pandel]

I'm sorry, but there is no documentation for the format.

Does that really mean "no documentation" at all or only nothing publicly available? Could you explain why Korg keeps it as a big secret? I think now that there'll be no more active development for the OASYS, Korg should be happy to see that there is some kind of active community development effort going on! With its outstanding capabilities and the myriads of possibilities the OASYS has to offer, a much smarter way of organizing progs and combis is needed.

I think, even if one would have to sign some kind of NDA or so, Korg shouldn't stop those efforts...

@Edit:
Is any detailed sysex/midi implementation documentation available?

[Daz]

If you only wish to work with PCG files in a simple way (moving programs or combis around for example) then you only need to know one thing. The PCG file uses a very RIFF chunk like format that you'll have seen used by MIDI, WAV, BMP files etc. If you look in your Parameter Guide you'll see the specification is given for KSF files. KSF files are also RIFF like, and it will give you a few clues as to how to work with PCG files which work similarly. You don't really need a document to work it out, unless you are looking at parameter details for individual programs/combis rather than just looking at groups of those.

I don't think Korg are trying to keep this stuff secret, it's more likely just a matter of resources. Should your stretched development team be working on developing new products or preparing documentation for third party developers ? The former right ? I wish there were more open documentation and I have got stuck with some of my own developments, because I don't have the time to chase or decode details. I have my actual work to do ! I do understand where Korg are coming from though, because I appreciate from personal experience the extra effort that is required to maintain high quality developer focused documentation.

Daz.

[pegnafroy]

And keep in mind that Dan Phillips is who wrote the most part of the manual, so the docu team is very, very narrow.

[pandel]

Many thanks, Daz! Regarding the RIFF like structure, I'll look into it.

I don't think Korg are trying to keep this stuff secret, it's more likely just a matter of resources. ... Wink I do understand where Korg are coming from though, because I appreciate from personal experience the extra effort that is required to maintain high quality developer focused documentation.

I absolutely understand and agree in general! And I would not expect them to prepare those documents now. I rather thought, they would exist somehow nearly as long as the OASYS exists. At least some basics, like the complete MIDI implementation or things like a file structure. Imagine the developer crew, which knows all the magic details, getting into severe trouble after having a nasty mushroom soup for diner before the OASYS was discontinued...

@pegnafroy
Yep, I know that ! I very much appreciate his work on the documentation!


But never mind! I understand that I have to investigate things by myself, even though I have to admit, that for my part I'm not very good at analysing file structures. If I have a format description, everything's ok, but if I have to find out everything by myself, I'm always afraid of doing something terribly wrong...

@Daz
But what I don't understand is that somehow some kind of information should exist, when I understand this correctly: http://www.korgforums.com/forum/phpBB2/ ... 8&start=48

Did you ever get the information you were looking for?

[Daz]

Here is some C++ snippets from the code I wrote from the Oasys that might help to show how things work.

Firstly here are RIFF like chunk id's for each blob of data in the PCG :

Code: Select all

// *** 
// ***	Oasys PCG chunk ids
// *** 

typedef DWORD CHKID;
#define DEFINE_CHKID(magic)  (magic)

const CHKID KORG = DEFINE_CHKID('KORG');
const CHKID PCG1 = DEFINE_CHKID('PCG1');

const CHKID DIV1 = DEFINE_CHKID('DIV1');
const CHKID INI1 = DEFINE_CHKID('INI1');
const CHKID INI2 = DEFINE_CHKID('INI2');

const CHKID PRG1 = DEFINE_CHKID('PRG1');
	const CHKID PBK1 = DEFINE_CHKID('PBK1');
	const CHKID MBK1 = DEFINE_CHKID('MBK1');

const CHKID CMB1 = DEFINE_CHKID('CMB1');
	const CHKID CBK1 = DEFINE_CHKID('CBK1');

const CHKID DKT1 = DEFINE_CHKID('DKT1');
	const CHKID DBK1 = DEFINE_CHKID('DBK1');

const CHKID WSQ1 = DEFINE_CHKID('WSQ1');
	const CHKID WBK1 = DEFINE_CHKID('WBK1');

const CHKID GLB1 = DEFINE_CHKID('GLB1');

Here are the structures for each chunk type :

Code: Select all

// ***
// ***	Oasys PCG File Structures
// ***

#pragma pack(1) 

struct PCGKORGHeader
{
     CHKID idKorg;         // 'KORG'
     BYTE  bProductId;     // 0x70
     BYTE  bFileType;      // 0x00 = PCG
     BYTE  bMajVer;        
     BYTE  bMinVer;        
     BYTE  abPad[ 8 ];     
     PCGKORGHeader() { memset( this, 0, sizeof(*this) ); }
};

struct PCGChunk
{
    CHKID idChunk;         
    DWORD cbChunk;    
	DWORD dwX;			   // not sure what this is -- entity count ?	     
    PCGChunk() { memset( this, 0, sizeof(*this) ); }
};

struct PCGBankHeader	   // used for all data types
{
    DWORD cItems;          // number of items (progs, combis, dkits)
    DWORD cbItem;          // size of each item
    DWORD dwBankId;        // 0x00-0x04 = A-E, 0x8000 = F, 0x20000 = U-A etc.
    PCGBankHeader() { memset( this, 0, sizeof(*this) ); }
};

#pragma pack()

Now here are the two functions to read/validate the file and then get references to all the banks in this file :

Code: Select all

bool CPCGFile::Read( const char* pszFile )
{
	// *** get file size 
	
	struct stat st;
	
	if( 0 != stat( pszFile, &st ) )
	{
		m_strRWError = "Failed to retrieve file status";
		return false;
	}
	
	size_t cbFile = st.st_size;
	
	// *** attempt to open file
	
	std::ifstream ifs( pszFile, std::ios::binary|std::ios::in );
	
	if( ! ifs )
	{
		m_strRWError = "Failed to open file for reading";
		return false;
	}
	
	// *** validate KORG and PCG1 headers
	
    PCGKORGHeader KorgHdr;

    ifs.read( (char*)&KorgHdr, sizeof(KorgHdr) );

    if( twiddle( KorgHdr.idKorg ) != KORG || KorgHdr.bFileType != 0x00  || ! ifs )                        
    {
        m_strRWError = "File does not have a valid KORG/PCG header";
        return false;
    }

    if( KorgHdr.bProductId != 0x70 ) // Oasys
    {
        m_strRWError = "PCG File is not in Korg Oasys format";
        return false;
    }

    PCGChunk pcg1;

    ifs.read( (char*)&pcg1, sizeof(pcg1) );

    if( twiddle( pcg1.idChunk ) != PCG1 || ! ifs )
    {
        m_strRWError = "PCG File does not have a valid PCG1 header";
        return false;
    }
	
	// *** rewind to the beginning of file and suck the whole thing into memory
	
	ifs.seekg( 0 );
	
	m_pbData = new BYTE[ cbFile ];
	ifs.read( (char*)m_pbData, (std::streamsize)cbFile );
	m_cbData = cbFile;	
	
	// *** now parse the bank data
	
	parseBanks();

	return true;
}

void CPCGFile::parseBanks()
{
	size_t iPos = 0;
	bool   fBanksDone;

	// *** jump over KORG and PCG1 header
	
	iPos += sizeof(PCGKORGHeader) + sizeof(PCGChunk);
	
	// *** loop round pulling in chunks/banks
	
	while( iPos <m_cbData>idChunk ) )
		{
		case PRG1:
		case CMB1:
		case DKT1:
		case WSQ1:
		
			fBanksDone = false;
		
			while( iPos <m_cbData>idChunk ) )
				{
					case PBK1:
					case MBK1:
					case CBK1:
					case WBK1:
					case DBK1:
					{
						iPos += sizeof(PCGChunk);
					
						PCGBankHeader bkhdr = *(PCGBankHeader*)&m_pbData[ iPos ];
						
						bkhdr.cbItem = twiddle( bkhdr.cbItem );
						bkhdr.cItems = twiddle( bkhdr.cItems );
						bkhdr.dwBankId = twiddle( bkhdr.dwBankId );
						
                                                // construct CPCGBank instace with pointer to beginning of bank data, the bank type, and the bank header which 
                                                // indicates the number of items in the bank and how large each is
						CPCGBank bank( &m_pbData[ iPos + sizeof(PCGBankHeader) ], twiddle(pBankChunk->idChunk), bkhdr );
 						
						m_banks.push_back( bank );
						
						iPos += twiddle( pBankChunk->cbChunk );
					}
					break;
					
					default:
						fBanksDone = true;
						break; // chunk is not a bank ... leave for outer loop
				}
			} // while( iPos < m_cbData ) ... reading banks
			break;
			
		default: // jump over chunks we don't care about
			iPos += twiddle( pChunk->cbChunk ); 
			break;	
		} // switch( pChunk->idChunk )
	
	} // while( iPos < m_cbData ) 
}

So the file consists of a bunch of chunks for each kind of data, but the ones we're specifically interested in are PRG1, CMB1, DKT1 and WSQ1 which contain the Program, Combi, Drumkit and Wavesequence banks. Once you're inside each of those chunk types you'll find a number of chunks for each bank. So a PRG1 chunk contains a number of PBK1 chunks, one for each bank of Programs in the PCG file. The data in the PBK1 chunk is data for all of the programs in that bank. You can slice it up using the cbItems member of the PCGBankHeader structure, that is the size of each item.

Just for larks note that EXi programs are not stored in PBK chunks, but MBK chunks. On the Triton MBK chunks were used to store MOSS programs, hence the M in the name. Exciting trivia, eh ?

Daz.

[Daz]

p.s. This kind of format is useful to know about. As I say it's not just PCG files that work this way, or even just music related files like WAV and AIFF. Many image file formats are similar, such as BMP, TIFF, and even some RAW camera files IIRC.

p.p.s As you'll see no sysex information is used in the code. You only need the sysex information once you start wanting to extract parameters from the data stored in those blobs in the file. The way that program/combi/dkit/wseq parameters are stored in the PCG chunks is very closely related to the format used when sending the exact same information using MIDI/Sysex. This is true of the Triton, M3 and Oasys.

Daz.

[Kevin Nolan]

Korg are are a fine company in many ways, but they do NOT understand the concept of professional support in the keyboard performance world.

They rarely offer in-depth tutorials on their products; they have a lousy service policy and infrastructure, they make absolutely no effort to publicise spare parts, they do not make service documentation available and have no comprehension of a global support network (and why so often you'll see 'special offers' from Korg US for example but with no cooperation to offer similar deals to other regions (that bugs me in particular because surely Korg should know that every offer made available to one region only constitutes a slap in the face to users in other reasons; but they don't get that)).

So while OASYS is great and Korg are an earnest company in so many ways - most especially in their commitment to new development and to keyboard players / synthesistis; they could do with dropping some of the apparent protectionist paranoia that seems to emanate from Korg Japan and that does affect their persona, but also their own capacity/desire to support their user community.

In particular, they should have a comprehensive web based spares and accessories portal (take a look at Yamaha US spares and accessories - its amazing); and they should release service documents for sale for all end-of-life instruments to service their well known flawed parts - faint lcd screens, flimsy switches, broken joy sticks and so on.

By comparison, Yamaha are awesome. Yamaha-Kemble UK are simply incredibly accommodating with the sale of accessories and service manuals and will even seek out parts in Yamaha Japan in particular if they are not available in the UK; while as I said the Yamaha US spares web portal is amazing (though I wish that was world wide too). But world wide Yamaha are great at providing service documentation and Korg should also follow suit in this regard


Kevin.

[Charlie]

@X-Trade: thanks for explaining the word - my fault having it written like that. I know what "diverged" means in general, but I didn't understand what Dan meant regarding the change in pcg-file-structure.

[pandel]

Need to use the 'Sina'ticons here

Dear Daz,

what an amazing answer and, I hope you believe me, it's much more than I expected ever!!!!!

So, as soon as I've time I'll give it a try!

Many, many thanks!

[danatkorg]

I don't think Korg are trying to keep this stuff secret, it's more likely just a matter of resources. Should your stretched development team be working on developing new products or preparing documentation for third party developers ? The former right ?

With regards to the PCG file format, that is exactly the case. I truly wish that we had documentation for this, but we simply do not.

Best regards,

Dan

[pandel]

Dan, as I said before, I highly appreciate your work on the documentation and absolutely believe that *you* personally would offer something if you had something.

I don't want to argue, I just thought, that the different development teams that worked on the parts of the OS would have had some basic doc stuff to coordinate what they were doing. I mean, I really don't believe the USA dev team has had to re-discover all the MIDI messages (Midiox ) introduced by the Japanese dev team to be sure not using them twice, only because the Japanese had not documented it...

And finally, I think, a product like the OASYS has to be accompanied by a complete (!) end user documentation, even if some work still has to be done! I would expect that from every high price product. (Besides, for me that includes the availability of a service manual after the product has come to an official end, too.)

Don't get me wrong! The op guide and the param guide are both exceptional and highly detailed. I love them, nothing to discuss here! And I understand the ressource problematic but it's a pity!

Have a nice weekend !