PSG Step 5: Getting off the SD Card

As mentioned, since recording for long periods over bluetooth is pretty risky, it makes a lot of sense to record onto the extremely stable micro SD card. The main OpenBCI SD Card Tutorial covers the basics of formatting the card and recording, and the process is very straightforward. However, in order to be maximally efficient, OpenBCI stores the data on the card in hexidecimal, so the resulting file has to be converted before it can be opened and viewed in the OpenBCI GUI or any other software. Their site does explain how to go about this conversion process, but there’s a but.

There are (or at least seem to be) two big problems here for polysomnographers. Firstly, the standard OpenBCI process described on the site for converting from the hexidecimal file on the SD card to a regular CSV file just doesn’t seem to work on the huge, long (12 hour) files required for a night’s recording, at least not on either of the two (decent) Macs I’ve tried. Attempting to do so generates an “out of memory” type error. Four hour recordings work fine, but not 12, and there are no options for anything in between.

I tried getting help on this issue from OpenBCI, who referred me to another forum, and in spite of the efforts of a friendly fellow named Andrew, it still just doesn’t work, for me at least – see this thread. If anyone out there routinely converts 12 hour recordings successfully using the regular process, let me know! My guess is that this just doesn’t come up for most Open BCI users…

But even once that problem is solved, there is a second second problem: Converting the CSV file produced by the Cyton into an EDF format that I can open with my awesome, open source “EDF Browser“. This is necessary to be able to score the file, crop it to a nice patch of REM activity and so on.

So I’m going to just set aside the first problem for the time being and come back to it later. Last night I did a four hour recording, resulting in a 266mb file. Four hours is obviously too short to record a full night’s sleep (I wonder why they don’t have an 8 hour option?), but it is better than nothing, and should still allow me to capture some REM periods. So, in order to get this file from the SD card to something readable in EDF browser, here are the (cumbersome) steps:


It starts like this, in hexidecimal on the SD Card
OpenBCI converts it, and I stuck on a bunch of zeros.
Not pretty, but it's in EDF!
  1. Launch the OpenBCI GUI from WITHIN the Processing environment – the translation process does not work in the standalone GUI. Again, the instructions on the OBCI site are quite clear – Launch Processing, then go to your sketchbook and launch the GUI from there.
  2. With the GUI launched, select Playback –> Convert SD card, and then select the file for conversion (typically OCBI_15.txt) or whatever.
  3. Very quickly this will place a converted file into a folder deep in the Processing Sketch folder. This location is NOT the same as listed on the OpenBCI website, BTW. (I think they might have just made a mistake). For me, it is located at Documents -> Processing -> OpenBCI_GUI -> OpenBCI_Gui (yes, again) ->Saved Data. I have a shortcut/alias for this folder in my “original scans” folder. The resulting file is called “SD Converted…” with a date and time stamp.

    So far so good – Now I should be able to open the file with the OpenBCI GUI, just as if I’d recorded it over bluetooth. But as cool and good looking at that GUI is, I can’t seem to fast-forward nor rewind… nor downsample or crop the data or anything else. For that I need EDF Browser. Fortunately, EDF browser has lots of translation abilities. Unfortunately, the OpenBCI CSV file is a pain in the butt to translate.

  4. Here’s a tricky part. The resulting CSV file contains those three accelerometer columns that are not recorded at the same rate as all the other data – you can see this in the first image on the right, those zeros that stick out every 10th row. EDF browser freaks out if you try to translate that file, because the number of columns is not consistent. Which sucks. Intuitively, I’d like to open the thing in Excel or whatever and just delete those last three columns, but the file is too large to open like that. So instead, I add a bunch of zeros to all the columns that don’t have accelerometer data in them. (!) This is achieved by opening the file in TextEdit and using Find/Replace. I replace “0.0,0.0,0.0 [Line Break]” with “o.0,0.0,0.0,0.0,0.0,0.0 [Line Break]”. This adds three columns of zeros to any row that ends with zeros. (Obviously if I were using more channels, this replacement would not be the same – I had three blank channels in my configuration last night, so I did it this way). This takes my machine a minute or two, but good old TextEdit does the trick eventually.
  5. Now that there is a CSV file with consistent columns, I can use the EDF translator. I go to EDF Browser, and selected “Convert ASCII to EDF”. This opens the central configuration window. For complete instructions on how to do this part, the User Manual is helpful. But for my setup (and for any OBCI setup) I chose:
    1. Separator = “,” (commas)
    2. 12 columns (The first is just an index, plus 8 for the main channels, plus those irritating 3 accelerometer columns)
    3. Frequency = 250hz (the header of the CSV file confirms this too)
    4. Data starts at line 9 (as is visible in the second image on the right)
    5. Label every column however you want, just don’t leave any blank (in my case I’m using five channels: EMG, EOG, Fp1-A1, Fp2-A1, O1-A1).
    6. UNCHECK those blank columns you don’t need (the little checkboxes on the left)
    7. Select “start”
    8. Choose the file you want to translate (maybe a bit counter-intuitive that you chose the file at the very end, but whatever!)
  6. This takes a bit of time (2-3 minutes?), so do not get impatient and start mashing “escape”. At some point there will be a prompt to save the file. However, even  after this prompt, it still takes another 2-3 minutes for the program to announce a successful save.
  7. This file can then be opened with EDF Browser. I opened my file and everything was dandy. Except that the data looks ugly as all get out – so now I ran the same notch filters (50hz since I’m in Europe) and bandpass filters (1-50) as appear in the OpenBCI GUI, and… better, but still kind of gross. So obviously I still have lots to learn about EDF browser and filters… I was not helped in this case by the fact that I stupidly attached the Bias cable to the AGND pin rather than Bias, so obviously there is more noise on the EEG than there should be! But this error isn’t relevant for the purposes of getting data from the SD card into EDF.

I’ve no idea if this will be helpful to anybody else, but at least now I’ve recorded this annoying process for posterity and  I’ll be able to look this up whenever I need to repeat it! I cannot help thinking there is an easier way to lop off those useless accelerometer recordings, or just not use them in the first place. If you stumble across this and have anything to share, please do get in touch! 

Having documented all this, there is, however, an easier way. Or at least a somewhat easier way, particularly if you are a skilled Windows or Linux user (I’m not really). It relies on using another bit of software, that developed by Dr. Frederik D. Weber: SpiSOP. In my next post I’ll go through how to get from data recorded on the SD card to viewing it in EDF browser using SpiSOP.