13.5) Batch Processing Of Data Files

We now have all the knowledge required to produce a script to batch process files. This is a powerful procedure and could save you a lot of time if your work involves repetitive tasks.

Let's consider that we have a series of data files named data_01.dat, data_02.dat, ..., data_99.dat. We want to cycle through these files and then load the data into a matrix which will hold all the individual data sets. How do we go about doing this?

[matlab]
>> % Create a list of data_??.dat files by piping the
>> % appropriate system command to file "files.txt"
>> if (isunix) % note this is shorthand for if (isunix==1)
>> unix('find . -type f -name data_??.dat > files.txt'); % find is a useful unix command
>> elseif (isdos)
>> system('dir data_??.dat /b > files.txt'); % dir with /b tag will just output file names
>> end
>> files=importdata('files.txt'); % now lets load the data
>> % create an output array and then populate it with the data
>> output=ones(360,180,99)*-999;
>> for i=1:length(files) % lets cycle through all the files
>> yr=str2num(files{i,1}(1,6:7)) % Extract the file number
>> output(:,:,yr)=importdata(files{i, 1})
>> end
>> output(output==-999)=NaN; % there wasn't a data file available for this year
>>
[/matlab]