Data Processing

Finally, I am done with the processing of all raw data. I have managed to create a single file with the following fields: Date, Day of the Week, Temperature, Hour, Day-ahead Price, Real-time Price, and Load. At this point in time, since the daily temperature of Philadelphia was not available till 30 June 2005, I had to change the range of dates as 1 June 2000 to 31 May 2005.

I used DayofWeek property of the DateTime structure to extract the correct day of week for a given date. Depending on the results, I am thinking of changing the Day of the Week to a binary datatype to hold if it is a weekday (Mon-Fri) or weekend(Sat, Sun).

The data is organized in my desktop as: Raw, Pre-processed and processed data.