What I love about automating things, is that once you figure out how to do something once, you can do it a million times more for only a tiny bit more work.
Only two weeks ago, I was struggling with matlab to download even one line of data from a tcpip socket. But once I figured out a few relevant commands, everything started falling into place.
- It all started when I figured out how to load a dll and connect to the socket.
- Eight days ago I figured out how to read a single line from the socket.
- Seven days ago I learned how to quickly parse the line into separate fields.
- Six days ago I solved how make sure the data was clean and downloaded a full day’s worth of data for one contract.
- Five days ago I went to the Del Mar Racetrack with my coworkers and relaxed the rest of the weekend in San Diego.
- Then yesterday I made some more incremental improvements and downloaded 8 days worth for 3 contracts.
- Today I finished an automated program that downloaded over 2 million lines from IQfeed, with the ability to queue up a pretty much unlimited amount for the future.
The limit imposed by our data provider (DTN IQfeed) is 30 days worth. I think I’ll schedule a download once every week. It’s great because now that I’ve finished with that program I can move on to bigger and better things, while the fruit of my labor grinds it out for me downloading massive amounts of data.