I was recently working on a project, the code was running slow and we weren't real sure why.
Here are some things to think about when trying to speed things up. You can use a profiling approach, and that can point where your slowdowns are. I did a bit of profiling but it wasn't helping, so I took another approach.
The first thing we were doing was reading content from a file. To me figuring out a baseline helps us understand, with the exception of using a new language or library that can read faster, once we were able to read the file in a silo, no other processing this was our baseline.
What I did was just write a simple Python script it read the file and threw away the output, I added some code to calculate how many lines per second we were reading roughly. It was fast.
However, overall, processing the file, which includes reading the file, doing some stuff, saving the results off, may have taken a considerable amount of time, but after baselining we realized we could at least get the lines wayyyyyyy faster than what our current code was doing. (We were using a different approach to read the file than what my baseline was, there were reasons for the alternative approach, but ultimately they weren't needed so this was a source of major speedup).
I may come back and give other tips but find a baseline so you know, if my code runs in hours, and I know that a particular step takes an hour, you know that whatever your final improvements shouldn't be faster than an hour (excluding any extreme optimizations, or language changes).
This also helps you guide, if for some reason that step takes a lot more than an hour fixing that step to reach that hour will get you some good gains.
Have y'all ever heard of Ahmdalls law?
The short version (which I'm butchering) is if a particular step only is a small part of processing then optimizing that step won't get you as good as gains compared to a step that is a major part of processing. Figure out what are your major steps, baseline those, and work from there find ways to move your code towards the baselines version as much as you can and you should see improvements.
