scrapy bits
Nerding Out
Building a module to view progress towards filling contracts at a glance
My help is on strike this morning, so we're not getting after it until they stop picketing.
It's not so dire, they just want to watch the Vikings play the Jets in London. So do I, but I vowed years ago to minimize the time in which I devote my full attention to the perpetual disappointment that is the Minnesota Vikings so while I watch the game I'm building a thing.
Over the years during harvest I get asked the same questions over and over again, daily. It's not as if I want to avoid the question, I just want to quickly and accurately answer it so my team can all be on the same page. The question is a variant of this: "how much grain do we need to bring to the elevator to fill contracts?" Also, "how full are the bins?" We'll save "how many acres did we harvest?" and "what was the yield?" for a later post.
Valid questions, all. The challenge in determining the answers to these questions is dealing with different data sources. Scale tickets from the elevator, and monitor data from the combine. Last year I built a thing to pull scale tickets and update my financials, and this year, I've made progress towards building a dashboard. It looks like this:
Do I need to blur the figures? I don't know if I care but it seems like maybe I should. Farmers in general are usually cagey about their numbers. Whatever, I might loosen my standards about this at some point but for now I'm just gonna blur the things.
Anyway, I have a monitor in my shop that shows this nifty little graphic, which I built using selenium in python to scrape elevator tickets from my local elevator into my financial database. Then I use SQL + pandas to query my financial db (GNUcash), and to make a pretty table I'm using Great Tables, which I just learned about a few weeks ago. It's dope.
Where was I going with this. OH! Building a thing while I watch the Vikings. Yeah, so this morning I thought it'd be cool to make selenium scrape John Deere Ops Center reports so I could automate the scraping of my daily production table. After some trial and error, I got it working to download the report. Now I just need to wrangle the XLSX spreadsheet with Pandas and plug it into my database. Sounds simple enough, right? It won't be. But I know it's possible now which means I have to do it.