No, this bot was not made to scalp, I take pride in my morals.

The Tech Stack

• Node
• Puppeteer
• Discord API

The name Marvin refers to the paranoid android from Hitchhiker's Guide to the Galaxy and the Useless Bot part is a nod to the butter bot from Rick and Morty. There isn't much to say about it, but I figured I'd add it anyway because it was fun to implement.

Enter the Pandemic

Supply chains were bust, people were bored and prices were at an all time high. If the first line didn't make it clear enough, scalping was also a very common practice. People would find ways to buy up products high in demand and low in supply and try to resell them in the aftermarket for extortionate prices. You had about as much luck trying to get anything you wanted at retail price as winning the lottery.

Marvin the Useless Bot was my personal solution for my own needs during the pandemic. As his name implies, he was useless. His only purpose was to track certain websites and send a push message to my Discord account if any notification state was changed about a product's availability. He did nothing else.

So if he was nothing more than a glorified push notification system, why is he worthy of a spot on the portfolio page? If my other projects don't showcase it enough, I have a tendency to take small ideas I have and blow them out of proportion. My personal world of feature creep.

In creating Marvin I gained solid exposure to Node and RESTful APIs. It was my first dive into working with JavaScript entirely on the backend. On top of that, I went down the rabbit hole that is web-scraping. I had previous professional experience building web-scraping scripts in Python for my job in the financial industries. If you're unfamiliar with web-scraping or "spidering", the idea is to collect information on the internet that exists on sites that may not provide a conveient means to access it. (Yes, this may be abused for nefarious reasons, but here we are strictly approaching this from an ethical standpoint.) Usually this means a site may not have an API setup for the data. If you are dealing with small or infrequent data, merely going onto that site and manually copying it might suffice. However, in the case of my job, I was dealing with daily economic data that was sometimes provided by the source on an almost hourly basis. For most of the data we acquired, we had partners, contracts and API keys to acquire the latest and greatest in the financial markets. However, when it comes to economic data in developing nations, particularly monetary data released by central banks, the data is often uploaded on static sites with little to no means to efficiently acquire it on a frequent basis. Enter our friendly neighborhood spider-bot. Assuming you have consent from the data provider to scrape their data, you can build automatic scripts that can go onto these sites and extract the data directly from the HTML it sits in. If you know what you're doing, you can easily acquire data from a provider the second it is released and leave a digital footprint as large as any other user visiting the site (or even less)! This is actually fantastic news for everyone involved, as this leads to less bandwidth and strain on the provider's servers and for your own client device! That was a long winded example about how web-scraping can be used in production and ethically. But for me, having a web-scraping library in JavaScript opened a world of possibilities. JavaScript is, after all, the language of the modern web. So having a single language to be able to fetch and wrangle the data felt like a massive boost in efficiency for me.

How He Operated

All the target links that I had Marvin track were static direct URLs that I checked out personally beforehand. This was done for many reasons. For one, Marvin didn't have much freedom, so recursively tunnelling to find what I was looking for was off the books. Second, although I had plans to expand his functionality, I pretty much held his hand in finding the elements on the page that I was interested in. When the data was fetched, it was stored in an SQLi instance where I could keep track of the target, the timestamp of each request, the response and the state of the element I was tracking. The gameplan for the database setup was for tracking long-term price changes. That way, deviations in retail prices months later could also be noted.

All Hail Efficiency and Morality

Marvin may have been useless, but he was a good boy. I developed him to be as noninvasive as possible. At his highest page request rate, he was limited to a single call to the target every five minutes. At his slowest, I limited him to one call every 24 hours. I even had plans to implement in a dynamic rate system, one that would increase the calls as changes to the site became more frequent and slowed down the calls if no data update was found between subsequent requests. Furthermore, I kept Marvin's link-depth to 2. He was only allowed to examine the target page and any links that pointed from it by one link click. I didn't have any reason to increase this number and my paranoia kept me from trying that even if I wanted to. With the number of links on a modern web page such as an e-commerce site, you can quickly get into hot-water with a roque bot.

It's 10PM, do you know where your bot is?

Gone too Soon

Marvin reached his EOL at the end of the pandemic. Rest in peace lil' buddy.*

*Unless I find another use for Marvin. We had a lot of memories together, but he needs more usage, I figured about extending his capabilities outside of Discord and opening him up to the public but not quite sure what I could do with him that isn't already readily available with other bots.