
# Using Data from Twitch and Amazon to Populate a DB for a Video Game Review Site Project

For our latest group assignment we were tasked with creating a game review site in just 3 days. The vague brief and tight timeline were designed to give us a taste of the poorly defined projects often found in the "real world".

As the group began to write the specification for the full stack MERN app, it was clear we would need a good amount of dummy data to show the app in all of its glory when presentation time came along. As the shape of the data would infer the collection model as well as the front end components this was a high priority task which I took on. My plan was a simple one, to find an api that served up video game data and fetch a suitable enough amount of data fo us to use in our app. I estimated it would take me less than an hour to have the data ready to populate our db. I would soon learn that this was a gross underestimate.

## Signing up to IGDB

A quick google lead me to Twitch's video game database, IGDB. It ws a vast collection of titles and data and combined with a generous usage policy, I didn't look any further. In order to use the service I would need a Twitch account to generate a Client ID and a Client Secret.

Unfortunately I made this a lot harder than it needed to be. When creating a Client Secret I was asked to provide an OAuth Redirect URL. It stated below the field that for local testing I could use `http://localhost` but unfortunately when I copied and pasted the url I unknowingly included a leading space which resulting in the form validation telling me I couldn't use localhost as a url.

After a good of hour of going round the houses on the twitch dev forums and even contemplating building the necessary endpoint I finally realized my mistake and could move on :-). I won't be making that mistake any time soon. Also note to future self: make sure you trim all whitespace before input validation.

## Fetching an oAuth Token

After finally getting my credentials, fetching an access token was much easier. I installed node-fetch and dotenev and added my `CLIENT_ID` and `CLIENT_SECRET` to my `.env`.

```bash
$ npm i node-fetch dotenv
```

```javascript
// dummyDataCreator.js
const fetch = require("fetch-node");
require(dotenv).config();
```

I then sent a request to Twitch's oAuth token endpoint as specified in the IGDB docs.

```javascript
// dummyDataCreator.js
const getAuthToken = async () => {
  const baseurl = "https://id.twitch.tv/oauth2/";
  const endpoint = "token";
  const query = `?client_id=${process.env.CLIENT_ID}&client_secret=${process.env.CLIENT_SECRET}&grant_type=client_credentials`;
  const url = baseurl + endpoint + query;
  return await fetch(url, {
    method: "POST",
  })
    .then((res) => res.json())
    .then((data) => data.access_token)
    .catch((err) => {
      throw new Error(`Unable to get auth token from twitch: ${err}`);
    });
};
```

## Getting Data from the API

When starting to query the api, I made the mistake of assuming that this api would be like all the others I had previously come across. At the time of writing I now realized if I had just read the docs from the start I could of avoided a lot of head scratching.

The api expects the queries to be placed in the body of a POST request. Also all fields must be specified to be returned. This can be done with a wildcard to get back all fields, but this did stump me for while.

```javascript
// example request
fetch("https://api.igdb.com/v4/games", {
  method: "Post",
  headers: {
    "Client-ID": clientId,
    Authorization: `Bearer ${authToken}`,
    Accept: "application/json",
  },
  body: `fields name,rating,summary; limit 3;`,
});
```

```javascript
// returns json:
[
  { id: 70, name: "Terra Nova: Strike Force Centauri", rating: 70 },
  { id: 40104, name: "Dogou Souken" },
  { id: 51663, name: "The Firemen 2: Pete & Danny" },
];
```

As you can see, not every game has all fields meaning the second and third objects do not have a `rating` property.

## Wrangling the Data

After the data was pulled from api I rebuilt the each game object, pruning unnecessary properties and editing others. Img urls were modified to point to the necessary optimized images. As not all fields were guaranteed to be returned, I used optional chaining when access the game properties.

```javascript
// building the object
.then((data) =>
    data.map((game) => {
      return {
        ...game,
        cover: reformatImgUrl(game.cover?.url, "t_cover_big"),
        screenshots: game.screenshots?.map((shot) =>
          reformatImgUrl(shot.url, "t_screenshot_big")
        ),
        genres: game.genres?.map((genre) => genre.name),
        involved_companies: game.involved_companies?.map(
          (company) => company.company.name
        ),
        release_dates: game.release_dates?.map((release) => ({
          timestamp: release.date,
          human: release.human,
        })),
      };
    })
  )
```

## Adding Reviews

When creating the components to leave a review, I wanted to add some review data purely for demo purposes. In reality reviews would be its own service with CRUD endpoints but due to time restrictions this wasn't possible.

To get the dummy data I found a university's repo of scraped amazon reviews. I then used a `Math.random` to add reviews to each video game object.

```javascript
const insertDummyReviews = () => {
  try {
    const numberOfReviews = Math.floor(Math.random() * 20 + 1);
    const reviewList = [];
    let i = 0;
    while (i < numberOfReviews) {
      reviewList.push(
        dummyReviews[Math.floor(Math.random() * dummyReviews.length)]
      );
      i++;
    }
  } catch (err) {
    throw new Error(`Unable to insert Dummy Review Data: ${err}`);
  }

  return reviewList;
};
```

## Populating the Database

With the data created all that was left to do was to populate the db. We were using Mongo Atlas to store the data and mongoose to connect to it. I was using mongoose and Atlas for the first time and they were both intuitive to use. After configuring the access options in Atlas I built a schema for the data.

```javascript
const mongoose = require("mongoose");

const Game = new mongoose.Schema({
  id: { type: String, required: true },
  cover: String,
  genres: [{ type: String }],
  involved_companies: [{ type: String }],
  name: String,
  rating: Number,
  release_dates: [{ timestamp: Number, human: String }],
  screenshots: [{ type: String }],
  summary: String,
  reviews: Array,
});

module.exports = mongoose.model("games", Game);
```

## Conclusion

This certainly isn't the first time I underestimated a task and the nature of the project did make me wonder if my time could be spent better else where. That being said that, once I was able to overcome the initial hurdles I was able to achieve my goals and learn something in the process. Knowing that apis can range in complexity will alow me to better estimate future tasks although as my experience grows I would like to think it would become less a problem.

Oh look, there I go again... Perhaps I haven't learned anything at all. 🤷‍♂️

```javascript
// dummyDataCreator.js

const fs = require("fs");
const fetch = require("node-fetch");
require("dotenv").config();

const getVideoGames = async (numberOfEntries = 20, options) => {
  const baseurl = "https://api.igdb.com/v4/";
  const endpoint = "games";
  const url = baseurl + endpoint;
  console.log(`Fetching video game data from endpoint: ${url}`);
  const data = await fetch(url, {
    method: "POST",
    headers: {
      "Client-ID": process.env.CLIENT_ID,
      Authorization: `Bearer ${await getAuthToken()}`,
      Accept: "application/json",
    },
    body: `fields name,rating,cover.url,follows,hypes,genres.name,screenshots.url,summary,involved_companies.company.name,release_dates.*;where release_dates.platform = 48 & hypes > 20;limit ${numberOfEntries};`,
  })
    .then((res) => res.json())
    .then((data) =>
      data.map((game) => {
        return {
          ...game,
          cover: reformatImgUrl(game.cover?.url, "t_cover_big"),
          screenshots: game.screenshots?.map((shot) =>
            reformatImgUrl(shot.url, "t_screenshot_big")
          ),
          genres: game.genres?.map((genre) => genre.name),
          involved_companies: game.involved_companies?.map(
            (company) => company.company.name
          ),
          release_dates: game.release_dates?.map((release) => ({
            timestamp: release.date,
            human: release.human,
          })),
          reviews: options?.dummyReviews
            ? insertDummyReviews(options?.dummyReviews)
            : null,
        };
      })
    )
    .catch((err) => {
      throw new Error(`Unable to get video game data: ${err}`);
    });

  return data;
};

const getAuthToken = async () => {
  const baseurl = "https://id.twitch.tv/oauth2/";
  const endpoint = "token";
  const query = `?client_id=${process.env.CLIENT_ID}&client_secret=${process.env.CLIENT_SECRET}&grant_type=client_credentials`;
  const url = baseurl + endpoint + query;
  return await fetch(url, {
    method: "POST",
  })
    .then((res) => res.json())
    .then((data) => data.access_token)
    .catch((err) => {
      throw new Error(`Unable to get auth token from twitch: ${err}`);
    });
};

const reformatImgUrl = (url, to, from = "t_thumb") => {
  if (!url) return null;
  return "https:" + url.replace(from, to);
};

const insertDummyReviews = () => {
  try {
    const numberOfReviews = Math.floor(Math.random() * 20 + 1);
    const reviewList = [];
    let i = 0;
    while (i < numberOfReviews) {
      reviewList.push(
        dummyReviews[Math.floor(Math.random() * dummyReviews.length)]
      );
      i++;
    }
  } catch (err) {
    throw new Error(`Unable to insert Dummy Review Data: ${err}`);
  }

  return reviewList;
};

module.exports = { getVideoGame };
```


For our latest group assignment we were tasked with creating a game review site in just 3 days.

Using Data from Twitch and Amazon to Populate a DB for a Video Game Review Site Project


# My Remote Bootcamp Experience: Twitter Clone Group Project

After 6 weeks of intense learning, our cohort was split into small groups for our first group project. Our brief was short and vague; to build a twitter clone front end with a mobile first approach. We were expected to use git but the rest how we would manage the project was up too us as a team. This surprised me a little as learning more about project management was one of my key aims of the course. We were strongly encouraged to use kanban boards and issue tickets however.

## Breaking The Project Down into Tasks

As a group, we decided to breakdown the project into smaller components, and I suggested we used Figma as a tool to help us do this. Figma worked really well, I was quickly able to make a number of boards at different sizes and screen capture the necessary device renders with the firefox devtools. With Figma's multiplayer mode we we able to discuss the images together as a team without the hassle of having to bounce around each others discord streams, something we have be come far too aware of working remotely.

We added a COMPONENTS column to our kanban board and started to create cards. We listed the visual elements needed for each card and discussed functionality. Figma gave us what we wanted to in terms of us being able to talk about the higher level components together, however it lacked any interactivity information and as we started to look at the site on our own devices the complexity grew. To temper the increased scope we added a NICE TO HAVE column to our project board.

## Adding CI/CD

After a little success deploying a small react app using github actions I was keen to further extend my CI/CD knowledge with this project. I created two actions; one to deploy to github actions on a merge to main and a to run eslint on creating a PR request. This took longer than I anticipated as I tripped over issues with creating the workflows. Once the linting was working I decided to park the deployment for now to start working on the components.

As the week went on, deployment was forgotten as other tasks took priority. This didn't really have too much of an impact on the project but in hindsight I wish I had got it running so that I could ensure that new pull requests would pass the build step before being allowed to merge into main and that we all had one source of truth in regards to the visual aspects of our project.

## Git Issues

As expected, we had our fair share of issues with version control. We would often need to pull in important changes to our main branch to our local branches. We began using `git merge` to do this, but later learned that `git rebase` was cleaner and more effective for out needs. I also learnt that you could rebase a local branch from a previous commit allowing you to overwrite unnecessary commits that were not anticipated at the the time.

With the deadline looming, our best intensions for writing well thought out commit messages went out the window. I believe that is a discipline that is best improved over time, however, the biggest mistake was letting ourselves create the time pressure in the first place.

## Managing Scope and Feature Creep

While we were carful at the beginning of the week to set realistic expectations of what could be done, we ultimately failed at limiting feature creep and our goal of avoiding a frantic and stressful race to the finish line.

As we went through the week our aim was to have everything done on our project board by thursday and by wednesday it looked like that was possible. However, by leaving some of the features that were proving difficult to implement and moving on to other tickets we had created a time bomb for ourselves.

These little tweaks soon piled up and as the more experienced of the group I tried to clear them. This also coincided with the other group members taking on extra functionality now the low hanging fruit had been completed.

A side effect of this was that they need more time from me to help them which slowed my progress at times.

I am really proud of what we achieved together as a group but we did find the final day stressful, something I hope to improve on and avoid in future projects.


After 6 weeks of intense learning, our cohort was split into small groups for our first group project.

My Remote Bootcamp Experience: Twitter Clone Group Project


# My Bootcamp Experience: What I learnt helping others learn CSS and HTML

In my second week of bootcamp we focused on HTML and CSS. After a solid day or two of lectures we were tasked with recreating a site. This forced us to focus on our ‘google fu’ and problem solving skills. Being someone with CSS and HTML experience, I was able to complete the assignment quickly, although not as quickly as I might have liked (more on that later). This left the rest of the week available to support my fellow students and for me to work on some personal projects. Here is what I learned.

## HTML is Hard To Read

I found that a lot of my fellow students' problems boiled down to them not correctly traversing the _dom_. When deciding what selectors to use to target an element or identifying suitable elements to use as flexbox containers / items, I would observe them skip levels of the _dom_ in their HTML file or not expand the relevant node in the browser dev tools. This surprised me, the _dom_ is like a file structure after all and everyone has experience navigating their machine’s file systems in explorer or finder yet this didn’t seem to translate when traversing the dom.

This made me realise just how much I forgot how foreign HTML markup looks to the uninitiated. A jumble of words, colours, and symbols with long lines that overflow out of the reader's sight, often starting from the middle of the page.

This got me thinking about selective attention and how much my peers were really seeing? With the brain receiving so much information and with no framework of how to filter and assign importance, it may be the case that the student totally misses the lone _ul_ tag because the brain has been looking for tags with classes.

As anyone in web dev knows, this is short lived with the idea that HTML is complicated becoming an idea that is just as foregin. It does make me wonder though - could tools such as the browser dev tools be redesigned to help combat this problem? Perhaps a "beginner mode"? Prettier makes this less of a problem in IDEs, but is there more that can be done? Or will this remain a right of passage every beginner must go through? Even through my limited code journey, There have been multiple moments of coming across code or a concept that my brain just can’t wrap its head around instantly. This can be super frustrating and even demoralizing, but the payoff from tackling the problem successfully is ever so sweet, making next time just a little easier.

## The concept of space ownership

Helping others can really cement concepts in your own head. One of these concepts for me this week was space ownership. This came to me as I was helping one of my fellow students remove the bullets in their unordered lists. After learning of the `list-style: none;` declaration, they would go on to add `padding: 0;` or `margin: 0;` to the same rule. To be fair, it makes perfect sense, why wouldn't these innocent beginners, not yet weary from all of CSS's many intricacies, think that it would not work? After all It worked when they wanted to bring a `<p>` tag to the right to create a margin. I explained that the parent, the `<ul>`, owned that space and to remove it we needed to target the `<ul>` element.

## I don't really know CSS

So far, my experience of working with CSS has always been on my own projects, where I have full control over the CSS and HTML. Sometimes, I have used frameworks like [Bootstrap](https://getbootstrap.com), [Bulma](https://bulma.io/) and [Tailwind](https://tailwindcss.com/). In [React ](https://reactjs.org/)I write CSS per component using styled-components or emotion. I can usually achieve the desired effect with a couple of clarifying googles. However, this week has taught me that I'm at a lower level than I thought I was. It's often the case that you don't know what you don't know and this has been the perfect example.

In the main assignment set for the week we were tasked to reproduce a site as close as possible to the original. This was given to us in pdf format along with the HTML file pre-written. It was explicitly stated that we could not edit the HTML file.

This single constraint gave me a lot more problems than I had anticipated. Not being able to add a container div, or add a class to an element forced me to come up with creative solutions to problems I have never had to solve so far.

When helping debugging student's stylesheets, I often struggled myself to get to the bottom of the issue quickly as they had used functionality I had not come across before, often a result of some creative googling.

All this is to say, that just because I can whip up a decent looking brochure page to spec, doesn't mean I don't still have a lot to learn.

## “Some people just don't get CSS” is unhelpful

It's a sentiment I've seen over and over again in reddit discussions, dev articles and even on our bootcamp and I really don't think it's helpful. To someone who has taken the time "to get it" this seems like a cop out. Perhaps I am one of the privileged ones whose brain 'works that way”, but I don't think so.

In my possibly naive opinion, I think CSS is a collection of rules that anyone is capable of understanding and mastering.

Having professionals in the field saying 'that their brain doesn't work that way' instead of 'its not something I have put the time in to understand' gives permission to any beginner to write off a whole technology far too early.

I wonder if this originates from the earlier days of the web. My first experience with web dev was [tables](https://thehistoryoftheweb.com/tables-layout-absurd/), [floats](https://css-tricks.com/all-about-floats/), and [GeoCities](https://en.wikipedia.org/wiki/Yahoo!_GeoCities) sites in the early 2000's. CSS was pretty clunky and built more with print media in mind. I appreciate where this mindset has come from although I doubt it was really true, even then.


In my second week of bootcamp we focused on HTML and CSS...

My Remote Bootcamp Experience: What I learnt helping others learn CSS and HTML


# My Remote Bootcamp Experience: My first week.

This week I started on a remote 3 month coding bootcamp with CodeNation. I've been self teaching for 18 months but have struggled to complete a portfolio of professional projects without support. Being given the opportunity to attend through a government pilot scheme, this was something I couldn't turn down.

## A Slow Start

Even going into this knowing it was a zero to hero type course, I was still surprised at the level of my fellow peers. Personally it took me many passes over a number of years to get the basics down, and here we were, going from data types to loops to functions all in the first week.

As someone who has the basics covered, I found this slow and frustrating, not that I was angry in anyway, just that I found it hard to keep focused and unable to channel the excitement that comes with starting something new.

The fact that this was also a remote course really didn't help matters. It was nice to have a personalised home set up, but the challenges of learning in groups over video conferencing software were acute in the first few days. Something the professional world has become far too aware of in the last year. I'm not sure CodeNation has found the right answer to this problem in terms of software as yet, then again, I don't know if the right solution even exists.

## Your Fellow Students Give the Most Value

Personally I found things really started to change for me once we started studying in groups, working through activities set. I was able to use my previous experience to help others in my group and gained confidence in my ability as well as really cementing certain ideas in my head at the same time.

Practicing reading others code, spotting problems, communicating ideas and going through my thought processes has been really enjoyable for me this week. It's really satisfying to help someone with a problem and then see them go on to help others with the same thing. This is something I will look to develop during the course and hope to continue in my professional career.

After so many months of solo study and lockdowns, having a group to work with has been really refreshing. Without the social cues from in person communication, it was pretty awkward at first when to know when to pipe up in voice chat, when to use the text chat and when to just shut up and give others a chance to speak. With voice chat you can only have one conversation at a time as a group and this doesn't suit everyone. If someone is feeling shy it’s easy for them to hide. Perhaps thats exactly what a shy person might want, however, it’s also difficult to recreate the experience of spontaneous little sub group chats in the background.

## Abstract Before Syntax

Watching others learn has been fascinating. It's made me think about the way I approach learning new topics. Following tutorials often gets you to a place with working code quickly, but I wonder if this just results in getting bogged down with syntax and platform idiosyncrasies instead of really getting to grips with understanding the topic you are focusing on.

This week fellow students have often looked to reach for syntax to solve a problem rather than trying to understand the problem itself.
For example, often instead of stepping through the program mentally to find a bug, the initial reaction is to expect that they have used the wrong set of syntax characters. This may be a perfectly logical thing to do on week one, but it’s helped me see gaps in my own approaches to learning more advance topics.

## Having Existing Knowledge Gives a Whole Different Perspective

As I have alluded to above, although the material covered in the first week was made up of fundamentals that I've been over many times, I still have learned loads. Helping others, as well as continuing relevant self taught topics in the in the background and some extra curricular comp sci material given to me by the head instructor has resulted in really productive week, increasing my confidence and strengthening my resolve to land my first dev role as soon as possible.


This week I started on a remote 3 month coding bootcamp with CodeNation...

Using Data from Twitch and Amazon to Populate a DB for a Video Game Review Site Project

My Remote Bootcamp Experience: Twitter Clone Group Project

My Remote Bootcamp Experience: What I learnt helping others learn CSS and HTML

My Remote Bootcamp Experience: My First Week