
If you want to build a Bible-related app, you probably need to store references to Bible passages like these in a database:

* John 3:16
* Eze 21
* SNG
* 2TI 1:2,3
* Romans 5:8-6:4

Storing these references as strings would be both inconsistent and inefficient. We can save space and keep our data uniform if we store them using only numbers.

I'm going to share the solution I used to build [My Bible Log](https://www.mybiblelog.com/), a free app that allows people to track their Bible reading. I'll explain the problems I encountered and how I solved them to create the Verse ID format used behind the scenes to store passage references.

## Passages as Text

To allow users to keep track of exactly which verses they read, I had to find a way to represent passages like these in a database:

* Genesis 1
* John 3:16
* 1 Peter
* 2 Peter 2-3
* Titus 2:11-3:11

Since the format is consistent, I could have just stored the text you see above. However, I wanted my solution to be as efficient as possible so it could scale. In programming, working with numbers is faster than working with text.

I also wanted to provide several features that would be much easier to implement if I could find a way to represent each passage as a single number. For example, I wanted to be able to run a report on which passages have been read already, and how many more verses the user had left to read.

## The Obvious Solution

There is one solution that might seem obvious. Why don't we start at the first verse of the Bible, call it verse `1`, and then count our way to the last verse of the Bible?

Well, we *could* do that, but different translations of the Bible actually include or exclude just a few verses (based on the prevalence of those verses in the original manuscripts), leaving different translations with different numbers of verses:

|Translation|Verse Count|
|---|---|
|KJV|31,102|
|NASB|31,102|
|NIV|31,173|
|ESV|31,102|

If we want our numbering system to be compatible with multiple Bible translations, we need to find another way to represent each verse as a unique number.

Also remember that some translations include verses in brackets with a footnote about why - often because it did not appear in the earliest available manuscripts. Since there is some ambiguity about which verses are included, even within a single translation, we can't use a rigid system of consecutive numbers to represent each individual verse.

## Passages as Numbers

My first step to storing passages as numbers was breaking each passage down into a set of variables that I could use to build a unique Verse ID number for a Bible verse.

These are the possible variables in any passage reference:

* Book
* Start Chapter
* Start Verse
* End Chapter
* End Verse

Not all passages have all of the parts above. For example, "Matthew" is a valid passage, referring to the whole book of Matthew. Similarly, "Mark 1" is valid, referring to the first chapter of Mark. Again, "Luke 4-7" is valid, referring to all verses within those three chapters of Luke.

You'll notice that the "start chapter" and "start verse" are implied to be 1 if they are not provided. If we don't state that we are starting in the middle of a book or chapter, we understand that we are starting at the beginning.

Similarly, if the "end chapter" is omitted, it is implied to be the final chapter of the book, and if the "end verse" is omitted, it is implied to be the final verse of the chapter. Without a specific stopping point, we read each book or chapter in its entirety.

So, even if we don't see a specific chapter or verse number in a passage, we can still determine those numbers and use them to build our Verse ID.

## Books as Numbers

You may have noticed that chapters and verses are already numbers, but books are known by their titles, which means we are still working with strings. We need a way to represent books like "Judges" and "Proverbs" as numbers.

This is simple enough - we just need to assign each book an arbitrary number. I started by assigning `1` to Genesis, and going through the standard order of Bible books until I reached book `66`, Revelation. I chose not to start at zero because I wanted these numbers to be at least somewhat decipherable at a glance, so I needed book `1` to refer to the first book of the Bible instead of the second.

## Verses as Numbers

If we have a verse like "Genesis 1:1", we can represent it with these variables:

|Variable|Value|
|---|---|
|`book`|`1`|
|`chapter`|`1`|
|`verse`|`1`|

Now, how could I represent this verse as a single number?

If I concatenated them, I would get `111`. The next verse would be `112`, and so on. However, it would be very unclear how to deconstruct this number and turn it back into three separate numbers to get the book, chapter, and verse.

What happens when we get to numbers like `5217`? Which option below would be correct?

|Variable|Book|Chapter|Verse|
|---|---|---|---|
|Option 1|`5`|`21`|`7`|
|Option 2|`5`|`2`|`17`|
|Option 3|`52`|`1`|`7`|

## Consistent Deconstruction

In order to reliably deconstruct a Verse ID number, we need to give each variable in the verse its own "space" in the number - a specific number of digits that always represent that variable.

As an example, if we want to represent two numbers, each with a range of 1-99, we need to give each number two digits (even if it sometimes only uses one).

We can represent two numbers with a max length of two digits each in a single four digit number like this:

|First Variable|Second Variable|Combined|
|---|---|---|
|1|1|0101|
|1|99|0199|
|55|7|5507|

Now we can be sure that `0199` means `01` and `99`, rather than `19` and `9`. Giving each number its own digits removes any ambiguity about what each digit represents.

We can easily construct and deconstruct these combination numbers in code:

```js
/**
 * Represents where the ones digit of the first number is,
 * in this case in the hundreds position of the combo number.
 * 
 * _ _ _ _
 *   ^
 */
const FIRST_OFFSET = 100;

const makeComboNumber = (first, second) => {
  return first * FIRST_OFFSET + second;
};

const parseComboNumber = (comboNumber) => {
  const first = Math.floor(comboNumber / FIRST_OFFSET);
  const second = comboNumber - first;
  return { first, second }; 
};

const comboNumber = makeComboNumber(1, 2); // 0102
const { first, second } = parseComboNumber(comboNumber); // { first: 1, second: 2 }
```

## Just Enough Space

It's important that each variable (book, chapter, and verse) gets enough space in the Verse ID. I had to determine how many digits each one could possibly need so I could be sure the variables wouldn't risk overlapping each other.

Books were easy - there are only 66 books in the Bible. The `book` variable gets 2 digits.

Finding the highest number of chapters in a book or verses in a chapter took a bit more work. It turns out Psalms is the longest book in the Bible with 150 chapters. Also, Psalm 119 is the longest chapter, boasting 176 verses. This means both `chapter` and `verse` need 3 digits.

I could now represent any individual verse in the Bible with just eight digits. For example, `01 001 001` will represent book 1, chapter 1, verse 1 - we just remove the spaces to get our final Verse ID number.

For example:

|Passage|Verse ID|
|---|---|
|Genesis 1:1|01001001|
|Genesis 22:16|01022016|
|John 3:16|43003016|

This means each verse can be represented as an integer, taking up only 4 bytes of space in memory! In contrast, "Genesis 22:16" would usually take up 14 bytes of space as a string. Creating Verse IDs has saved us memory and storage space.

## Handling Leading Zeroes

You'll notice that any Verse ID from the first 9 books of the Bible will have a leading zero. Leading zeroes can get weird in programming languages.

For example, JavaScript treats numbers with a leading zero as octal numbers, so they'll get interpreted as a *different* decimal number (decimal numbers, a.k.a. base-10 numbers, are what we use every day):

|Octal Number|Decimal (actual) Counterpart|
|---|---|
|01|1|
|010|8|
|0100|64|
|045|37|
|01001001|262657|

To help ensure that we never run into issues with leading zeroes, I decided to add a `1` to the beginning of each Verse ID.

These are the updated Verse ID values from the table above:

|Passage|Verse ID|
|---|---|
|Genesis 1:1|101001001|
|Genesis 22:16|101022016|
|John 3:16|143003016|

Because they are still within the range of a typical `int` in memory, they still only take up 4 bytes of space!

## Serializing and Deserializing

This is the code I use to "make" and "parse" Verse ID values:

```js
Bible.makeVerseId = (book, chapter, verse) => {
  const verseId = 100000000 + book * 1000000 + chapter * 1000 + verse;
  return verseId;
};

Bible.parseVerseId = (verseId) => {
  verseId -= 100000000;
  const book = Math.floor(verseId / 1000000);
  verseId -= book * 1000000;
  const chapter = Math.floor(verseId / 1000);
  verseId -= chapter * 1000;
  const verse = verseId;
  return { book, chapter, verse };
};
```

I also have additional code for validating that a Verse ID represents a real verse in the Bible to ensure data integrity.

## Verse Ranges as Numbers

So far we've solved the problem of how to store a single verse reference as a number... but most of the time people read multiple verses together. Users want to track the whole passage that they read as a single unit, not as a list of individual verses.

To solve this, I simply store two Verse ID numbers together in the database: `startVerseId` and `endVerseId`.

Technically, we could create a longer "Passage ID" number to represent an entire passage. This would take all 5 variables I mentioned at the beginning of this article:

* Book (2 digits)
* Start Chapter (3 digits)
* Start Verse (3 digits)
* End Chapter (3 digits)
* End Verse (3 digits)

However, this comes to a total of 14 digits, plus one if we still want a leading `1` to keep things consistent.

Signed integers can only support values up to 10 digits long (specifically, values up to 2,147,483,647). To store larger numbers, we would need to use more than 4 bytes. In fact, a "bigint" (big integer), the next size up for number storage, takes 8 bytes - the same as two separate integers. So at that point we might as well just store two separate integers and not worry about handling another special number format.



Looking to optimize your code? You'll need to test it.

By testing different approaches, you can compare them fairly and choose the most performant option.

You don't need any special tools to do this - your browser console can compare the speed of individual functions easily!

## A Reusable Benchmark Utility Function

I wrote a utility to make it easy for you to test the performance of your functions. After several iterations and improvements, this is the code:

```js
const benchmark = ({
  runCount = 10_000_000,
  getArgs = () => [],
  testFunctions = {},
}) => {
  const test = (fn) => {
    const start = performance.now();
    for (let i = 0; i < runCount; i++) {
      const args = getArgs();
      fn(...args);
    }
    const end = performance.now();
    const runtime = end - start;
    return runtime;
  };

  const results = {};
  for (const funcName in testFunctions) {
    const fn = testFunctions[funcName];
    results[funcName] = test(fn);
  }
  return results;
};
```

To use this utility, you'll need to provide the functions you want to test in the `testFunctions` object.

Also, provide an optional `getArgs` function to generate the arguments for you tests. You can have `getArgs` return the same value for each test or random arguments. I prefer to randomize the inputs to make sure the JavaScript engine isn't somehow optimizing the code based on a static and predictable argument, because I know the arguments will *not* be static or predictable in production!

## Testing Performance in Practice

Here's how to use it:

```js
// functions to test
const getHighestNumber1 = (numberArray) => {
  let highest = numberArray[0];
  for (let i = 1; i < numberArray.length; i++) {
    if (numberArray[i] > highest) {
      highest = numberArray[i];
    }
  }
  return highest;
};

const getHighestNumber2 = (numberArray) => {
  let highest = numberArray[0];
  for (let i = 1; i < numberArray.length; i++) {
    let num = numberArray[i];
    if (num > highest) {
      highest = num;
    }
  }
  return highest;
};

const getHighestNumber3 = (numberArray) => {
  let highest = numberArray[0];
  numberArray.forEach(num => {
    if (num > highest) {
      highest = num;
    }
  });
  return highest;
};

const getHighestNumber4 = (numberArray) => {
  return Math.max(...numberArray);
};

// benchmark the functions!
const results = benchmark({
  runCount: 10_000_000,
  getArgs: () => {
    const numberArray = [];
    for (let i = 0; i < 10; i++) {
      numberArray.push(Math.random() * 100);
    }
    return [numberArray];
  },
  testFunctions: {
    getHighestNumber1,
    getHighestNumber2,
    getHighestNumber3,
    getHighestNumber4,
  },
});
```

I suggest running the test several times in your browser console to be sure of the results.

After running the tests above, it is clear that `getHighestNumber1` and `getHighestNumber2` are roughly equal in performance, while `getHighestNumber3` is just a bit slower. Looks like that `Array.prototype.forEach` call comes at a cost!

Finally, we see that `getHighestNumber4` lags behind significantly, taking 2x to 3x as long to execute as the other options. Even though `Math.max` is built into JavaScript, it has worse performance than writing your own `for` loop and `if` statement!

## A Word of Caution

Although the example results above make `Math.max` look pretty bad, I wouldn't stop using `Math.max`. Speaking from experience, it's usually more important to be able to read your code easily than it is to squeeze every last millisecond of performance out of it.

A `for` loop and `if` statement might be a tad faster, but `Math.max` is far more concise and make your intent immediately clear. Before you try to optimize all of your code, remember that network calls (to APIs, databases, etc.) take far more time than a local function call. Be sure you are optimizing where it matters, and only when it matters!

## Other Considerations

In my utility function, I called `getArgs` within the innermost `for` loop, which also calls the function to be tested. The advantage of this is that each time the test function is called, it will have completely new arguments. This can help prevent the JavaScript engine from optimizing the code in a way that doesn't represent a production environment.

However, one disadvantage of calling `getArgs` at that location is that it makes the test slower. The call time of `getArgs` ends up as part of the test... for each iteration. Since `getArgs` is the same for each of your test functions, and since it is called the same number of times, this is still a fair approach. But it is slower.

Below, I labelled two alternative locations for calling `getArgs`.

```js
const benchmark = ({
  runCount = 10_000_000,
  getArgs = () => [],
  testFunctions = {},
}) => {
  const args = getArgs(); // Alternative 1

  const test = (fn) => {
    const args = getArgs(); // Alternative 2
    const start = performance.now();
    for (let i = 0; i < runCount; i++) {
      const args = getArgs(); // Current Location
      fn(...args);
    }
    const end = performance.now();
    const runtime = end - start;
    return runtime;
  };

  const results = {};
  for (const funcName in testFunctions) {
    const fn = testFunctions[funcName];
    results[funcName] = test(fn);
  }
  return results;
};
```

## Alternative 1

The first alternative is to call `getArgs` outside of the `test` function. By doing this, all of your tests will use the same set of arguments. However, this increases the chances that the JavaScript engine will perform optimizations that skew your test results. For example, it may realize that the arguments are constant and that the function is deterministic, and therefore cache the function return value. Also, this means your test functions don't have a chance to actually run against a variety of arguments, which would better represent their performance in a production setting.

## Alternative 2

The second alternative is to call `getArgs` right before starting the timer for each function. This might decrease the chances of the engine performing its own unrealistic optimizations, but it still doesn't provide a variety of inputs for your functions to represent the unpredictable nature of a production environment.

## My Pick

Overall, I prefer the current location of `getArgs` and I think it helps ensure the test results are representative of which function will have better performance in production.


These are online tools that I've used in the past - or bookmarked just in case!

Most are actually totally free, though a few may just have a nice free trial.

I update this list from time to time as tools fall in and out of favor.

|Tool|Purpose|
|---|---|
|[30 seconds of code](https://www.30secondsofcode.org/js/p/1)|Huge collection of snippets in various programming languages|
|[AI. Image Enlarger](https://imglarger.com/)|Enlarge and enhance images with AI|
|[answer](https://github.com/answerdev/answer)|Open source knowledge base / Q&A community software|
|[Animista](https://animista.net/)|A collection of awesome copy/paste CSS animations|
|[Awesome Node.js](https://github.com/sindresorhus/awesome-nodejs)|A convenient list of many great Node.js modules and resources|
|[Code Beautify](https://codebeautify.org/)|Tools for formatting and beautifying all kinds of code (with many links to other tools and utilities)|
|[Color Hunt](https://colorhunt.co/)|Discover color palettes|
|[CSV2JSON](https://csvjson.com/csv2json)|Convert CSV to JSON|
|[ChatGPT](https://chat.openai.com/chat)|AI language model chat bot|
|[Dream Studio](https://beta.dreamstudio.ai/home)|AI image generation|
|[fffuel](https://fffuel.co/)|Many color tools and SVG generators for gradients, patterns, textures, shapes and backgrounds|
|[Free Online Photo Editor](https://www.freeonlinephotoeditor.com/)|Online photo editing|
|[Favicon Generator](https://www.favicon-generator.org/)|Favicon generator/converter (with links to many other tools)|
|[Google Dataset Search](https://datasetsearch.research.google.com/)|Many free datasets|
|[HTML Tidy](https://htmltidy.net/)|Correct / tidy up HTML code|
|[HTML2Jade](https://html2jade.org/)|Convert HTML to Jade/Pug|
|[Iconmonstr](https://iconmonstr.com/)|Free SVG icons|
|[InfoByIP](https://www.infobyip.com/ipbulklookup.php)|Bulk IP lookup tool|
|[Let's Debug](https://letsdebug.net/)|Let's Encrypt debugger|
|[ngrok](https://ngrok.com/)|Put localhost sites on the internet|
|[npm trends](https://npmtrends.com)|View and compare NPM package popularity over time|
|[Online Tone Generator](https://www.szynalski.com/tone-generator/)|Online tone generator|
|[Photopea](https://www.photopea.com/)|Online photo editing|
|[ray.so](https://ray.so/)|Turn code snippets into beautiful, customizable images|
|[RegExr](https://regexr.com/)|Build and test regular expressions|
|[Regex101](https://regex101.com/)|Build and test regular expressions|
|[removebg](https://www.remove.bg/)|Remove image backgrounds.|
|[Stable Diffusion](https://huggingface.co/spaces/stabilityai/stable-diffusion)|AI image generation|
|[Simple Icons](https://simpleicons.org/)|Free SVG icons for popular brands|
|[storyset](https://storyset.com/)|Free, customizable, animated illustrations (SVG and PNG).|
|[TempMail](https://temp-mail.org/en/)|Temporary email|
|[Terms of Service; Didn't Read (Tos;DR)](https://tosdr.org/)|Get a bullet-point summary of many sites' Terms of Service|
|[transform](https://transform.tools/)|Convert between many data types (SVG, HTML, JSON, JSX, YAML, etc.)|
|[uiGradients](https://uigradients.com/)|Many different ready-to-go CSS gradients|


This article is part of a series of ChatGPT experiments, in which I give ChatGPT a prompt, share its response, and offer some reflections.

## Prompt #1

Can you write a React component in Typescript that will take two numbers as props and display a math equation showing the numbers being added together, along with their sum?

## Response #1

Sure! Here's an example of a simple React component written in TypeScript that takes two numbers as props and displays a math equation showing the numbers being added together, along with their sum:

```ts[class="line-numbers"]
import React from 'react';

interface Props {
  num1: number;
  num2: number;
}

const Addition: React.FC<Props> = ({ num1, num2 }) => {
  const sum = num1 + num2;

  return (
    <div>
      <p>{`${num1} + ${num2} = ${sum}`}</p>
    </div>
  );
};

export default Addition;
```

You can use this component by passing in two numbers as props, like this:

```ts[class="line-numbers"]
import React from 'react';
import Addition from './Addition';

const App: React.FC = () => {
  return (
    <div>
      <Addition num1={5} num2={3} />
    </div>
  );
};

export default App;
```

## Prompt #2

Could you write several tests using Jest to make sure the above component works as expected with different inputs?

## Response #2

Yes, here are a few tests using Jest that you can use to verify that the Addition component works as expected with different inputs:

```ts[class="line-numbers"]
import React from 'react';
import { render } from '@testing-library/react';
import Addition from './Addition';

describe('Addition component', () => {
  it('should display the correct sum for positive numbers', () => {
    const { getByText } = render(<Addition num1={5} num2={3} />);
    const sum = getByText('5 + 3 = 8');
    expect(sum).toBeInTheDocument();
  });

  it('should display the correct sum for negative numbers', () => {
    const { getByText } = render(<Addition num1={-5} num2={-3} />);
    const sum = getByText('-5 + -3 = -8');
    expect(sum).toBeInTheDocument();
  });

  it('should display the correct sum for a mixture of positive and negative numbers', () => {
    const { getByText } = render(<Addition num1={5} num2={-3} />);
    const sum = getByText('5 + -3 = 2');
    expect(sum).toBeInTheDocument();
  });
});
```

## Reflections

Again, I am impressed!

ChatGPT can write JSX in Typescript, providing an interface for component props, and giving props intuitive names.

The component is *exactly* what I asked for and the code is *crystal clear*.

Additionally, ChatGPT offered an example of how to use the component without being prompted!

When I asked for some Jest tests to go along with the new component, ChatGPT didn't let me down.
It had enough sense to test different combinations of positive and negative numbers, and it knew exactly which additional testing library to import and which of its methods to use.

Like previous ChatGPT experiments, this one has helped convince me that ChatGPT is a perfect developer sidekick.

What developer doesn't wish they could just focus on solving problems, leaving the actual code implementation to someone else?
If you give ChatGPT your blueprints, it will build your project for you, piece by piece. You just have to handle quality control and gluing the pieces together.



This article is part of a series of ChatGPT experiments, in which I give ChatGPT a prompt, share its response, and offer some reflections.

## Prompt

> Write a cookie recipe that uses szechuan green peppercorns and has mala tingly spicy heat.

## Prompt Notes

Obviously "Szechuan" should be capitalized since it is a proper noun, but I used my phone to submit this prompt and let the capitalization slide. I've kept the prompt in its original format for science. ChatGPT understood the word and always capitalized it.

Also, it looks like the preferred spelling of "Szechuan" is technically "Sichuan", but I used the first spelling that came to mind and ChatGPT went with it. I'll be using the "Szechuan" spelling for the rest of this article for consistency.

## Response

Here is a recipe for Szechuan Green Peppercorn Cookies with Mala Tingly Spicy Heat:

Ingredients:

* 2 cups all-purpose flour
* 1 teaspoon baking powder
* 1/2 teaspoon salt
* 1/2 teaspoon Szechuan green peppercorns, crushed
* 1/2 teaspoon dried chili flakes
* 1 cup unsalted butter, softened
* 3/4 cup granulated sugar
* 1 large egg
* 1 teaspoon vanilla extract

Instructions:

1. Preheat the oven to 350°F (175°C). Line a large baking sheet with parchment paper.
1. In a medium bowl, whisk together the flour, baking powder, salt, Szechuan green peppercorns, and dried chili flakes.
1. In a large bowl, beat the butter and sugar together with an electric mixer until light and fluffy. Add the egg and vanilla extract and beat until well combined.
1. Gradually add the flour mixture to the butter mixture and mix until just combined.
1. Using a cookie scoop or spoon, drop the dough by rounded tablespoonfuls onto the prepared baking sheet.
1. Bake for 12-15 minutes or until the edges are lightly golden.
1. Let the cookies cool on the baking sheet for 5 minutes before transferring to a wire rack to cool completely.

These cookies will have a spicy, tingly heat from the Szechuan green peppercorns and chili flakes. Enjoy!

## Reflections

This was impressive.

I'm a huge fan of the tingly sensation that Szechuan peppercorns give you. I also enjoy the classic combination of Szechuan peppercorns and chili peppers, which is known as "Mala". I wanted to see if ChatGPT would know to include chili pepper as an ingredient if I never used the word "chili" in the prompt.

I was pleasantly surprised to see ChatGPT churn out a fully formatted recipe that would ostensibly create actual cookies (I haven't tested it yet myself). I was even more pleased to see that dried chili flakes made their way into the recipe without me ever mentioning chili peppers.

ChatGPT even knew that the peppercorns would have to be crushed to be used in a cookie recipe!

I Googled for similar recipes after running this prompt, and as I suspected, there isn't a popular cookie recipe with these ingredients. ChatGPT didn't copy someone else's work - it combined an awareness of the mala spice combination with an awareness of cookie composition to generate this unique original recipe.

I'm looking forward to trying these sometime!



This article is part of a series of ChatGPT experiments, in which I give ChatGPT a prompt, share its response, and offer some reflections.

## Prompt

> Can you write a Rust program that will scan a directory and all subdirectories and identify duplicate files within them?

## Response

Yes, here is a simple program in Rust that scans a directory and all its subdirectories and identifies duplicate files based on their content:

```rust[class="line-numbers"]
use std::collections::HashMap;
use std::fs;
use std::path::Path;
use std::hash::{Hash, Hasher};
use std::fs::File;
use std::io::Read;

fn hash_file<T: Hash>(file_path: &Path) -> Result<T, std::io::Error> {
    let mut hasher = std::collections::hash_map::DefaultHasher::new();
    let mut file = File::open(file_path)?;
    let mut buffer = [0; 1024];

    while let Ok(bytes_read) = file.read(&mut buffer) {
        if bytes_read == 0 {
            break;
        }
        hasher.write(&buffer[..bytes_read]);
    }

    Ok(hasher.finish())
}

fn find_duplicates<T: Hash>(root: &Path) -> Result<HashMap<T, Vec<String>>, std::io::Error> {
    let mut file_hashes = HashMap::new();

    for entry in fs::read_dir(root)? {
        let entry = entry?;
        let path = entry.path();

        if path.is_dir() {
            continue;
        }

        let hash = hash_file(&path)?;
        file_hashes.entry(hash).or_default().push(
            path.to_string_lossy().to_string()
        );
    }

    let duplicates = file_hashes
        .into_iter()
        .filter_map(|(_, files)| {
            if files.len() > 1 {
                Some((files[0].clone(), files[1..].to_vec()))
            } else {
                None
            }
        })
        .collect();

    Ok(duplicates)
}

fn main() -> Result<(), std::io::Error> {
    let duplicates = find_duplicates(Path::new("."))?;

    for (file, duplicates) in duplicates {
        println!("{} is duplicated in:", file);
        for duplicate in duplicates {
            println!("\t{}", duplicate);
        }
    }

    Ok(())
}
```

## Reflections

This experiment is a follow-up to a ChatGPT prompt that asked for the same solution in Go. I wanted to try it in Rust because I figured there would be less Rust code for ChatGPT to train on, so I could get a feel for how well ChatGPT performs with less reference material. Also, a Rust version of this script should be even faster and more memory efficient than the Go version.

I was initially impressed by the output, partly because ChatGPT offered a full response to the prompt on the first try. With some other code-generation prompts, ChatGPT stopped responding halfway through "typing" the code so I had to re-submit the prompt and let ChatGPT try again.

### The Good

I appreciate that ChatGPT is aware of:

* how to import different Rust modules cleanly in a single import block
* how to reference those modules from the rest of the code
* how to give variables and functions meaningful names
* how to follow indentation conventions in Rust (four spaces per indentation level)
* how to identify duplicate files by comparing their hashes
* how to use all of the different characters in the expansive Rust syntax

Clearly, ChatGPT has a strong grasp of where characters should go - there isn't a stray character anywhere. At first glance the code looks complete: all opening brackets, braces, and parentheses have a closing partner.

### The Bad

Unfortunately, ChatGPT seems to have a limited understanding of Rust's type system and how to use generics. The code fails to compile because type annotations are incorrect or missing altogether in specific places.

Additionally, this code would only check the files in the immediate directory. It was my intent to check all files in subdirectories as well, which I think is a more common/realistic use case.

Finally, this code wants to use `std::hash::Hasher` to hash files. This is a built-in hasher that can change between Rust versions, making it difficult to keep types consistent. Also, there's no guarantee that the underlying hashing algorithm used will be secure or performant, as it can change between versions as well.

### The Fix

To address these issues and actually get the program to compile, I relied on several sources of information:

* Google (to find other sources)
* Stack Overflow
* The Rust Language Docs
* Specific Rust Crate Docs
* GitHub Copilot
* VS Code `rust-analyzer` extension
* ChatGPT (for more specific Rust questions)

Even though ChatGPT didn't give me a complete or valid solution, it did give me a solid starting point. As they say, you can't edit a blank page - it's better to start somewhere and iterate from there than to expect perfection on the first try. I found that ChatGPT did a good job of getting me past the "blank page" stage of the project and on to actually iterating toward my goal.

In the end, I was able to use the general framework provided by ChatGPT to create a working program. You can find it on my GitHub here: [Rust: Find Duplicate Files](https://github.com/AaronSmithX/rust_find_duplicate_files). It uses the BLAKE3 algorithm for fast and secure hashing, and it iterates through subdirectories as my original prompt requested.



This article is part of a series of ChatGPT experiments, in which I give ChatGPT a prompt, share its response, and offer some reflections.

## Prompt

> Write a Node.js application that uses Express to provide a REST API. There should be an API endpoint for each basic math operation: addition, subtraction, multiplication, division, and modulus. Each endpoint should receive the input numbers as query parameters and return a single number as a response.

## Response

```js[class="line-numbers"]
const express = require("express");
const app = express();

app.get("/add", (req, res) => {
  const a = Number(req.query.a);
  const b = Number(req.query.b);
  const result = a + b;
  res.send({ result });
});

app.get("/subtract", (req, res) => {
  const a = Number(req.query.a);
  const b = Number(req.query.b);
  const result = a - b;
  res.send({ result });
});

app.get("/multiply", (req, res) => {
  const a = Number(req.query.a);
  const b = Number(req.query.b);
  const result = a * b;
  res.send({ result });
});

app.get("/divide", (req, res) => {
  const a = Number(req.query.a);
  const b = Number(req.query.b);
  const result = a / b;
  res.send({ result });
});

app.get("/modulus", (req, res) => {
  const a = Number(req.query.a);
  const b = Number(req.query.b);
  const result = a % b;
  res.send({ result });
});

app.listen(3000, () => {
  console.log("Math API running on port 3000");
});
```

This application creates an instance of the Express application, and sets up endpoints for each of the five basic math operations. The endpoints receive the input numbers as query parameters, perform the specified operation, and return the result as a single number in a JSON response. The application listens on port 3000 and logs a message to the console when it is ready to receive requests.

## Reflections

This was impressive, and while the code may look a little repetitive, it is very clean and has no unnecessary abstractions.

I did have to submit this same prompt to ChatGPT twice because it stopped responding after spitting out 4 lines of code the first time. However, it finished writing the program smoothly the second time.

I ran the code and tested all of the endpoints just to enjoy seeing them work. The code is so clean and simple that I didn't expect any problems. All five math operations work smoothly.

One thing I really appreciated was the explanation of how this solution works. ChatGPT gave me an overview of the program in plain English that played on my initial prompt while offering specific details about the implementation.

The only "error" with this output is that the code section was labeled as SCSS in the ChatGPT interface. However, the actual syntax highlighting seemed to be JavaScript. I can't say how much of the code labeling and highlighting is controlled by ChatGPT itself and how much of that functionality belongs to the UI in front of ChatGPT, so I can't count this as a miss on ChatGPT's part.

Overall, this was a very promising experiment, and I look forward to using ChatGPT to quickly prototype and scaffold APIs in the future.



This article is part of a series of ChatGPT experiments, in which I give ChatGPT a prompt, share its response, and offer some reflections.

## Prompt

> Can you write a golang program that will scan a directory and all subdirectories and identify duplicate files within them?

## Response

Yes, here's a simple program in Go that scans a directory and all its subdirectories and identifies duplicate files based on their content:

```go[class="line-numbers"]
package main

import (
	"crypto/sha256"
	"fmt"
	"io/ioutil"
	"os"
	"path/filepath"
)

func hashFile(filePath string) ([]byte, error) {
	data, err := ioutil.ReadFile(filePath)
	if err != nil {
		return nil, err
	}

	hash := sha256.Sum256(data)
	return hash[:], nil
}

func findDuplicates(root string) (map[string][]string, error) {
	fileHashes := make(map[string][]string)
	err := filepath.Walk(root, func(path string, info os.FileInfo, err error) error {
		if info.IsDir() {
			return nil
		}

		hash, err := hashFile(path)
		if err != nil {
			return err
		}

		fileHashes[string(hash)] = append(fileHashes[string(hash)], path)
		return nil
	})
	if err != nil {
		return nil, err
	}

	duplicates := make(map[string][]string)
	for _, files := range fileHashes {
		if len(files) > 1 {
			duplicates[files[0]] = files[1:]
		}
	}

	return duplicates, nil
}

func main() {
	duplicates, err := findDuplicates(".")
	if err != nil {
		fmt.Println(err)
		return
	}

	for file, duplicates := range duplicates {
		fmt.Printf("%s is duplicated in:\n", file)
		for _, duplicate := range duplicates {
			fmt.Printf("\t%s\n", duplicate)
		}
	}
}
```

## Reflections

This was impressive. I compiled and ran this program to ensure that it worked as expected. There were no hiccups or unexpected errors - this code compiles and executes just fine.

I did have to submit this same prompt to ChatGPT twice because it stopped responding after a few lines the first time. However, it finished writing the program smoothly the second time.

I appreciate that ChatGPT is aware of:

* how to import different Go modules cleanly in a single import block
* how to use those modules correctly
* how to give variables and functions meaningful names
* how to follow error handling conventions in Go
* how to follow indentation conventions in Go (one actual hard tab per indentation level)
* how to identify duplicate files by comparing their hashes

As a developer, I also realize some of ChatGPT's limitations, or at least its lack of intuition:

* There is no console output to declare that the program has started.
* There is no progress reporting while the program is running.
* The program can run successfully with no output if the directory has no duplicates.
* The program always scans the current directory, and cannot take a directory as a command line argument.

All of the above would be useful features that I would want to include in this tool. However, I did not specify them in the ChatGPT requirements, so I can't fault ChatGPT for not including them. They are easy enough to add myself.

This is still a *huge* time saver for me. I've written a program with the same functionality in Node.js, but this is a case where Go's inherent speed (from compiling down to machine code) is really important.

I wanted to rewrite my program in Go at some point, but I didn't have the depth of knowledge in Go to write it as cleanly or quickly as ChatGPT. Now I can just copy/paste this solution and start using it to be productive!



This topic comes up enough in conversation that I wanted to have a resource ready to share with people after talking about it.

## The roadmap

Becoming a software engineer is a journey of acquiring new skills.

There are two simple road markers on your journey:

1. When you open your first code file and start making changes, you are a developer.
2. When you get paid to write code, you are a *professional* developer.

I share this because *imposter syndrome* is a huge problem for would-be developers. For whatever reason, people think you have to be really special in some way or another to learn how to code, and they think you have to have *all* of the skills before you're actually a developer. That's just not true.

In fact, nobody has all the skills. We all have different skills. You just need enough relevant skills to be valuable to a company.

To become a developer, focus on learning new skills. Plenty of these skills are also useful for non-programming jobs and even your personal life.

## Learn JavaScript to become a front-end developer

I would personally suggest learning JavaScript as your first real programming language for three main reasons:

1. It can do practically anything.
2. It's pretty easy to pick up.
3. There are real jobs out there for it.

You might be familiar, but JavaScript is a very flexible, dynamic language. You don't have to deal with a ton of strict rules while learning it, so you can get right to the point. It's also easy to run - we've all used a web browser, and web browsers run JavaScript. There's no install or setup required.

You will want to get familiar with HTML and CSS before jumping into JavaScript, but they are not *real* programming languages.
They're simple enough to pick up in a few hours, so jump right in!

When I started the self-taught route to programming, I cut my teeth on HTML, CSS, and some JavaScript with the free resources at [W3Schools](https://www.w3schools.com/js/default.asp).

These days, I would probably start with MDN (Mozilla Developer Network) instead. MDN is a great reference website for all things web. This is their [JavaScript tutorial](https://developer.mozilla.org/en-US/docs/Learn/JavaScript).

## Learn Node.js to become a full-stack developer

Node.js (or just "Node") is a JavaScript runtime. Basically, it's an executable file that sits on your computer and runs JavaScript. Inside it is the same JavaScript engine that's inside the Chrome browser: V8. Node lets you use JavaScript skills outside of the browser.

If you learn Node, you'll be able to create more than just websites with JavaScript. You can create web servers and automation scripts as well. You can also create desktop apps. I have done all of the above for different jobs.

W3Schools and MDN both seem to have decent Node.js tutorials:

* [W3Schools Node.js Tutorial](https://www.w3schools.com/nodejs/default.asp)
* [MDN Node.js/JavaScript Tutorial](https://developer.mozilla.org/en-US/docs/Learn/Server-side/Express_Nodejs)

## Learn to use the terminal to work more efficiently

When you see TV hackers doing their thing, all keyboard and no mouse, they're using the *terminal* to get things done just by typing commands. Using the terminal is very efficient. Many development workflows are built around using the terminal, so you'll find yourself using it regularly.

If you are on Windows, you will need to install a program to give you access to the same terminal commands that Linux and Mac have. Those are the standard commands. You can install [Git Bash](https://gitforwindows.org/) to gain access to those commands. You will need to install Git anyway, so this is practically a required tool for Windows development.

I recommend following the [Command line crash course](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Understanding_client-side_tools/Command_line) from MDN. Remember that if you already have Git Bash installed, you won't need to install the "Windows Subsystem for Linux" that they recommend, which has a steeper learning curve.

## Learn Git to be able to work on a team

Git is a distributed version control system. It lets us save and share our code easily, and protects us against losing progress. We can use it to easily revert our code to previous versions and back up our code on a separate server.

You WILL need to learn and use Git for your developer job. Thankfully, the basic Git workflow is pretty easy to grasp.

W3Schools has a nice Git tutorial that will also set you up with a GitHub account. You definitely want a GitHub account. It's free. Here's the tutorial: [Git Tutorial](https://www.w3schools.com/git/default.asp?remote=github).

## Learn to build on other people's work

We stand on the shoulders of giants, as they say. There are many, many engineers of different kinds who have come before us and given us the technology we have today. That includes all of the code that has already been written, and much of that code is available for us to import into our own projects.

Most languages have a *package manager* that allows you to basically install other people's code into your own projects. For Node.js, the package manager is NPM (the "Node Package Manager"). NPM gets installed by the Node.js installer, so if you have Node you already have NPM.

The MDN tutorial on building a Node.js web app has a good section on using [NPM](https://developer.mozilla.org/en-US/docs/Learn/Server-side/Express_Nodejs/development_environment#using_npm).

You can find all kinds of packages to install on the [NPM website](https://www.npmjs.com/).

It can be tempting to build something from scratch just because you could theoretically do it yourself. In fact, this is sometimes a great way to learn new things! However, if you are trying to actually get things done and add new features to a project, that approach is a huge waste of time.

Get comfortable with building on the code that has already been written. Don't reinvent the wheel out of pride.

## Learn Google to solve any problem with ease

Googling is a skill, and it's one of the skills that all good programmers have in common.

I've mentioned before that all programmers have different skills. This is part of the reason we all need to be good at Google: for whatever problem we're having, there's a good chance other developers with more skills in that particular area have already solved that problem. We just need to find their answers.

This article on [21 Tips and Tricks to Master the Art of Googling as a Developer](https://www.makeuseof.com/21-tips-and-tricks-to-master-the-art-of-googling-as-a-developer/) has some great advice on how to make the most of Google.

You should also know that many answers are found on [Stack Overflow](https://stackoverflow.com/), which is *the* question and answer site for programmers. You can use Google to search Stack Overflow by adding `site:stackoverflow.com` to your search query.

## Do projects to learn and demonstrate your skills

After following a tutorial or two, it's time to try your hand at a real project. Having a few projects under your belt will give you something to talk about with recruiters and interviewers. You can also build a personal portfolio site to show them off. I highly recommend this.

Overall, the approach I would suggest is to pick a project that you sort of understand and just start building it. Use Google, Stack Overflow, and MDN to answer your questions as you go. Break large problems into a list of specific questions, then find the answers to those questions. You'll always learn more than you set out to learn in the first place.

## Learn a framework to become more valuable

You can accomplish more and write less code by using a framework. You should probably create a demo project for a few frameworks just to have some familiarity with them and what they can do for you.

I recommend trying Express for backend development and either React or Vue for frontend:

* [Express](https://expressjs.com/) (the most popular Node.js framework for building web servers)
* [React](https://reactjs.org/) (frontend framework for building websites - uses a syntax called "JSX")
* [Vue](https://vuejs.org/) (frontend framework for building websites - uses a template system)

You can easily find the official website and many tutorials for each framework with Google.

## Make a developer portfolio site

If you want to make a good first impression and stick out from the crowd, I suggest making a personal portfolio site. You can easily host one for free on [GitHub Pages](https://pages.github.com/).

Your portfolio website is a sales tool, and you are the product. It should focus on introducing your skills, highlighting your experience, and sharing any relevant personal interests.

I recommend:

* Including projects you've worked on.
* Adding screenshots of your websites/apps (not your source code).
* Linking to your GitHub profile so people can see the quality of your code.
* Linking to your LinkedIn so people can connect with you.

I've seen some developers share their resumes on their personal websites for anyone to download. I wouldn't advise this, as resumes usually contain contact info that doesn't belong on the open web.

## Practice for code interviews with katas

The developer interview process often includes at least one coding challenge. You will be asked to solve a problem with code while the interviewer watches (and often acts as a resource - they want to see what you'd be like on a team).

To practice for these challenges, I recommend using CodeWars. It's a website with many katas (fancy word for "code challenges") that you can solve on your own. [Here's CodeWars](https://www.codewars.com/).

CodeWars is GREAT because it runs your code against a list of tests and shows you which ones passed and any that still need work. It's very rewarding.

I also created my own kata learning site for JavaScript called [N2JS](https://www.n2js.com/) ("into JS" or "into JavaScript"). It has 100+ lessons and also runs your code against a list of tests. Instead of giving you random challenges like CodeWars that may require skills you don't yet have, N2JS has lessons to teach you new JavaScript concepts before giving you a kata where you can use them.

I created N2JS with a self-guided high school computer class in mind, so keep that in mind if the pacing doesn't match your learning speed. You can always jump straight into more complex challenges on CodeWars.

## Enroll in a coding bootcamp to get massive experience very quickly

This step is optional.

Coding bootcamps may be a serious investment, but they're far more economical than a college degree - especially for programming.

Many university computer science programs are slow-paced, outdated, and padded with content that is irrelevant to the job market. You know, academic stuff.

Bootcamps, on the other hand, give you a full immersion experience into the world of software development, turning you out into the workforce in around 3-4 months.
Some of them are full time, in-person programs. Others are asynchronous and online. Still others are hybrids.

I recommend looking for a bootcamp that offers as much in-person or at least realtime collaboration as possible.
You will want to be able to talk about your experience working on a team of developers when it comes time to find a job.

If you're worried about financing, there are options out there.
Individual bootcamps can point you to financing partners that have worked well for other students.

Full disclosure: I've taught at a coding bootcamp, so I might be biased toward the idea as a whole.
However, I don't get a commission for sending you anywhere. It's just an investment worth considering.

## Apply for jobs

When you're ready to look for software-focused roles, it can help to look for "associate" or "apprenticeship" positions.
These positions are often designed to be entry-level, and it's understood that you'll be doing some learning on the job.

Make sure you're ready to present yourself in the best possible light. Career changers can feel compelled to tell their personal story of becoming a developer, as if they need to justify themselves and what they're doing. Skip that. Jump to what you can do, what you have done, and what you will do for the company.

Be bold. Speak on the forward edge of truth.

It definitely helps to have a tidy LinkedIn profile, a polished and proofread resume, and a personal portfolio site.
Every piece of evidence that points to your skills and professionalism is going to get you closer to the work you want to do.

Additionally, it helps to keep a spreadsheet of the jobs you've applied for. Job searches *feel* like they go from 0% to 100% the instant you accept a job offer.

In reality, your job search progresses steadily and includes many things: your learning, your practice, your professional presentation, and of course your actual job applications and interviews.

Keeping a list of those job applications helps you see the progress you're making and the work you're putting in.

Continue to improve your skills and apply for jobs, and suddenly you will find yourself at 100%.

## Further Reading

If you'd like to read another take on the subject, check out this external article on [Why Learning to Code is so Damn Hard](https://www.thinkful.com/blog/why-learning-to-code-is-so-damn-hard/). Be advised: like plenty of other good content on the internet, it does end in a sales pitch.

## Good luck!

I'm sure there's plenty more to be said about learning to code, and I might have left some gaps in the info above, but I hope this gives you an idea of where to start!



I've mentioned before that I've been automating and scraping websites for a long time. It's not always pretty, but sometimes it's absolutely necessary. In this post I want to share a few tips and tricks for automating React apps.

## The Irony

There's  irony here. Often, we scrape websites because they are old and unmaintained and never had an API… we simply have no other choice. However, React is a more modern framework. Check out this timeline:

* 2006 – Facebook releases its app platform and API
* 2009 – Node.js is released, allowing API servers to be written in JavaScript
* 2013 – Facebook releases React

Any website that uses React could also have been created with a convenient API for data access. In 2006, APIs were already popular enough that Facebook offered one, and then in 2009 Node.js enabled developers to use JavaScript to build API servers. This means any devs who built a React app could have also built an API server.

To be fair, most React apps probably *are* making HTTP requests to get things done, but that doesn't mean there's an API there for you to consume. The backend of a site is often built to only handle the needs of the frontend, with things like authentication and session management tightly coupling the two together. If that's the case, it can be better to scrape the React frontend than to try to interact directly with the server.

## Typical Automation

Automating a website means getting a computer to use the website as if it were a person, clicking on things and entering text and copying data from the page.

There are different tools for doing this like Selenium and Puppeteer, which have their own methods to call. We can also scrape a website without any external tools. For example, we can run a script in the browser console. We can also create a bookmarklet to [load an external JavaScript file](/articles/easily-load-an-external-script-using-javascript/) and run it. In this case, we will use plain vanilla JavaScript to move through the website and find data.

We can find elements with CSS selectors or [XPath expressions](/articles/automate-and-scrape-a-website-with-xpath-expressions/).

If we want to click on a link or a button, we can use the `element.click()` method.

If we want to enter text into a form, we can simply find the `<input>` or `<textarea>` element and set its value: element.value = "robot user input".

The above approach also works for selecting an option in a `<select>` dropdown.

However, there are unique challenges with automating React apps that make these usual approaches ineffective. React has a reactivity system (hence the name) that listens for user input events and then uses them to update its internal state data.

This reactivity system listens for specific events that aren't fired when we change an element's value directly or call an element's `click()` method. The page may appear to respond as usual, but React's internal state will not reflect the changes we made on the page.

Eventually, our automation will likely fail because React is not updating in response to our input.

## Clicking in a React app

For many websites, using `element.click()` is enough to click the element from JavaScript. In fact, we can still use that method on React components to follow links and submit some forms.

However, many React components require a click event to function properly. Maybe a switch component will only toggle when clicked. Maybe a dropdown or menu will only open when clicked. In these cases, we need to trigger React's reactivity system.

To do so, we will dispatch a `MouseEvent` three times, once for each type of event that fires when a user clicks an element:

* mousedown
* click
* mouseup

This ensures that React will recognize the click no matter what specific mouse events it was listening for. If React was expecting all three to happen in order, we satisfy that requirement:

```javascript[class="line-numbers"]
/**
 * Simulates a mouse click event in the browser in a way that React will recognize.
 * @param {HTMLElement} element 
 */
const simulateMouseClick = (element) => {
  const mouseClickEvents = ['mousedown', 'click', 'mouseup'];
  mouseClickEvents.forEach(mouseEventType =>
    element.dispatchEvent(
      new MouseEvent(mouseEventType, {
        view: window,
        bubbles: true,
        cancelable: true,
        buttons: 1,
      }),
    ),
  );
};
```

## React app mouse hover

There are times when we need to hover our cursor over an element to proceed with automation. For example, we may need to move our mouse over a dropdown component in order for a menu to appear.

On some websites, such a menu will always exist but will be hidden by CSS classes. In that case we can still click links in the menu even if they are hidden. With React, however, those menu options won't even be rendered unless the menu is activated by a mouse hover event.

In order to simulate a hover event, we will dispatch a `mousemove` event in the middle of our target element. React's event listeners will pick up this event and recognize that we are hovering over the element:

```javascript
/**
 * Simulates mouse movement across an element in a way that React will recognize.
 * @param {HTMLElement} element 
 */
const simulateMouseHover = (element) => {
  const x = element.offsetLeft + element.offsetWidth / 2;
  const y = element.offsetTop + element.offsetHeight / 2;
​
  element.dispatchEvent(
    new MouseEvent('mousemove', {
      view: window,
      bubbles: true,
      cancelable: true,
      buttons: 1,
      screenX: x,
      screenY: y,
      clientX: x,
      clientY: y,
    }),
  );
};
```

## React app text entry

Entering text in a React app is very similar to how we would enter text in any website. We find the form input element we want to update, and we set its `value` directly.

However, React's two-way data binding overrides the native value setter function. Without that setter function, our updates may appear in the browser but they won't actually update the element's `value` property, and React won't pick up on them either.

For example, if we enter text into a React form and then submit that form, React won't actually have our data. It will appear in the browser, but React will have an empty form since it didn't "hear" the input events that should have fired.

To get around this, we need to do two things:

1. Get the original setter function and call it to make sure the element's value is actually updated
1. Dispatch an input event from that element so React "hears" the change happen and pulls the element's value into its internal state

This is what that looks like:

```javascript
/**
 * Enters text in an input field in a way that React will respond to.
 * Directly setting an input value works, but React will not copy the value to its internal model.
 * The value would disappear if the component is re-rendered, and even if it didn't it would never
 * be submitted in an API request.
 * @param {HTMLInputElement} input 
 * @param {string} text 
 */
const simulateTextEntry = (input, text) => {
  const nativeInputValueSetter = Object.getOwnPropertyDescriptor(window.HTMLInputElement.prototype, 'value').set;
  nativeInputValueSetter.call(input, text);
​
  const event = new Event('input', { bubbles: true });
  input.dispatchEvent(event);
};
```

Reference: [https://stackoverflow.com/a/46012210](https://stackoverflow.com/a/46012210)



I've been scraping websites for a long time. One of my first "real" programming projects used Java and Selenium to automate a website and scrape data. I later used Node.js to pull data from many different education-related websites so teachers could see it all in one place.

In this post, I want to share some tips for using XPath expressions to find page elements and data. Scroll to the bottom for two helper functions that make it just as easy to use XPath expressions as CSS selectors!

## Why Scrape?

In a perfect world, every site would be regularly maintained and would have an API you could use to efficiently get the data you need. In reality, we often need to use an older legacy site to access the data we need. There is no API, and the website we need to use is clunky and time consuming.

We might use web scraping to make sure people aren't doing work that a computer can do automatically. For example, instead of having a person search for individual records manually, we can make a list of records we want to search and build an automation that finds those records. If you have multiple sources of data, you can scrape them concurrently with an automation.

In summary, automated web scraping is always more efficient than manual searching and data entry.

## What is an XPath Expression?

XPath stands for "XML Path Language." Since HTML is an XML-like language, we can use XPath to find specific HTML nodes. This is similar to using a CSS Selector to find an element on the page, but it can be much more powerful.

For example, this is how we would find the same element using a CSS selector and XPath expression:

Description|CSS Selector|XPath Expression
---|---|---
H1 Element|`h1`|`//h1`
Paragraph within a section|`section p`|`//section//p`
Anchor element with title "home"|`a[title="home"]`|`//a(@title="home")`
Second direct descendant list item in an ordered list|`ol > li:nth-of-type(2)`|`//ol/li[2]`
Paragraph with exact text "lorem ipsum"|impossible|`//p[text()="lorem ipsum"]`
Paragraph containing text "lorem ipsum"|impossible|`//p[contains(text(), "lorem ipsum")]` 

> XPath expressions give us more power than just CSS selectors.

As you can see, XPath expressions have capabilities beyond CSS selectors. For example, XPath expressions can find an element based on its text content.

## How to use XPath Expressions in JavaScript

Most developers quickly learn about the `querySelector` and `querySelectorAll` methods for querying the DOM with CSS selectors. For example:

```javascript[class="line-numbers"]
// find a heading
const heading = document.querySelector('h1');
​
// find all paragraphs
const paragraphs = document.querySelectorAll('p');
```

Similarly, the `document.evaluate` function allows us to search the DOM using an XPath:

```javascript[class="line-numbers"]
// find a heading
const heading = document.evaluate(
  '//h1',
  document,
  null,
  XPathResult.ANY_UNORDERED_NODE_TYPE,
  null,
).singleNodeValue;
​
// find all paragraphs and log their text content
const paragraphs = document.evaluate(
  '//p',
  document,
  null,
  XPathResult.ORDERED_NODE_SNAPSHOT_TYPE,
  null,
);
​
for (let i = 0; i < paragraphs.snapshotLength; i++) {
  const text = paragraphs.snapshotItem(i).textContent;
  console.log(text);
}
```

## document.evaluate parameters

The `document.evaluate` function receives several parameters. This makes it seem more complicated than the `querySelector` and `querySelectorAll` functions. This is the price we pay for the power of XPath.

I'll be showing you a wrapper function to make XPath much easier to use. Since you won't have to worry about the details, I'm going to skip most of them. You can always check the documentation over at MDN if you want to know more:

[https://developer.mozilla.org/en-US/docs/Web/API/Document/evaluate](https://developer.mozilla.org/en-US/docs/Web/API/Document/evaluate)

### Search within a parent element using XPath Context Nodes

One thing you *do* want to be aware of is how you can search within a specific element using an XPath expression. You can do this with the `contextNode` parameter of `document.evaluate`. It is the second parameter.

In the examples above, the context node was always the `document` itself. However, it can be any element you provide. Calling `document.evaluate` with a specific context node will only return matches within that node.

This is similar to how we can call `querySelector` on the `document` itself, but we can also call it on a specific element within the document:

```javascript[class="line-numbers"]
// searches the whole document for paragraphs
const allParagraphs = document.querySelectorAll('p');
​
// searches only for paragraphs inside the section element
const section = document.querySelector('section');
const sectionParagraphs = section.querySelectorAll('p');
```

This is what it looks like to search a specific element for descendants using XPath:

```javascript[class="line-numbers"]
// searches only for paragraphs inside the section element
const section = document.querySelector('section');
​
const sectionParagraphs = document.evaluate(
  '//p',
  section, // <-- notice the change!
  null,
  XPathResult.ORDERED_NODE_SNAPSHOT_TYPE,
  null,
);
​
for (let i = 0; i < sectionParagraphs.snapshotLength; i++) {
  const text = sectionParagraphs.snapshotItem(i).textContent;
  console.log(text);
}
```

## Mix and Match XPath Expressions and CSS Selectors

You can use XPath expressions and CSS selectors in your code. Both of them return references to DOM elements, and both of them use those same element references to search for descendant nodes.

## Simple document.evaluate wrappers

Using `document.evaluate` is more verbose than using a CSS selector. However, we can simplify things by creating a wrapper function for `document.evaluate` that mimics `querySelector` and `querySelectorAll`.

### Find one element by XPath easily

You can use this function to easily find a single document node using an XPath. This is similar to using the `querySelector` function:

```javascript[class="line-numbers"]
/**
* Shorthand for calling `document.evaluate` to get a single element via XPath.
* @param {*} xpathExpression 
* @param {*} contextNode 
* @returns 
*/
const getNodeByXpath = (xpathExpression, contextNode = document) => document.evaluate(
  xpathExpression,
  contextNode,
  null,
  XPathResult.ANY_UNORDERED_NODE_TYPE,
  null,
).singleNodeValue;
​
// From root of document
const heading = getNodeByXpath('//h1');
​
// From specific ancestor element ("context node")
const section = document.querySelector('section');
const sectionHeading = getNodeByXpath('//h2', section);
```

### Find multiple elements by XPath easily

You can use this function to easily find multiple document nodes using an XPath. This is similar to using the `querySelectorAll` function:

```javascript[class="line-numbers"]
/**
* Shorthand for calling `document.evaluate` to get multiple element via XPath.
* Returns an array of nodes, which may be empty.
* @param {*} xpathExpression 
* @param {*} contextNode 
* @returns 
*/
const getNodesByXpath = (xpathExpression, contextNode = document) => {
  const result = document.evaluate(
    xpathExpression,
    contextNode,
    null,
    XPathResult.ORDERED_NODE_SNAPSHOT_TYPE,
    null,
  );
​
  const nodes = [];
  for (let i = 0; i < result.snapshotLength; i++) {
    const node = result.snapshotItem(i);
    nodes.push(node);
  }
  return nodes;
};
​
// From root of document
const paragraphs = getNodesByXpath('//p');
​
// From specific ancestor element ("context node")
const section = document.querySelector('section');
const sectionParagraphs = getNodesByXpath('//p', section);
```

## References

For help building your own XPath expressions, check out this cheatsheet:

[https://devhints.io/xpath](https://devhints.io/xpath)



I created [N2JS](https://www.n2js.com/) ("into JavaScript") so I could teach people how to code through *katas*, or code challenges. As a former high school teacher and coding bootcamp instructor, I wanted to improve the way people first learn how to code.

I've developed an [Essential JavaScript](https://www.n2js.com/courses/essential-javascript) course with over 100 lessons to help people learn to code from the very beginning. Anyone can take this course for free to learn programming skills at their own pace.

Programming concepts are broken down into bite-sized chunks. After learning a new concept, you'll be tested with a kata. When you submit your solution, a series of automated tests will check to see if you passed the kata. This gives users real-time feedback on their learning and problem solving.



Most of the time HTML images can take care of themselves. By default, an `<img>` element will load an image file and display it at its natural width on the page. If the file is 200px wide and 400px tall, that's how big it will be when it loads.

If we need to adjust the size of the image, or how it is cropped or stretched or otherwise positioned, we can use CSS.

However, there are still times when we need to use JavaScript to manually work with an image, and often we need the image's natural dimensions to do so. For example, we may need to paint the image on an [HTML canvas](https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API).

Getting this information is easy, as the [HTMLImageElement](https://developer.mozilla.org/en-US/docs/Web/API/HTMLImageElement) API includes a `width` and `height` property that expose the natural dimensions of the image file. Let's see those properties in action in JavaScript:

```javascript[class="line-numbers"]
const myImage = document.querySelector('img.my-image')
const { width, height } = myImage
console.log(`The image is ${width}px wide and ${height}px tall.`)
```

## Determine any image file's size

There are cases when we need to know the size of an image that hasn't been loaded yet. When we first load a web page, the HTML is available before any image data has loaded. The HTML is parsed into the DOM, and the page is rendered. Many times we see the blink where the page has rendered with no image and the image finishes loading later.

This delay can cause problems if we need to use JavaScript to determine the size or behavior of elements around the image. Before the image has loaded, its `naturalHeight` and `naturalWidth` are both zero. If our code tries to lay out a web page based on the size and aspect ratio of the image before it loads, it will treat the image as if it had no size at all.

To get around this, we have to wait for the image data to be available in the browser. Once the image is loaded we can execute our JavaScript to resize and reposition elements based on the image.

Here's an example:

```javascript[class="line-numbers"]
// Returns the original dimensions of an image file via callback
function getImgSize(imgSrc, callback) {
  const newImg = new Image();
​
  newImg.onload = function() {
    const height = newImg.height;
    const width = newImg.width;
    callback({ width, height });
  };
​
  newImg.src = imgSrc; // this must be done AFTER setting onload
}
```

This function takes the URL of an image (`imgSrc`) and waits until the image has loaded. At that point, the `callback` function is called with the image's natural width and height.

To accomplish this, we first create an [HTMLImageElement](https://developer.mozilla.org/en-US/docs/Web/API/HTMLImageElement/Image). This is the same element that is created in the DOM by an HTML `<img>` tag.

`HTMLImageElement`s have a `width` and `height` property. If a `width` and `height` attribute exist on an `<img>` element in HTML, those values will be returned. That means they are not the natural width and height of  the image data being displayed. However, if an image does **not** have a `width` and `height` attribute manually set, it will return the natural width and height of the image data.

It is important that we wait for the image data to load in the browser. We do this by listening for the `onload` event. This event fires as soon as the image data is available in the browser. We have to be careful here – the image data might already be cached in the browser from a previous request, meaning the `onload` event will fire as soon as the image has an `src`. To ensure our `onload` callback is actually called, we have to add it *before* we give the image an `src`, which triggers the image data to load (from the browser cache or if necessary from a new request).

## Other Applications

Another way we can take advantage of the `onload` event listener is to preload multiple images in sequence without overwhelming the connection to the server. For example:

```javascript[class="line-numbers"]
// This is a Promise-based async version
const loadImage = (imageSrc) => new Promise(resolve => {
  const image = new Image();
  image.onload = () => {
    const height = image.height;
    const width = image.width;
    resolve({ image, width, height });
  };
  image.src = imageSrc;
});
​
const imageUrls = ['image1.png', 'image2.png', 'image3.png'];
​
const run = async () => {
  for (const imageUrl of imageUrls) {
    const { image, width, height } = await loadImage(imageUrl);
    // do something with `image`, `width`, and `height`
  }
};
​
run();
```



In programming, context is important. Part of that context is the currently executing script file.

For example, I might need to know which directory the current JavaScript file is running in so I can refer to other resources using a relative path. Relative paths are more convenient to work with than typing out absolute paths for everything. Also, if I were using absolute paths and I moved a file or directory in my code project, I would have to update all of the absolute paths inside. So there are valid reasons for wanting to know which script file we are currently in.

Node.js makes it extremely easy to determine the current file or directory with `__filename` and `__dirname`, respectively.

Fortunately, it's not too difficult to find the same information in the browser using `document.currentScript`. This property returns the `<script>` element that is currently being executed.

```javascript[class="line-numbers"]
console.log(document.currentScript.src)
```

## Caveats

There are a few caveats to using `document.currentScript`. First, the property only references its containing `<script>` element if the code is executing synchronously. That means we can't use it in callbacks and event handlers. Fortunately, this is easy to get around:

```javascript[class="line-numbers"]
const currentScript = document.currentScript
​
function myCallback() {
  console.log(currentScript.src)
}
​
setTimeout(myCallback, 1000)
```

By saving a reference to the script file when the code runs synchronously, we are able to reference the `document.currentScript` property later in our asynchronous code.

Another caveat is that `document.currentScript` doesn't work in [JavaScript modules](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules). That's okay. We can just use the `import.meta` property there instead.

## Internet Explorer

`document.currentScript` is supported by all modern browsers, but doesn't have support in Internet Explorer. You can still use this snippet to achieve the same effect:

```javascript[class="line-numbers"]
var currentScript;
if (document.currentScript) {
  currentScript = document.currentScript
} else {
  var scripts = document.getElementsByTagName('script')
  currentScript = scripts[scripts.length - 1]
}
console.log('Script located at: ' + currentScript.src)
```

We will query for all script elements on the page and then grab the last one by its index. With few exceptions, the currently executing script will be the last script element added to the DOM, as any later script elements haven't been loaded yet.

One rare exception where this won't work is if you dynamically append a script within the DOM, such as in the head of the document, after page load. If there were script elements in the body of the HTML document, the last one would be returned instead of the currently executing script that was added later.

The good news is that if you are manually appending scripts to the DOM, you can ensure those dynamically added scripts are being added at the end of the body where this script will successfully identify them in Internet Explorer.



Usually when we need to include a JavaScript file on an HTML page we just do this:

```html[class="line-numbers"]
<script src="the-javascript-file.js"></script>
```

And with modern JavaScript maybe we throw an `async` or `defer` [attribute](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/script#Attributes) on that script tag for a little extra performance. Better yet, we could set `type="module"` to use the [JavaScript module system](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules).

If we are using JavaScript modules, we can include other JavaScript module files directly by using an `import` statement:

```javascript[class="line-numbers"]
import otherModule from '/other/module.js'
```

However, there are times when none of these options are available. For example, if we don't have access to edit the original HTML markup being served, we are forced to load JavaScript dynamically.

Real world use cases for this include [bookmarklets](https://support.mozilla.org/en-US/kb/bookmarklets-perform-common-web-page-tasks) and [web extensions](https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions).

## Loading JavaScript dynamically

A `<script>` element can be created and appended to the DOM just like any other HTML element. For example:

```javascript[class="line-numbers"]
const script = document.createElement('script')
script.src = '/my/script/file.js'
document.head.append(script)
```

Once a script element has been appended to the DOM, it will be executed. This means that inline scripts will have their contents interpreted and executed as JavaScript just as we would expect if they had been part of the HTML when it was first loaded. Similarly, external script files will be loaded and executed.

Here's an inline example:

```javascript[class="line-numbers"]
const inlineScript = document.createElement('script')
script.innerHTML = 'alert("Inline script loaded!")'
document.head.append(script)
```

As you can see, it's easy to create and append new script elements, allowing us to include any number of external JavaScript files dynamically after a page has loaded.

## Determining when a JavaScript file is loaded

The real challenge isn't loading the file – it's knowing when the file has finished loading. For example, maybe we have code that uses a library like jQuery or AngularJS or Vue (listed in order of ancientness, not preference). We need to make sure the library is loaded before we execute our own code, otherwise our code will break.

We could do something silly like call `setInterval` and continually check if the library has loaded by looking for its global window variable:

```javascript[class="line-numbers"]
const jqueryScript = document.createElement('script')
jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.min.js'
document.head.append(jqueryScript)
​
const jqueryCheckInterval = setInterval(() => {
  if (typeof window.jQuery !== 'undefined') {
    clearInterval(jqueryCheckInterval)
    // do something with jQuery here
  }
}, 10)
```

However, this code is ugly and wastes resources. Instead, we should listen directly for the script element to fire its `onload` event:

```javascript[class="line-numbers"]
const jqueryScript = document.createElement('script')
jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.min.js'
jqueryScript.onload = () => {/* do something with jQuery */}
document.head.append(jqueryScript)
```

We've already cut the size of our code in half, making it much easier to read and work with. It's also slightly more performant.

The code would be even easier to read if we used [Promises](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise), which would allow us to chain multiple scripts together to load one after the other. Here's a function we can use:

```javascript[class="line-numbers"]
/**
 * Loads a JavaScript file and returns a Promise for when it is loaded
 */
const loadScript = src => {
  return new Promise((resolve, reject) => {
    const script = document.createElement('script')
    script.type = 'text/javascript'
    script.onload = resolve
    script.onerror = reject
    script.src = src
    document.head.append(script)
  })
}
```

Notice we have also introduced error handling by listening for the script element's `onerror` event.

Here's what the script looks like in action:

```javascript[class="line-numbers"]
loadScript('https://code.jquery.com/jquery-3.4.1.min.js')
  .then(() => loadScript('https://code.jquery.com/ui/1.12.1/jquery-ui.min.js'))
  .then(() => {
    // now safe to use jQuery and jQuery UI, which depends on jQuery
  })
  .catch(() => console.error('Something went wrong.'))
```

## Keeping Things Fresh

The above script works great for libraries and modules that never change, such as those loaded from a [CDN](https://developer.mozilla.org/en-US/docs/Glossary/CDN). Once the script is loaded, the browser will automatically cache it. The next time the script is needed, the browser will reuse the copy it saved earlier. This saves bandwidth and makes the page load faster.

This built-in behavior is actually a problem for scripts that change. For example, you might want to create a bookmarklet with just enough code to load an external script, allowing that script to do all of the heavy lifting. That script might change over time as you add new features. If you use the above `loadScript` function, those new features might not show up because the browser has already cached your script, and it now reuses that cached version instead of checking your server.

To ensure your script is actually loaded from your server each time, you can add a meaningless query value to the end of the script URL. As long as this value is different each time the script is loaded, it will cause the browser to treat the URL as a new resource and load it directly from the server each time.

Here's what that can look like in code:

```javascript[class="line-numbers"]
/**
 * Loads a JavaScript file and returns a Promise for when it is loaded
 */
const loadScriptNoCache = src => {
  return new Promise((resolve, reject) => {
    const url = new URL(src)
    url.searchParams.set('random', Math.random())
    
    const script = document.createElement('script')
    script.type = 'text/javascript'
    script.onload = resolve
    script.onerror = reject
    script.src = url.toString()
    document.head.append(script)
  })
}
```

## Dueling with dinosaurs

If you don't have access to the original HTML source of the page you're working with, there's a chance you're facing other limitations as well. For example, you could be forced to work with Internet Explorer.

IE may be old and behind the times, but thankfully we can accommodate it with just a few modifications. First, we need to drop the Promises API and go back to using callbacks. Second, we need to account for IE's unique way of handling script load events. Namely, IE doesn't fire an `onload` event, but it does give scripts an `onreadystatechange` event just like XMLHttpRequests.

Here's the callback-based version that works with Internet Explorer as well as other browsers:

```javascript[class="line-numbers"]
/**
 * Plays well with historic artifacts
 */
function loadScript(src, callback) {
​
  var script = document.createElement('script')
  script.type = 'text/javascript'
​
  // IE
  if (script.readyState) {
    script.onreadystatechange = function () {
      if (script.readyState === 'loaded' || script.readyState === 'complete') {
        script.onreadystatechange = null
        callback()
      }
    }
  }
  // Others
  else {
    script.onload = callback
  }
​
  script.src = src
  document.head.appendChild(script)
}
```



Heroku's documentation for its Golang buildpack isn't up to date, and it's unclear which files we really need to get started. I'm going to lay out the bare essentials for creating a minimal Golang Heroku app. I will explain the purpose of each and every file so we can assert that we do in fact have a bare-bones starter project for Go on Heroku.

## History of the Heroku Go buildpack

Heroku supports many languages with its [list of official buildpacks](https://devcenter.heroku.com/articles/buildpacks#officially-supported-buildpacks). One of the platform's more recent additions was the [official Go buildpack](https://elements.heroku.com/buildpacks/heroku/heroku-buildpack-go).

Before there was an official buildpack for Golang, we could still create Go apps on Heroku using a [custom buildpack](https://elements.heroku.com/buildpacks) created by [a member of the community](https://elements.heroku.com/buildpacks/kr/heroku-buildpack-go). Unfortunately, this was a more complicated option and had no official support from Heroku.

Since we can now use the official Heroku Go buildpack, we should take that approach. However, Heroku's [Go buildpack documentation](https://devcenter.heroku.com/articles/getting-started-with-go) and [coordinating demo app](https://github.com/heroku/go-getting-started) are still set up to use the old custom buildpack. For example, they still have a `Dockerfile` and `Makefile`. With the new buildpack, we don't need to manually interact with Docker or Make!

Heroku's Go demo app also has a `/vendor` folder for dependencies. The Go language has evolved and there is an official tool for managing dependencies: [Go Modules](https://blog.golang.org/using-go-modules). The official Heroku Go buildpack supports Go Modules, so we no longer need to use a 3rd party tool for vendoring.

Basically, Heroku's Go buildpack is far ahead of its documentation. This leads to a lot of dead code that we would be blindly copying if we started from Heroku's old example project. Let's cut that out and see how little it takes to get up and running on Heroku with Golang.

## Setting up

You will of course need to [install Go](https://golang.org/doc/install#install) on your machine.

To follow along, you will also need to install the [Heroku Toolbelt](https://blog.heroku.com/the_heroku_toolbelt). This will allow you to use the Heroku CLI to set up and deploy our minimal Golang app.

Since the Heroku CLI uses Git as part of its deployment process, you will also need to have Git installed. Windows users can [install Git Bash](https://gitforwindows.org/).

## Create a Golang web server

Start by creating your project directory:

```shell[class="line-numbers"]
mkdir minimal-golang-heroku-app
```

Within that directory, create a `main.go` file:

```go[class="line-numbers"]
package main
​
import (
    "fmt"
    "net/http"
)
​
func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprint(w, "Hello World!")
    })
​
    http.ListenAndServe(":80", nil)
}
```

This is the minimum viable Go web server. All we are doing is listening for a request to the root path and responding with "Hello World!" If you run the code, you can open your browser to `localhost` to see the "Hello World!" response come back:

```go[class="line-numbers"]
go run main.go
```

## Set up a Git repository

We will deploy our project to Heroku by using Git, so let's initialize it as a Git repository and commit our changes.

Run these commands inside the project folder:

```shell[class="line-numbers"]
git init
git add .
git commit -m "Initial commit"
```

## Create a Heroku app

Now that we have a web server, let's turn it into a Heroku app. We will set things up using the Heroku CLI while in the project directory:

```shell[class="line-numbers"]
heroku create
```

That command does two things. First, it creates an empty application on Heroku. It also creates an empty Git repository on Heroku to go along with the app. Finally, if your local project is a Git repository, this command will set the new Heroku repository as a remote named `heroku`.

You will see output like this:

```html[class="line-numbers"]
$ heroku create
Creating app... done, ⬢ evening-ocean-54721
https://evening-ocean-54721.herokuapp.com/ | https://git.heroku.com/evening-ocean-54721.git
```

If you check the list of apps on your [Heroku dashboard](https://dashboard.heroku.com/apps) after running that command, you will see the new app has been created.

## Push to Heroku

The only step left is to actually push the code from our computer up to our Heroku app so we can see it on the web. However, there is a catch.

Try pushing your app up to Heroku. Remember you need to set the upstream branch in Git since this is your first push:

```shell[class="line-numbers"]
git push -u heroku master
```

Heroku will reject your code at this point with these error messages:

```html[class="line-numbers"]
No default language could be detected for this app.
HINT: This occurs when Heroku cannot detect the buildpack to use for this application automatically.
See https://devcenter.heroku.com/articles/buildpacks
```

In order to build and run your code, Heroku needs to know which buildpack to use. We can [manually set the buildpack](https://devcenter.heroku.com/articles/buildpacks#setting-a-buildpack-on-an-application) using the CLI, the GUI, or an `app.json` file. If we don't manually set a buildpack, Heroku will look for clues to determine which language to use, but it will stop if it can't figure it out.

I prefer for Heroku to intuit the correct buildpack, as it saves me a step (and potential troubleshooting) when deploying an app for the first time.

## Initialize a Go module

Fortunately, it's easy to signal to Heroku that this is a Golang app. We just need to [initialize it as its own Go module](https://github.com/golang/go/wiki/Modules#quick-start):

```shell[class="line-numbers"]
go mod init github.com/AaronSmithX/minimal-golang-heroku-app
```

This will create a `go.mod` file, which declares that the directory is a Go Module. Technically, you can name your module whatever you want. However, it's convention to name a module to match its Git repository.

Before we install any dependencies, our `go.mod` file is very minimal:

```html[class="line-numbers"]
module github.com/AaronSmithX/minimal-golang-heroku-app
​
go 1.12
```

As you may have guessed, Heroku will look for a `go.mod` file in new projects to see if it should use the Go buildpack. Let's commit our `go.mod` file and try pushing to Heroku again (we still need to set our upstream branch since our last push was rejected):

```html[class="line-numbers"]
git add go.mod
git commit -m "Added go.mod"
git push -u heroku master
```

This time, Heroku will accept our commit. It will successfully build our minimal Golang application and run it as a [dyno](https://www.heroku.com/dynos).

Unfortunately, we have one more problem to solve. If you try to view your application online (you can find the URL in the `git push` output or on your Heroku dashboard), you'll see Heroku's "Application Error" screen instead of our "Hello World!" message.

## Using Heroku environment variables

This is an easy fix. Since Heroku apps share a machine with other apps, they are assigned an arbitrary port to listen on. In other words, our app isn't going to be listening on port 80, which is what it tries to do by default. It also won't be listening to the same port each time it is started (and Heroku does love to restart its dynos).

To get the `PORT` environment variable from Heroku, update `main.go`:

```go[class="line-numbers"]
package main
​
import (
    "fmt"
    "net/http"
    "os"
)
​
func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprint(w, "Hello World!")
    })
​
    http.ListenAndServe(":"+os.Getenv("PORT"), nil)
}
```

We are importing the `os` package at the top, then using `os.Getenv("PORT")` to read the `PORT` environment variable. Notice we add a colon (:) in front of the port to get the correct syntax.

Let's test our app by passing in a `PORT` value:

```html[class="line-numbers"]
PORT=8080 go run main.go
```

If you visit `localhost:8080`, you will see the app running.

Now make one final commit and push:

```html[class="line-numbers"]
git add main.go
git commit -m "Use PORT"
git push
```

And with that, Heroku will accept our changes, successfully build our app, and finally deploy it to the web for all to see.

## Final notes

You can see the complete example [on my GitHub](https://github.com/AaronSmithX/minimal-golang-heroku-app).

One thing we didn't do is install dependencies. Heroku doesn't require any extra steps here – if you know how to work with [Go Modules](https://github.com/golang/go/wiki/Modules), you already know how to manage dependencies on Heroku.



This technique is a little bizarre and probably doesn't have many direct applications. I'm going to share it because it lets me talk about regular expressions, and it's food for thought about other ways to use them.

## The problem

We have a DOM element that we want to serialize (basically, we want to store it as a string). The problem is, there is a *ton* of extra white space in the element's HTML markup. If we stored that white space in a JavaScript variable we would be wasting resources. If we have to store many elements like this, the problem is even bigger!

We need a way to remove space from a string, but not just any space. We only want to remove white space characters that occur between HTML elements.

```html[class="line-numbers"]
<p>We want to keep all space between these words.</p>
​
<p>We want to remove the blank line above this paragraph.</p>
​
<span>Also, </span>          <span>the space between these two spans</span>
```

## A simple solution

Here's the answer:

```javascript[class="line-numbers"]
function condenseHTML(elem) {
    const html = elem.outerHTML
    const condensedHtml = html.replace(/>\s+</g, '><')
    return condensedHtml
}
```

Let's walk through this:

* Our function accepts a single argument: an HTMLElement.
* We then get the HTML that represents this element and its children using the element's outerHTML property.
* Next, we use a regular expression to execute a search and replace operation.

Let's talk about that regular expression:

* The regular expression looks for one or more white space characters (\s matches white space characters, + matches one or more).
* Specifically, we are looking for white space between > and < characters. In other words, white space that appears after the end of  one HTML element and before the beginning of the next. Notice that we are matching the > and < as well.
* Also notice that the regular expression has the g flag at the end, which stands for "global". This means it can find multiple matches in the same string.
* We will then replace all the matches by using the string's replace method.

Since our regular expression matched the `>` and `<` characters, they will also be replaced. That's why we are replacing each individual match with `><`, as otherwise the angular brackets would be lost. Now the HTML elements remain valid but there will no longer be space between them.

## Caveats

This approach isn't perfect. Technically, we are removing space between elements that really should be there. See this example:

```html[class="line-numbers"]
<span>We still need</span>         <span>space here.</span>
```

The space between these two spans will all be removed, which means the words "need" and "space" will be smashed together when viewed in the browser. To be extra safe, our solution could replace multiple spaces with a single space instead of replacing them all with no spaces.

Remember that in HTML, multiple spaces usually condense down to a single visual space (unless the element's style has `white-space` set to `pre`, `pre-wrap`, or `break-spaces`).

## Extended learning

If you'd like to get more practice using regular expressions, I recommend checking out [regex101.com](https://regex101.com/).

If you're interested in further optimizing HTML serialization, try using a [DOMParser](https://developer.mozilla.org/en-US/docs/Web/API/DOMParser) to parse the HTML source into a live [Document](https://developer.mozilla.org/en-US/docs/Web/API/Document). You can then use the context of each [Element](https://developer.mozilla.org/en-US/docs/Web/API/Element) and [Text](https://developer.mozilla.org/en-US/docs/Web/API/Text) node to determine which spaces are safe to remove.



Pro tip: you can make the white pixels in your image files look transparent using just CSS!

To do this, use the `mix-blend-mode` property:

```scss[class="line-numbers"]
img.clear-white {
  mix-blend-mode: multiply;
}
```

That snippet will cause all images with the "clear-white" class to display as though their white pixels were transparent.

This technique is especially useful if you are working in an environment that doesn't have great tools for controlling transparency in the actual image files.

The `mix-blend-mode` property is currently supported by Chrome, Firefox, and Safari. Since Microsoft has [adopted the Chromium project for Edge](https://blogs.windows.com/windowsexperience/2018/12/06/microsoft-edge-making-the-web-better-through-more-open-source-collaboration/), maybe we will see support in Edge soon.

For more great info on CSS `mix-blend-mode` and how to use it, check out [this article on CSS Tricks](https://css-tricks.com/almanac/properties/m/mix-blend-mode/).



When people first learn a programming language, they often start by creating small programs that run in the terminal (a.k.a. "console" a.k.a. "command prompt"). The program will give some text output, then the user will type some text input, then the program will read it and give some more output.

One such exercise could be: create a program that asks the user for their name, allows them to type their name, then greets them by name.

There are many programming tutorials that teach Python and Ruby and other scripting languages, and many of them start here, with simple console programs. On the other hand, Node.js tutorials usually start by creating a simple web server. That's super cool, but it means most Node.js developers aren't very familiar with how the terminal works in their own programming language.

## Interacting with the terminal

First, we need a way to write to the terminal and also a way to read data that was entered in the terminal:

* We will use process.stdin to read data that was typed in the terminal.
* We will use process.stdout to write data out to the console.

`stdin` is the standard input stream, and `stdout` is the standard output stream. This pattern exists across programming languages. To simplify, these [streams](https://nodejs.org/api/stream.html#stream_stream) allow data to pass in and out of the program through the terminal.

Here's a one-liner example of how to write to `stdout`:

```javascript[class="line-numbers"]
process.stdout.write('Hello world!')
```

And here's a simple example of how to read from `stdin`:

```javascript[class="line-numbers"]
process.stdin.once('data', data => {
  console.log('Data entered: ' + data)
  process.exit()
})
```

We added a one-time event listener for a `'data'` event from the stream, which happens when someone types input and presses Enter.

Since `stdin` is a `Readable` stream, it begins in paused mode. Adding a `'data'` listener [switches the stream to flowing mode](https://nodejs.org/api/stream.html#stream_two_reading_modes). When `stdin` is in flowing mode, it listens for data from the terminal.

Once the `'data'` event triggers, we can work with the input from the console. In this case, we will just log some output to the console. Notice how the global `console` instance is already configured to write to `stdout`!

Also notice how we had to call `process.exit()` to exit the program. Since we left the `process.stdin` stream open, the program was still listening for user input. Calling `process.exit()` is one way to terminate the program. A more proper approach would be to close the stream by calling `process.stdin.pause()` in the same place.

## Keeping things organized

Since Node.js is asynchronous, things are going to get a little weird. When you ask the user for input, Node.js doesn't wait for them to type something. It just keeps on working. So it's important that we keep things as organized as possible so we don't lose track of what's happening.

Fortunately, the built-in `readline` module simplifies things for us:

```javascript[class="line-numbers"]
const readline = require('readline')
​
const rl = readline.createInterface({
  input: process.stdin,
  output: process.stdout
})
​
rl.question('What is your name?', nameAnswer => {
  console.log(`Nice to meet you, ${nameAnswer}.`)
})
```

Once we create a `readline` interface, we don't have to worry about using `process.stdin` and `process.stdout` directly.

Notice that `readline` uses callbacks. If we want to ask several questions in sequence, we have to juggle those callbacks somehow:

```javascript[class="line-numbers"]
rl.question('What is your favorite color?', colorAnswer => {
  console.log(`I like ${colorAnswer} too.`)
​
  rl.question(`What shade of ${colorAnswer} is best?`, shadeAnswer => {
    console.log(`Wow, ${shadeAnswer} is also my favorite!`)
    
    rl.close()
  })
})
```

Of course, JavaScript has a solution for this. We can use Promises to avoid nesting callbacks ad nauseam:

```javascript[class="line-numbers"]
const question = prompt => {
  return new Promise((resolve, reject) => {
    rl.question(prompt + '\n', resolve)
  })
}
```

Conveniently, we have abstracted away all interaction with the standard input and output streams. That leaves you to focus on the higher-level concern of what text should display and what to do with the text typed by the user. We have also abstracted away the pyramid of callbacks, so we can more easily see line-by-line how the terminal interaction will go.

## Putting it all together

Here's a final complete example of our setup, which is inside an `async` function so we can use `await` to inline our Promises:

```javascript[class="line-numbers"]
const readline = require('readline')
​
const rl = readline.createInterface({
  input: process.stdin,
  output: process.stdout
})
​
const question = prompt => {
  return new Promise((resolve, reject) => {
    rl.question(prompt, resolve)
  })
}
​
(async () => {
  const nameAnswer = await question('What is your name?')
  console.log(`Nice to meet you, ${nameAnswer}.`)
  
  const whereAnswer = await question('Where are you from?')
  console.log(`I hear it's nice in ${whereAnswer}.`)
  
  rl.close()
})()
```



[My Bible Log](https://www.mybiblelog.com/) is a free online Bible reading tracker.

It's a [Progressive Web App,](https://developer.mozilla.org/en-US/docs/Web/Progressive_web_apps) so you can use it as a website or install it on your device as a regular app.

I created it because I couldn't find a good online tool to track Bible reading. There were plenty of checklists that I could print off, but I wanted something more flexible and precise. My goal with this project is to help people read the whole Bible at their own pace, following their own plan.

The Bible is a huge book. Technically, it's an anthology of 66 books. Those 66 books were later organized into chapters and verses, with a verse being basically a sentence.

There are 31,102 verses in the Bible. That's a lot of reading.

If you want to read the Bible in a year you need to read 86 verses a day. Now that's more manageable. And if you know that some days and times work better than others, you can read more during those times and let it all balance out. It's up to you.

Most Bible reading plans tell you what to read and when you should read it. In contrast, you tell My Bible Log what you read and when you read it. This is your personal Bible reading tracker. Have it your way.

## Getting Started

There's no need to register for an account or verify your email. You can use Facebook or Google to sign in and start tracking your Bible reading immediately. Just go to [MyBibleLog.com](https://www.mybiblelog.com/).



In my last post I showed [how to create file downloads with JavaScript](/articles/create-file-downloads-with-plain-browser-javascript/).

That's only part of the picture. If we want our users to be able to continue working with their data after they've downloaded it, we also need to support file uploads.

## Why not use another storage option?

The specific use case we are talking about is a project where we don't have access to any sort of backend server or database, but we still need a way for our users to save their work. One option is to just store the data in the browser, but all of our in-browser options have significant limitations:

* If we used session storage, the data would disappear as soon as the browser was closed. In most cases, we need the data to persist between browser sessions.
* We could use HTTP cookies to store data, but cookies have a maximum size of  4096 bytes per domain. When it comes to storing data, that's probably not enough. Also, since cookies are transferred to the server with every request, we would be bogging down our interactions with the server. Wait! There is no server! In that case, our users are likely viewing our HTML file using the file:// protocol, which means cookies can't be set in the first place.
* We could step up our game with local storage or IndexedDB, which both have reasonable size limits and can store data across browser sessions. The major problem here is that they are browser-specific. Our users can't use their data in other browsers on other computers. They also can't export their data to be used in other programs.

If we allow users to just save data to the filesystem, none of the above drawbacks apply. Files don't disappear when the browser is closed or the user clears their cookies. There is practically no size limit. Most importantly, files are portable and can be used in other browsers, on other computers, and even in other applications.

## Reading an uploaded file

In order to read an uploaded file, we will first need an HTML form to which we can attach our "upload." Of course, we aren't really uploading anything since we don't have a server. We just need a file `<input>` element to let our JavaScript read a file from the local filesystem.

```html[class="line-numbers"]
<input type="file" id="file-to-read">
```

Well that was about as easy as it gets. That one-liner will allow us to select a file on our computer to "upload." Now let's add a button that we can click to initiate reading the selected file.

```html[class="line-numbers"]
<button onclick="readFileAsText()">Load Selected File</button>
```

Piece of cake. This is just a regular button with an event listener. When we click it, the `readFileAsText` function is called.

Let's finish up by defining that function:

```html[class="line-numbers"]
<script>
  const readFileAsText = function() {
    const fileToRead = document.getElementById('file-to-read').files[0]
    const fileReader = new FileReader()
    
    fileReader.addEventListener('load', function(fileLoadedEvent) {
      const textFromFileLoaded = fileLoadedEvent.target.result
      console.log(textFromFileLoaded)
    })
​
    fileReader.readAsText(fileToRead, 'UTF-8')
  }
</script>
```

## Step by step, in English

* The first thing we need to do is get a reference to the file that the user selected. We do that by finding the file input, in this case by its ID. Then we access its files property. Since a file input can allow multiple upload files to be selected, the files property is an array. We only care about the first item, so we will grab the item at index zero.
* We will then create a FileReader, which allows us to asynchronously read data from a File or Blob.
* It is important to realize what "asynchronous" means here. It means the FileReader will mind its own business while it is reading and won't block the rest of our code from running. We need to add an event listener for its load event so we can work with the result when it is finished reading.
* In our event listener we need to reference the FileReader because it now has a populated result property. We can always do this by using event.target. (Since this particular example has a traditional function and not an arrow function we could also use this to refer to the FileReader.)
* We are now free to do whatever we want with the contents of the file, which are a string. In this example, the file contents are logged to the console.
* Remember that we haven't actually told the FileReader to read the file yet – so far we have only told it what to do when it finishes reading a file. It is important to attach event listeners first. However, in order to get the FileReader to read a file in the first place, we need to call its readAsText method on the file from our `<input>`. This method causes the result to be a string. The FileReader class has other methods that cause result to be different types of data.

## Managing the data

If you are going to allow your users to save and load data from your app, I recommend serializing that data in JSON form. If you do, the data will have a clear structure and will be easy to deserialize and load back into your application. It will also be easier for other applications and custom scripts to consume the data.



Data is the heart of many applications. Users interact with data constantly, and the real value of many web apps is the simplicity with which they allow users to interact with their data.

One way to make data management more convenient is to just allow users to download the data they're working with. In a world overrun with cloud computing, this sounds very old school. Still, there's a time and place for everything.

We have a secret weapon for creating file downloads using only browser technology: the anchor tag's [download attribute](https://developer.mozilla.org/en-US/docs/Web/API/HTMLAnchorElement/download). Check it out:

```html[class="line-numbers"]
<a href="/files/517d5e8f158e.txt"
   download="pretty-file-name.txt">
  Download
</a>
```

Notice several details about the above code snippet:

* This is a regular anchor tag, the same one we use to create hyperlinks
* It has an href attribute pointing to a file on the server
* It has a download attribute that specifies a filename that will be suggested when the file is downloaded

When I click this link, I am initiating a download of the file `/files/nvc89340nc.txt` on the server. When the "Save" dialog appears, it will suggest that I save the file as `pretty-file-name.txt`.

That's great, but remember our use case: we don't have a server. We want to take data that was created and manipulated in JavaScript and allow users to download it onto their personal machines.

There are two good ways to do that, depending on your use case. The first is to use a [Blob ](https://developer.mozilla.org/en-US/docs/Web/API/Blob)and corresponding [object URL](https://developer.mozilla.org/en-US/docs/Web/API/URL/createObjectURL). The second is to use a [Data URL](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URIs). Either option will allow you to turn JavaScript strings into files that can be downloaded by clicking a link.

## Blob and Object URL

First, the code:

```javascript[class="line-numbers"]
// Generates a text file download named `filename` with contents of `dataString`.
const generateDownload = (filename, dataString) => {
  const a = document.createElement('a')
  document.body.appendChild(a)
  a.style = 'display: none'
  const blob = new Blob([dataString], {type: 'octet/stream'}),
        url = URL.createObjectURL(blob)
  a.href = url
  a.download = filename
  a.click()
  window.webkitURL.revokeObjectURL(url)
  a.parentElement.removeChild(a)
}
​
```

In this example, we first create an anchor to click and a Blob to contain our data. Then we create an object URL that references our Blob and assign that URL to our link's `href` attribute. Finally, now that we have our data and a URL that points to it, we can "click" our anchor in JavaScript. This initiates the file download.

Afterward, we will clean up by revoking our object URL and removing our temporary link element.

## Data URL

This option is more straightforward. As in, it uses less technologies and they are probably easier to understand:

```javascript[class="line-numbers"]
// Generates a text file download named `filename` with contents of `dataString`.
const generateJSONDownload = (filename, dataString) => {
  const data = 'text/json;charset=utf-8,' + encodeURIComponent(dataString)
  const link = document.createElement('a')
  a.style.display = 'none'
​
  const attrDownload = document.createAttribute('download')
  attrDownload.value = filename
  link.setAttributeNode(attrDownload)
​
  const attrHref = document.createAttribute('href')
  attrHref.value = 'data:' + data
  link.setAttributeNode(attrHref)
​
  document.body.append(link)
  link.click()
  link.remove()
}
```

URLs that begin with `data:` are Data URLs. They allow us to embed data into a URL. Check the [MDN page](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URIs#Syntax) for more examples.

In this particular example we will create a JSON file download. Data URLs contain the MIME type of their embedded data, which is `text/json` in the snippet below.

We will use [`encodeURIComponent`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent) to ensure our data can be safely added to the URL. Then we apply that URL to our anchor element's `href`, add a `download` attribute, and click it. Viola! The string of data we provided will now download onto our machine.

## Use Cases

This technique is especially valuable for single file HTML tools and bookmarklets, which both have limitations for how data can be stored and retrieved.



If you have some experience with Ruby, you're probably familiar with the Ruby range syntax:

```ruby[class="line-numbers"]
(7..11)
```

A Ruby range is an enumerable object with a starting and ending value. The range above will iterate from 7 up to 11. Super easy!

Let's see what that looks like in JavaScript:

```javascript[class="line-numbers"]
for (let i = 7; i <= 11; i++) {
    //
}
```

Oh, right. JavaScript doesn't have ranges.

But wait… if the above `for` loop gets the job done, does JavaScript even *need* ranges? After all, Ruby is known for having a lot of shiny features that might secretly be bad ideas.

Well, check this out:

```ruby[class="line-numbers"]
(7..11).include? 9 # true
​
my_range = (13..31)
my_range.include? 12 # false
my_range.include? 30 # true
```

Notice how the range can have its own methods because it's an actual object? And we can store it and use it later? We can't do that with JavaScript, and sometimes shiny new features like ranges have an actual purpose.

In this case, an actual range object makes it easier to read and manage our code since we are encapsulating the starting and ending value into a single object.

It would be nice if we could just do this in JavaScript:

```javascript[class="line-numbers"]
for (let i of Range(7, 11)) {
  console.log(i)
}
​
const myRange = Range(4, -8)
if (myRange.includes(-1)) {
  console.log('We got a negative one')
}
```

Fortunately, it's not hard to make our own JavaScript ranges. There are different ways of approaching this problem, including actual JavaScript [iterators and generators](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Iterators_and_Generators).

To keep things simple, I'm just going to define a function that creates arrays of integers based on the `from` and `to` arguments I pass in:

```javascript[class="line-numbers"]
const Range = (from, to) => {
  const increment = from < to ? 1 : -1
  let difference = Math.abs(to - from) + 1
  const result = []
  while (difference) {
    result.push(from)
    from += increment
    difference--
  }
  return result
}
```

Since this allocates an entire array in memory, it could get expensive if you are creating many ranges or very large ranges. You should definitely consider whether a regular `for` loop will get the job done before using this snippet.

On the other hand, if you have a script you pull out once a year that iterates over arbitrary ranges of integers, this is a great way to make it easier to revisit and maintain.



[Promises](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise) were a huge step forward for JavaScript. Finally, a mess of nested callbacks could be replaced with a fluent API.

Still, the built-in Promise has some limitations. As a replacement for callback functions, Promises are designed to wrap and represent a single function. When you create a Promise you hand control over to it by giving it two callbacks: `resolve` and `reject`. Then you wait until the function calls one of them:

```javascript[class="line-numbers"]
new Promise((resolve, reject) => {
  // do something asynchronous here
  // call `resolve` or `reject` when finished
})
.then(result => {
  // this code can't execute until `resolve` or `reject` is called above
})
```

This means a standard Promise is simply a more organized way to handle callback results and errors.

That's a problem because it limits *abstraction*. If a Promise is only a callback object, we limit it to that particular use case. But Promises can be a more flexible means of control flow. They represent pending action, and they indicate when it is finished.

A Promise can exist on its own without wrapping a callback function. To make this happen, we just need to pull its `resolve` and `reject` methods outside the scope of its callback:

```javascript[class="line-numbers"]
const defer = () => {
​
  let resolve, reject
​
  const deferred = new Promise((res, rej) => {
    resolve = res
    reject = rej
  })
​
  deferred.resolve = resolve
  deferred.reject = reject
​
  return deferred
}
```

We now have a `Promise` with two additional methods: `resolve` and `reject`. We can use these together with `then` and `catch` to control flow any way we want.

Let's break this down. `defer` is a factory function. It creates special `Promise` objects, so to use it we would assign the result to a variable:

```javascript[class="line-numbers"]
const myPromise = defer()
```

We can attach callbacks to `myPromise` just like any other `Promise`:

```javascript[class="line-numbers"]
myPromise
  .then(result => {
    console.log('It finished!', result)
  })
  .catch(err => {
    console.warn('Something went wrong', err)
  })
```

Finally, anywhere in our program that `myPromise` is in scope, we can `resolve` or `reject` it:

```javascript[class="line-numbers"]
if (err) {
  myPromise.reject(err)
}
else {
  myPromise.resolve(value)
}
```

You might have noticed this looks a lot like a [jQuery Deferred Object](https://api.jquery.com/category/deferred-object/). This pattern gives us much of the same utility value without a library.

I'll leave it to you to decide if it fits your current work.



The best way to teach technology is by using technology. Gone are the days of heavy, boring textbooks – our era is one of interactive, self-paced, engaging instruction!

I have a few suggestions for how to make your digital curriculum as effective as possible.

Use questions as headings:

* What is Go?
* What is the difference between Go and Golang?
* Why should I learn Go?

Use short paragraphs.

They should be bite-size chunks of 1-3 sentences that are easy to distinguish on mobile devices.

Utilize lists, which can:

1. break a larger idea into smaller points
1. group related facts together
1. help readers remember important information

Use tables, which are great for organizing reference material and clearly comparing items property by property.

Include plenty of code snippets. It's especially helpful to compare and contrast code snippets:

* What would happen if we added this?
* What would happen if we removed this?
* What's another way to solve this problem?
* How would we fix this error?
* Which of these approaches is more performant?
* Which of these approaches is easier to read and maintain?

Embed interactives in the instructional content. These could be completely unique learning tools, or they could just be an embedded sandbox that the reader can play with:

* https://stackblitz.com/docs
* https://hexo.io/docs/tag-plugins.html

Not all interactive content needs to be embedded. Students can read your instructional content from one tab or screen while following your instructions in another:

* https://flems.io/
* https://codesandbox.io/s/?utm_source=landingpage
* https://stackblitz.com/
* https://jsfiddle.net/

Additional interactives could include:

* Multiple choice learning checks with guided feedback (feedback on both correct and incorrect responses that steers the learner toward better understanding)
* Interactive checklists that save state (localStorage) so learners can better manage their learning.
* Terminology review flashcards

Check yourself with these three questions:

* Do I clearly explain what my readers are supposed to be learning so they can check their knowledge?
* Could a complete beginner understand this tutorial? If not, is my target audience clear?
* Do I link to external references for things I expect my readers to already understand or to learn elsewhere?

Finally, break your content into manageable pages! It is natural for readers to measure their learning progress by how many pages they've "completed" on your site. Break your concepts into separate pages to meet this expectation.


I work as a programmer, but I was trained to be a high school English teacher. I still love languages, and I still love to teach.

If you're interested in leveraging technology to share knowledge in any capacity, I might have a few resources for you.

## Resources

### [Structuring Code Tutorials](/articles/structuring-code-tutorials/)

Suggestions for writing your own online content, with specific attention given to coding tutorials.

### [Digital Learning Tools](/articles/an-overview-of-digital-learning-tools/)

An overview of digital learning tools, why they're valuable, and how to start using them.

### [Creating New Tools](/articles/creating-new-digital-learning-tools/)

An introduction to creating your own, fully custom learning tools.


Are you interested in creating your own, fully custom digital learning tools?

It's not as difficult or time consuming as you might think, and it's easy to get started!

## The languages of the web

Digital learning tools are usually built with the same technology as regular websites. It's all very connected. Everything is a stepping stone to something else, so no matter where you start you will find it easy to grow in any direction.

You already know you're going to be writing some code. Websites are made of three main languages:

1. HTML stands for HyperText Markup Language. It is 0% intimidating and technically not even a programming language. HTML says "This is a heading, this is a paragraph, and this is a button." Dead simple. You'll get the hang of it in an hour.
1. CSS stands for Cascading Style Sheets. It is also not technically a programming language. CSS says "This paragraph is in Times New Roman" and "This button turns blue when your mouse is over it." CSS is a little more complicated than HTML. You'll get the hang of it within a day, but then you'll spend the whole week having fun playing around with it.
1. JavaScript is my favorite programming language. It's also the only actual programming language in this list. Compared to HTML and CSS, JavaScript will actually take some time to learn. But remember, those two were easy as cake so that doesn't mean much. Also, I believe in you – I learned JavaScript from the internet (in high school), and you can too. And if you're feeling panicky at all, remember this: most webpages don't need any JavaScript!

There are two resources I like to point people toward for learning HTML, CSS, and JavaScript:

1. w3schools.com is perfect for getting started with HTML and CSS. They also have a JavaScript tutorial, and many other helpful tutorials on how to build web pages.
1. The Mozilla Developer Network (MDN) has tutorials and documentation on HTML, CSS, and JavaScript. Professional developers use this website all the time because it has very clear information on pretty much everything related to building websites.

## Publishing / hosting your new tools

Once you've built a website, you'll need a way to get it online. There are many ways to do this, but most of them cost money.

[GitHub Pages](https://pages.github.com/) is my favorite free way to host static content. GitHub is **the** website where programmers share code for free and work on code together. It's a neat place, and everyone is welcome.

You'll need to register for a GitHub account first. Then you can follow [this documentation](https://help.github.com/en/articles/configuring-a-publishing-source-for-github-pages) for how to set up your site.

Again, there are many ways to host a website on the internet, and if this is too techy or foreign to you, just Google around for another option! If you work for a school, you can probably set up a website through them.


Digital teaching tools take many forms. The most well-known is probably video. Sites like [Khan Academy](https://www.khanacademy.org/) help learners focus on specific skills and pick them up quickly. Apps like [Duolingo](https://www.duolingo.com/) take a similar approach to self-directed learning.

We get a *ton* of value out of these tools. Really – schools that leverage these tools are equipping their students better than schools that don't. Digital interfaces are universal, and navigating them should be second nature. What's more, there are plenty of things that can be done digitally that simply *can't* be done manually at the same speed or scale.

Before anyone gets the wrong idea, let me clarify: these tools do **not** replace teachers. These tools help both teachers and students. Would you want to be educated by an AI? (Hint: There's no such thing as real artificial intelligence.) I'm not at all suggesting that we could have less teachers if only we had more or better technology. I am absolutely claiming that with the help of the right technology, teachers can make a much bigger impact.

## Value added experiences

Digital learning tools deserve more investment for a variety of reasons:

1. Like a book that is written once and read by many people many times, high-quality digital learning tools are developed once and used by many learners
1. They can be easily distributed at a relatively low cost as there is no manufacturing involved
1. While in-person instruction has its own incredible value, the ability for each individual student to learn at his or her own pace is also incredibly valuable
1. Any learner can re-watch, re-read, or re-try digital content, which saves a lot of time with questions
1. These tools make information easier to navigate – students can easily search for and bookmark important content, creating a reference they are more likely to use
1. Digital tools can adapt to the learner's needs, such as adjusting the difficulty of review questions to match the learner's ability level
1. Digital tools save teachers valuable time with administration and bookkeeping, allowing them to devote more of their attention to quality instruction
1. Digital tools can help teachers efficiently assess student learning, which allows them to do so more regularly, which allows them to create a more personalized learning experience
1. Digital tools can compete in a world of constant stimulation and immediate gratification, providing an engaging experience for young digital natives
1. Digital tools enable collaboration and competition, even remotely, providing a more engaging classroom experience

We have this technology at our fingertips. The question is: how do we take advantage of it?

## Finding existing tools

The easiest way to leverage education technology is to just use what already exists. Many of these tools are free!

I already mentioned a few examples of great learning tools above. The table below lists a few more, along with the specific needs they address.

Tool|Price|Purpose
---|---|---
Duolingo|Free|An app and website for learning languages.
Khan Academy|Free|High-quality, organized, focused video tutorials on a variety of topics, with a special focus on math and science.
Quizlet|Free|Interactive study tools like flash cards, word matching, spelling, and multiple-choice review. 

There are plenty more where those came from. Try Googling some of these terms:

* Free learning tools
* Online teaching tools
* Educational video sites
* Classroom apps

If all else fails, you can always build your own!

## Build your own digital learning tools

Programming may not be your cup of tea, but you don't have to be a computer wizard (or a huge nerd) to make your own learning tools.

Some of the free sites listed above will let you create new experiences tailored to your specific classroom. For example, you can create flashcards for your specific vocabulary, or a practice quiz for your own concepts.

If you're a little more media savvy, you might want to create your own videos to really help concepts sink in. My favorite is still [Cry Me a Verb](https://www.youtube.com/watch?v=yup8ifNVcKI) from when I was in high school Spanish.

Alternatively, you can have your students use these tools to create their own review activities! As they say, the best way to learn something is to teach it.

But what if you're a little more hardcore than that? What if you want to actually **build** something? A totally fresh, from scratch, new-this-year learning experience? Whether you can see it in your head or you just want the skills, I'm happy to point you in the right direction.

Check out my article on [Creating New Tools](/articles/creating-new-digital-learning-tools/).


Aaron Smith

Articles