AWS Password Cracking with Rook – Making Progress

I’ve been doing some more AWS Password Cracking recently, but with a different tool!

AWS Password Cracking – Introduction

If you read my last post about Coalfire NPK, then you know that I’ve been in the market for cloud password crackers.

This time, I tried out Rook from JumpsecLabs.

While far simpler than NPK, this was closer to what I was looking for as far as simplicity and configurations.

Installation and Configuration

First, I created a new EC2 host to serve as my bastion/Terraform runner.

doyler@mbp:~$ ssh -i ~/.ssh/crackingRig-TerraLaunch.pem [email protected]
The authenticity of host '1.2.3.4 (1.2.3.4)' can't be established.
ECDSA key fingerprint is SHA256:em6jS7Pao26UWQ/g8EFFsoNvBgKLOVjdY4oYG+O3BkA.
Are you sure you want to continue connecting (yes/no)? yes

...

ubuntu@ip-1-2-3-4:~$ 

Next, I installed Terraform to the server.

ubuntu@ip-1-2-3-4:~$ terraform --version
Terraform v0.12.24

Next, I updated my Python to version 3 as well.

ubuntu@ip-1-2-3-4:~$ python3 --version
Python 3.6.9

With the pre-requisites installed, it was time to get cracking!

AWS Password Cracking – More Prerequisites

Unfortunately, when I tried to run the tool, I was still missing some prerequisites.

ubuntu@ip-1-2-3-4:~/tools/Rook$ python rook.py --check-id
Traceback (most recent call last):
  File "rook.py", line 6, in 
    import boto3
ModuleNotFoundError: No module named 'boto3'

I installed the other prerequisites, hoping this would solve my problem.

ubuntu@ip-1-2-3-4:~/tools/Rook$ pip install boto3
ubuntu@ip-1-2-3-4:~/tools/Rook$ pip install bson
ubuntu@ip-1-2-3-4:~/tools/Rook$ pip install json_util

Unfortunately, this was not the correct bson, so I had to fix the modules one more time.

pip uninstall bson
pip uninstall pymongo
pip install pymongo

Next, I had to fix where Terraform was located/where the script was looking.

ubuntu@ip-1-2-3-4:~/tools/Rook$ python rook.py --check-id


     �-��-��-��-��-��-��--  �-��-��-��-��-��-��--  �-��-��-��-��-��-��-- �-��-��--  �-��-��--
     �-��-��"══�-��-��--�-��-��"═══�-��-��--�-��-��"═══�-��-��--�-��-��' �-��-��"╝
     �-��-��-��-��-��-��"╝�-��-��'   �-��-��'�-��-��'   �-��-��'�-��-��-��-��-��"╝ 
     �-��-��"══�-��-��--�-��-��'   �-��-��'�-��-��'   �-��-��'�-��-��"═�-��-��-- 
     �-��-��'  �-��-��'╚�-��-��-��-��-��-��"╝╚�-��-��-��-��-��-��"╝�-��-��'  �-��-��--
     ╚═╝  ╚═╝ ╚═════╝  ╚═════╝ ╚═╝  ╚═╝
Terraform AWS instances for cracking hashes
    
[!] Terraform executable is not present. The file can be downloaded from here https://releases.hashicorp.com/terraform/0.12.10/terraform_0.12.10_linux_amd64.zip
[!] Exiting...

Finally, I used gimme-aws-creds to configure the AWS credentials that I would need.

ubuntu@ip-1-2-3-4:~/tools/Rook$ cat ~/.aws/credentials 
[default]
aws_access_key_id = (key_id)
aws_secret_access_key = (access_key)
aws_session_token =  (session_token)
aws_security_token = (security_token)

Up Next? Debugging!

First, when I tried to run the script, I was getting a weird error.

ubuntu@ip-1-2-3-4:~/tools/Rook$ python rook.py --check-id


     �-��-��-��-��-��-��--  �-��-��-��-��-��-��--  �-��-��-��-��-��-��-- �-��-��--  �-��-��--
     �-��-��"══�-��-��--�-��-��"═══�-��-��--�-��-��"═══�-��-��--�-��-��' �-��-��"╝
     �-��-��-��-��-��-��"╝�-��-��'   �-��-��'�-��-��'   �-��-��'�-��-��-��-��-��"╝ 
     �-��-��"══�-��-��--�-��-��'   �-��-��'�-��-��'   �-��-��'�-��-��"═�-��-��-- 
     �-��-��'  �-��-��'╚�-��-��-��-��-��-��"╝╚�-��-��-��-��-��-��"╝�-��-��'  �-��-��--
     ╚═╝  ╚═╝ ╚═════╝  ╚═════╝ ╚═╝  ╚═╝
Terraform AWS instances for cracking hashes
    
[!] No AWS credentials present. Running aws configure now. Region should be eu-west-2
Traceback (most recent call last):
  File "rook.py", line 145, in 
    presetup()
  File "rook.py", line 50, in presetup
    subprocess.call("aws", "configure")
  File "/usr/lib/python3.6/subprocess.py", line 287, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/lib/python3.6/subprocess.py", line 629, in __init__
    raise TypeError("bufsize must be an integer")
TypeError: bufsize must be an integer

Next, I updated the .tf files to use us-west-2, and I updated the Python script to point to the proper credentials file.

def presetup():
    home = str(Path.home())
    if os.path.exists('/usr/local/bin/terraform') is False:
        print("[!] Terraform executable is not present. The file can be downloaded from here https://releases.hashicorp.com/terraform/0.12.10/terraform_0.12.10_linux_amd64.zip")
        print("[!] Exiting...")
        sys.exit(1)
    if os.path.exists(home + '/.aws/credentials') is False:
        print("[!] No AWS credentials present. Running aws configure now. Region should be eu-west-2")
        subprocess.call("aws", "configure")
        print("[!] You can now run Rook!")
        sys.exit(0)
    if os.path.exists('./.terraform/') is False:
        print("[+] Performing first time Terraform setup. Please wait while correct packages are built.")
        subprocess.call(['terraform', 'init'])
        print("[+] Terraform initialisation complete, please re-run to create your instance.")
        sys.exit(0)

I also did not realize that there wasn’t a main.tf file, so I copied over my modified example file.

ubuntu@ip-1-2-3-4:~/tools/Rook$ cp terraform_files/spot-cracking.tf ./main.tf

Now, when I ran the script, I was able to successfully initialize Terraform.

ubuntu@ip-1-2-3-4:~/tools/Rook$ python rook.py --check-id


     �-��-��-��-��-��-��--  �-��-��-��-��-��-��--  �-��-��-��-��-��-��-- �-��-��--  �-��-��--
     �-��-��"══�-��-��--�-��-��"═══�-��-��--�-��-��"═══�-��-��--�-��-��' �-��-��"╝
     �-��-��-��-��-��-��"╝�-��-��'   �-��-��'�-��-��'   �-��-��'�-��-��-��-��-��"╝ 
     �-��-��"══�-��-��--�-��-��'   �-��-��'�-��-��'   �-��-��'�-��-��"═�-��-��-- 
     �-��-��'  �-��-��'╚�-��-��-��-��-��-��"╝╚�-��-��-��-��-��-��"╝�-��-��'  �-��-��--
     ╚═╝  ╚═╝ ╚═════╝  ╚═════╝ ╚═╝  ╚═╝
Terraform AWS instances for cracking hashes
    
[+] Performing first time Terraform setup. Please wait while correct packages are built.

... <snip> ...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
[+] Terraform initialisation complete, please re-run to create your instance.

Unfortunately, I had to update the region in the variables file as well.

ubuntu@ip-1-2-3-4:~/tools/Rook$ python rook.py --check-id


     �-��-��-��-��-��-��--  �-��-��-��-��-��-��--  �-��-��-��-��-��-��-- �-��-��--  �-��-��--
     �-��-��"══�-��-��--�-��-��"═══�-��-��--�-��-��"═══�-��-��--�-��-��' �-��-��"╝
     �-��-��-��-��-��-��"╝�-��-��'   �-��-��'�-��-��'   �-��-��'�-��-��-��-��-��"╝ 
     �-��-��"══�-��-��--�-��-��'   �-��-��'�-��-��'   �-��-��'�-��-��"═�-��-��-- 
     �-��-��'  �-��-��'╚�-��-��-��-��-��-��"╝╚�-��-��-��-��-��-��"╝�-��-��'  �-��-��--
     ╚═╝  ╚═╝ ╚═════╝  ╚═════╝ ╚═╝  ╚═╝
Terraform AWS instances for cracking hashes
    
Traceback (most recent call last):

... <snip> ...

  File "/home/ubuntu/.local/lib/python3.6/site-packages/botocore/regions.py", line 135, in _endpoint_for_partition
    raise NoRegionError()
botocore.exceptions.NoRegionError: You must specify a region.

Now, when I ran the script, it properly picked up and authenticated with the credentials.

ubuntu@ip-1-2-3-4:~/tools/Rook$ python rook.py --check-id


     �-��-��-��-��-��-��--  �-��-��-��-��-��-��--  �-��-��-��-��-��-��-- �-��-��--  �-��-��--
     �-��-��"══�-��-��--�-��-��"═══�-��-��--�-��-��"═══�-��-��--�-��-��' �-��-��"╝
     �-��-��-��-��-��-��"╝�-��-��'   �-��-��'�-��-��'   �-��-��'�-��-��-��-��-��"╝ 
     �-��-��"══�-��-��--�-��-��'   �-��-��'�-��-��'   �-��-��'�-��-��"═�-��-��-- 
     �-��-��'  �-��-��'╚�-��-��-��-��-��-��"╝╚�-��-��-��-��-��-��"╝�-��-��'  �-��-��--
     ╚═╝  ╚═╝ ╚═════╝  ╚═════╝ ╚═╝  ╚═╝
Terraform AWS instances for cracking hashes
    
[+] AWS Identity keypair name is: crackingRig-RayBurp

Next, I ran into an issue with my zone, and needed to update it to us-west-2b.

ubuntu@ip-1-2-3-4:~/tools/Rook$ python rook.py -t p3.16xlarge -f /home/ubuntu/hashes/dbhashes.txt -m 30 -i rook-crackingPrivate -s /home/ubuntu/.ssh/rook-crackingPrivate.pem --spot 7.34 --debug

... <snip> ...

aws_spot_instance_request.rook-spot: Creating...

Error: Error requesting spot instances: InvalidParameterValue: Invalid availability zone: [us-west-2]
	status code: 400, request id: 6ea0d9bf-f83e-49f7-a814-33fdb1d82ada

  on main.tf line 33, in resource "aws_spot_instance_request" "rook-spot":
  33: resource "aws_spot_instance_request" "rook-spot" {

After fixing my zone, I also had to update the AMI that I wanted to use.

ubuntu@ip-1-2-3-4:~/tools/Rook$ python rook.py -t p3.16xlarge -f /home/ubuntu/hashes/dbhashes.txt -m 30 -i rook-crackingPrivate -s /home/ubuntu/.ssh/rook-crackingPrivate.pem --spot 7.34 --debug

... <snip> ...

Error: Error requesting spot instances: InvalidAMIID.NotFound: The image id '[ami-0ee246e709782b1be]' does not exist
	status code: 400, request id: 5bcec225-3df9-4ea6-99a1-c74b752e735c

  on main.tf line 33, in resource "aws_spot_instance_request" "rook-spot":
  33: resource "aws_spot_instance_request" "rook-spot" {

But Wait, There’s More (Debugging)!

Thinking that I had finally solved my problems, I ran the script yet again.

My next problem was my own fault, and I didn’t reference a snapshot to use.

ubuntu@ip-1-2-3-4:~/tools/Rook$ python rook.py -t p3.16xlarge -f /home/ubuntu/hashes/dbhashes.txt -m 30 -i rook-crackingPrivate -s /home/ubuntu/.ssh/rook-crackingPrivate.pem --spot 7.34 --debug

... <snip> ...

Error: Error requesting spot instances: InvalidSnapshotID.NotFound: The snapshot ID 'snap-' does not exist
	status code: 400, request id: fcf1a8b2-0f69-41f0-9fa7-fb3d26a0c069

  on main.tf line 33, in resource "aws_spot_instance_request" "rook-spot":
  33: resource "aws_spot_instance_request" "rook-spot" {

After commenting out the snapshot section entirely (don’t worry, that will bite me later), I ran into issues with my bid price.

ubuntu@ip-1-2-3-4:~/tools/Rook$ python rook.py -t p3.16xlarge -f /home/ubuntu/hashes/dbhashes.txt -m 30 -i rook-crackingPrivate -s /home/ubuntu/.ssh/rook-crackingPrivate.pem --spot 7.34 --debug

... <snip> ...

  ProductDescription: "Linux/UNIX",
  SpotInstanceRequestId: "sir-6tdibs2q",
  SpotPrice: "7.340000",
  State: "open",
  Status: {
    Code: "pending-evaluation",
    Message: "Your Spot request has been submitted for review, and is pending evaluation.",
    UpdateTime: 2020-04-03 19:18:12 +0000 UTC
  },
  Type: "one-time"
}) to resolve: unexpected state 'price-too-low', wanted target 'fulfilled'. last error: %!s()

  on main.tf line 33, in resource "aws_spot_instance_request" "rook-spot":
  33: resource "aws_spot_instance_request" "rook-spot" {

price-too-low: Your Spot request price of 7.34 is lower than the minimum required Spot request fulfillment price of 7.344. 

While I was annoyed that AWS blocked me over four-hundredths of a cent, I verified the current prices.

ubuntu@ip-1-2-3-4:~/tools/Rook$ python rook.py --check-spot p3.16xlarge


     �-��-��-��-��-��-��--  �-��-��-��-��-��-��--  �-��-��-��-��-��-��-- �-��-��--  �-��-��--
     �-��-��"══�-��-��--�-��-��"═══�-��-��--�-��-��"═══�-��-��--�-��-��' �-��-��"╝
     �-��-��-��-��-��-��"╝�-��-��'   �-��-��'�-��-��'   �-��-��'�-��-��-��-��-��"╝ 
     �-��-��"══�-��-��--�-��-��'   �-��-��'�-��-��'   �-��-��'�-��-��"═�-��-��-- 
     �-��-��'  �-��-��'╚�-��-��-��-��-��-��"╝╚�-��-��-��-��-��-��"╝�-��-��'  �-��-��--
     ╚═╝  ╚═╝ ╚═════╝  ╚═════╝ ╚═╝  ╚═╝
Terraform AWS instances for cracking hashes
    
[+] Getting spot prices for p3.16xlarge instances in eu-west-2b.

[
    {
        "AvailabilityZone": "us-west-2c",
        "InstanceType": "p3.16xlarge",
        "ProductDescription": "Linux/UNIX",
        "SpotPrice": "7.344000",
        "Timestamp": 1585861737.0
    },
    {
        "AvailabilityZone": "us-west-2a",
        "InstanceType": "p3.16xlarge",
        "ProductDescription": "Linux/UNIX",
        "SpotPrice": "7.344000",
        "Timestamp": 1585861737.0
    },
    {
        "AvailabilityZone": "us-west-2b",
        "InstanceType": "p3.16xlarge",
        "ProductDescription": "Linux/UNIX",
        "SpotPrice": "7.344000",
        "Timestamp": 1585861737.0
    }
]

After fixing my bid price, I received an error about my wordlists and a missing volume (ah yes, that snapshot…).

ubuntu@ip-1-2-3-4:~/tools/Rook$ python rook.py -t p3.8xlarge -f /home/ubuntu/hashes/dbhashes.txt -m 30 -i rook-crackingPrivate -s /home/ubuntu/.ssh/rook-crackingPrivate.pem --spot 7.344 --debug

... <snip> ...

aws_spot_instance_request.rook-spot (remote-exec): mount: /words: wrong fs type, bad option, bad superblock on /dev/xvdb, missing codepage or helper program, or other error.
aws_spot_instance_request.rook-spot (remote-exec): nohup: appending output to '/home/ubuntu/nohup.out'

When I connected to the instance, hashcat had failed because it could not find the wordlists.

AWS Password Cracking – EBS Volumes and Making Progress

I realized that I needed to setup an EBS volume, so that my wordlists would be persistent across instances.

First, I followed the AWS EBS guide.

I created this volume from my initial Terraform runner, as I figured I could update them as needed as well.

root@ip-1-2-3-4:/# lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0     7:0    0 89.1M  1 loop /snap/core/8268
loop1     7:1    0   18M  1 loop /snap/amazon-ssm-agent/1480
loop2     7:2    0 93.8M  1 loop /snap/core/8935
loop3     7:3    0   18M  1 loop /snap/amazon-ssm-agent/1566
xvda    202:0    0    8G  0 disk 
�""�"�xvda1 202:1    0    8G  0 part /
xvdf    202:80   0   50G  0 disk 
root@ip-1-2-3-4:/# file -s /dev/xvdf
/dev/xvdf: data
root@ip-1-2-3-4:/# mkfs -t xfs /dev/xvdf
meta-data=/dev/xvdf              isize=512    agcount=4, agsize=3276800 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=0, rmapbt=0, reflink=0
data     =                       bsize=4096   blocks=13107200, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=6400, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
root@ip-1-2-3-4:/# mount /dev/xvdf /words

Final TF Files

In the end, this was my main.tf file.

locals {
  hashcmd 	= "${var.cmdp1} ${var.hashmode} ${var.cmdp2}"
}

data "http" "myip" {
  url = "https://ipinfo.io/ip"
}

# create and run a cracking instance
provider "aws" {
  profile	= "default"
  region	= "us-west-2"
}

resource "aws_security_group" "rook_security" {
  description	= "Allow inbound SSH."
  
  egress {
    from_port	= 0
    to_port	= 0
    protocol	= "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port	= 22
    to_port	= 22
    protocol	= "tcp"
    cidr_blocks	= ["${var.whitelistip}", "${chomp(data.http.myip.body)}/32"]
  }
}

resource "aws_spot_instance_request" "rook-spot" {
  spot_price		= "${var.spotprice}"
  spot_type		= "one-time"
  availability_zone	= "us-west-2b"
  wait_for_fulfillment	= true
  ami			= "${var.ami}"
  instance_type		= "${var.itype}"
  key_name		= "${var.identity}"
  security_groups	= ["${aws_security_group.rook_security.name}"]
   
  ebs_block_device {
    device_name	= "/dev/xvdb"
    volume_size = 50
    volume_type = "gp2"
    snapshot_id = "${var.snapid}"
  }

  connection {
    type	= "ssh"
    host	= "${aws_spot_instance_request.rook-spot.public_ip}"
    user	= "ubuntu"
    private_key = "${file("${var.sshkeyfile}")}"
  }
  
  provisioner "file" {
    source	= "files/blacklist-nouveau.conf"
    destination	= "/tmp/blacklist-nouveau.conf"
  }

  provisioner "file" {
    source	= "files/nouveau-kms.conf"
    destination	= "/tmp/nouveau-kms.conf"
  }
  
  provisioner "file" {
    source	= "files/hashes.txt"
    destination	= "/tmp/hashes.txt"
  }

  provisioner "remote-exec" {
    inline = [
    "sudo apt update",
    "sudo DEBIAN_FRONTEND=noninteractive apt upgrade -y -o Dpkg::Options::=\"--force-confdef\" -o Dpkg::Options::=\"--force-confold\"",
    "sudo DEBIAN_FRONTEND=noninteractive apt install -y -o Dpkg::Options::=\"--force-confdef\" -o Dpkg::Options::=\"--force-confold\" build-essential linux-headers-4.15.0-1040-aws gcc screen linux-image-extra-virtual git make",
    "sudo apt update && sudo apt upgrade -y && sudo apt install build-essential -yq",
    "sudo cp /tmp/blacklist-nouveau.conf /etc/modprobe.d/.",
    "sudo cp /tmp/nouveau-kms.conf /etc/modprobe.d/.",
    "sudo cp /tmp/hashes.txt /opt/.",
    "sudo update-initramfs -u",
    "sudo reboot &",
    ]
    # sshd process may exit before the reboot completes, preventing it from
    # returning the scripts exit status
    # allow_missing_exit_status = true
    # Option doesn't appear to be working currently...
  }

  provisioner "remote-exec" {
    # new remote exec to connect back after restart
    inline = [
    "sudo wget -P /opt/ ${var.nvidia}",
    "sudo wget -P /opt/ ${var.hashcat}",
    "sudo /bin/bash /opt/NVIDIA-Linux-x86_64-410.104.run --ui=none --no-questions --silent -X",
    "sudo mkdir /opt/hashcat/",
    "sudo tar -xvf /opt/hashcat-5.1.0.tar.gz -C /opt/",
    "cd /opt/hashcat-5.1.0 && sudo make",
    "sudo mkdir /words/",
    "sudo mount /dev/xvdb /words/",
    "${local.hashcmd}",
    "sleep 1",
    ]
  }
}

resource "null_resource" "local" {
  provisioner "local-exec" {
    command = "echo ssh -i ${var.sshkeyfile} ubuntu@${aws_spot_instance_request.rook-spot.public_ip}"
  }
}

Additionally, this is what my variables.tf file looked like.

variable "snapid" {
  default	= "snap-SNAPIDHERE"
}

variable "nvidia" {
  default	= "http://us.download.nvidia.com/tesla/410.104/NVIDIA-Linux-x86_64-410.104.run"
}

variable "hashcat" {
  default	= "https://hashcat.net/files/hashcat-5.1.0.tar.gz"
}

variable "ami" {
  default	= "ami-0d1cd67c26f5fca19"
}

variable "itype" {
  default	= "p3.16xlarge"
}

variable "identity" {
  default	= "user"
}

variable "whitelistip" {
  default	= "0.0.0.0/32"
}

variable "sshkeyfile" {
  default	= "/home/user/.ssh/user"
}

variable "spotprice" {
  default	= null
}

variable "cmdp1" {
  default 	= "nohup sudo screen -dmS hashcat bash  -c 'sudo /opt/hashcat-5.1.0/hashcat -a 0 -m"
}

variable "cmdp2" {
  default = "/opt/hashes.txt /words/rockyou.txt /words/int_discovered_pw.txt /words/crackstation.txt -r /words/OneRuleToRuleThemAll.rule -o 00cracked.txt; exec bash' &"
}

variable "hashmode" {
  type		= number
  default	= "1000"
}

Running and Cracking

After finally updating it again, I was able to begin a successful run!

Note that I occasionally receive capacity not available errors, but that just takes some patience or a higher bid price.

ubuntu@ip-1-2-3-4:~/tools/Rook$ python rook.py -t p3.16xlarge -f /home/ubuntu/hashes/dbhashes.txt -m 30 -i rook-crackingPrivate -s /home/ubuntu/.ssh/rook-crackingPrivate.pem --spot 7.40 --debug

... <snip> ...

null_resource.local (local-exec): ssh -i /home/ubuntu/.ssh/rook-crackingPrivate.pem [email protected]
null_resource.local: Creation complete after 0s [id=3081094914902258810]

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

After Terraform created the instance, I SSHed into it and checked the hashcat status.

ubuntu@ip-1-2-3-4:~/tools/Rook$ ssh -i /home/ubuntu/.ssh/rook-crackingPrivate.pem [email protected]
The authenticity of host '5.6.7.8 (5.6.7.8)' can't be established.
ECDSA key fingerprint is SHA256:mDcIDE4vV/lcHDFjVofemK8qcGm8UwG1r31hGWVDF/w.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '5.6.7.8' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 18.04.4 LTS (GNU/Linux 4.15.0-1057-aws x86_64)

Last login: Fri Apr  3 20:14:40 2020 from 1.2.3.4
ubuntu@ip-5-6-7-8:~$ ps aux | grep hashcat
root      43479  0.0  0.0  32016  2856 ?        Ss   20:15   0:00 SCREEN -dmS hashcat bash -c sudo /opt/hashcat-5.1.0/hashcat -a 0 -m 30 /opt/hashes.txt /words/rockyou.txt /words/int_discovered_pw.txt /words/crackstation.txt -r /words/OneRuleToRuleThemAll.rule -o 00cracked.txt; exec bash
ubuntu    43751  0.0  0.0  14856  1100 pts/0    S+   20:17   0:00 grep --color=auto hashcat
ubuntu@ip-5-6-7-8:~$ screen -r
There is no screen to be resumed.
ubuntu@ip-5-6-7-8:~$ sudo su -
root@ip-5-6-7-8:~# screen -ls
There is a screen on:
	43479.hashcat	(04/03/20 20:15:08)	(Detached)
1 Socket in /run/screen/S-root.
root@ip-5-6-7-8:~# screen -r
[detached from 43479.hashcat]
root@ip-5-6-7-8:~# screen -r

As you can see, hashcat was running, and blazed through my keyspace using the 8 Tesla GPUs.

hashcat (v5.1.0) starting...

nvmlDeviceGetFanSpeed(): Not Supported

nvmlDeviceGetFanSpeed(): Not Supported

nvmlDeviceGetFanSpeed(): Not Supported

nvmlDeviceGetFanSpeed(): Not Supported

nvmlDeviceGetFanSpeed(): Not Supported

nvmlDeviceGetFanSpeed(): Not Supported

nvmlDeviceGetFanSpeed(): Not Supported

nvmlDeviceGetFanSpeed(): Not Supported

OpenCL Platform #1: NVIDIA Corporation
======================================
* Device #1: Tesla V100-SXM2-16GB, 4032/16130 MB allocatable, 80MCU
* Device #2: Tesla V100-SXM2-16GB, 4032/16130 MB allocatable, 80MCU
* Device #3: Tesla V100-SXM2-16GB, 4032/16130 MB allocatable, 80MCU
* Device #4: Tesla V100-SXM2-16GB, 4032/16130 MB allocatable, 80MCU
* Device #5: Tesla V100-SXM2-16GB, 4032/16130 MB allocatable, 80MCU
* Device #6: Tesla V100-SXM2-16GB, 4032/16130 MB allocatable, 80MCU
* Device #7: Tesla V100-SXM2-16GB, 4032/16130 MB allocatable, 80MCU
* Device #8: Tesla V100-SXM2-16GB, 4032/16130 MB allocatable, 80MCU

Skipping invalid or unsupported rule in file /words/OneRuleToRuleThemAll.rule on line 8210: ^o^�...^Ă^o^t
Skipping invalid or unsupported rule in file /words/OneRuleToRuleThemAll.rule on line 42459: ^a^�...^Ă^e^s^a^r^t^n^o^c
Hashes: 10 digests; 10 unique digests, 1 unique salts
Bitmaps: 16 bits, 65536 entries, 0x0000ffff mask, 262144 bytes, 5/13 rotates
Rules: 51995

...

Candidates.#1....: [Copying]
Candidates.#2....: [Copying]
Candidates.#3....: [Copying]
Candidates.#4....: [Copying]
Candidates.#5....: [Copying]
Candidates.#6....: [Copying]
Candidates.#7....: todo -> todo
Candidates.#8....: [Copying]
Hardware.Mon.#1..: Temp: 46c Util:  0% Core:1387MHz Mem: 877MHz Bus:16
Hardware.Mon.#2..: Temp: 51c Util:  0% Core:1530MHz Mem: 877MHz Bus:16
Hardware.Mon.#3..: Temp: 47c Util:  0% Core:1530MHz Mem: 877MHz Bus:16
Hardware.Mon.#4..: Temp: 50c Util:  0% Core:1530MHz Mem: 877MHz Bus:16
Hardware.Mon.#5..: Temp: 50c Util:  0% Core:1530MHz Mem: 877MHz Bus:16
Hardware.Mon.#6..: Temp: 52c Util:  0% Core:1530MHz Mem: 877MHz Bus:16
Hardware.Mon.#7..: Temp: 51c Util: 41% Core:1530MHz Mem: 877MHz Bus:16
Hardware.Mon.#8..: Temp: 51c Util:  0% Core:1312MHz Mem: 877MHz Bus:16

Started: Fri Apr  3 20:15:08 2020
Stopped: Fri Apr  3 20:17:07 2020

Also, I wanted to compare it to my work Mac, which didn’t even come close.

doyler@mbp:~$ hashcat -b -m 30
hashcat (v5.1.0-1397-g7f4df9eb) starting in benchmark mode...

Benchmarking uses hand-optimized kernel code by default.
You can use it in your cracking session by setting the -O option.
Note: Using optimized kernel code limits the maximum supported password length.
To disable the optimized kernel code in benchmark mode, use the -w option.

OpenCL API (OpenCL 1.2 (Jun 23 2019 21:50:55)) - Platform #1 [Apple]
====================================================================
* Device #1: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz, skipped
* Device #2: Intel(R) UHD Graphics 630, 384/1536 MB allocatable, 24MCU
* Device #3: AMD Radeon Pro 560X Compute Engine, 1024/4096 MB allocatable, 16MCU

Benchmark relevant options:
===========================
* --optimized-kernel-enable

Hashmode: 30 - md5(utf16le($pass).$salt)

Speed.#2.........:   520.7 MH/s (92.55ms) @ Accel:8 Loops:1024 Thr:256 Vec:1
Speed.#3.........:  3677.9 MH/s (62.01ms) @ Accel:128 Loops:512 Thr:256 Vec:1
Speed.#*.........:  4198.6 MH/s

Started: Thu Apr  2 21:14:11 2020
Stopped: Thu Apr  2 21:14:18 2020

In the end, I cracked 8/10 of my demo MD5 hashes in just under 2 minutes!

root@ip-5-6-7-8:/opt/hashcat-5.1.0# cat 00cracked.txt
(hash1)::(password1)
(hash2)::(password2)
(hash3)::(password3) 
(hash4)::(password4)  
(hash5)::(password5)
(hash6)::(password6)
(hash7)::(password7) 
(hash8)::(password8)

AWS Password Cracking – Conclusion

While it took a bit to configure, I’m really liking Rook.

I plan to use this instead of NPK going forward.

We will also be using cloud-cracking for the time being, as it should be more cost effective (and easier to expense) than a physical cracking rig.

That said, I still want to build my own tooling, as there were some shortfalls here as well.

If you have any suggestions, or want to help me with Terraform, then let me know!