CrashDump analysis automation using SlackBot, python, cdb from Windows

I’m going to show how to automate dump analysis using a Slackbot but using telegram bot is also quite same.

In order automate dump analysis, Visual Studio or WinDbg can be used in theory, however writing a script which behaves manipulate other application which has GUI interface, emulating keyboard and mouse inputs, and reading the specific text within edit control from running application going to be a challenge. You might accomplish it using Windows Automation API, but Windows Debugger Toolset provides a better way command line-based utility such as cdb.exe.

CDB.EXE

cdb.exe is located same directory where WinDbg.exe resides.


Following page has basic instructions how to open a dump file with cdb.exe from command line.

https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/analyzing-a-user-mode-dump-file-with-cdb


cdb -y SymbolPath -i ImagePath -z DumpFileName

You can find out more command line options for cdb.exe:

https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/cdb-command-line-options

Let’s see how it works with simple demo. The directory contains demo app and its PDB symbol.


Run the cdb.exe with ‘-y’ option to specify the dump to open with.

& “C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\cdb.exe” -z crashme.dmp

The result shows exactly same what we’ve seen from WinDBG.



PS D:\workspace\playground\crashme\Release> & “C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\cdb.exe” -z crashme.dmp

Microsoft (R) Windows Debugger Version 10.0.14321.1024 X86

Copyright (c) Microsoft Corporation. All rights reserved.

Loading Dump File [D:\workspace\playground\crashme\Release\crashme.dmp]

User Mini Dump File with Full Memory: Only application data is available

************* Symbol Path validation summary **************

Response                         Time (ms)     Location

Deferred                                       srv*d:\symbols*https://msdl.microsoft.com/download/symbols

Symbol search path is: srv*d:\symbols*https://msdl.microsoft.com/download/symbols

Executable search path is:

Windows 10 Version 16299 MP (8 procs) Free x86 compatible

Product: WinNt, suite: SingleUserTS

Built by: 10.0.16299.15 (WinBuild.160101.0800)

Machine Name:

Debug session time: Mon Nov 27 22:54:12.000 2017 (UTC + 9:00)

System Uptime: 2 days 17:55:58.676

Process Uptime: 0 days 0:00:18.000

……

This dump file has an exception of interest stored in it.

The stored exception information can be accessed via .ecxr.

(3fac.21d0): Access violation – code c0000005 (first/second chance not available)

*** WARNING: Unable to verify checksum for crashme.exe

eax=00000000 ebx=aabbccdd ecx=00000000 edx=00000000 esi=742d80e8 edi=02a85998

eip=00311015 esp=006ffa98 ebp=006ffaa8 iopl=0         nv up ei pl zr na pe nc

cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010244

crashme!main+0x15:

00311015 c700ddccbbaa    mov     dword ptr [eax],0AABBCCDDh ds:002b:00000000=????????

0:000>

cdb.exe successfully launched and waits user input. Press ‘q <enter>’ to return to the command line.

PS D:\workspace\playground\crashme\Release> & “C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\cdb.exe” -z crashme.dmp

Microsoft (R) Windows Debugger Version 10.0.14321.1024 X86

Copyright (c) Microsoft Corporation. All rights reserved.

Loading Dump File [D:\workspace\playground\crashme\Release\crashme.dmp]

User Mini Dump File with Full Memory: Only application data is available

************* Symbol Path validation summary **************

Response                         Time (ms)     Location

Deferred                                       srv*d:\symbols*https://msdl.microsoft.com/download/symbols

Symbol search path is: srv*d:\symbols*https://msdl.microsoft.com/download/symbols

Executable search path is:

Windows 10 Version 16299 MP (8 procs) Free x86 compatible

Product: WinNt, suite: SingleUserTS

Built by: 10.0.16299.15 (WinBuild.160101.0800)

Machine Name:

Debug session time: Mon Nov 27 22:54:12.000 2017 (UTC + 9:00)

System Uptime: 2 days 17:55:58.676

Process Uptime: 0 days 0:00:18.000

……

This dump file has an exception of interest stored in it.

The stored exception information can be accessed via .ecxr.

(3fac.21d0): Access violation – code c0000005 (first/second chance not available)

*** WARNING: Unable to verify checksum for crashme.exe

eax=00000000 ebx=aabbccdd ecx=00000000 edx=00000000 esi=742d80e8 edi=02a85998

eip=00311015 esp=006ffa98 ebp=006ffaa8 iopl=0         nv up ei pl zr na pe nc

cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010244

crashme!main+0x15:

00311015 c700ddccbbaa    mov     dword ptr [eax],0AABBCCDDh ds:002b:00000000=????????

0:000> q

quit:

PS D:\workspace\playground\crashme\Release>

You can use ‘-c’ to pass initial command runs followed by opening the dump. Each command is separated by ‘;’ so that enables carry out multiple command at once. For example, “u @eip; r; q” will show the disassembly since @EIP points, prints out registers and quit the cdb.exe itself then.

PS D:\workspace\playground\crashme\Release> & “C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\cdb.exe” -z crashme.dmp -c “u @eip; r; q”


PS D:\workspace\playground\crashme\Release> & “C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\cdb.exe” -z crashme.dmp -c “u @eip; r; q”

Microsoft (R) Windows Debugger Version 10.0.14321.1024 X86

Copyright (c) Microsoft Corporation. All rights reserved.

Loading Dump File [D:\workspace\playground\crashme\Release\crashme.dmp]

User Mini Dump File with Full Memory: Only application data is available

************* Symbol Path validation summary **************

Response                         Time (ms)     Location

Deferred                                       srv*d:\symbols*https://msdl.microsoft.com/download/symbols

Symbol search path is: srv*d:\symbols*https://msdl.microsoft.com/download/symbols

Executable search path is:

Windows 10 Version 16299 MP (8 procs) Free x86 compatible

Product: WinNt, suite: SingleUserTS

Built by: 10.0.16299.15 (WinBuild.160101.0800)

Machine Name:

Debug session time: Mon Nov 27 22:54:12.000 2017 (UTC + 9:00)

System Uptime: 2 days 17:55:58.676

Process Uptime: 0 days 0:00:18.000

……

This dump file has an exception of interest stored in it.

The stored exception information can be accessed via .ecxr.

(3fac.21d0): Access violation – code c0000005 (first/second chance not available)

*** WARNING: Unable to verify checksum for crashme.exe

eax=00000000 ebx=aabbccdd ecx=00000000 edx=00000000 esi=742d80e8 edi=02a85998

eip=00311015 esp=006ffa98 ebp=006ffaa8 iopl=0         nv up ei pl zr na pe nc

cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010244

crashme!main+0x15:

00311015 c700ddccbbaa    mov     dword ptr [eax],0AABBCCDDh ds:002b:00000000=????????

0:000> cdb: Reading initial command ‘u @eip; r; q’

crashme!main+0x15:

00311015 c700ddccbbaa    mov     dword ptr [eax],0AABBCCDDh

0031101b 5b              pop     ebx

0031101c 58              pop     eax

0031101d 33c0            xor     eax,eax

0031101f 5b              pop     ebx

00311020 8be5            mov     esp,ebp

00311022 5d              pop     ebp

00311023 c3              ret

eax=00000000 ebx=aabbccdd ecx=00000000 edx=00000000 esi=742d80e8 edi=02a85998

eip=00311015 esp=006ffa98 ebp=006ffaa8 iopl=0         nv up ei pl zr na pe nc

cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010244

crashme!main+0x15:

00311015 c700ddccbbaa    mov     dword ptr [eax],0AABBCCDDh ds:002b:00000000=????????

quit:

PS D:\workspace\playground\crashme\Release>

We now have figured out how to open a crash dump with initial analysis command from command line, it’s time to write the script automate those.

Let’s make the python virtual environment first.

PS D:\workspace\playground\crashme\Release> python -m venv venv

PS D:\workspace\playground\crashme\Release> .\venv\Scripts\activate

(venv) PS D:\workspace\playground\crashme\Release>


Launching cdb.exe and Gettings its output from Python script.

You can use the subprocess python module for spawning other process and getting its output. Please see the following:

        process = subprocess.Popen(cdb_param, stdout=subprocess.PIPE)

        for line in process.stdout:

            output_line = line.decode(‘utf-8’, ‘ignore’)

            #print(output_line, end=”)

            output_str_list.append(output_line)

In case you are wondering how to pass remaining options including -z dump file itself and initial command with -u…

import subprocess

import sys

exe_path = r”C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\cdb.exe”

arguments = [‘-z’, ‘crashme.dmp’, ‘-c’, ‘u @eip; r; q’]

def run_command(command):

    output = subprocess.check_output(command)

    print(output.decode(‘utf-8’))

if __name__ == “__main__”:

    run_command([exe_path] + arguments)

Let’s name the script as ‘cdb.py’ and now we’re ready to launch the cdb.exe and getting its result:

D:\workspace\playground\crashme\Release>python cdb.py

Microsoft (R) Windows Debugger Version 10.0.14321.1024 X86

Copyright (c) Microsoft Corporation. All rights reserved.

Loading Dump File [D:\workspace\playground\crashme\Release\crashme.dmp]

User Mini Dump File with Full Memory: Only application data is available

************* Symbol Path validation summary **************

Response                         Time (ms)     Location

Deferred                                       srv*d:\symbols*https://msdl.microsoft.com/download/symbols

Symbol search path is: srv*d:\symbols*https://msdl.microsoft.com/download/symbols

Executable search path is:

Windows 10 Version 16299 MP (8 procs) Free x86 compatible

Product: WinNt, suite: SingleUserTS

Built by: 10.0.16299.15 (WinBuild.160101.0800)

Machine Name:

Debug session time: Mon Nov 27 22:54:12.000 2017 (UTC + 9:00)

System Uptime: 2 days 17:55:58.676

Process Uptime: 0 days 0:00:18.000

……

This dump file has an exception of interest stored in it.

The stored exception information can be accessed via .ecxr.

(3fac.21d0): Access violation – code c0000005 (first/second chance not available)

*** WARNING: Unable to verify checksum for crashme.exe

eax=00000000 ebx=aabbccdd ecx=00000000 edx=00000000 esi=742d80e8 edi=02a85998

eip=00311015 esp=006ffa98 ebp=006ffaa8 iopl=0         nv up ei pl zr na pe nc

cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010244

crashme!main+0x15:

00311015 c700ddccbbaa    mov     dword ptr [eax],0AABBCCDDh ds:002b:00000000=????????

0:000> cdb: Reading initial command ‘u @eip; r; q’

crashme!main+0x15:

00311015 c700ddccbbaa    mov     dword ptr [eax],0AABBCCDDh

0031101b 5b              pop     ebx

0031101c 58              pop     eax

0031101d 33c0            xor     eax,eax

0031101f 5b              pop     ebx

00311020 8be5            mov     esp,ebp

00311022 5d              pop     ebp

00311023 c3              ret

eax=00000000 ebx=aabbccdd ecx=00000000 edx=00000000 esi=742d80e8 edi=02a85998

eip=00311015 esp=006ffa98 ebp=006ffaa8 iopl=0         nv up ei pl zr na pe nc

cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010244

crashme!main+0x15:

00311015 c700ddccbbaa    mov     dword ptr [eax],0AABBCCDDh ds:002b:00000000=????????

quit:

D:\workspace\playground\crashme\Release>


Other useful initial commands may include:

  • !analyze -v
  • ecxr;kb
  • ecxr;kP;dv /t/V
  • etc…

Connecting the cdb.py to SlackBot

This post isn’t a comprehensive bot writing manual, so here describes only essential steps.

First, you need to create a new bot user from the Integration page. Visit the ‘Administration > Manage apps > Custom Integrations’.


I named it a ‘debugbot’ here.


If it’s successfully created, it’ll show a API token. It’s a secret so don’t share publicly (the ‘debugbot’ no longer exist).


You can also modify the bot’s name and profile picture here.


https://github.com/slackapi/python-slackclient
is the official repo of the Slack’s Python bot client. Clone the slack-client library.

(venv) D:\workspace\playground\crashme\Release>git clone https://github.com/slackapi/python-slackclient.git

Cloning into ‘python-slackclient’…

remote: Counting objects: 1320, done.

remote: Compressing objects: 100% (24/24), done.

remote: Total 1320 (delta 15), reused 24 (delta 10), pack-reused 1286

Receiving objects: 100% (1320/1320), 878.80 KiB | 359.00 KiB/s, done.

Resolving deltas: 100% (797/797), done.

(venv) D:\workspace\playground\crashme\Release>cd python-slackclient

(venv) D:\workspace\playground\crashme\Release\python-slackclient>pip install -r requirements.txt

Obtaining file:///D:/workspace/playground/crashme/Release/python-slackclient (from -r requirements.txt (line 3))

Collecting websocket-client<1.0a0,>=0.35 (from slackclient==1.0.9->-r requirements.txt (line 3))

  Downloading websocket_client-0.44.0-py2.py3-none-any.whl (199kB)

    100% |████████████████████████████████| 204kB 799kB/s

Collecting requests<3.0a0,>=2.11 (from slackclient==1.0.9->-r requirements.txt (line 3))

  Downloading requests-2.18.4-py2.py3-none-any.whl (88kB)

    100% |████████████████████████████████| 92kB 738kB/s

Collecting six<2.0a0,>=1.10 (from slackclient==1.0.9->-r requirements.txt (line 3))

  Downloading six-1.11.0-py2.py3-none-any.whl

Collecting chardet<3.1.0,>=3.0.2 (from requests<3.0a0,>=2.11->slackclient==1.0.9->-r requirements.txt (line 3))

  Downloading chardet-3.0.4-py2.py3-none-any.whl (133kB)

    100% |████████████████████████████████| 143kB 172kB/s

Collecting certifi>=2017.4.17 (from requests<3.0a0,>=2.11->slackclient==1.0.9->-r requirements.txt (line 3))

  Downloading certifi-2017.11.5-py2.py3-none-any.whl (330kB)

    100% |████████████████████████████████| 337kB 297kB/s

Collecting urllib3<1.23,>=1.21.1 (from requests<3.0a0,>=2.11->slackclient==1.0.9->-r requirements.txt (line 3))

  Downloading urllib3-1.22-py2.py3-none-any.whl (132kB)

    100% |████████████████████████████████| 133kB 197kB/s

Collecting idna<2.7,>=2.5 (from requests<3.0a0,>=2.11->slackclient==1.0.9->-r requirements.txt (line 3))

  Downloading idna-2.6-py2.py3-none-any.whl (56kB)

    100% |████████████████████████████████| 61kB 72kB/s

Installing collected packages: six, websocket-client, chardet, certifi, urllib3, idna, requests, slackclient

  Running setup.py develop for slackclient

Successfully installed certifi-2017.11.5 chardet-3.0.4 idna-2.6 requests-2.18.4 six-1.11.0 slackclient urllib3-1.22 websocket-client-0.44.0


Then you need to invite the bot to read text from the client coming from the Slack channel. I created a dedicated ‘#autoanalysis’ channel and invited the bot for example.




Run the following command from a command line to set the API token to ‘SLACK_API’ environment variable for convenient.

SET SLACK_API_TOKEN=xoxb-282029623751-BVtmnS3BQitmjZvjpQL7PSGP

Then, write a basic Slack bot client code as following. Please refer to the full documentation more detail from https://slackapi.github.io/python-slackclient/ and https://python-slackclient.readthedocs.io/en/latest/basic_usage.html#sending-a-message. It basically does WebAPI call with ‘chat.postMessage’ command described here https://api.slack.com/methods/chat.postMessage.

import os

from slackclient import SlackClient

slack_token = os.environ[“SLACK_API_TOKEN”]

sc = SlackClient(slack_token)

ret = sc.api_call(

  “chat.postMessage”,

  channel=”#autoanalysis”,

  text=”Hello from Python! :tada:”

)

print(ret)

I added ‘print(ret)’ last line to see what the Slack server returned. The return object also contains the API call result.

(venv) D:\workspace\playground\crashme\Release>python slack.py

{‘ok’: True, ‘channel’: ‘C88K023NZ’, ‘ts’: ‘1512317953.000112’, ‘message’: {‘text’: ‘Hello from Python! :tada:’, ‘username’: ‘bot’, ‘bot_id’: ‘B892XRL93’, ‘type’: ‘message’, ‘subtype’: ‘bot_message’, ‘ts’: ‘1512317953.000112’}}

If everything goes well, you’ll see the small snippet prints that string to the Slack channel as below:


Read a command and do analysis from the Bot

Using the WebAPI based ‘api_call’ was simple to use but it’s not enough to receive real time messages others talking from the channel. For that purpose, the SlackClient provides Real Time Messaging API(RTM API).

http://slackapi.github.io/python-slackclient/real_time_messaging.html#

Next snippet shows how to connect to the bot channel and keep printing messages in real time while it’s connected.

import os

import time

from slackclient import SlackClient

slack_token = os.environ[“SLACK_API_TOKEN”]

sc = SlackClient(slack_token)

if sc.rtm_connect():

    while True:

        print( sc.rtm_read())

        time.sleep(1)

else:

    print( “Connection Failed”)

It’ll keep running until pressing ‘Ctrl+C'(or disconnected)

(venv) D:\workspace\playground\crashme\Release>python slack.py

[]

[{‘type’: ‘hello’}]

[{‘type’: ‘reconnect_url’, ‘url’: ‘wss://mpmulti-7vr5.lb.slack-msgs.com/websocket/_gkSQP2x_PoYBPCsYMqpryRjT7ggXh4AfVyTT8Gz1MJ44xZLpLHg_iPHzIFEXxPBS0TwuH9gy9Nqe3PdqE5ikWUlYsF2Ior3GTdI-eZ83GadFvzf9b0KTGFvfVUK_GROj06QwQkEiy3zlVguHGmPRThTTau6u0J2ws69qF4Yv1Q=’}]

[{‘type’: ‘presence_change’, ‘presence’: ‘active’, ‘user’: ‘U8A0VJBN3’}]

[]

[]

[]

[]

Traceback (most recent call last):

  File “slack.py”, line 11, in <module>

    time.sleep(1)

KeyboardInterrupt

(venv) D:\workspace\playground\crashme\Release>


You can filter out only messages by following:

import os

import time

from slackclient import SlackClient

slack_token = os.environ[“SLACK_API_TOKEN”]

sc = SlackClient(slack_token)

if sc.rtm_connect():

    while True:

        json_response = sc.rtm_read()

        for json_item in json_response:

            if “type” in json_item and json_item[“type”] == “message”:

                print(json_item[“text”])

        time.sleep(1)

else:

    print( “Connection Failed”)

Let’s put all together. It waits until someone type ‘analyze’, run the cdb.exe and finally upload its output as a text attachment.

import os

import time

import subprocess

import sys

import io

exe_path = r”C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\cdb.exe”

arguments = [‘-z’, ‘crashme.dmp’, ‘-c’, ‘u @eip; r; q’]

def run_command(command):

    output = subprocess.check_output(command)

    return output.decode(‘utf-8’)

from slackclient import SlackClient

slack_token = os.environ[“SLACK_API_TOKEN”]

sc = SlackClient(slack_token)

if sc.rtm_connect():

    while True:

        json_response = sc.rtm_read()

        for json_item in json_response:

            if “type” in json_item and json_item[“type”] == “message”:

                if json_item[“text”].lower() == “analyze”:

                    result = run_command([exe_path] + arguments)

                    ret = sc.api_call(“files.upload”, filename=”cdb_analyze_result.txt”,channels=json_item[“channel”],file=io.BytesIO(str.encode(result)))

                    if not ‘ok’ in ret or not ret[‘ok’]:

                        print(ret[‘error’])

        time.sleep(1)

else:

    print( “Connection Failed”)

The cdb’s output showed up in that channel successfully.


Let’s separate the upload part as a different function named ‘upload_text’ for simplicity.

import os

import time

import subprocess

import sys

import io

exe_path = r”C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\cdb.exe”

arguments = [‘-z’, ‘crashme.dmp’, ‘-c’, ‘u @eip; r; q’]

def run_command(command):

    output = subprocess.check_output(command)

    return output.decode(‘utf-8’)

def upload_text(text, file_name, channel):

    ret = sc.api_call(“files.upload”, filename=file_name,channels=channel,file=io.BytesIO(str.encode(text)))

    if not ‘ok’ in ret or not ret[‘ok’]:

        print(ret[‘error’])

from slackclient import SlackClient

slack_token = os.environ[“SLACK_API_TOKEN”]

sc = SlackClient(slack_token)

if sc.rtm_connect():

    while True:

        json_response = sc.rtm_read()

        for json_item in json_response:

            if “type” in json_item and json_item[“type”] == “message”:

                if json_item[“text”].lower() == “analyze”:

                    result = run_command([exe_path] + arguments)

                    upload_text(result, “cdb_analyze_result.txt”, json_item[“channel”])

        time.sleep(1)

else:

    print( “Connection Failed”)

More tests!


Here’s the last sample slightly improved to handle errors and command reading.

import os

import time

import subprocess

import sys

import io

exe_path = r”C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\cdb.exe”

arguments = [‘-c’, ‘u @eip; r; q’]

def run_command(command):

    try:

        output = subprocess.check_output(command)

        return output.decode(‘utf-8’)

    except subprocess.CalledProcessError as e:

        return ‘analysis failed with {}, {}’.format(e.returncode, e.output.decode(‘utf-8’))

def upload_text(text, file_name, channel):

    ret = sc.api_call(“files.upload”, filename=file_name,channels=channel,file=io.BytesIO(str.encode(text)))

    if not ‘ok’ in ret or not ret[‘ok’]:

        print(ret[‘error’])

from slackclient import SlackClient

slack_token = os.environ[“SLACK_API_TOKEN”]

sc = SlackClient(slack_token)

if sc.rtm_connect():

    while True:

        json_response = sc.rtm_read()

        for json_item in json_response:

            if “type” in json_item and json_item[“type”] == “message”:

                args = json_item[“text”].split()

                if args[0].startswith(‘!’):

                    if len(args) > 1 and args[0].lower() == “!analyze”:

                        result = run_command([exe_path] + [‘-z’, args[1]] + arguments)

                        upload_text(result, “cdb_analyze_result.txt”, json_item[“channel”])

                    else:

                        ret = sc.api_call(“chat.postMessage”, channel=json_item[“channel”],text=”Unknown command”)

        time.sleep(1)

else:

    print( “Connection Failed”)

Thanks for reading.

Heejune.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s