I’m going to show how to automate dump analysis using a Slackbot but using telegram bot is also quite same.
In order automate dump analysis, Visual Studio or WinDbg can be used in theory, however writing a script which behaves manipulate other application which has GUI interface, emulating keyboard and mouse inputs, and reading the specific text within edit control from running application going to be a challenge. You might accomplish it using Windows Automation API, but Windows Debugger Toolset provides a better way command line-based utility such as cdb.exe.
CDB.EXE
cdb.exe is located same directory where WinDbg.exe resides.
Following page has basic instructions how to open a dump file with cdb.exe from command line.
cdb -y SymbolPath -i ImagePath -z DumpFileName
You can find out more command line options for cdb.exe:
https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/cdb-command-line-options
Let’s see how it works with simple demo. The directory contains demo app and its PDB symbol.
Run the cdb.exe with ‘-y’ option to specify the dump to open with.
& “C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\cdb.exe” -z crashme.dmp
The result shows exactly same what we’ve seen from WinDBG.
| PS D:\workspace\playground\crashme\Release> & “C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\cdb.exe” -z crashme.dmp Microsoft (R) Windows Debugger Version 10.0.14321.1024 X86 Copyright (c) Microsoft Corporation. All rights reserved. Loading Dump File [D:\workspace\playground\crashme\Release\crashme.dmp] User Mini Dump File with Full Memory: Only application data is available ************* Symbol Path validation summary ************** Response Time (ms) Location Deferred srv*d:\symbols*https://msdl.microsoft.com/download/symbols Symbol search path is: srv*d:\symbols*https://msdl.microsoft.com/download/symbols Executable search path is: Windows 10 Version 16299 MP (8 procs) Free x86 compatible Product: WinNt, suite: SingleUserTS Built by: 10.0.16299.15 (WinBuild.160101.0800) Machine Name: Debug session time: Mon Nov 27 22:54:12.000 2017 (UTC + 9:00) System Uptime: 2 days 17:55:58.676 Process Uptime: 0 days 0:00:18.000 …… This dump file has an exception of interest stored in it. The stored exception information can be accessed via .ecxr. (3fac.21d0): Access violation – code c0000005 (first/second chance not available) *** WARNING: Unable to verify checksum for crashme.exe eax=00000000 ebx=aabbccdd ecx=00000000 edx=00000000 esi=742d80e8 edi=02a85998 eip=00311015 esp=006ffa98 ebp=006ffaa8 iopl=0 nv up ei pl zr na pe nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010244 crashme!main+0x15: 00311015 c700ddccbbaa mov dword ptr [eax],0AABBCCDDh ds:002b:00000000=???????? 0:000> |
cdb.exe successfully launched and waits user input. Press ‘q <enter>’ to return to the command line.
| PS D:\workspace\playground\crashme\Release> & “C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\cdb.exe” -z crashme.dmp Microsoft (R) Windows Debugger Version 10.0.14321.1024 X86 Copyright (c) Microsoft Corporation. All rights reserved. Loading Dump File [D:\workspace\playground\crashme\Release\crashme.dmp] User Mini Dump File with Full Memory: Only application data is available ************* Symbol Path validation summary ************** Response Time (ms) Location Deferred srv*d:\symbols*https://msdl.microsoft.com/download/symbols Symbol search path is: srv*d:\symbols*https://msdl.microsoft.com/download/symbols Executable search path is: Windows 10 Version 16299 MP (8 procs) Free x86 compatible Product: WinNt, suite: SingleUserTS Built by: 10.0.16299.15 (WinBuild.160101.0800) Machine Name: Debug session time: Mon Nov 27 22:54:12.000 2017 (UTC + 9:00) System Uptime: 2 days 17:55:58.676 Process Uptime: 0 days 0:00:18.000 …… This dump file has an exception of interest stored in it. The stored exception information can be accessed via .ecxr. (3fac.21d0): Access violation – code c0000005 (first/second chance not available) *** WARNING: Unable to verify checksum for crashme.exe eax=00000000 ebx=aabbccdd ecx=00000000 edx=00000000 esi=742d80e8 edi=02a85998 eip=00311015 esp=006ffa98 ebp=006ffaa8 iopl=0 nv up ei pl zr na pe nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010244 crashme!main+0x15: 00311015 c700ddccbbaa mov dword ptr [eax],0AABBCCDDh ds:002b:00000000=???????? 0:000> q quit: PS D:\workspace\playground\crashme\Release> |
You can use ‘-c’ to pass initial command runs followed by opening the dump. Each command is separated by ‘;’ so that enables carry out multiple command at once. For example, “u @eip; r; q” will show the disassembly since @EIP points, prints out registers and quit the cdb.exe itself then.
PS D:\workspace\playground\crashme\Release> & “C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\cdb.exe” -z crashme.dmp -c “u @eip; r; q”
| PS D:\workspace\playground\crashme\Release> & “C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\cdb.exe” -z crashme.dmp -c “u @eip; r; q” Microsoft (R) Windows Debugger Version 10.0.14321.1024 X86 Copyright (c) Microsoft Corporation. All rights reserved. Loading Dump File [D:\workspace\playground\crashme\Release\crashme.dmp] User Mini Dump File with Full Memory: Only application data is available ************* Symbol Path validation summary ************** Response Time (ms) Location Deferred srv*d:\symbols*https://msdl.microsoft.com/download/symbols Symbol search path is: srv*d:\symbols*https://msdl.microsoft.com/download/symbols Executable search path is: Windows 10 Version 16299 MP (8 procs) Free x86 compatible Product: WinNt, suite: SingleUserTS Built by: 10.0.16299.15 (WinBuild.160101.0800) Machine Name: Debug session time: Mon Nov 27 22:54:12.000 2017 (UTC + 9:00) System Uptime: 2 days 17:55:58.676 Process Uptime: 0 days 0:00:18.000 …… This dump file has an exception of interest stored in it. The stored exception information can be accessed via .ecxr. (3fac.21d0): Access violation – code c0000005 (first/second chance not available) *** WARNING: Unable to verify checksum for crashme.exe eax=00000000 ebx=aabbccdd ecx=00000000 edx=00000000 esi=742d80e8 edi=02a85998 eip=00311015 esp=006ffa98 ebp=006ffaa8 iopl=0 nv up ei pl zr na pe nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010244 crashme!main+0x15: 00311015 c700ddccbbaa mov dword ptr [eax],0AABBCCDDh ds:002b:00000000=???????? 0:000> cdb: Reading initial command ‘u @eip; r; q’ crashme!main+0x15: 00311015 c700ddccbbaa mov dword ptr [eax],0AABBCCDDh 0031101b 5b pop ebx 0031101c 58 pop eax 0031101d 33c0 xor eax,eax 0031101f 5b pop ebx 00311020 8be5 mov esp,ebp 00311022 5d pop ebp 00311023 c3 ret eax=00000000 ebx=aabbccdd ecx=00000000 edx=00000000 esi=742d80e8 edi=02a85998 eip=00311015 esp=006ffa98 ebp=006ffaa8 iopl=0 nv up ei pl zr na pe nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010244 crashme!main+0x15: 00311015 c700ddccbbaa mov dword ptr [eax],0AABBCCDDh ds:002b:00000000=???????? quit: PS D:\workspace\playground\crashme\Release> |
We now have figured out how to open a crash dump with initial analysis command from command line, it’s time to write the script automate those.
Let’s make the python virtual environment first.
| PS D:\workspace\playground\crashme\Release> python -m venv venv PS D:\workspace\playground\crashme\Release> .\venv\Scripts\activate (venv) PS D:\workspace\playground\crashme\Release> |
Launching cdb.exe and Gettings its output from Python script.
You can use the subprocess python module for spawning other process and getting its output. Please see the following:
process = subprocess.Popen(cdb_param, stdout=subprocess.PIPE)
for line in process.stdout:
output_line = line.decode(‘utf-8’, ‘ignore’)
#print(output_line, end=”)
output_str_list.append(output_line)
In case you are wondering how to pass remaining options including -z dump file itself and initial command with -u…
| import subprocess import sys exe_path = r”C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\cdb.exe” arguments = [‘-z’, ‘crashme.dmp’, ‘-c’, ‘u @eip; r; q’] def run_command(command): output = subprocess.check_output(command) print(output.decode(‘utf-8’)) if __name__ == “__main__”: run_command([exe_path] + arguments) |
Let’s name the script as ‘cdb.py’ and now we’re ready to launch the cdb.exe and getting its result:
D:\workspace\playground\crashme\Release>python cdb.py
Microsoft (R) Windows Debugger Version 10.0.14321.1024 X86
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [D:\workspace\playground\crashme\Release\crashme.dmp]
User Mini Dump File with Full Memory: Only application data is available
************* Symbol Path validation summary **************
Response Time (ms) Location
Deferred srv*d:\symbols*https://msdl.microsoft.com/download/symbols
Symbol search path is: srv*d:\symbols*https://msdl.microsoft.com/download/symbols
Executable search path is:
Windows 10 Version 16299 MP (8 procs) Free x86 compatible
Product: WinNt, suite: SingleUserTS
Built by: 10.0.16299.15 (WinBuild.160101.0800)
Machine Name:
Debug session time: Mon Nov 27 22:54:12.000 2017 (UTC + 9:00)
System Uptime: 2 days 17:55:58.676
Process Uptime: 0 days 0:00:18.000
……
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(3fac.21d0): Access violation – code c0000005 (first/second chance not available)
*** WARNING: Unable to verify checksum for crashme.exe
eax=00000000 ebx=aabbccdd ecx=00000000 edx=00000000 esi=742d80e8 edi=02a85998
eip=00311015 esp=006ffa98 ebp=006ffaa8 iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010244
crashme!main+0x15:
00311015 c700ddccbbaa mov dword ptr [eax],0AABBCCDDh ds:002b:00000000=????????
0:000> cdb: Reading initial command ‘u @eip; r; q’
crashme!main+0x15:
00311015 c700ddccbbaa mov dword ptr [eax],0AABBCCDDh
0031101b 5b pop ebx
0031101c 58 pop eax
0031101d 33c0 xor eax,eax
0031101f 5b pop ebx
00311020 8be5 mov esp,ebp
00311022 5d pop ebp
00311023 c3 ret
eax=00000000 ebx=aabbccdd ecx=00000000 edx=00000000 esi=742d80e8 edi=02a85998
eip=00311015 esp=006ffa98 ebp=006ffaa8 iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010244
crashme!main+0x15:
00311015 c700ddccbbaa mov dword ptr [eax],0AABBCCDDh ds:002b:00000000=????????
quit:
D:\workspace\playground\crashme\Release>
Other useful initial commands may include:
- !analyze -v
- ecxr;kb
- ecxr;kP;dv /t/V
- etc…
Connecting the cdb.py to SlackBot
This post isn’t a comprehensive bot writing manual, so here describes only essential steps.
First, you need to create a new bot user from the Integration page. Visit the ‘Administration > Manage apps > Custom Integrations’.
I named it a ‘debugbot’ here.
If it’s successfully created, it’ll show a API token. It’s a secret so don’t share publicly (the ‘debugbot’ no longer exist).
You can also modify the bot’s name and profile picture here.
https://github.com/slackapi/python-slackclient
is the official repo of the Slack’s Python bot client. Clone the slack-client library.
| (venv) D:\workspace\playground\crashme\Release>git clone https://github.com/slackapi/python-slackclient.git Cloning into ‘python-slackclient’… remote: Counting objects: 1320, done. remote: Compressing objects: 100% (24/24), done. remote: Total 1320 (delta 15), reused 24 (delta 10), pack-reused 1286 Receiving objects: 100% (1320/1320), 878.80 KiB | 359.00 KiB/s, done. Resolving deltas: 100% (797/797), done. (venv) D:\workspace\playground\crashme\Release>cd python-slackclient (venv) D:\workspace\playground\crashme\Release\python-slackclient>pip install -r requirements.txt Obtaining file:///D:/workspace/playground/crashme/Release/python-slackclient (from -r requirements.txt (line 3)) Collecting websocket-client<1.0a0,>=0.35 (from slackclient==1.0.9->-r requirements.txt (line 3)) Downloading websocket_client-0.44.0-py2.py3-none-any.whl (199kB) 100% |████████████████████████████████| 204kB 799kB/s Collecting requests<3.0a0,>=2.11 (from slackclient==1.0.9->-r requirements.txt (line 3)) Downloading requests-2.18.4-py2.py3-none-any.whl (88kB) 100% |████████████████████████████████| 92kB 738kB/s Collecting six<2.0a0,>=1.10 (from slackclient==1.0.9->-r requirements.txt (line 3)) Downloading six-1.11.0-py2.py3-none-any.whl Collecting chardet<3.1.0,>=3.0.2 (from requests<3.0a0,>=2.11->slackclient==1.0.9->-r requirements.txt (line 3)) Downloading chardet-3.0.4-py2.py3-none-any.whl (133kB) 100% |████████████████████████████████| 143kB 172kB/s Collecting certifi>=2017.4.17 (from requests<3.0a0,>=2.11->slackclient==1.0.9->-r requirements.txt (line 3)) Downloading certifi-2017.11.5-py2.py3-none-any.whl (330kB) 100% |████████████████████████████████| 337kB 297kB/s Collecting urllib3<1.23,>=1.21.1 (from requests<3.0a0,>=2.11->slackclient==1.0.9->-r requirements.txt (line 3)) Downloading urllib3-1.22-py2.py3-none-any.whl (132kB) 100% |████████████████████████████████| 133kB 197kB/s Collecting idna<2.7,>=2.5 (from requests<3.0a0,>=2.11->slackclient==1.0.9->-r requirements.txt (line 3)) Downloading idna-2.6-py2.py3-none-any.whl (56kB) 100% |████████████████████████████████| 61kB 72kB/s Installing collected packages: six, websocket-client, chardet, certifi, urllib3, idna, requests, slackclient Running setup.py develop for slackclient Successfully installed certifi-2017.11.5 chardet-3.0.4 idna-2.6 requests-2.18.4 six-1.11.0 slackclient urllib3-1.22 websocket-client-0.44.0 |
Then you need to invite the bot to read text from the client coming from the Slack channel. I created a dedicated ‘#autoanalysis’ channel and invited the bot for example.
Run the following command from a command line to set the API token to ‘SLACK_API’ environment variable for convenient.
SET SLACK_API_TOKEN=xoxb-282029623751-BVtmnS3BQitmjZvjpQL7PSGP
Then, write a basic Slack bot client code as following. Please refer to the full documentation more detail from https://slackapi.github.io/python-slackclient/ and https://python-slackclient.readthedocs.io/en/latest/basic_usage.html#sending-a-message. It basically does WebAPI call with ‘chat.postMessage’ command described here https://api.slack.com/methods/chat.postMessage.
| import os from slackclient import SlackClient slack_token = os.environ[“SLACK_API_TOKEN”] sc = SlackClient(slack_token) ret = sc.api_call( “chat.postMessage”, channel=”#autoanalysis”, text=”Hello from Python! :tada:” ) print(ret) |
I added ‘print(ret)’ last line to see what the Slack server returned. The return object also contains the API call result.
| (venv) D:\workspace\playground\crashme\Release>python slack.py {‘ok’: True, ‘channel’: ‘C88K023NZ’, ‘ts’: ‘1512317953.000112’, ‘message’: {‘text’: ‘Hello from Python! :tada:’, ‘username’: ‘bot’, ‘bot_id’: ‘B892XRL93’, ‘type’: ‘message’, ‘subtype’: ‘bot_message’, ‘ts’: ‘1512317953.000112’}} |
If everything goes well, you’ll see the small snippet prints that string to the Slack channel as below:
Read a command and do analysis from the Bot
Using the WebAPI based ‘api_call’ was simple to use but it’s not enough to receive real time messages others talking from the channel. For that purpose, the SlackClient provides Real Time Messaging API(RTM API).
http://slackapi.github.io/python-slackclient/real_time_messaging.html#
Next snippet shows how to connect to the bot channel and keep printing messages in real time while it’s connected.
| import os import time from slackclient import SlackClient slack_token = os.environ[“SLACK_API_TOKEN”] sc = SlackClient(slack_token) if sc.rtm_connect(): while True: print( sc.rtm_read()) time.sleep(1) else: print( “Connection Failed”) |
It’ll keep running until pressing ‘Ctrl+C'(or disconnected)
| (venv) D:\workspace\playground\crashme\Release>python slack.py [] [{‘type’: ‘hello’}] [{‘type’: ‘reconnect_url’, ‘url’: ‘wss://mpmulti-7vr5.lb.slack-msgs.com/websocket/_gkSQP2x_PoYBPCsYMqpryRjT7ggXh4AfVyTT8Gz1MJ44xZLpLHg_iPHzIFEXxPBS0TwuH9gy9Nqe3PdqE5ikWUlYsF2Ior3GTdI-eZ83GadFvzf9b0KTGFvfVUK_GROj06QwQkEiy3zlVguHGmPRThTTau6u0J2ws69qF4Yv1Q=’}] [{‘type’: ‘presence_change’, ‘presence’: ‘active’, ‘user’: ‘U8A0VJBN3’}] [] [] [] [] Traceback (most recent call last): File “slack.py”, line 11, in <module> time.sleep(1) KeyboardInterrupt (venv) D:\workspace\playground\crashme\Release> |
You can filter out only messages by following:
| import os import time from slackclient import SlackClient slack_token = os.environ[“SLACK_API_TOKEN”] sc = SlackClient(slack_token) if sc.rtm_connect(): while True: json_response = sc.rtm_read() for json_item in json_response: if “type” in json_item and json_item[“type”] == “message”: print(json_item[“text”]) time.sleep(1) else: print( “Connection Failed”) |
Let’s put all together. It waits until someone type ‘analyze’, run the cdb.exe and finally upload its output as a text attachment.
| import os import time import subprocess import sys import io exe_path = r”C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\cdb.exe” arguments = [‘-z’, ‘crashme.dmp’, ‘-c’, ‘u @eip; r; q’] def run_command(command): output = subprocess.check_output(command) return output.decode(‘utf-8’) from slackclient import SlackClient slack_token = os.environ[“SLACK_API_TOKEN”] sc = SlackClient(slack_token) if sc.rtm_connect(): while True: json_response = sc.rtm_read() for json_item in json_response: if “type” in json_item and json_item[“type”] == “message”: if json_item[“text”].lower() == “analyze”: result = run_command([exe_path] + arguments) ret = sc.api_call(“files.upload”, filename=”cdb_analyze_result.txt”,channels=json_item[“channel”],file=io.BytesIO(str.encode(result))) if not ‘ok’ in ret or not ret[‘ok’]: print(ret[‘error’]) time.sleep(1) else: print( “Connection Failed”) |
The cdb’s output showed up in that channel successfully.
Let’s separate the upload part as a different function named ‘upload_text’ for simplicity.
| import os import time import subprocess import sys import io exe_path = r”C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\cdb.exe” arguments = [‘-z’, ‘crashme.dmp’, ‘-c’, ‘u @eip; r; q’] def run_command(command): output = subprocess.check_output(command) return output.decode(‘utf-8’) def upload_text(text, file_name, channel): ret = sc.api_call(“files.upload”, filename=file_name,channels=channel,file=io.BytesIO(str.encode(text))) if not ‘ok’ in ret or not ret[‘ok’]: print(ret[‘error’]) from slackclient import SlackClient slack_token = os.environ[“SLACK_API_TOKEN”] sc = SlackClient(slack_token) if sc.rtm_connect(): while True: json_response = sc.rtm_read() for json_item in json_response: if “type” in json_item and json_item[“type”] == “message”: if json_item[“text”].lower() == “analyze”: result = run_command([exe_path] + arguments) upload_text(result, “cdb_analyze_result.txt”, json_item[“channel”]) time.sleep(1) else: print( “Connection Failed”) |
More tests!
Here’s the last sample slightly improved to handle errors and command reading.
| import os import time import subprocess import sys import io exe_path = r”C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\cdb.exe” arguments = [‘-c’, ‘u @eip; r; q’] def run_command(command): try: output = subprocess.check_output(command) return output.decode(‘utf-8’) except subprocess.CalledProcessError as e: return ‘analysis failed with {}, {}’.format(e.returncode, e.output.decode(‘utf-8’)) def upload_text(text, file_name, channel): ret = sc.api_call(“files.upload”, filename=file_name,channels=channel,file=io.BytesIO(str.encode(text))) if not ‘ok’ in ret or not ret[‘ok’]: print(ret[‘error’]) from slackclient import SlackClient slack_token = os.environ[“SLACK_API_TOKEN”] sc = SlackClient(slack_token) if sc.rtm_connect(): while True: json_response = sc.rtm_read() for json_item in json_response: if “type” in json_item and json_item[“type”] == “message”: args = json_item[“text”].split() if args[0].startswith(‘!’): if len(args) > 1 and args[0].lower() == “!analyze”: result = run_command([exe_path] + [‘-z’, args[1]] + arguments) upload_text(result, “cdb_analyze_result.txt”, json_item[“channel”]) else: ret = sc.api_call(“chat.postMessage”, channel=json_item[“channel”],text=”Unknown command”) time.sleep(1) else: print( “Connection Failed”) |
Thanks for reading.
Heejune.
it's compiling 
One thought on “CrashDump analysis automation using SlackBot, python, cdb from Windows”