Introducing Kernel Composure
Parsing Kernel Panic logs on macOS 10.15.x
In this post we will be explaining how we have created an application for macOS to help administrators quickly parse logs generated by kernel panics on macOS 10.15. Kernel panics can result in a less than ideal experience for users, but administrators can now keep their cool and stay composed when troubleshooting by using Encore’s Kernel Composure. The code that makes all of this work is at the bottom of this article.
Background
For those unfamiliar, kernel panics on macOS are the equivalent of the Windows “Blue Screen of Death”. It’s a full crash of your computer followed by a black screen with the message “You need to restart your computer. Hold down the Power button for several seconds or press the Restart button.” displayed in several different languages. While kernel panics are becoming less common due to Apple’s limiting of kernel extensions, they are unfortunately still a reality for macOS administrators.
When a macOS computer kernel panics, a log is generated and stored in /Library/Logs/DiagnosticReports in the format of Kernel_{date of error}_{computer name}. For most recent versions of OS X/macOS, those files were easy to open up and quickly determine what caused the issue. Beginning with macOS 10.15, the format of the logs was drastically changed. While this has made it harder to open them and review, it has made them much easier to automatically parse.
Overview of Solution
Our parser is written entirely in Python using v3 syntax and relies on the following modules: * PyQT5 * py2app * darkdetect * qdarkstyle * argparse - Command line version only
The Python script to parse these logs can be run directly from the command line, but macOS does not include Python 3 by default. In order to bundle up our Python framework and all of our dependencies, we use the excellent tool py2app. This allows us to build a complete application which means the administrator does not need to install any dependencies to use it. Additionally, we are able to use PyQT5 to build a GUI to make everything more accessible to those who aren’t comfortable with the command line. You can even deploy the application to your users to help you gather initial information on kernel panics before digging in.
This isn’t supposed to be necessary according to https://www.learnpyqt.com/blog/macos-mojave-dark-mode-support-pyqt5122/, but we found that this was not our experience. So in order to provide support for those who prefer to work in Dark Mode on macOS 10.14+, we use darkdetect to check which theme you’re on and qdarkstyle to properly give our app a Dark Mode theme.
Code AKA The Meat
Command Line Version
Let’s start with the completely standalone version. This is in Py3 syntax, but only requires argparse as an additional module. It does no formatting and only presents the information in a more readable fashion. This could be written in pure Py2 syntax using sys.argv, but as Python is slated to be removed from macOS in a future version, we believe it’s worth moving to Python 3 now.
#!/usr/bin/env python3
import argparse
import json
import re
# Parses a given log file for every match of a specific regex
def parse_log_file(log_file_path, regex):
match_list = []
with open(log_file_path, "r") as log_file:
content = log_file.read().splitlines()
for line in content:
for match in re.finditer(regex, line, re.S):
match_text = match.group()
match_list.append(match_text)
return match_list
def main():
kernel_panic = parse_log_file(log_file_path, regex)
initial_info = json.loads(kernel_panic[0])
for key, value in initial_info.items():
print(f"{key}: {value}")
panic_json = json.loads(kernel_panic[1])
print(panic_json["macOSPanicString"])
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Kernel Panic Log Path")
parser.add_argument(
"log_file_path",
type=str,
help="Kernel Panic Log Path i.e. /Library/Logs/DiagnosticReports/Kernel_2020-03-09-135527_Test-Mac.panic",
)
args = parser.parse_args()
regex = r"\{(.*?)\}"
log_file_path = f"{args.log_file_path}"
main()
GUI Version
By using PyQT5, we can add a GUI to our Python script. This is fully cross-platform, so as long as the modules are installed and you can get the .panic files transferred over, this can even be given to Windows administrators for usage.
#!/usr/bin/env python3
import json
import os
import re
import sys
import darkdetect
import qdarkstyle
from PyQt5.QtCore import QDir
from PyQt5.QtWidgets import (
QApplication,
QFileDialog,
QLabel,
QPushButton,
QTextEdit,
QVBoxLayout,
QWidget,
)
class kernelcomposure(QWidget):
def __init__(self, parent=None):
super(kernelcomposure, self).__init__(parent)
layout = QVBoxLayout()
self.btn = QPushButton("Browse for Kernel Panic log")
self.btn.clicked.connect(self.getfile)
layout.addWidget(self.btn)
self.le = QLabel("Kernel Composure")
self.contents = QTextEdit()
self.contents.setReadOnly(True)
layout.addWidget(self.contents)
self.setLayout(layout)
self.setWindowTitle("Kernel Composure")
# Parses a given log file for every match of a specific regex
def parse_log_file(self, log_file_path, regex):
match_list = []
with open(log_file_path, "r") as log_file:
content = log_file.read().splitlines()
for line in content:
for match in re.finditer(regex, line, re.S):
match_text = match.group()
match_list.append(match_text)
return match_list
def getfile(self):
fname = QFileDialog.getOpenFileName(
self, "Open file", "/", "Kernel Panic Files (*.panic)"
)
log_file_path = fname[0]
if not os.path.exists(log_file_path):
self.contents.setText("Please select a valid Kernel Panic log.")
else:
self.parsefile(fname[0])
def parsefile(self, log_file_path):
regex = r"\{(.*?)\}"
kernel_panic = self.parse_log_file(log_file_path, regex)
try:
initial_info = json.loads(kernel_panic[0])
for key, value in initial_info.items():
if key == list(initial_info.keys())[0]:
panic_info = (
f"<b>{key}:</b> <br> {value} <br>"
)
else:
panic_info = f"{panic_info} <br> <b>{key}:</b> <br> {value} <br>"
panic_json = json.loads(kernel_panic[1])
macOSPanicString = panic_json["macOSPanicString"]
panic_info = (
f"{panic_info} <br> <b>Kernel Extensions in backtrace:</b> <br>"
)
backtrace = False
for line in macOSPanicString.splitlines():
if "Kernel Extensions in backtrace" in line:
backtrace = True
continue
elif "BSD process name corresponding to current thread" in line:
backtrace = False
process_name = line.split(":", 1)[1]
panic_info = f"{panic_info} <b>Last process before crash:</b> {process_name} <br>"
continue
elif backtrace and "dependency" not in line:
panic_info = f"{panic_info} {line.strip()} <br>"
elif backtrace and "dependency" in line:
panic_info = (
f"{panic_info} {line.strip()} <br>"
)
html_panic_string = macOSPanicString.replace("\n", "<br>")
panic_info = (
f"{panic_info} <br> <b>Full Panic String:</b> <br> {html_panic_string}"
)
self.contents.setText(panic_info)
except IndexError:
self.contents.setText(
"Currently I can only parse Kernel Panic logs generated from macOS 10.15.x"
)
def main():
app = QApplication(sys.argv)
ex = kernelcomposure()
ex.resize(950, 500)
if len(sys.argv) > 1:
ex.parsefile(sys.argv[1])
if darkdetect.isDark():
ex.setStyleSheet(qdarkstyle.load_stylesheet_pyqt5())
ex.show()
sys.exit(app.exec_())
if __name__ == "__main__":
main()
Python Application
Next we come to the native macOS application. This allows us to provide a native GUI and a fully standalone application designed to be easy to use for anyone. In order for py2app to build the application, we first need to generate a setup.py file. A base version can be generated from within the working directly by using py2applet which is included within the py2app module. A good breakdown of using it yourself is found here. Our setup.py is as follows:
"""
This is a setup.py script generated by py2applet
Usage:
python setup.py py2app
"""
from setuptools import setup
APP = ["./src/main/python/main.py"]
DATA_FILES = []
OPTIONS = {
"argv_emulation": True,
"iconfile": "kp.icns",
"plist": {
"CFBundleName": "Kernel Composure",
"CFBundleDisplayName": "Kernel Composure",
"CFBundleGetInfoString": "Kernel Composure",
"CFBundleIdentifier": "com.encore.kernelcomposure",
"CFBundleVersion": "0.2.0",
"CFBundleShortVersionString": "0.2.0",
"NSHumanReadableCopyright": u"Copyright © 2020, Encore Technologies, All Rights Reserved",
"CFBundleTypeExtensions": ["panic"],
},
}
setup(
app=APP,
data_files=DATA_FILES,
options={"py2app": OPTIONS},
setup_requires=["py2app"],
)
After generating your setup.py, you finish your application build by running:
python setup.py py2app
This will result in your .app being built within the dist folder of your working directory.
Swift application
Finally, we decided to rewrite the app in Swift to stay entirely within the native Apple tools. One of the biggest reasons for this change is that the app doesn’t need to include the entire Python 3 framework inside of it. This brought the size of the app down from ~135 MB to less than 400 KB while maintaining all of the same functionality. This size went up after changing the deployment target from macOS 10.15 to macOS 10.13 due to some needed Swift inclusions, but is still much smaller than the comparable Python app. Due to the nature of storyboards, it’s hard to accurately show all of the code within this blogpost, but the AppDelegate.swift and ViewController.swift are shown below:
AppDelegate
// AppDelegate.swift
// Kernel Composure
//
// Created by Tyler Sparr on 5/17/20.
// Copyright © 2020 Encore Technologies. All rights reserved.
//
import Cocoa
@NSApplicationMain
class AppDelegate: NSObject, NSApplicationDelegate {
var filename: String = ""
// Accepts a file via "Open With" and does an automatic parsing run
func application(_ sender: NSApplication, openFile filename: String) -> Bool {
self.filename = filename
let fileUrl = URL(fileURLWithPath: filename)
guard let main = NSApp.mainWindow?.contentViewController as! ViewController? else { return true }
main.automatic_run(fileUrl)
return true
}
func applicationDidFinishLaunching(_ aNotification: Notification) {
// Insert code here to initialize your application
}
func applicationWillTerminate(_ aNotification: Notification) {
// Insert code here to tear down your application
}
}
ViewController
//
// ViewController.swift
// Kernel Composure
//
// Created by Tyler Sparr on 5/17/20.
// Copyright © 2020 Encore Technologies. All rights reserved.
//
import Cocoa
class ViewController: NSViewController {
// Defines our button at the top of the window
@IBOutlet weak var kernelPanicSelection: NSButton!
// Defines the box where we output our text
@IBOutlet weak var kernelPanicText: NSTextView!
// Action connected to our button, opens file selection
@IBAction func handleKernelPanicSelection(_ sender: Any) {
let dialog = NSOpenPanel();
dialog.title = "Choose the Kernel Panic log";
dialog.showsResizeIndicator = true;
dialog.showsHiddenFiles = false;
dialog.allowsMultipleSelection = false;
dialog.canChooseDirectories = false;
dialog.allowedFileTypes = ["panic"];
if (dialog.runModal() == NSApplication.ModalResponse.OK) {
let result = dialog.url
if (result != nil) {
basic_parse_kernel_panic(file_name: result!)
}
} else {
// User clicked on "Cancel"
return
}
}
// Action connected to View - > Show Advanced menu button
@IBAction func handleAdvancedViewSelection(_ sender: Any) {
// Check if there's already text in our view
let panic_string = (kernelPanicText.textStorage as NSAttributedString?)?.string
// If there's text, pull the filename and reparse gathering and displaying all info
if panic_string != nil {
let lines = panic_string!.split(whereSeparator: \.isNewline)
let filename = lines[1].trimmingCharacters(in: .whitespaces)
let fileUrl = URL(fileURLWithPath: filename)
advanced_parse_kernel_panic(file_name: fileUrl)
}
else {
return
}
}
// Action connected to Help -> Kernel Composure Help menu button
@IBAction func getMoreInfo(_ sender: Any) {
let url = URL(string: "https://developer.apple.com/library/archive/technotes/tn2063/_index.html")!
NSWorkspace.shared.open(url)
}
// This is called from AppDelegate.swift if the file is opened by "Open With"
// rather than via our GUI button
func automatic_run(_ filename: URL) {
basic_parse_kernel_panic(file_name: filename)
}
// Make sure we start off with a blank, uneditable view that supports rich text
func setupUI() {
kernelPanicText.string = ""
kernelPanicText.isEditable = false
kernelPanicText.isRichText = true
}
// Reads the file line by line and adds the lines to an array
func loadFile(file_path:URL) -> Array<String>{
var panic_log_lines: [String] = []
let path:String = file_path.path
let reader = LineReader(path: path)
for line in reader! {
panic_log_lines.append(line)
}
return panic_log_lines
}
// Parse the first line of the Kernel Panic log and return it as a JSON object
func get_initial_info(panic_log_lines:Array<String>) -> JSON{
let initial_info = panic_log_lines[0]
let json = JSON.init(parseJSON:initial_info)
return(json)
}
// Parse the second line of the Kernel Panic log and return it as a JSON object
func get_panic_string(panic_log_lines:Array<String>) -> JSON{
let panic_string = panic_log_lines[1]
let json = JSON.init(parseJSON:panic_string)
return(json)
}
// Get the kernel extensions in backtrace via brute force text parsing
func get_backtrace(panic_string:JSON) -> Array<String>{
var process_info: [String] = []
let panic_string = panic_string["macOSPanicString"].stringValue
let lines = panic_string.split(whereSeparator: \.isNewline)
// Find where the backtrace starts
let backtraceIndex = lines.firstIndex(of: " Kernel Extensions in backtrace:")!
for line in lines {
// Find where the backtrace ends
if line.contains("BSD process name corresponding to current thread") {
let process_name_line = lines.firstIndex(of: line)
// Copy everything in between the two lines to get our full backtrace
let backtrace_values = lines[backtraceIndex..<process_name_line!]
for line in backtrace_values {
// Bold the first line in the backtrace
if backtrace_values.first == line {
let stripped_line = line.trimmingCharacters(in: .whitespaces)
process_info.append("<b>\(stripped_line)</b>")
}
// Indent all dependencies
if line.contains("dependency"){
let stripped_line = line.trimmingCharacters(in: .whitespaces)
process_info.append(" \(stripped_line)<br>")
} else {
let stripped_line = line.trimmingCharacters(in: .whitespaces)
process_info.append("\(stripped_line)<br>")
}
}
// Get the process name by splitting by colon
let process_name = line.components(separatedBy: ":")
let process_name_txt = process_name[1]
process_info.append("<br><br><b>Last process before crash:</b> \(process_name_txt)<br>")
}
}
return process_info
}
// Build our basic output with all of the necessary HTML formatting
func build_initial_output(file_name: URL, initial_info:JSON, panic_string:JSON, process_info:Array<String>) -> NSAttributedString{
var initial_output: [String] = []
let path:String = file_name.path
initial_output.append("<b>filename:</b> <br> \(path)<br>")
let timestamp = initial_info["timestamp"].stringValue
initial_output.append("<b>timestamp:</b> <br> \(timestamp)<br>")
let bug_type = initial_info["bug_type"].stringValue
initial_output.append("<b>bug_type:</b> <br> \(bug_type)<br>")
let os_version = initial_info["os_version"].stringValue
initial_output.append("<b>os_version:</b> <br> \(os_version)<br>")
let initial_output_text = initial_output.joined(separator: "\n")
var process_text = process_info.joined(separator: ",")
process_text = process_text.replacingOccurrences(of: ",", with: "", options: NSString.CompareOptions.literal, range:nil)
let full_panic_text = initial_output_text + "<br><br>" + process_text
// We need to convert from String to Data to use HTML formatting
let panic_html = full_panic_text.data(using: .utf8)!
// Tell it we want our Data to be parsed as HTML and make it mutable
let formatted_panic = NSMutableAttributedString(html: panic_html, documentAttributes: nil)!
// We need to get the full length so we can apply formatting to the whole string
let panic_range = NSMakeRange(0, formatted_panic.length)
// This is necessary for automatic Dark Mode support
formatted_panic.addAttribute(.foregroundColor, value: NSColor.textColor, range: panic_range)
return formatted_panic
}
// Build our entire output with necessary HTML formatting
func build_full_output(file_name: URL, initial_info:JSON, panic_string:JSON, process_info:Array<String>) -> NSAttributedString{
var initial_output: [String] = []
let path:String = file_name.path
initial_output.append("<b>filename:</b> <br> \(path)<br>")
let timestamp = initial_info["timestamp"].stringValue
initial_output.append("<b>timestamp:</b> <br> \(timestamp)<br>")
let bug_type = initial_info["bug_type"].stringValue
initial_output.append("<b>bug_type:</b> <br> \(bug_type)<br>")
let os_version = initial_info["os_version"].stringValue
initial_output.append("<b>os_version:</b> <br> \(os_version)<br>")
let initial_output_text = initial_output.joined(separator: "\n")
var process_text = process_info.joined(separator: ",")
process_text = process_text.replacingOccurrences(of: ",", with: "", options: NSString.CompareOptions.literal, range:nil)
var panic_string = panic_string["macOSPanicString"].stringValue
panic_string = panic_string.replacingOccurrences(of: "\n", with: "<br>", options: NSString.CompareOptions.literal, range:nil)
while panic_string.hasSuffix("<br>"){
panic_string = String(panic_string.dropLast(4))
}
let full_panic_text = initial_output_text + "<br><br>" + process_text + "<br><br>" + "<b>Full Panic String:</b><br>" + panic_string
// We need to convert from String to Data to use HTML formatting
let panic_html = full_panic_text.data(using: .utf8)!
// Tell it we want our Data to be parsed as HTML and make it mutable
let formatted_panic = NSMutableAttributedString(html: panic_html, documentAttributes: nil)!
// We need to get the full length so we can apply formatting to the whole string
let panic_range = NSMakeRange(0, formatted_panic.length)
// This is necessary for automatic Dark Mode support
formatted_panic.addAttribute(.foregroundColor, value: NSColor.textColor, range: panic_range)
return formatted_panic
}
// Combines all of our basic parsing functions together for initial output
func basic_parse_kernel_panic(file_name: URL) {
let panic_text_array = loadFile(file_path: file_name)
let initial_info = get_initial_info(panic_log_lines: panic_text_array)
if initial_info != JSON.null {
let panic_string = get_panic_string(panic_log_lines: panic_text_array)
let process_info = get_backtrace(panic_string: panic_string)
let initial_output = build_initial_output(file_name: file_name, initial_info: initial_info, panic_string: panic_string, process_info: process_info)
kernelPanicText.textStorage?.setAttributedString(initial_output)
} else {
kernelPanicText.string = "Currently I can only parse Kernel Panic logs generated from macOS 10.15.x. Please select a different log."
return
}
}
// Combines all parsing functions together for full output
func advanced_parse_kernel_panic(file_name: URL) {
let panic_text_array = loadFile(file_path: file_name)
let initial_info = get_initial_info(panic_log_lines: panic_text_array)
if initial_info != JSON.null {
let panic_string = get_panic_string(panic_log_lines: panic_text_array)
let process_info = get_backtrace(panic_string: panic_string)
let full_output = build_full_output(file_name: file_name, initial_info: initial_info, panic_string: panic_string, process_info: process_info)
kernelPanicText.textStorage?.setAttributedString(full_output)
} else {
kernelPanicText.string = "Currently I can only parse Kernel Panic logs generated from macOS 10.15.x. Please select a different log."
return
}
}
override func viewDidLoad() {
super.viewDidLoad()
// Do any additional setup after loading the view.
// Default function, we're not using
}
// Just does our initial UI setup
override func viewWillAppear() {
super.viewWillAppear()
setupUI()
}
override var representedObject: Any? {
didSet {
// Update the view, if already loaded.
// Default function, we're not using
}
}
}
Summary
This was a real problem we needed to solve for our team and as strong supporters and contributors to the open source community, we wanted to make sure everyone could benefit from our work. We hope the community finds it useful. You can download the latest release at https://github.com/EncoreTechnologies/swift-KernelComposure/releases and we encourage comments, issues, PRs, etc.
Share this post
Twitter
Google+
Facebook
Reddit
LinkedIn
StumbleUpon
Email